How to run tests #38

certik · 2020-03-27T20:38:19Z

Here is how I build Enzyme. How do I execute tests?

$ cd enzyme
$ mkdir build
$ cd build
$ cmake .. 
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
LLVM_SHLIBEXT=.so
found llvm dir /home/ondrej/repos/Enzyme/enzyme/build
found llvm lit /home/ondrej/repos/Enzyme/enzyme/build
CMAKE_PREFIX_PATH /home/ondrej/repos/Enzyme/enzyme/build
-- Linker detection: GNU ld
found llvm include directory here: /usr/lib/llvm-6.0/include
found llvm definitions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
found llvm version 6
first llvm include directory/usr/lib/llvm-6.0/include
found enzyme sources ActiveVariable.cppEnzyme.cppEnzymeLogic.cppFunctionUtils.cppGradientUtils.cppTypeAnalysis.cppUtils.cppSCEV/ScalarEvolutionExpander.cpp
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ondrej/repos/Enzyme/enzyme/build
$ make -j4
Scanning dependencies of target intrinsics_gen
[  0%] Built target intrinsics_gen
Scanning dependencies of target LLVMEnzyme-6
[ 11%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/ActiveVariable.cpp.o
[ 22%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/Enzyme.cpp.o
[ 33%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/EnzymeLogic.cpp.o
[ 44%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/FunctionUtils.cpp.o
/home/ondrej/repos/Enzyme/enzyme/Enzyme/ActiveVariable.cpp: In function ‘void addCallRemovingCycle(std::vector<llvm::CallInst*>&, llvm::CallInst*)’:
/home/ondrej/repos/Enzyme/enzyme/Enzyme/ActiveVariable.cpp:186:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
                 if (newtrace.size()-1-j == i) break;
                     ~~~~~~~~~~~~~~~~~~~~^~~~
[ 55%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/GradientUtils.cpp.o
[ 66%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/TypeAnalysis.cpp.o
/home/ondrej/repos/Enzyme/enzyme/Enzyme/TypeAnalysis.cpp: In member function ‘ValueData ValueData::KeepForCast(const llvm::DataLayout&, llvm::Type*, llvm::Type*) const’:
/home/ondrej/repos/Enzyme/enzyme/Enzyme/TypeAnalysis.cpp:75:50: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         if (pair.first[0] != -1 && pair.first[0] < tosize) {
[ 77%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/Utils.cpp.o
[ 88%] Building CXX object Enzyme/CMakeFiles/LLVMEnzyme-6.dir/SCEV/ScalarEvolutionExpander.cpp.o
[100%] Linking CXX shared module LLVMEnzyme-6.so
[100%] Built target LLVMEnzyme-6
$ ctest 
*********************************
No test configuration file found!
*********************************
Usage

  ctest [options]

$ make check-enzyme-integration
[  0%] Built target intrinsics_gen
[ 90%] Built target LLVMEnzyme-6
Scanning dependencies of target check-enzyme-integration
[100%] Running enzyme integration tests
/bin/sh: 1: ../../: Permission denied
test/Integration/CMakeFiles/check-enzyme-integration.dir/build.make:57: recipe for target 'test/Integration/CMakeFiles/check-enzyme-integration' failed
make[3]: *** [test/Integration/CMakeFiles/check-enzyme-integration] Error 126
CMakeFiles/Makefile2:296: recipe for target 'test/Integration/CMakeFiles/check-enzyme-integration.dir/all' failed
make[2]: *** [test/Integration/CMakeFiles/check-enzyme-integration.dir/all] Error 2
CMakeFiles/Makefile2:303: recipe for target 'test/Integration/CMakeFiles/check-enzyme-integration.dir/rule' failed
make[1]: *** [test/Integration/CMakeFiles/check-enzyme-integration.dir/rule] Error 2
Makefile:203: recipe for target 'check-enzyme-integration' failed
make: *** [check-enzyme-integration] Error 2

The text was updated successfully, but these errors were encountered:

wsmoses · 2020-03-27T20:40:05Z

make check-enzyme should run tests (I'll make a quick readme).

certik · 2020-03-27T20:41:22Z

This is what I get:

$ make check-enzyme
[  0%] Built target intrinsics_gen
[ 90%] Built target LLVMEnzyme-6
Scanning dependencies of target check-enzyme
[100%] Running enzyme regression tests
/bin/sh: 1: ../../: Permission denied
test/Enzyme/CMakeFiles/check-enzyme.dir/build.make:57: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme' failed
make[3]: *** [test/Enzyme/CMakeFiles/check-enzyme] Error 126
CMakeFiles/Makefile2:269: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/all' failed
make[2]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/all] Error 2
CMakeFiles/Makefile2:276: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/rule' failed
make[1]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/rule] Error 2
Makefile:190: recipe for target 'check-enzyme' failed
make: *** [check-enzyme] Error 2

certik · 2020-03-27T20:49:33Z

But if I run things manually then it works:

ondrej@pn1707483:~/.../test/Enzyme(tx-cc)$ opt-6.0 < sin.ll -load ../../build/Enzyme/LLVMEnzyme-6.so  -enzyme -enzyme_preopt=false -O3 -S 
; ModuleID = '<stdin>'
source_filename = "<stdin>"

; Function Attrs: nounwind readnone
define double @tester(double %x) local_unnamed_addr #0 {
entry:
  %0 = tail call fast double @llvm.sin.f64(double %x)
  ret double %0
}

; Function Attrs: nounwind readnone
define double @test_derivative(double %x) local_unnamed_addr #0 {
entry:
  %0 = tail call fast double @llvm.cos.f64(double %x) #2
  ret double %0
}

; Function Attrs: nounwind readnone speculatable
declare double @llvm.cos.f64(double) #1

; Function Attrs: nounwind readnone speculatable
declare double @llvm.sin.f64(double) #1

; Function Attrs: nounwind readnone
define double @preprocess_tester(double %x) local_unnamed_addr #0 {
entry:
  %0 = tail call fast double @llvm.sin.f64(double %x)
  ret double %0
}

attributes #0 = { nounwind readnone }
attributes #1 = { nounwind readnone speculatable }
attributes #2 = { nounwind }

wsmoses · 2020-03-27T20:52:02Z

Can you post the output of make check-enzyme VERBOSE=1, I'm wondering what exact command is being run. Offhand, my first guess is there's a file path with a space that isn't being quoted properly.

certik · 2020-03-27T20:53:05Z

Sure:

$ make check-enzyme VERBOSE=1
/home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -S/home/ondrej/repos/Enzyme/enzyme -B/home/ondrej/repos/Enzyme/enzyme/build --check-build-system CMakeFiles/Makefile.cmake 0
make -f CMakeFiles/Makefile2 check-enzyme
make[1]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
/home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -S/home/ondrej/repos/Enzyme/enzyme -B/home/ondrej/repos/Enzyme/enzyme/build --check-build-system CMakeFiles/Makefile.cmake 0
/home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_progress_start /home/ondrej/repos/Enzyme/enzyme/build/CMakeFiles 10
make -f CMakeFiles/Makefile2 test/Enzyme/CMakeFiles/check-enzyme.dir/all
make[2]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f CMakeFiles/intrinsics_gen.dir/build.make CMakeFiles/intrinsics_gen.dir/depend
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
cd /home/ondrej/repos/Enzyme/enzyme/build && /home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build/CMakeFiles/intrinsics_gen.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f CMakeFiles/intrinsics_gen.dir/build.make CMakeFiles/intrinsics_gen.dir/build
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
make[3]: Nothing to be done for 'CMakeFiles/intrinsics_gen.dir/build'.
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
[  0%] Built target intrinsics_gen
make -f Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build.make Enzyme/CMakeFiles/LLVMEnzyme-6.dir/depend
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
cd /home/ondrej/repos/Enzyme/enzyme/build && /home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme/Enzyme /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build/Enzyme /home/ondrej/repos/Enzyme/enzyme/build/Enzyme/CMakeFiles/LLVMEnzyme-6.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build.make Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
make[3]: Nothing to be done for 'Enzyme/CMakeFiles/LLVMEnzyme-6.dir/build'.
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
[ 90%] Built target LLVMEnzyme-6
make -f test/Enzyme/CMakeFiles/check-enzyme.dir/build.make test/Enzyme/CMakeFiles/check-enzyme.dir/depend
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
cd /home/ondrej/repos/Enzyme/enzyme/build && /home/ondrej/ext/cmake-3.16.2-Linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/ondrej/repos/Enzyme/enzyme /home/ondrej/repos/Enzyme/enzyme/test/Enzyme /home/ondrej/repos/Enzyme/enzyme/build /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme/CMakeFiles/check-enzyme.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
make -f test/Enzyme/CMakeFiles/check-enzyme.dir/build.make test/Enzyme/CMakeFiles/check-enzyme.dir/build
make[3]: Entering directory '/home/ondrej/repos/Enzyme/enzyme/build'
[100%] Running enzyme regression tests
cd /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme && ../../ -v /home/ondrej/repos/Enzyme/enzyme/build/test/Enzyme
/bin/sh: 1: ../../: Permission denied
test/Enzyme/CMakeFiles/check-enzyme.dir/build.make:57: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme' failed
make[3]: *** [test/Enzyme/CMakeFiles/check-enzyme] Error 126
make[3]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
CMakeFiles/Makefile2:269: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/all' failed
make[2]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/all] Error 2
make[2]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
CMakeFiles/Makefile2:276: recipe for target 'test/Enzyme/CMakeFiles/check-enzyme.dir/rule' failed
make[1]: *** [test/Enzyme/CMakeFiles/check-enzyme.dir/rule] Error 2
make[1]: Leaving directory '/home/ondrej/repos/Enzyme/enzyme/build'
Makefile:190: recipe for target 'check-enzyme' failed
make: *** [check-enzyme] Error 2

wsmoses · 2020-03-27T20:56:25Z

Ah, it wasn't able to find llvm-lit.

Do you see a file at the following location or somewhere similar on your machine: /usr/lib/llvm-6/build/utils/lit/lit.py

If so, add this to your cmake (with the appropriate path):

-DLLVM_EXTERNAL_LIT=/usr/lib/llvm-6/build/utils/lit/lit.py

certik · 2020-03-27T21:05:34Z

Unfortunately I don't have any utils subdirectory nor any lit.py. I can see it is some kind of a testing tool: https://github.com/llvm-mirror/llvm/tree/2c4ca6832fa6b306ee6a7010bfb80a3f2596f824/utils/lit

wsmoses · 2020-03-27T21:09:08Z

Assuming you're on ubuntu, can you try installing llvm-6-tools and llvm-6-dev ?

certik · 2020-03-27T21:19:46Z

That worked. The packages are called llvm-6.0-tools and llvm-6.0-dev. I use Ubuntu 18.04. Here is the cmake command:

cmake -DLLVM_EXTERNAL_LIT=/usr/lib/llvm-6.0/build/utils/lit/lit.py ..

and I uploaded the test results here: https://gist.github.com/certik/f3202b6bc907481e460084cc19ff9905.

Some pass and some fail.

wsmoses · 2020-03-27T21:29:28Z

That makes sense, we stopped testing against LLVM 6 ("officially supporting" / testing versions 7-9) because it wasn't possible to write some tests in a way that was compatible for LLVM 6 and later versions (since there was a change to memcpy where it took an extra argument and a few other things). That said, functionally Enzyme should still work on LLVM 6 (since it was just hard to write tests that were compatible with all versions of LLVM) and skimming through your output that seems to be the case.

certik · 2020-03-27T21:41:57Z

I can upgrade, I only used LLVM 6 since it was already installed. I installed llvm-9-tools and llvm-9-dev and reconfigured:

$ cmake -DLLVM_EXTERNAL_LIT=/usr/lib/llvm-9/build/utils/lit/lit.py ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
LLVM_SHLIBEXT=.so
found llvm dir /home/ondrej/repos/Enzyme/enzyme/build
found llvm lit /usr/lib/llvm-9/build/utils/lit/lit.py
CMAKE_PREFIX_PATH /home/ondrej/repos/Enzyme/enzyme/build
CMake Error at /usr/lib/llvm-9/lib/cmake/llvm/LLVMExports.cmake:1323 (message):
  The imported target "yaml-bench" references the file

     "/usr/lib/llvm-9/bin/yaml-bench"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/usr/lib/llvm-9/lib/cmake/llvm/LLVMExports.cmake"

  but not all the files it references.

Call Stack (most recent call first):
  /usr/lib/llvm-9/cmake/LLVMConfig.cmake:245 (include)
  CMakeLists.txt:42 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/ondrej/repos/Enzyme/enzyme/build/CMakeFiles/CMakeOutput.log".

It could be that LLVM 9 is broken in Ubuntu. So I installed LLVM 8 and reconfigured with:

cmake -DLLVM_DIR=/usr/lib/llvm-8 -DLLVM_EXTERNAL_LIT=/usr/lib/llvm-8/build/utils/lit/lit.py ..

and that seems to work. Looks like there is only one failure now: https://gist.github.com/certik/3c0482cf7886b92d293342ee41b2e82c

Update: that failure was caused by myself, I modified one of the tests to play with it. When I removed my modifications, everything passes:

Testing Time: 1.71s
  Expected Passes    : 130
  Expected Failures  : 8

wsmoses · 2020-03-27T21:55:11Z

Ah excellent, yeah that failure seemed odd to me, but glad to see it was resolved!

certik · 2020-03-27T21:59:34Z

Thanks!

certik closed this as completed Mar 27, 2020

ZuseZ4 mentioned this issue Sep 5, 2022

Assertion `addingType' failed. #827

Closed

FROL256 mentioned this issue Jan 17, 2024

Bugreport for complex codebase (probably some math problem) #1613

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to run tests #38

How to run tests #38

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020 •

edited

Loading

certik commented Mar 27, 2020 •

edited

Loading

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020 •

edited

Loading

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

How to run tests #38

How to run tests #38

Comments

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020 • edited Loading

certik commented Mar 27, 2020 • edited Loading

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020 • edited Loading

wsmoses commented Mar 27, 2020

certik commented Mar 27, 2020

wsmoses commented Mar 27, 2020 •

edited

Loading

certik commented Mar 27, 2020 •

edited

Loading

certik commented Mar 27, 2020 •

edited

Loading