Does multisource work? #1462

davidedelvento · 2023-10-03T21:48:05Z

I am unable to get multisource work with current version of Enzyme, or finding any up-to-date documentation about it.

For older versions, one could do what's described at https://github.com/EnzymeAD/Enzyme-Tutorial/blob/main/9_multisource/Makefile but that does not work anymore as described at EnzymeAD/Enzyme-Tutorial#6

Is it something which is currently broken, or is it just missing from the documentation? Either way, can it be fixed?

TIA!

ZuseZ4 · 2023-10-03T22:08:15Z

Hi. Could you please give the concrete error messages you are reading and your compiler version, is it build from source? MultisourceAD works for C++, but if you have functions spanning multiple compilation units (in general .cpp files), you will need to use LDEnzyme, not ClangEnzyme. Depending on your system the docs might need some updates, but for the projects I work on it works, so without more information unfortunately I can't say whats the issue

ZuseZ4 · 2023-10-03T22:13:56Z

Ah my bad, I saw you opened an issuenon the tutorial repo.
As a rule of thumb, ClangEnzyme works as a clangPlugin on a single .cpp file, LLVMEnzyme works as a opt plugin on a single .bc or .ll file and LLDEnzyme works as an LLD (not ld!) plugin on multiple files after they have been merged using lto. This Info should probably go into the docs, please feel free to update them if you have some time. I did fix the cmake template a while ago to acount for the LLDEnyzme, since it is the one I mostly use.
Here is the link to the PR: EnzymeAD/CMake-Template#2

davidedelvento · 2023-10-03T22:48:37Z

Many thanks @ZuseZ4

Unfortunately that still does not work.

I barely changed the Makefile to adapt it to my environment:

[/home/davide 9_multisource]$ git diff --word-diff Makefile
diff --git a/9_multisource/Makefile b/9_multisource/Makefile
index d0bbe84..1020812 100644
--- a/9_multisource/Makefile
+++ b/9_multisource/Makefile
@@ -4,10 +4,10 @@ clean:
        rm -f *.o *.ll *.exe

%.o: %.c
        [-../dockerscript.sh clang-12-]{+clang+} -c -fuse-ld=lld -flto [-/host/$^-]{+$^+} -O2 -ffast-math -o [-/host/$@-]{+$@+}

lib.exe: myblas.o multisource.o
        [-../dockerscript.sh clang-12-]{+clang+} -fuse-ld=lld -flto [-/host/myblas.o /host/multisource.o-]{+myblas.o multisource.o+} -O2 -ffast-math -o [-/host/$@ -Wl,-mllvm=-load=/Enzyme/enzyme/build/Enzyme/LLDEnzyme-12.so-]{+$@ -v -Wl,-mllvm=-load=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so+}

run-%: %.exe
        [-../dockerscript.sh /host/$^-]{+./$^+} 3

(I added the -v per error message suggestion) and then trying to utilize it fails with undefined symbol: __enzyme_autodiff

[/home/davide 9_multisource]$ make clean && make
rm -f *.o *.ll *.exe
clang -c -fuse-ld=lld -flto myblas.c -O2 -ffast-math -o myblas.o
clang-16: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
clang -c -fuse-ld=lld -flto multisource.c -O2 -ffast-math -o multisource.o
clang-16: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
clang -fuse-ld=lld -flto myblas.o multisource.o -O2 -ffast-math -o lib.exe -v -Wl,-mllvm=-load=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so
clang version 16.0.2
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-8.5.0/llvm-16.0.2-tuhjitgwjb27yyj5oloa4aeqgmmsy6zx/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
 "/home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-8.5.0/llvm-16.0.2-tuhjitgwjb27yyj5oloa4aeqgmmsy6zx/bin/ld.lld" -pie --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o lib.exe /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/Scrt1.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/8/crtbeginS.o /usr/lib/gcc/x86_64-redhat-linux/8/crtfastmath.o -L/home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-8.5.0/llvm-16.0.2-tuhjitgwjb27yyj5oloa4aeqgmmsy6zx/bin/../lib/x86_64-unknown-linux-gnu -L/usr/lib/gcc/x86_64-redhat-linux/8 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/lib -L/usr/lib -plugin-opt=mcpu=x86-64 -plugin-opt=O2 -L/home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-8.5.0/llvm-16.0.2-tuhjitgwjb27yyj5oloa4aeqgmmsy6zx/lib myblas.o multisource.o -mllvm=-load=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/8/crtendS.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
ld.lld: error: undefined symbol: __enzyme_autodiff
>>> referenced by ld-temp.o
>>>               lto.tmp:(main)
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:10: lib.exe] Error 1

As you can see I am using spack for some of the installs, but that's irrelevant (those installation are solid and I use them for many other things with no problems). The only thing that I'm uncertain is the picking up of the (system-provided) install of gcc. Is that what one would expect?

Finally, perhaps useless information and normal way in which Enzyme works, since it needs to do some "magic" to create those missing symbols, but in case it's helpful

[/home/davide 9_multisource]$ nm ./../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so | grep enzyme_autodiff
[/home/davide 9_multisource]$ nm ./../Enzyme/enzyme/build/Enzyme/*.so | grep enzyme_autodiff
[/home/davide 9_multisource]$

wsmoses · 2023-10-06T16:20:37Z

Duplicate of #1389 (comment)

Unfortunately upstream LLVM 16+ dropped support for the old pass manager, and doesn't have support for plugins in the new pass manager yet. If you use LLVM < that version LLDEnzyme should be fine.

Upstream issue in LLVM: llvm/llvm-project#65085 (comment)

wsmoses · 2023-10-06T16:21:08Z

Oh sorry to clarify, what OS are you on? The above comment only applies to macos. Linux should be fine.

davidedelvento · 2023-10-06T16:24:19Z

The example above is on Linux RH8.8 and as you see it's not fine. I probably made some mistake myself in compiling or utilizing Enzyme, but I can't figure out what.

I have LLVM 14 installed and I could recompile Enzyme against that, but I was hoping to use a newer version (I don't have LLVM 15 installed).

wsmoses · 2023-10-06T16:26:34Z

Can you run test-cmake on your Enzmye build. That will test ClangEnzyme and LLDEnzyme (multisource). Does that fail or suceed?

davidedelvento · 2023-10-06T16:41:35Z

Sure, happy to run those tests, but which ones?

Do you mean run ninja check-enzyme? I don't have any test-cmake binary in my installation

Thanks!

wsmoses · 2023-10-06T16:48:37Z

ninja test-cmake (

Enzyme/enzyme/test/CMakeLists.txt

Line 16 in 07f3e71

    
           add_custom_target(test-cmake COMMAND rm -rf ${CMAKE_CURRENT_BINARY_DIR}/test_cmake && mkdir ${CMAKE_CURRENT_BINARY_DIR}/test_cmake && cd ${CMAKE_CURRENT_BINARY_DIR}/test_cmake &&  ${CMAKE_COMMAND} -S ${CMAKE_CURRENT_BINARY_DIR}/test_cmake ${CMAKE_CURRENT_SOURCE_DIR}/test_find_package -DEnzyme_DIR=${CMAKE_BINARY_DIR} -DCMAKE_C_COMPILER=$<TARGET_FILE:clang> && ${CMAKE_COMMAND} --build ${CMAKE_CURRENT_BINARY_DIR}/test_cmake DEPENDS ClangEnzymeFlags LLDEnzymeFlags)

)

davidedelvento · 2023-10-06T17:08:29Z

Just this?

[0/2] Re-checking globbed directories...
[3/3] cd /home/davide/repositories/ENZYME/Enzyme/enzyme/build-v0.0.86-with-llvm-v16.0..../home/davide/repositories/ENZYME/Enzyme/enzyme/build-v0.0.86-with-llvm-v16.0.2/test/test_cmake
-- The C compiler identification is Clang 16.0.2
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/sw/spack-v0.20.1-b/opt/spack/linux-rhel8-icelake/gcc-8.5.0/llvm-16.0.2-tuhjitgwjb27yyj5oloa4aeqgmmsy6zx/bin/clang-16 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
found dir /home/davide/repositories/ENZYME/Enzyme/enzyme/build-v0.0.86-with-llvm-v16.0.2
found 1
imported targets
-- Configuring done
-- Generating done
-- Build files have been written to: /home/davide/repositories/ENZYME/Enzyme/enzyme/build-v0.0.86-with-llvm-v16.0.2/test/test_cmake
[ 25%] Building C object CMakeFiles/enzyme_test_1.dir/main.c.o
[ 50%] Linking C executable enzyme_test_1
[ 50%] Built target enzyme_test_1
[ 75%] Building C object CMakeFiles/enzyme_test_2.dir/main.c.o
[100%] Linking C executable enzyme_test_2
[100%] Built target enzyme_test_2

tgymnich · 2023-10-06T17:17:40Z

I am using LLVM 14 with LTO. Here is a snippet from my CMake config:

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -flto")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -Wl,--lto-legacy-pass-manager,--disable-new-dtags,-mllvm=-load=/local/Enzyme/build/Enzyme/LLDEnzyme-14.so,-mllvm=-enzyme-attributor=0")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld -Wl,--lto-legacy-pass-manager,--disable-new-dtags,-mllvm=-load=/local/Enzyme/build/Enzyme/LLDEnzyme-14.so,-mllvm=-enzyme-attributor=0")

wsmoses · 2023-10-06T17:19:47Z

Ok the fact that the previous test worked means that lldenzyme successfully compiled a multi source program on your system. Can you make a zip of your mini repo and upload it?

…

On Fri, Oct 6, 2023 at 12:17 PM Tim Gymnich ***@***.***> wrote: I am using LLVM 14 with LTO. Here is a snippet from my CMake config: set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -Wl,--lto-legacy-pass-manager,--disable-new-dtags,-mllvm=-load=/local/Enzyme/build/Enzyme/LLDEnzyme-14.so,-mllvm=-enzyme-attributor=0,-mllvm=-enzyme-max-type-offset=25,-mllvm=-enzyme-max-type-depth=2,-mllvm=-enzyme-max-int-offset=10,-mllvm=-enzyme-loose-types=false")set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld -Wl,--lto-legacy-pass-manager,--disable-new-dtags,-mllvm=-load=/local/Enzyme/build/Enzyme/LLDEnzyme-14.so,-mllvm=-enzyme-attributor=0,-mllvm=-enzyme-max-type-offset=25,-mllvm=-enzyme-max-type-depth=2,-mllvm=-enzyme-max-int-offset=10,-mllvm=-enzyme-loose-types=false") — Reply to this email directly, view it on GitHub <#1462 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTUXG5MG2KP7JMTZEVDH3X6A4L5AVCNFSM6AAAAAA5RUHTROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRGEZTSNJZGQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

davidedelvento · 2023-10-06T17:39:56Z

@tgymnich

I am using LLVM 14 with LTO. Here is a snippet from my CMake config

Thanks. The --lto-legacy-pass-manager is unknown argument to v16 of LLD. Removing that and leaving all the rest causes the same issue

@wsmoses

Ok the fact that the previous test worked means that lldenzyme successfully compiled a multi source program on your system.

Fantastic, where is it? Looking at enzyme_test_1.dir (and 2) they look single-source to me.

Can you make a zip of your mini repo and upload it?

Sure I can, but

What do you mean by "mini-repo"?
For the teaching to fish rather than giving a fish series, perhaps I should/could look on that test that worked and try to figure it out myself what I am doing differently/wrong?

Thanks again!

wsmoses · 2023-10-06T17:40:30Z

Ah I see the problem. Since you're manually adding the flags to your Makefile you're missing some flags that depend on LLVM Version (these are automatically added with LLDEnzymeFlags in the cmake template here:

Enzyme/enzyme/Enzyme/CMakeLists.txt

Line 165 in 07f3e71

set(LTO_LEGACY_MANAGER_VERSIONS 13 14)

)

Can you also add -Wl,-mllvm=-load-pass-plugin=/Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so

davidedelvento · 2023-10-06T17:59:50Z

My Makefile (diff from the original in a comment above, perhaps the --word-diff makes it poorly readable) does have a -Wl,-mllvm=-load=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so already. I tried adding or replacing it with the slightly different -load-pass-plugin both prefixing it with comma as you pasted and with space as some other people mentioned that some versions of LLVM want.

In all cases it always fail with

ld.lld: error: -mllvm: ld.lld: Unknown command line argument '-load-pass-plugin=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so'.  Try: 'ld.lld --help'
ld.lld: Did you mean '--dfa-instr-limit=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so'?
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:12: lib.exe] Error 1

for reference, this is

$ ld.lld --version
LLD 16.0.2 (compatible with GNU linkers)

wsmoses · 2023-10-06T18:56:10Z

Sorry can you try with two dashes (-Wl,-mllvm=--load-pass-plugin=/Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so

davidedelvento · 2023-10-06T20:10:30Z

Thanks for keep trying, and for your patience with me, really appreciated.

That didn't work either, but I digging into lld documentation I found the correct syntax which fixed the issue. That is

-Wl,--load-pass-plugin=../../Enzyme/enzyme/build/Enzyme/LLDEnzyme-16.so

For other poor souls struggling with the same problem, and landing on this issue in the future, do note the absence of -mllvm=

As far as I'm concerned, this can be closed. I'm not closing it myself in case we want to keep it around as a reminder to put this into the documentation, as @ZuseZ4 suggested. I'll try to find time to make such a PR myself.

ZuseZ4 · 2024-01-31T23:36:57Z

@davidedelvento Ping in case you have some time to update the docs to cover your issue.

davidedelvento · 2024-02-01T00:15:53Z

Sorry this fell through the cracks. Yes, I will update the docs from https://github.com/EnzymeAD/www/blob/www/content/getting_started/ making sure they work with LLVM 16 and make a PR for needed changes. Let me know if you intended something different instead

ZuseZ4 · 2024-02-01T00:21:29Z

Awesome, thanks! Nothing else, I mostly hope that step by step we will get to cover all Enzyme+LLVM versions.

davidedelvento · 2024-02-01T00:30:54Z

we will get to cover all Enzyme+LLVM versions.

So you'd prefer me leaving what is there already for LLVM 15 and perhaps with a prefix or tab option selecting version having the LLVM 16 in addition not instead, correct?

ZuseZ4 · 2024-02-01T00:34:39Z

Yes, we officially support LLVM 11-18, so please don't delete comments for older versions.
Just add your note and the version for which it works (as far as you know).

davidedelvento · 2024-02-01T00:50:22Z

Sounds good. The nice tab-based switching among languages seems more complicated than I hoped in hugo so I will just make regular code sections and leave the code-toggle to someone who knows their way in hugo better than I

ZuseZ4 · 2024-02-01T00:55:11Z

Yes no worries, any improvement is welcome. We had been working on a completely new website for Enzyme Core using a newer template, but no one had the time to finish that. Please also assign me or wsmoses as a reviewer, to make sure that the PR doesn't get lost.

davidedelvento · 2024-02-03T04:33:59Z

@ZuseZ4 perhaps you have been already notified by either the PR itself or the automatic comment/mention in this thread, but I guess it can't hurt to further ping you about it explicitly.

Hopefully EnzymeAD/www#23 is a decent enough improvement.

Thanks again to everybody for maintaining such a great project!

davidedelvento · 2024-02-13T23:16:38Z

This is resolved and the above-mentioned documentation hopefully will help others from stumbling on the same block.

wsmoses closed this as not planned Won't fix, can't repro, duplicate, stale Oct 6, 2023

wsmoses reopened this Oct 6, 2023

davidedelvento mentioned this issue Feb 3, 2024

Llvm16 pr EnzymeAD/www#23

Open

davidedelvento closed this as completed Feb 13, 2024

MilesCranmer pushed a commit to MilesCranmer/Enzyme that referenced this issue Jul 24, 2024

Fix unused shadow return type (EnzymeAD#1462)

d230f66

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does multisource work? #1462

Does multisource work? #1462

davidedelvento commented Oct 3, 2023

ZuseZ4 commented Oct 3, 2023

ZuseZ4 commented Oct 3, 2023 •

edited

Loading

davidedelvento commented Oct 3, 2023 •

edited

Loading

wsmoses commented Oct 6, 2023

wsmoses commented Oct 6, 2023 •

edited

Loading

davidedelvento commented Oct 6, 2023 •

edited

Loading

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

tgymnich commented Oct 6, 2023 •

edited

Loading

wsmoses commented Oct 6, 2023 via email

davidedelvento commented Oct 6, 2023 •

edited

Loading

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

ZuseZ4 commented Jan 31, 2024

davidedelvento commented Feb 1, 2024

ZuseZ4 commented Feb 1, 2024

davidedelvento commented Feb 1, 2024

ZuseZ4 commented Feb 1, 2024

davidedelvento commented Feb 1, 2024

ZuseZ4 commented Feb 1, 2024 •

edited

Loading

davidedelvento commented Feb 3, 2024

davidedelvento commented Feb 13, 2024

Does multisource work? #1462

Does multisource work? #1462

Comments

davidedelvento commented Oct 3, 2023

ZuseZ4 commented Oct 3, 2023

ZuseZ4 commented Oct 3, 2023 • edited Loading

davidedelvento commented Oct 3, 2023 • edited Loading

wsmoses commented Oct 6, 2023

wsmoses commented Oct 6, 2023 • edited Loading

davidedelvento commented Oct 6, 2023 • edited Loading

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

tgymnich commented Oct 6, 2023 • edited Loading

wsmoses commented Oct 6, 2023 via email

davidedelvento commented Oct 6, 2023 • edited Loading

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

wsmoses commented Oct 6, 2023

davidedelvento commented Oct 6, 2023

ZuseZ4 commented Jan 31, 2024

davidedelvento commented Feb 1, 2024

ZuseZ4 commented Feb 1, 2024

davidedelvento commented Feb 1, 2024

ZuseZ4 commented Feb 1, 2024

davidedelvento commented Feb 1, 2024

ZuseZ4 commented Feb 1, 2024 • edited Loading

davidedelvento commented Feb 3, 2024

davidedelvento commented Feb 13, 2024

ZuseZ4 commented Oct 3, 2023 •

edited

Loading

davidedelvento commented Oct 3, 2023 •

edited

Loading

wsmoses commented Oct 6, 2023 •

edited

Loading

davidedelvento commented Oct 6, 2023 •

edited

Loading

tgymnich commented Oct 6, 2023 •

edited

Loading

davidedelvento commented Oct 6, 2023 •

edited

Loading

ZuseZ4 commented Feb 1, 2024 •

edited

Loading