Undefined symbols for __enzyme_autodiff(...)

[chrislupp]

Hi,

I've been trying to assess how to use Enzyme together with more complex codes (that use Eigen) and created a pretty simple C++ test program:

#include <iostream>

#include <Eigen/Dense>
using Eigen::VectorXd;

void __enzyme_autodiff(...);


double foo( VectorXd v )
{
    double out = 0.0;

    for( int i = 0 ; i < v.rows() ; i++ )
        out += v(i) * v(i);

    return out;
}



int main()
{
    size_t n = 3;

    VectorXd v(n);
    VectorXd dv(n);

    for( int i = 0; i < n ; i++)
    {
        v(i) = i;
        dv(i) = 0.0;
    }

#ifdef __clang__
    __enzyme_autodiff(foo, enzyme_dup, &v, &dv);
#endif

    std::cout << foo(v) << std::endl << std::endl;

    std::cout << dv << std::endl;


    return 0;
}

I've been compiling it with the following:

clang++ test.cpp -I/path/to/eigen -S -emit-llvm -o input.ll -O3 -fno-vectorize -fno-slp-vectorize -fno-unroll-loops
opt input.ll -load=ClangEnzyme-11.dylib -enzyme -o output.ll -S
clang++ output.ll -O3 -o matrix

I've tried to base all of the above on other examples (some posted as issues here). Several of the other examples compile and link properly. Unfortunately, it seems that the autodiffed function for my example never makes it into the executable, because I get a missing symbol error on linking:

Undefined symbols for architecture x86_64:
  "__enzyme_autodiff(...)", referenced from:
      _main in output-c75457.o
ld: symbol(s) not found for architecture x86_64

That's kind of perplexing to me, because other examples I build don't seem to have this issue. I've also tried more explicit declarations of the autodiff functions, but no luck. Am I missing something fundamental here, or are there any pointers you could give me to resolve this. Thanks!

[wsmoses]

What version of llvm/clang are you using?

[chrislupp]

sorry, forgot to add that. Clang 11.1.0

[wsmoses]

No worries, ok that rules out the new pass manager causing disruption.

What version of Enzyme did you use (and how did you build/install it)?

Also which clang are you using (apple installed version, homebrew, or something distinct)? Just to double check, mind pasting the output of clang++ --version

[chrislupp]

sorry for the delay in responding, I've only been working on this in the evenings.

I believe that I am using the apple installed clang version. The output from clang++ --version is:

clang version 11.1.0
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin

I am using Enzyme 0.0.9 (I installed it a few weeks ago and haven't updated since). I installed it via homebrew on macOS Big Sur.

[chrislupp]

A brief update, I've updated enzyme to 0.0.11 and clang 12.0.0. And I'm pretty sure it's clang via homebrew not apple's version. Unfortunately, the missing symbol issue is still happening post update.

[wsmoses]

Hi Chris,

Do you mind trying to add -lowerinvoke prior to calling Enzyme within opt, then trying again?

opt input.ll -lowerinvoke -load= ClangEnzyme-11.dylib -enzyme -o output.ll -S

[wsmoses]

Also note that it appears homebrew has updated to LLVM 12, but has the default Enzyme install on LLVM 11, so you may also need to be explicit in using the right Clang (I'll update homebrew to keep enzyme at 12 as well shortly).

$ /usr/local/Cellar/llvm\@11/11.1.0_1/bin/clang++ test.cpp -I /usr//local/Cellar/eigen/3.3.9/include/eigen3 -S -emit-llvm -o input.ll -O3 -fno-vectorize -fno-slp-vectorize -fno-unroll-loops
$ /usr/local/Cellar/llvm\@11/11.1.0_1/bin/opt input.ll -lowerinvoke -load=LLVMEnzyme-11.dylib -enzyme -o output.ll -S
$ /usr/local/Cellar/llvm\@11/11.1.0_1/bin/clang++ output.ll -O3 -o matrix
$ ./matrix
5

0
2
4

[chrislupp]

Thanks. The use of -lowerinvoke solved the missing symbols issue.

It might be worth noting that I switched to a docker image that I created as an Enzyme playground. Not sure if you think something like that is useful for others to experiment around with. But if so, I could try to find a way to share it (dockerfile pull request or an image on dockerhub).

[wsmoses]

Following up here quickly (apologies on the delay as had a paper hackathon this weekend). The lowerinvoke should now no longer be needed regardless.

As do your docker playground that would definitely be useful!

If you build one, would you mind adding it to the website (the www branch of this repo)?

[chrislupp]

No problem, know the feeling regarding papers. I just wanted to close the issue as it wasn't a problem any more.

Yeah, I can add a brief description/instructions to the website (aside from putting an image on dockerhub). But I will have to check if I have to run the dockerfile/image and the text for the website through the release process at work first. So it could potentially take a few weeks.

[chrislupp]

Not sure if you use Docker, but I just pushed an image to Docker Hub set up with Enzyme for C/C++. It can be found under chrislupp/enzyme-playground. I'm putting together a brief guide for the website and will file a pull request when done.