Python symbols are resolved at link-time instead of run-time

[olofk]

I'm adding cocotb support for the verilator backend in Edalize, using the excellent https://docs.cocotb.org/en/stable/custom_flows.html guide.

With this, however, I'm hitting issues of unresolved symbols during link-time, like:

/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: /home/olof/.local/lib/python3.11/site-packages/cocotb/libs/libcocotb.so: undefined reference to `PyObject_CallFunctionObjArgs'
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: /home/olof/.local/lib/python3.11/site-packages/cocotb/libs/libcocotb.so: undefined reference to `PyGILState_Ensure'
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: /home/olof/.local/lib/python3.11/site-packages/cocotb/libs/libcocotb.so: undefined reference to `Py_Initialize'
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: /home/olof/.local/lib/python3.11/site-packages/cocotb/libs/libcocotb.so: undefined reference to `PyErr_Print'
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: /home/olof/.local/lib/python3.11/site-packages/cocotb/libs/libcocotb.so: undefined reference to `Py_IsInitialized'

As I understand it, the cocotb libs are loaded at runtime, so we shouldn't need to bother checking this at link-time. Adding -Wl,--unresolved-symbols=ignore-all to LDFLAGS seems to get around this, but I'm not sure it's the best solution. I guess there is some common way to do this for projects that load stuff at runtime, but it's a bit outside of my wheelhouse.

[ktbarrett]

I think -Wl,--unreoslved-symbols=ignore-in-shared-libs would work better. Ideally we could constrain that to only the symbols in libpython, but I guess the larger question is why you would be running into this problem? The libpython you dynamically link against should be available at compile time for the linker to check to see if there are unresolved symbols so we don't have to do this.

For background, we use LIBPYTHON_LOC to dynamically load a specific libpython as we can't trust LD_LIBRARY_PATH: many simulators muck with it. embed.cpp in this repo is a stub that dynamically loads that libpython before dynamically entering code that is linked against it so we can avoid loading libpython by dependency before we load the one that we want.

[olofk]

That -Wl,--unreoslved-symbols=ignore-in-shared-libs option seems indeed better, but I also agree that it's strange that I'm eccountering this issue in the first place. I'm on Gentoo, so that could perhaps hint at some differences in the environment, but I had a bug report from another Edalize user who encountered the same issue, and I suspect it's a low chance they are also a Gentoo user.

I guess this bug report technically isn't a cocotb issue, but I'm still curious to know if other users are hitting the same issue or if there's some magic in the cocotb makefiles that I haven't seen that addresses this.

[ktbarrett]

I'm still curious to know if other users are hitting the same issue

This is the first I'm hearing about it. We can keep this issue open to potentially gather other users having issues.

Have you set up /etc/ld.so.conf to include the directory your libpython is in and then run ldconfig?

[olofk]

Yep.

lrwxrwxrwx 1 root root      20 apr 12 19:53 libpython3.11.so -> libpython3.11.so.1.0
-rwxr-xr-x 1 root root 4968480 apr 12 19:54 libpython3.11.so.1.0
lrwxrwxrwx 1 root root      20 apr 12 19:58 libpython3.12.so -> libpython3.12.so.1.0
-rwxr-xr-x 1 root root 6220128 apr 12 19:58 libpython3.12.so.1.0

and that dir is /etc/ld.so.conf. The more I look at this, the less I understand why this is working on other systems without explicitly passing -lpython

[ktbarrett]

And why distutils doesn't add -lpython itself considering we are using its Extension class.

Perhaps this is something cocotb can do. We would need to compute the libpython name (includes the Python version) and add that to the libraries argument of Extension for those libraries that depend upon Python.

[mballance]

I've noticed that newer Python versions do not link extensions against libpython. While this should make extensions a bit more flexible (ideally, not strictly tied to a single Python version), not doing so also introduces some challenges. One approach you could take with Verilator is to explicitly link the simulation-image binary against libpython. I suppose the good news is that cocotb-config makes it easy to identify libpython. On a related note, is there any interest in enhancing the Verilator runtime 'main' to support dynamically loading a PLI/VPI library? The goal would be to enable cocotb to be directly loaded by a simulation image produced by `verilator --binary`, and remove the need for cocotb to develop and maintain verilator.cpp.

…

-Matthew

On Thu, May 16, 2024 at 5:07 PM Kaleb Barrett ***@***.***> wrote: And why distutils doesn't add -lpython itself considering we are using its Extension class. Perhaps this is something cocotb can do. We would need to compute the libpython name (includes the Python version) and add that to the libraries argument of Extension for those libraries that depend upon Python. — Reply to this email directly, view it on GitHub <#3892 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKHLBPA24EECDYMLJZG6LLZCVC33AVCNFSM6AAAAABHTVQU6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJWGQYDEOBZGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[olofk]

Hmm.. shouldn't we simply add a cocotb-config --ldflags that returns basically -Wl,-rpath,$(output of cocotb-config --lib-dir) -L$(output of cocotb-config --lib-dir) -lcocotbvpi_verilator -lgpi -lcocotb -lgpilog -lcocotbutils -lpython$(correct version identified by find_libpython) ?

[ktbarrett]

@olofk I don't think anyone is manually building cocotb sources, so I don't understand the point of that.

There might be an assumption here that wasn't communicated. What are you building where it's failing to link? I'm thinking you're talking about using pip install ... to build a development version of cocotb. Are you actually talking about linking libcocotbvpi_verilator against the verilated binary?

[olofk]

Ah no. I'm using a pip-installed 1.8.1 This is for having Edalize launch simulations with cocotb. Basically doing https://docs.cocotb.org/en/stable/custom_flows.html#verilator . But my problem is that these instructions don't work for me unless I ignore unresolved symbols or explicitly pass -lpython3.11 in the LDFLAGS.

I'm attaching an example build directory that was created by fusesoc run my FuseSoC+cocotb example. In the attached archive I have added -Wl,--unresolved-symbols=ignore-in-shared-libsbut it would be very interesting to see if other people can actually make this build with that option removed.

The idea with adding a --ldflags option to cocotb-config was more to avoid passing all these flags manually to Verilator.
fuseocotb.tar.gz

[ktbarrett]

Ah. I thought it was weird you were linking all those libraries in your project. The libraries were rearchitected to avoid this problem, but the custom flows docs weren't updated.

See what's actually linked here.

[olofk]

Yes! That works just fine. So the problem was actually overlinking rather than underlinking :) PR for docs here #3901 Closing this now.