Unable to pass python callback to C++ #175
Comments
|
Yes, looks like the same bug. My problem remains reproducing the error. Do you have more details, e.g. platform, compiler version, etc.? |
|
A few updates. First, the bug persists even after installing Second, I'm using Arch Linux with both gcc and clang installed: Details hereAnything else? |
|
@wlav, I found a related issue root-project/root#12294 (comment) BTW, is it possible to point cling to a custom toolchain, system and STL headers, and compiled objects? |
|
The fix for root#12294 is already included in the release, so that can't be it. For the toolchain, in principle yes: Cling picks up the compiler paths from the system compiler, so either change |
|
@wlav not sure if it helps, but I was testing a different variant from the example @dyershov provided. Error traceLine 19 seems weird. Tmpl_name="<" ? Currently using: |
|
Nice, a fully reproducible crash. :) Thanks! Fixed in repo. (Problem wasn't that traceback line 19, but the code assuming a string type while it had |
|
Now I'm getting a different error. Shouldn't the Python int have been converted to C++ int before getting to the following prototype? Not sure when do these conversions happen, so I'm guessing. |
|
Interesting, as I didn't (and don't) get that result. However, I can see how that could happen. Presumed fix is in repo. |
|
Prototype error went away, but the TypeErrors persist. |
|
Right, because of |
|
Hmm. Testing a bit more, found something that may be of interest. Running the same example: Tried getting the generated code for that function wrapper that failed to be materialized. Generated codeThen defined that manually. Python codeIt still fails the exact same way (expected), but the symbol materialization failure list is different. When it is on a different cppdef call, it prints the original list. |
|
Thought it would be nice to call the wrapper function directly and got the following. GLIBCXX_USE_CXX11_ABI shenanigans? |
|
I can reproduce this to differing extents in ubuntu and fedora docker containers. The ubuntu one is set up as follows: # docker run -t -i --rm ubuntu bash
apt-get update && apt-get install -y python3 python3-pip python3-dev build-essential wget
python3 -V
pip3 -V
pip3 install cppyyNow, running the examples from above and my Observer code from #189 works without warnings, but loading ogdf-python yields a similar problem: On fedora the issue is easier to reproduce # docker run -t -i --rm fedora bash
sudo dnf install -y python3 python3-pip python3-devel gcc g++ cmake wget
python3 -V
pip3 -V
pip3 install cppyy
mv /usr/local/lib64/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so /usr/local/lib/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so # fix #191
python3 -c "import cppyy" # worksThe example from above here yields: >>> call_int_int(add, 3, 7)
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap1Eii }) }
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: int ::call_int_int(int(*)(int,int) f, int i1, int i2) =>
TypeError: could not convert argument 1Running my Observer code: # wget https://gist.githubusercontent.com/N-Coder/c1fafdd5ff2aae1a134852416e9e3587/raw/f7189e70f5df36cba36cfc6da36332c965ddc125/test.cpp
# python3
Python 3.11.5 (main, Aug 28 2023, 00:00:00) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cppyy import *
>>> import cppyy.gbl as c
>>>
>>> include("test.cpp")
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { $.cling-module-142.__inits.0, _ZN10MyObserverD1Ev, _ZN10MyObserverD0Ev, _ZTV10MyObserver, _ZN8Observer7observeEP10Observable, _ZN10MyObserver12onRegisteredEP10Observable, _ZTI10MyObserver, long_lived_obs, _ZN8ObserverD1Ev, _ZN8Observer12onRegisteredEP10Observable, __orc_init_func.cling-module-142, _ZNSt7__cxx114listIP8ObserverSaIS2_EED1Ev, _GLOBAL__sub_I_cling_module_142, _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag, _ZTS10MyObserver, _ZN10MyObserverD2Ev, _ZTI8Observer, _ZTS8Observer, _ZTV8Observer, main, _ZN10MyObserverC2Ev, _ZN8ObserverD2Ev, _ZN8ObserverD0Ev, __cxx_global_var_initcling_module_142_, _ZN10MyObserver11onThingDoneERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE, __clang_call_terminate }) }
True
>>>
>>> class MyObserver(c.Observer):
... def onThingDone(self, msg):
... print(id(self), "onThingDone", msg)
... def onRegistered(self, prev):
... print(id(self), "onRegistered", "was", prev, "is", self.observed)
...
>>> obj = c.Observable()
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-142 }) }
>>> obs = MyObserver()
cling JIT session error: Failed to materialize symbols: { (main, { _ZN8Observer7observeEP10Observable }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZTVN16__cppyy_internal11Dispatcher1E }) }
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: none of the 2 overloaded methods succeeded. Full details:
none of the 3 overloaded methods succeeded. Full details:
Dispatcher1::Dispatcher1(const __cppyy_internal::Dispatcher1& other) =>
TypeError: takes at least 1 arguments (0 given)
Dispatcher1::Dispatcher1(const Observer& a0) =>
TypeError: takes at least 1 arguments (0 given)
__cppyy_internal::Dispatcher1 constructor failed
none of the 3 overloaded methods succeeded. Full details:
Dispatcher1::Dispatcher1(const __cppyy_internal::Dispatcher1& other) =>
TypeError: takes at least 1 arguments (0 given)
Dispatcher1::Dispatcher1(const Observer& a0) =>
TypeError: takes at least 1 arguments (0 given)
__cppyy_internal::Dispatcher1 constructor failed
>>> exit()
cling JIT session error: Failed to materialize symbols: { (main, { _ZN8Observer7observeEP10Observable }) }And loading ogdf-python: |
|
It seems that some fedora update now broke cppyy version 2.4, so now downgrading is also no longer an option and I currently cannot install a working cppyy version (2.4 due to below error, 3.0 due to the "Failed to materialize symbols" problems). # docker run -t -i --rm fedora bash
sudo dnf update -y
sudo dnf install -y python3 python3-pip python3-devel gcc g++ cmake wget
pip3 install "cppyy<3"
mv /usr/local/lib64/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so /usr/local/lib/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so # fix #191
python3 -c "import cppyy"OutputThere seem to be compile errors due to the "(Re-)building pre-compiled headers" and also some after "No precompiled header available (failed to build)" and lastly everything breaks down with |
This was referenced Oct 9, 2023
|
@wlav found a workaround in https://github.com/sxs-collaboration/spectre/pull/5222/files#diff-093aadf224e5fee0d33ae1810f2f1c23304fb5ca398ba6b96c4e7918e0811729 It is apparently a bug caused by the PCH (I've been hitting a few with C++20, so I'm not surprised). Still checking other cases. Hopefully fixes them all. :D Update: for all Python callbacks I've tried, this seems to do the trick. Other symbol materialization issues related to static initialization are a different problem. |
Gabrielcarvfer
added a commit
to Gabrielcarvfer/ns-3-dev
that referenced
this issue
Oct 19, 2023
Gabrielcarvfer
added a commit
to Gabrielcarvfer/ns-3-dev
that referenced
this issue
Oct 20, 2023
|
I can not reproduce the problem with all provided docker images/examples, but where I can, I confirm that the fix found by @Gabrielcarvfer solves the issue (many, many, thanks for that!). I simplified it to: in This has been a long run ... with this one, the Mac issue, and the gcc12 problem resolved (famous last words :) ), I can finally cut a release. Again, many thanks! |
|
Just to add to my own optimistic message: likely the reason that it was not reproducible with some docker images, is that the two offending methods were introduced only with gcc12. |
|
Should all be good now with release 3.1.0. Feel free to reopen if you find otherwise. |
|
Bad news, I still get this with ogdf-python on my Fedora 38... Good news is that seems that even though these warnings, many things seem to work again, although that needs further testing. My Observer example from here is not working: |
|
:( |
|
Can you check the output of: I'm curious (hoping, rather) whether it's something as simple as me having the cutoff incorrect of the GLIBCXX where the problem started. |
|
Got an error on Ubuntu 22.04 with GCC 11.4.0 But it is working for me on Ubuntu 23.04 |
|
That error is coming from: where that |
|
Looks like issue with |
|
In fact, if I remove all other workarounds and just add this at the top of the script (after importing cppyy), then things run fine for me on the Fedora docker image (don't know yet about the others): Edit: even better: I can add the above to the |
|
@wlav I could create a PR to set up GitHub Actions CI in this repo to continuously perform (at least smoke) tests on different OSes and versions (say latest and oldest supported Ubuntu, Debian and Fedora via Docker and also MacOs and Windows). Test coverage may be extendable but just checking for exceptions or warnings on startup or with a handful of simple use-cases (eg from the docs or from GitHub issues to check for regressions) might be a good start. It might be a little more challenging to bring all things together and in-sync with the split across different packages and repos that we have here, so I'd initially gear this towards checking (almost) released versions for problems on individual OSes or trying new things based on a released configuration instead of testing everything live alongside development. How would that sound to you? If you're open to this (you can still easily ignore the CI if you don't care for that) I'd happily draft a PR to set this up. |
|
Oh, and the cppdef seems to have fixed it! While there still is a warning for Edit: interestingly, turning off the declaration of |
|
For CI, yes that may work, but it's a bit convoluted to build from source when not everything is released b/c of the way pip deals (or rather not deals) with dependencies. cppyy 4.0.0 (based on clang-repl) should be consolidated, but is some way off (upstream says that about half the tests succeed; I haven't worked on it myself yet). Building the full backend (which includes LLVM) is also costly and doesn't work on platforms such as manylinux which insists on using a pre-C++11 ABI. :( Maybe something that can run the full test-suite manually just prior to a release? (Note that the test-suite is only 100% error free on Linux and Mac; there are a handful of errors on Windows left.) For the static initializer, there is this: cms-sw/cmssw#43077 (comment) so upstream is working on that (and they have to care alot about CMS software ;) ). (Yes, I've seen your edit, but there's something real still there.) I'm going to cut a new release later today (a fix in clingwrapper is easy to release, which is why it exists as a separate package, but ironically enough this is the first time since its existence I'm making use of that). I just want to get this one done as well: #191, since I'm now on this Fedora image anyway where it's easy to reproduce. |
|
Release 3.1.1 works for me (famous last words) on that Fedora docker image ... |
|
Things are looking very good now on Fedora. Install works flawlessly, I only get the warning from the ogdf initializer, and the As you are already building cppyy-cling wheels in CI and the rest is easy to install, I threw together this script which should work on debian, ubuntu and fedora docker containers to set up all components with their latest repo versions. Currently, this fails with the following error on all three systems: So it seems the four components are currently somehow in an inconsistent state. Still, I'll set up a CI pipeline around this next and ensure that there are options to only run the versions from PyPi and to manually trigger such runs. I'll open a new issue for this once there is further progress... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
My issue might be related to #156
I installed
cppyy==3.0.0inside a virtual environment usingvirtualenvwrapper. The following code fromcppyyexamples fails:The error messages are:
The text was updated successfully, but these errors were encountered: