Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"size changed, may indicate binary incompatibility" #3042

Open
pitrou opened this issue Jul 15, 2019 · 7 comments

Comments

@pitrou
Copy link
Contributor

commented Jul 15, 2019

What is that and how do I silence this warning?

../../pyarrow-test-3.6/lib/python3.6/site-packages/pyarrow/tests/test_cython.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   ValueError: pyarrow.lib.DictionaryMemo size changed, may indicate binary incompatibility. Expected 184 from C header, got 160 from PyObject
lib.pxd:75: ValueError
@pitrou

This comment has been minimized.

Copy link
Contributor Author

commented Jul 15, 2019

Note the wheel was built with Cython 0.29.8 and is being tested with Cython 0.29.12.

@pitrou

This comment has been minimized.

Copy link
Contributor Author

commented Jul 16, 2019

@scoder What would this be? We are not the only ones to get the issue apparently: apache/arrow#4884 (comment)

@pitrou

This comment has been minimized.

Copy link
Contributor Author

commented Jul 16, 2019

Also I can't reproduce locally, even with those exact Cython versions.

@pitrou

This comment has been minimized.

Copy link
Contributor Author

commented Jul 16, 2019

Update: I managed to reproduce locally... but only with manylinux1, not manylinux2010!

Here is a manylinux1-produced wheel:

Python 3.7.3 (default, Apr  3 2019, 19:16:38) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.lib.DictionaryMemo.__basicsize__
160

Here is a manylinux2010-produced wheel:

Python 3.7.3 (default, Apr  3 2019, 19:16:38) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.lib.DictionaryMemo.__basicsize__
184
@mbrobbel

This comment has been minimized.

Copy link

commented Jul 16, 2019

Update: I managed to reproduce locally... but only with manylinux1, not manylinux2010!

We're using the published pyarrow manylinux1 wheel when building a manylinux2010 wheel of our project which explains the value error.

@pitrou

This comment has been minimized.

Copy link
Contributor Author

commented Jul 16, 2019

Ok and I finally understand: Cython is flagging a real issue here. We're putting a C++ DictionaryMemo structure in our Python DictionaryMemo extension type. But the size of the C++ structure depends on the C++ (and libstdc++) compiler ABI, so loading it on a more recent compiler may break. Instead, we should only store a pointer like we do on other extension types.

So, sorry for my initial annoyance and thanks to Cython for flagging this!

@syedmustafa54

This comment has been minimized.

Copy link

commented Jul 16, 2019

How can we create binary @pitrou

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.