ujson import error on alpine 3.9 #326

ngaya-ll · 2019-02-06T00:03:24Z

After installing ujson in alpine 3.9, Python raises an ImportError when attempting to import the package:

ImportError: Error relocating /usr/lib/python2.7/site-packages/ujson.so: strreverse: symbol not found

Possibly related to #180 as the symptoms are similar.

Full terminal output showing steps to reproduce:

~ $ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:b3dbf31b77fd99d9c08f780ce6f5282aba076d70a513a8be859d8d3a4d0c92b8
Status: Image is up to date for alpine:latest
~ $ docker run --rm -it alpine sh
/ # apk add py-pip python python-dev gcc musl-dev
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/25) Installing binutils (2.31.1-r2)
(2/25) Installing gmp (6.1.2-r1)
(3/25) Installing isl (0.18-r0)
(4/25) Installing libgomp (8.2.0-r2)
(5/25) Installing libatomic (8.2.0-r2)
(6/25) Installing libgcc (8.2.0-r2)
(7/25) Installing mpfr3 (3.1.5-r1)
(8/25) Installing mpc1 (1.0.3-r1)
(9/25) Installing libstdc++ (8.2.0-r2)
(10/25) Installing gcc (8.2.0-r2)
(11/25) Installing musl-dev (1.1.20-r3)
(12/25) Installing libbz2 (1.0.6-r6)
(13/25) Installing expat (2.2.6-r0)
(14/25) Installing libffi (3.2.1-r6)
(15/25) Installing gdbm (1.13-r1)
(16/25) Installing ncurses-terminfo-base (6.1_p20190105-r0)
(17/25) Installing ncurses-terminfo (6.1_p20190105-r0)
(18/25) Installing ncurses-libs (6.1_p20190105-r0)
(19/25) Installing readline (7.0.003-r1)
(20/25) Installing sqlite-libs (3.26.0-r3)
(21/25) Installing python2 (2.7.15-r3)
(22/25) Installing py-setuptools (40.6.3-r0)
(23/25) Installing py2-pip (18.1-r0)
(24/25) Installing pkgconf (1.6.0-r0)
(25/25) Installing python2-dev (2.7.15-r3)
Executing busybox-1.29.3-r10.trigger
OK: 171 MiB in 39 packages
/ # pip install -U ujson
Collecting ujson
  Downloading https://files.pythonhosted.org/packages/16/c4/79f3409bc710559015464e5f49b9879430d8f87498ecdc335899732e5377/ujson-1.35.tar.gz (192kB)
    100% |████████████████████████████████| 194kB 10.3MB/s 
Installing collected packages: ujson
  Running setup.py install for ujson ... done
Successfully installed ujson-1.35
/ # python -c 'import ujson'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: Error relocating /usr/lib/python2.7/site-packages/ujson.so: strreverse: symbol not found

The text was updated successfully, but these errors were encountered:

streamnsight · 2019-02-08T07:44:25Z

pip install git+git://github.com/esnme/ultrajson.git
fixed it for me.

but there should really be a new release cut.

stef · 2019-02-26T15:01:53Z

the wheel contains only the lib built against glibc not against musl libc and this messes up the pip install. indeed cloning the repo and running python setup.py install fixes the problem and rebuilds the library against musl.

stef · 2019-02-26T15:03:25Z

what i meant to say is really, that a new release will not fix this. the wheels need to be fixed so that they also contain binaries built against musl libc.

ngaya-ll · 2019-02-26T21:40:13Z

Notice that the pip output shows that it is using setup.py install to build the package, not the binary wheel. This seems to be an issue with the 1.35 source package.

I was able to successfully install ujson from the current master branch after installing a couple more build dependencies:

/ # apk add g++ git
(1/8) Installing libc-dev (0.7.1-r0)
(2/8) Installing g++ (8.2.0-r2)
(3/8) Installing ca-certificates (20190108-r0)
(4/8) Installing nghttp2-libs (1.35.1-r0)
(5/8) Installing libssh2 (1.8.0-r4)
(6/8) Installing libcurl (7.64.0-r1)
(7/8) Installing pcre2 (10.32-r1)
(8/8) Installing git (2.20.1-r0)
Executing busybox-1.29.3-r10.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 241 MiB in 47 packages
/ # pip install git+git://github.com/esnme/ultrajson.git
Collecting git+git://github.com/esnme/ultrajson.git
  Cloning git://github.com/esnme/ultrajson.git to /tmp/pip-req-build-iHSigB
Installing collected packages: ujson
  Found existing installation: ujson 1.35
    Uninstalling ujson-1.35:
      Successfully uninstalled ujson-1.35
  Running setup.py install for ujson ... done
Successfully installed ujson-2.0
/ # python -c 'import ujson'

So the issue seems to be fixed on master.

stef · 2019-03-03T00:06:46Z

So the issue seems to be fixed on master.

when you pip install from git, then there is no wheel, and instead it builds the c dependency. the problem is not in the source on master, but the problem is in the pip wheel package, which is built without musl support. so it is not fixed on master, as it cannot be fixed there, there's nothing wrong on master. the problem is with the pip package.

ngaya-ll · 2019-03-07T19:53:48Z

@stef As I explained in my previous comment, the problem has nothing to do with wheels, as pip is installing from source in both cases. To confirm this, I added the --no-binary=ujson flag to the pip install command and got the same results:

When installing version 1.35 from source on Alpine 3.9, the ImportError issue occurs.
When installing from master from source on Alpine 3.9, the issue does not occur.

stef · 2019-03-07T20:11:09Z

to quote your original report:

Installing collected packages: ujson
Running setup.py install for ujson ... done
Successfully installed ujson-1.35
/ # python -c 'import ujson'
Traceback (most recent call last):
File "", line 1, in
ImportError: Error relocating /usr/lib/python2.7/site-packages/ujson.so: strreverse: symbol not found

it says "setup.py install" and then it says that the lib you supposedly built on alpine has a glibc dependency. that is impossible, if indeed setup.py install built this on alpine. that missing symbol comes from the badly build wheel.

ngaya-ll · 2019-03-08T21:09:24Z

I see the same issue (with slightly different output) when I clone the repository, check out the v1.35 tag and run python setup.py install directly.

/ # git clone git://github.com/esnme/ultrajson.git -b v1.35
Cloning into 'ultrajson'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 1692 (delta 2), reused 2 (delta 2), pack-reused 1689
Receiving objects: 100% (1692/1692), 7.81 MiB | 9.78 MiB/s, done.
Resolving deltas: 100% (909/909), done.
Note: checking out '3723f573d3c27b2f33d93baa4345576502dcabb5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

/ # cd ultrajson/
/ultrajson # python setup.py install
running install
running bdist_egg
running egg_info
creating ujson.egg-info
writing ujson.egg-info/PKG-INFO
writing top-level names to ujson.egg-info/top_level.txt
writing dependency_links to ujson.egg-info/dependency_links.txt
writing manifest file 'ujson.egg-info/SOURCES.txt'
reading manifest file 'ujson.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ujson.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'ujson' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/python
creating build/temp.linux-x86_64-2.7/lib
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/include/python2.7 -c ./python/ujson.c -o build/temp.linux-x86_64-2.7/./python/ujson.o -D_GNU_SOURCE
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/include/python2.7 -c ./python/objToJSON.c -o build/temp.linux-x86_64-2.7/./python/objToJSON.o -D_GNU_SOURCE
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/include/python2.7 -c ./python/JSONtoObj.c -o build/temp.linux-x86_64-2.7/./python/JSONtoObj.o -D_GNU_SOURCE
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/include/python2.7 -c ./lib/ultrajsonenc.c -o build/temp.linux-x86_64-2.7/./lib/ultrajsonenc.o -D_GNU_SOURCE
./lib/ultrajsonenc.c:156:23: warning: 'g_hexChars' is static but used in inline function 'Buffer_AppendShortHexUnchecked' which is not static
   *(outputOffset++) = g_hexChars[(value & 0x000f) >> 0];
                       ^~~~~~~~~~
./lib/ultrajsonenc.c:155:23: warning: 'g_hexChars' is static but used in inline function 'Buffer_AppendShortHexUnchecked' which is not static
   *(outputOffset++) = g_hexChars[(value & 0x00f0) >> 4];
                       ^~~~~~~~~~
./lib/ultrajsonenc.c:154:23: warning: 'g_hexChars' is static but used in inline function 'Buffer_AppendShortHexUnchecked' which is not static
   *(outputOffset++) = g_hexChars[(value & 0x0f00) >> 8];
                       ^~~~~~~~~~
./lib/ultrajsonenc.c:153:23: warning: 'g_hexChars' is static but used in inline function 'Buffer_AppendShortHexUnchecked' which is not static
   *(outputOffset++) = g_hexChars[(value & 0xf000) >> 12];
                       ^~~~~~~~~~
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/include/python2.7 -c ./lib/ultrajsondec.c -o build/temp.linux-x86_64-2.7/./lib/ultrajsondec.o -D_GNU_SOURCE
creating build/lib.linux-x86_64-2.7
gcc -shared -Wl,--as-needed build/temp.linux-x86_64-2.7/./python/ujson.o build/temp.linux-x86_64-2.7/./python/objToJSON.o build/temp.linux-x86_64-2.7/./python/JSONtoObj.o build/temp.linux-x86_64-2.7/./lib/ultrajsonenc.o build/temp.linux-x86_64-2.7/./lib/ultrajsondec.o -L/usr/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/ujson.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/ujson.so -> build/bdist.linux-x86_64/egg
creating stub loader for ujson.so
byte-compiling build/bdist.linux-x86_64/egg/ujson.py to ujson.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying ujson.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying ujson.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying ujson.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying ujson.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/ujson-1.35-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing ujson-1.35-py2.7-linux-x86_64.egg
Copying ujson-1.35-py2.7-linux-x86_64.egg to /usr/lib/python2.7/site-packages
Adding ujson 1.35 to easy-install.pth file

Installed /usr/lib/python2.7/site-packages/ujson-1.35-py2.7-linux-x86_64.egg
Processing dependencies for ujson==1.35
Finished processing dependencies for ujson==1.35
/ultrajson # python -c 'import ujson'
/usr/lib/python2.7/site-packages/ujson-1.35-py2.7-linux-x86_64.egg/ujson.py:3: UserWarning: Module ujson was already imported from /usr/lib/python2.7/site-packages/ujson-1.35-py2.7-linux-x86_64.egg/ujson.pyc, but /ultrajson is being added to sys.path
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/ujson.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/ujson.py", line 6, in __bootstrap__
ImportError: Error relocating /root/.cache/Python-Eggs/ujson-1.35-py2.7-linux-x86_64.egg-tmp/ujson.so: strreverse: symbol not found

ngaya-ll · 2019-03-08T21:36:38Z

This can be fixed by adding the static modifier to the Buffer_AppendShortHexUnchecked and strreverse functions in lib/ultrajsonenc.c. Would it be possible to release a 1.35.1 version with this patch?

ngaya-ll · 2019-03-08T21:44:22Z

This was fixed on master with #238

ngaya-ll · 2019-03-08T22:01:29Z

The root cause was a change to C99 inline semantics in GCC 5, as described in #237.

stef · 2019-03-08T22:09:22Z

good job ;)

rocketbitz · 2019-03-19T18:48:02Z

Is a new release going to be cut for this?

ujson v1.35 can't work vs Alpine 3.9 [1]. upper-constraints are overriden to include the patch needed [2]. It fixes rally testcases (gnocchi). [1] ultrajson/ultrajson#326 [2] ultrajson/ultrajson@d25e024 Change-Id: Iac9975b0669d5975c63d76ddefb64058e5183369 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>

ujson v1.35 can't work vs Alpine 3.9 [1]. upper-constraints are overriden to include the patch needed [2]. It fixes rally testcases (gnocchi). [1] ultrajson/ultrajson#326 [2] ultrajson/ultrajson@d25e024 Change-Id: Iac9975b0669d5975c63d76ddefb64058e5183369 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com> (cherry picked from commit c6eb6ec)

Speedup docker release images build with targeted builds See merge request noc/noc!2358 (cherry picked from commit 5da383d) 2db26a6 define environmets 19d9bc7 Merge branch 'master' of code.getnoc.com:noc/noc 716aee0 Add mongo-rs healthcheck. launch migrations on hc. Remove all other definitions except used 03b2699 Merge 36a3ab1 Merge branch 'master' of code.getnoc.com:noc/noc ebbf12f Merge branch 'master' of code.getnoc.com:noc/noc 0571089 Merge branch 'master' of code.getnoc.com:noc/noc 463e39d Merge branch 'master' of code.getnoc.com:noc/noc efcb50a Speedup docker release images build with targeted builds 553e4b3 Revert to alpine 3.8. Because of ultrajson/ultrajson#326

rmacian · 2019-07-31T16:28:01Z

Get rid of it using the python image based on alpine. I was trying to install python 3.7.3 in the vanilla alpine10 and was facing this problem. I created a new one with the 3.7.3-alpine3.10 tag from https://hub.docker.com/_/python
and I could do a pip install ujson without any problems

ngaya-ll · 2019-08-06T01:13:46Z

@rmacian Interesting. It seems that the alpine package for python has different compiler flags than those used by the python docker image.

Alpine 3.10 python:

gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/include/python2.7 -c ./lib/ultrajsonenc.c -o build/temp.linux-x86_64-2.7/./lib/ultrajsonenc.o -D_GNU_SOURCE

Python docker image:

gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DTHREAD_STACK_SIZE=0x100000 -fPIC -I./python -I./lib -I/usr/local/include/python2.7 -c ./lib/ultrajsonenc.c -o build/temp.linux-x86_64-2.7/./lib/ultrajsonenc.o -D_GNU_SOURCE

If you add -O3 to the compiler flags in the Alpine 3.10 case, the import error goes away. So perhaps the extra optimizations result in some inlining that avoids the linking error.

hugovk · 2020-02-27T22:04:11Z

A new release will be out soon, see #352.

It looks like this was fixed in master by #238, so I'll close this issue. Thanks!

rocketbitz · 2020-02-27T22:05:56Z

much appreciated

smrutisenapati · 2021-05-24T16:43:38Z

working solution: pip install git+git://github.com/esnme/ultrajson.git@6cf6c7f
It has the commit id of the fix, with no other breaking changes in version of 1.35

YifengMao mentioned this issue Jul 20, 2019

Server revamp splunk/eventgen#266

Merged

freeseacher pushed a commit to nocproject/noc that referenced this issue Jul 20, 2019

Revert to alpine 3.8. Because of ultrajson/ultrajson#326

553e4b3

li-wu mentioned this issue Oct 31, 2019

Fix oom caused by ujson splunk/eventgen#336

Merged

mrmachine mentioned this issue Dec 21, 2019

ImportError ujson.so undefined symbol Buffer_AppendShortHexUnchecked #346

Closed

ccaputo mentioned this issue Feb 22, 2020

ujson-1.35 with gcc-9.2.0 results in "undefined symbol: Buffer_AppendShortHexUnchecked" irrdnet/irrd#283

Closed

hugovk closed this as completed Feb 27, 2020

ghost mentioned this issue May 2, 2020

Bump alpine and nginx to latest stable version & Fix deps for build amplify nginxinc/docker-nginx-amplify#24

Closed

li-wu mentioned this issue May 6, 2020

fix ujson version splunk/eventgen#389

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ujson import error on alpine 3.9 #326

ujson import error on alpine 3.9 #326

ngaya-ll commented Feb 6, 2019

streamnsight commented Feb 8, 2019

stef commented Feb 26, 2019

stef commented Feb 26, 2019

ngaya-ll commented Feb 26, 2019 •

edited

Loading

stef commented Mar 3, 2019

ngaya-ll commented Mar 7, 2019

stef commented Mar 7, 2019

ngaya-ll commented Mar 8, 2019 •

edited

Loading

ngaya-ll commented Mar 8, 2019

ngaya-ll commented Mar 8, 2019

ngaya-ll commented Mar 8, 2019 •

edited

Loading

stef commented Mar 8, 2019

rocketbitz commented Mar 19, 2019

rmacian commented Jul 31, 2019

ngaya-ll commented Aug 6, 2019

hugovk commented Feb 27, 2020

rocketbitz commented Feb 27, 2020

smrutisenapati commented May 24, 2021

ujson import error on alpine 3.9 #326

ujson import error on alpine 3.9 #326

Comments

ngaya-ll commented Feb 6, 2019

streamnsight commented Feb 8, 2019

stef commented Feb 26, 2019

stef commented Feb 26, 2019

ngaya-ll commented Feb 26, 2019 • edited Loading

stef commented Mar 3, 2019

ngaya-ll commented Mar 7, 2019

stef commented Mar 7, 2019

ngaya-ll commented Mar 8, 2019 • edited Loading

ngaya-ll commented Mar 8, 2019

ngaya-ll commented Mar 8, 2019

ngaya-ll commented Mar 8, 2019 • edited Loading

stef commented Mar 8, 2019

rocketbitz commented Mar 19, 2019

rmacian commented Jul 31, 2019

ngaya-ll commented Aug 6, 2019

hugovk commented Feb 27, 2020

rocketbitz commented Feb 27, 2020

smrutisenapati commented May 24, 2021

ngaya-ll commented Feb 26, 2019 •

edited

Loading

ngaya-ll commented Mar 8, 2019 •

edited

Loading

ngaya-ll commented Mar 8, 2019 •

edited

Loading