cannot pip3 install orjson in Docker currently (alpine linux)

[systemcrash]

I get this:

Collecting orjson
  Downloading https://files.pythonhosted.org/packages/9c/6e/14a622d6b2615573bbd1c404dc16f60a153474d20e91ab13ab49dde0867c/orjson-3.0.2.tar.gz (649kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /usr/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-vugd6617 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- maturin>=0.8.1,<0.9:
  Collecting maturin<0.9,>=0.8.1
    Downloading https://files.pythonhosted.org/packages/a7/df/a9060d8e4639feec0e8ab98081e5b9e7bd302db29b666cc79c2c6baf4397/maturin-0.8.1.tar.gz (78kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
  Collecting toml~=0.10.0 (from maturin<0.9,>=0.8.1)
    Using cached https://files.pythonhosted.org/packages/9f/e1/1b40b80f2e1663a6b9f497123c11d7d988c0919abbf3c3f2688e448c5363/toml-0.10.1-py2.py3-none-any.whl
  Installing collected packages: toml, maturin
    Running setup.py install for maturin: started
      Running setup.py install for maturin: finished with status 'error'
      Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-pyhjev5y/maturin/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-89n4jgda/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-vugd6617 --compile:
      running install
      warning: unused manifest key: package.edition
      error: failed to parse lock file at: /tmp/pip-install-pyhjev5y/maturin/Cargo.lock

      Caused by:
        invalid serialized PackageId for key `package.dependencies`
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-install-pyhjev5y/maturin/setup.py", line 101, in <module>
          zip_safe=False,
        File "/tmp/pip-build-env-5zcgcduk/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-pyhjev5y/maturin/setup.py", line 59, in run
          ["cargo", "rustc", "--bin", "maturin", "--", "-C", "link-arg=-s"]
        File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cargo', 'rustc', '--bin', 'maturin', '--', '-C', 'link-arg=-s']' returned non-zero exit status 101.

When called from docker-compose build with alpine linux 3.7

Not possible to distribute orjson without the need to compile?

If I try on latest alpine linux:

...
Collecting orjson
  Downloading orjson-3.0.2.tar.gz (649 kB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpv96ooomy
         cwd: /tmp/pip-install-se40dy4c/orjson
    Complete output (14 lines):
    💥 maturin failed
      Caused by: Cargo metadata failed. Do you have cargo in your PATH?
      Caused by: Error during execution of `cargo metadata`: error: failed to run `rustc` to learn about target-specific information

    Caused by:
      process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z mutable-noalias --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
    --- stderr
    error: the option `Z` is only accepted on the nightly compiler



    Checking for Rust toolchain....
    Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-lpe0im7j --manylinux=off --strip=on`
    Error: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-lpe0im7j', '--manylinux=off', '--strip=on']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.8/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpv96ooomy Check the logs for full command output.

[systemcrash]

OR does rust not have a recent 'stable' build which understands mutable-noalias?

[itamarst]

Alpine Linux cannot use wheels from PyPI. This isn't an orjson-specific issue, any binary wheel will fail to install. You're better off just not using Alpine for this reason (https://pythonspeed.com/articles/alpine-docker-python/ talks about this in more detail.)

[systemcrash]

K. Article seems to know what they're talking about, but there goes my 10MB container for a 350MB Debian container, just to run ... orjson.

What tells you that PyPl / wheels are to blame in those logs? ( for my edification )

I'll run json for now since I'm not yet processing 2^32 records. python3 runs fine in Alpine for me.

[itamarst]

It's not the logs, it's external knowledge. Most Linux distributions use glibc, Alpine Linux uses musl, so Alpine's Python disables binary wheels since they are built for glibc and can therefore be incompatible.

[magusd]

damn, lost 4 hours trying to fix this in order to save a few hundred MBs using alpine.
just switched to buster and everything went fine. thanks @itamarst

[bluebrown]

Just want to mention that if you copy the stuff compiled into site-packages and sometimes also binaries from /user/local/bin into a second stage alpine image, the resulting image won't be bigger since you just dropped all compile dependencies, again. You just need to make sure you install the slim version of the dependencies again in the final image.

So, result can in fact be around 100 MB. Wile to compile stage, image can rack up to 1 Gb+. Yes, the build time will be significantly higher, but we have layer caching. It is OK for a production image, IMO.

FROM python:3.9-alpine as builder

RUN apk add --no-cache gcc g++ musl-dev rust cargo patchelf

RUN python -m pip install orjson


FROM python:3.9-alpine

COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages

RUN apk add --no-cache libstdc++

CMD ["python", "-c", "import orjson; print(orjson.dumps({'number': 1}))"]

$ docker build -t test .
$ docker run test
b'{"number":1}'

With this Dockerfile the builder stage is as big as 1.28 GB, but the final image is only 58 MB. The first build took around 10 minutes on my machine. But after caching, when changing lets say only the program to run, it only takes seconds.

Edit: since recently, we seem to need to add patchelf ourselves. See #227. I have added it to the dockerfile.

[gitpushdashf]

I think cibuildwheel now has support for musllinux wheels.

[testing12331]

Just want to mention that if you copy the stuff compiled into site-packages and sometimes also binaries from /user/local/bin into a second stage alpine image, the resulting image won't be bigger since you just dropped all compile dependencies, again. You just need to make sure you install the slim version of the dependencies again in the final image.

So, result can in fact be around 100 MB. Wile to compile stage, image can rack up to 1 Gb+. Yes, the build time will be significantly higher, but we have layer caching. It is OK for a production image, IMO.

FROM python:3.9-alpine as builder

RUN apk add --no-cache gcc g++ musl-dev rust cargo patchelf

RUN python -m pip install orjson


FROM python:3.9-alpine

COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages

RUN apk add --no-cache libstdc++

CMD ["python", "-c", "import orjson; print(orjson.dumps({'number': 1}))"]

$ docker build -t test .
$ docker run test
b'{"number":1}'

With this Dockerfile the builder stage is as big as 1.28 GB, but the final image is only 58 MB. The first build took around 10 minutes on my machine. But after caching, when changing lets say only the program to run, it only takes seconds.

Edit: since recently, we seem to need to add patchelf ourselves. See #227. I have added it to the dockerfile.

Hi I am using python 3.6 in my project and would like to switch from simplejson to orjson.
I basically copied your dockerfile and try to use it with python 3.6 and orjson version 3.6.1.
However, I got an error and can't resolve it. Can you please help me?

#6 790.1   Installing build dependencies: still running...
#6 792.4   Installing build dependencies: finished with status 'done'
#6 792.4   Getting requirements to build wheel: started
#6 792.6   Getting requirements to build wheel: finished with status 'done'
#6 792.6     Preparing wheel metadata: started
#6 792.7     Preparing wheel metadata: finished with status 'error'
#6 792.7     ERROR: Command errored out with exit status 1:
#6 792.7      command: /usr/local/bin/python /usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp4jsrqn_t
#6 792.7          cwd: /tmp/pip-install-f35tk3jx/orjson_764280cc4d1840efb2332dde17a8f0fe
#6 792.7     Complete output (6 lines):
#6 792.7     💥 maturin failed
#6 792.7       Caused by: pyproject.toml is invalid
#6 792.7       Caused by: pyproject.toml is not PEP 517 compliant: invalid type: string "on", expected a boolean for key `tool.maturin.strip` at line 12 column 9
#6 792.7     Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-i24q9wda', '--interpreter', '/usr/local/bin/python']' returned non-zero exit status 1.
#6 792.7     Checking for Rust toolchain....
#6 792.7     Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-i24q9wda --interpreter /usr/local/bin/python`

FROM python:3.6-alpine as builder

RUN apk add --no-cache gcc g++ musl-dev rust cargo patchelf

RUN python -m pip install orjson==3.6.1


FROM python:3.6-alpine

COPY --from=builder /usr/local/lib/python3.6/site-packages /usr/local/lib/python3.6/site-packages

RUN apk add --no-cache libstdc++

CMD ["python", "-c", "import orjson; print(orjson.dumps({'number': 1}))"]

[ThreeRandomWords]

This is a response specifically to the previous comment about orjson 3.6.1 triggering the PEP 517 compliance error. This closed issue thread has the top hit in results that Google displays to me when searching for that error message for the last couple of days.

I do not have a solution that will get 3.6.x working, I present this workaround. For me the error occurs specifically with orjson 3.6.x when using Python 3.11.7 with pip 24.1.2 on RHEL8, and doesn't occur with orjson 3.7 or later in the same environment. My project also works using orjson 3.6.x in a Docker container based on python:3.8 official image (Debian 'bookworm'), with cargo installed via apt-get.

Since updating requirements.txt with orjson >3.6, <4 works for me, I've updated my requirements.txt in various project to use that version. I haven't the time to test later versions or dig into differences between Debian Bookworm, RHEL8, various different versions of Python, glibc, etc.

WFM, YMMV.

Sincere apologies to the orjson developers for adding this off-topic comment to an ancient closed thread.