Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pyo3-0.20.0 on ppc32: no AtomicI64 in sync::atomic #3614

Closed
mgorny opened this issue Nov 30, 2023 · 6 comments · Fixed by #3619
Closed

pyo3-0.20.0 on ppc32: no AtomicI64 in sync::atomic #3614

mgorny opened this issue Nov 30, 2023 · 6 comments · Fixed by #3619
Labels
bug

Comments

@mgorny
Copy link

mgorny commented Nov 30, 2023

Bug Description

When building a package (html-py-ever example from setuptools-rust) using pyo3 0.20.0 on ppc32, I'm getting the following error:

error[E0432]: unresolved import `std::sync::atomic::AtomicI64`
 --> /home/mgorny/.cargo/registry/src/index.crates.io-9850fa1df3136423/pyo3-0.20.0/src/impl_/pymodule.rs:6:25
  |
6 | use std::sync::atomic::{AtomicI64, Ordering};
  |                         ^^^^^^^^^
  |                         |
  |                         no `AtomicI64` in `sync::atomic`
  |                         help: a similar name exists in the module: `AtomicI32`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `pyo3` (lib) due to previous error

Steps to Reproduce

git clone https://github.com/PyO3/setuptools-rust/
cd setuptools-rust/examples/html-py-ever
python -m build -w

Backtrace

Running `rustc --crate-name pyo3 --edition=2021 /home/mgorny/.cargo/registry/src/index.crates.io-9850fa1df3136423/pyo3-0.20.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=135 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="indoc"' --cfg 'feature="macros"' --cfg 'feature="pyo3-macros"' --cfg 'feature="unindent"' -C metadata=c3280b3684a9dc16 -C extra-filename=-c3280b3684a9dc16 --out-dir /home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps -L dependency=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps --extern cfg_if=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libcfg_if-dd66dfbd15e1f200.rmeta --extern indoc=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libindoc-330c4043caad1819.so --extern libc=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/liblibc-bd25ee420bfedf7b.rmeta --extern memoffset=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libmemoffset-80b40800efcae77e.rmeta --extern parking_lot=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libparking_lot-166554cbb51c52ff.rmeta --extern pyo3_ffi=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libpyo3_ffi-d8ceef02bf595e0b.rmeta --extern pyo3_macros=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libpyo3_macros-b6c77a4bfb6a873a.so --extern unindent=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libunindent-a0d1a359c22931b4.rmeta --cap-lints allow --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3_8 --cfg Py_3_9 --cfg Py_3_10 --cfg Py_3_11 --cfg thread_local_const_init`
error[E0432]: unresolved import `std::sync::atomic::AtomicI64`
 --> /home/mgorny/.cargo/registry/src/index.crates.io-9850fa1df3136423/pyo3-0.20.0/src/impl_/pymodule.rs:6:25
  |
6 | use std::sync::atomic::{AtomicI64, Ordering};
  |                         ^^^^^^^^^
  |                         |
  |                         no `AtomicI64` in `sync::atomic`
  |                         help: a similar name exists in the module: `AtomicI32`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `pyo3` (lib) due to previous error

Caused by:
  process didn't exit successfully: `rustc --crate-name pyo3 --edition=2021 /home/mgorny/.cargo/registry/src/index.crates.io-9850fa1df3136423/pyo3-0.20.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=135 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="indoc"' --cfg 'feature="macros"' --cfg 'feature="pyo3-macros"' --cfg 'feature="unindent"' -C metadata=c3280b3684a9dc16 -C extra-filename=-c3280b3684a9dc16 --out-dir /home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps -L dependency=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps --extern cfg_if=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libcfg_if-dd66dfbd15e1f200.rmeta --extern indoc=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libindoc-330c4043caad1819.so --extern libc=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/liblibc-bd25ee420bfedf7b.rmeta --extern memoffset=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libmemoffset-80b40800efcae77e.rmeta --extern parking_lot=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libparking_lot-166554cbb51c52ff.rmeta --extern pyo3_ffi=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libpyo3_ffi-d8ceef02bf595e0b.rmeta --extern pyo3_macros=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libpyo3_macros-b6c77a4bfb6a873a.so --extern unindent=/home/mgorny/setuptools-rust/examples/html-py-ever/target/release/deps/libunindent-a0d1a359c22931b4.rmeta --cap-lints allow --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3_8 --cfg Py_3_9 --cfg Py_3_10 --cfg Py_3_11 --cfg thread_local_const_init` (exit status: 1)
error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101

ERROR Backend subprocess exited when trying to invoke build_wheel

Your operating system and version

Gentoo Linux, ppc32 nspawn container on POWER9

Your Python version (python --version)

Python 3.11.6

Your Rust version (rustc --version)

rustc 1.71.1 (eb26296b5 2023-08-03)

Your PyO3 version

0.20.0

How did you install python? Did you use a virtualenv?

Gentoo package. Tested inside a venv (with another venv layer via build) and outside.

Additional Info

I know that C programs using 64-bit atomics need to explicitly link -latomic here. I don't know how that maps to Rust.
@mgorny mgorny added the bug label Nov 30, 2023
@mgorny mgorny changed the title pyo3-0.20.0 on ppc32: no AtomicI64 in sync::atomic`` pyo3-0.20.0 on ppc32: no \AtomicI64\ in \sync::atomic\`` Nov 30, 2023
@mgorny mgorny changed the title pyo3-0.20.0 on ppc32: no \AtomicI64\ in \sync::atomic\`` pyo3-0.20.0 on ppc32: no AtomicI64 in sync::atomic Nov 30, 2023
@messense
Copy link
Member

messense commented Dec 1, 2023

IMO in Rust we could use the portable-atomic crate.

@davidhewitt
Copy link
Member

Does it work if we have that as an optional dependency? Alternatively we could use AtomicBool and a OnceLock pair, which I actually used in my original implementation.

@mgorny
Copy link
Author

mgorny commented Dec 1, 2023

If you give me a patch, I can test it on this system.

@davidhewitt
Copy link
Member

I will try to put something together by end of Sunday. Afraid I'm a little bit behind on OSS this week; was in hospital with my son the two weeks before.

@adamreichold adamreichold mentioned this issue Dec 2, 2023
Merged
@adamreichold
Copy link
Member

I created #3619 as I agree with @messense on using portable-atomic to resolve this. @mgorny Could you give it a try?

An alternative I see is safely casting the interpreter ID to 32-bits (using TryInto) and use that instead of a full 64-bit interpreter ID because 4 billion subinterpreters should be enough for everybody?

@mgorny
Copy link
Author

mgorny commented Dec 2, 2023

Thank you! I can confirm that the setuptools-rust example builds against your branch on that machine, and the tests pass.

@jefferyto jefferyto mentioned this issue Dec 4, 2023
Closed
@antonijn antonijn mentioned this issue Feb 11, 2024
Closed
@davidhewitt davidhewitt mentioned this issue Apr 15, 2024
Closed
arnout pushed a commit to buildroot/buildroot that referenced this issue May 7, 2024
@RomainNaour @tpetazzoni
package/python-rpds-py: bump to version 0.18.1
26b0540 
As reported by Antoine Coutant on the Buildroot mailing list [1]:

rpds depends on the crate pyo3. This crate uses

  "std::sync::atomic::AtomicI64"

which doesn't exist for 32 bits targets.
That leads to an error as reported by Buildroot Gitlab-ci.

An issue have been opened on pyo3's github [2].
A fix have been accepted by pyo3 [3] and merged in pyo3 0.20.3 release
[4]. This pyo3 0.20.3 uses crate portable-atomic [5] to provide atomic
types and compare and swap operations for all targets that can use std.

python-rpds-py 0.18.1 is the first release that use pyo3 0.20.3.

Before python-rpds-py 0.18.1 release fixing this issue would require
that we be able to patch a package during the Cargo vendoring step [6].

[1] http://lists.busybox.net/pipermail/buildroot/2024-February/684867.html
[2]: PyO3/pyo3#3614
[3]: PyO3/pyo3#3619
[4]: https://github.com/PyO3/pyo3/blob/v0.20.3/CHANGELOG.md
[5]: https://github.com/taiki-e/portable-atomic
[6]: http://lists.busybox.net/pipermail/buildroot/2024-February/684886.html

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/6783811842 (TestPythonPy3Dtschema)
https://gitlab.com/buildroot.org/buildroot/-/jobs/6783811860 (TestPythonPy3FlaskExpectsJson)

Reported-by: Antoine Coutant <antoine.coutant@smile.fr>
Cc: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use portable-atomic for targets which lack 64-bit atomics used to check interpreter ID. PyO3/pyo3
4 participants
@mgorny @messense @davidhewitt @adamreichold