building rust-cpython

[ali5h]

i am trying to build pyo3 and rust-cpython with bazel (https://github.com/ali5h/bazel_rust_python).

I finally got it to a state the crates can be built, but the main code cannot. Need help making this work. Currently on the sample repo posted above this happens for cpython sample:

$  bazel build //rust_cpython_test/... 
INFO: Analyzed target //rust_cpython_test:lib (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /.../rust_cpython/rust_cpython_test/BUILD:5:1: error executing shell command: '/bin/bash -c CARGO_MANIFEST_DIR=$(pwd)/rust_cpython_test external/rust_linux_x86_64/bin/rustc "$@" --remap-path-prefix="$(pwd)"=__bazel_redacted_pwd  rust_cpython_test/src/lib.rs --crate-name=lib -...' failed (Exit 1) bash failed: error executing command /bin/bash -c 'CARGO_MANIFEST_DIR=$(pwd)/rust_cpython_test external/rust_linux_x86_64/bin/rustc "$@" --remap-path-prefix="$(pwd)"=__bazel_redacted_pwd' '' rust_cpython_test/src/lib.rs '--crate-name=lib' ... (remaining 27 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
error[E0432]: unresolved import `cpython`
 --> rust_cpython_test/src/lib.rs:1:5
  |
1 | use cpython::{py_fn, py_module_initializer, PyResult, Python};
  |     ^^^^^^^ maybe a missing crate `cpython`?

error: cannot determine resolution for the macro `py_module_initializer`
 --> rust_cpython_test/src/lib.rs:5:1
  |
5 | py_module_initializer!(rust2py, |py, m| {
  | ^^^^^^^^^^^^^^^^^^^^^
  |
  = note: import resolution is stuck, try simplifying macro imports

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0432`.
Target //rust_cpython_test:lib failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.140s, Critical Path: 0.08s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Similarly for pyo3:

$ bazel build //pyo3_test/...
INFO: Analyzed target //pyo3_test:lib (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /.../rust_cpython/pyo3_test/BUILD:5:1: error executing shell command: '/bin/bash -c CARGO_MANIFEST_DIR=$(pwd)/pyo3_test external/rust_linux_x86_64/bin/rustc "$@" --remap-path-prefix="$(pwd)"=__bazel_redacted_pwd  pyo3_test/src/lib.rs --crate-name=lib --crate-type=cdyl...' failed (Exit 1) bash failed: error executing command /bin/bash -c 'CARGO_MANIFEST_DIR=$(pwd)/pyo3_test external/rust_linux_x86_64/bin/rustc "$@" --remap-path-prefix="$(pwd)"=__bazel_redacted_pwd' '' pyo3_test/src/lib.rs '--crate-name=lib' ... (remaining 40 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
error[E0433]: failed to resolve: maybe a missing crate `pyo3`?
 --> pyo3_test/src/lib.rs:1:5
  |
1 | use pyo3::prelude::*;
  |     ^^^^ maybe a missing crate `pyo3`?

error[E0432]: unresolved import `pyo3`
 --> pyo3_test/src/lib.rs:2:5
  |
2 | use pyo3::wrap_pyfunction;
  |     ^^^^ maybe a missing crate `pyo3`?

error: cannot find attribute `pymodule` in this scope
  --> pyo3_test/src/lib.rs:11:3
   |
11 | #[pymodule]
   |   ^^^^^^^^

error: cannot determine resolution for the macro `wrap_pyfunction`
  --> pyo3_test/src/lib.rs:13:19
   |
13 |     m.add_wrapped(wrap_pyfunction!(sum_as_string))?;
   |                   ^^^^^^^^^^^^^^^
   |
   = note: import resolution is stuck, try simplifying macro imports

error: cannot find attribute `pyfunction` in this scope
 --> pyo3_test/src/lib.rs:4:3
  |
4 | #[pyfunction]
  |   ^^^^^^^^^^

error[E0412]: cannot find type `PyResult` in this scope
   --> pyo3_test/src/lib.rs:6:41
    |
6   |   fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    |                                           ^^^^^^^^ help: an enum with a similar name exists: `Result`

error[E0412]: cannot find type `Python` in this scope
  --> pyo3_test/src/lib.rs:12:20
   |
12 | fn string_sum(_py: Python, m: &PyModule) -> PyResult<()> {
   |                    ^^^^^^ not found in this scope

error[E0412]: cannot find type `PyModule` in this scope
  --> pyo3_test/src/lib.rs:12:32
   |
12 | fn string_sum(_py: Python, m: &PyModule) -> PyResult<()> {
   |                                ^^^^^^^^ not found in this scope

error[E0412]: cannot find type `PyResult` in this scope
   --> pyo3_test/src/lib.rs:12:45
    |
12  |   fn string_sum(_py: Python, m: &PyModule) -> PyResult<()> {
    |                                               ^^^^^^^^ help: an enum with a similar name exists: `Result`

error: aborting due to 9 previous errors

Some errors have detailed explanations: E0412, E0432, E0433.
For more information about an error, try `rustc --explain E0412`.
Target //pyo3_test:lib failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.158s, Critical Path: 0.13s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

I am afraid i am missing something obvious here, bc errors are similar.

[dfreese]

looks like you need to specify edition = 2018 in your rust_repository_set in your WORKSPACE

edit: if there's a spot in the rules_rust documentation would have helped, definitely open a PR. The docs can use some work.

[ali5h]

yes, that is it. Thanks. pyo3 works now, cpyhon example fails with

error: expected type, found `$`
  --> rust_cpython_test/src/lib.rs:5:1
   |
5  | / py_module_initializer!(rust2py, |py, m| {
6  | |     m.add(py, "__doc__", "This module is implemented in Rust.")?;
7  | |     m.add(
8  | |         py,
...  |
12 | |     Ok(())
13 | | });
   | |___^
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

[ali5h]

rust_library output for cdylib is useless for use in python, it has some generated name, like bazel-out/k8-fastbuild-ST-../bin/pyo3_test/liblib--1109082208.so, which cannot be import in python.

[dfreese]

That would be better opened as an issue in rules_rust if it doesn't actually work with python. That being said, bazel-out/k8-fastbuild-.../bin is a part of bazel's building and sandboxing process, so you may need post processing steps to use a library without bazel's knowledge.

[ali5h]

yes, i made it work by some genrule, but cannot make rust-cpython work. Any idea why this is failing?

error: expected type, found `$`
  --> rust_cpython_test/src/lib.rs:5:1
   |
5  | / py_module_initializer!(rust2py, |py, m| {
6  | |     m.add(py, "__doc__", "This module is implemented in Rust.")?;
7  | |     m.add(
8  | |         py,
...  |
12 | |     Ok(())
13 | | });
   | |___^
   |
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

[dfreese]

Not off hand. That looks to be a specific issue related to the usage of rust_cpython macro, which I'm not familiar with.

[ali5h]

the problem is I can build the code with cargo, but not with bazel.

[dfreese]

Thanks for clarifying. It looks like rust_cpython, and maybe some of its dependencies, use the build.rs file to set some cfg flags for rustc. This isn't fully supported yet. #166 may help, but that was after 0.3.3 and there hasn't been a release yet.

There's a broader discussion of this in #41.

[ali5h]

yes, they do, I explicitly avoided running the build.rs bc they run bunch of local commands that depend on the env. But I want to specify those configs in bazel. For example, things like python version or where the python headers/binary are. I want through the logs generated by cargo build and added all (i am probably missing some) the flags and config to bazel builds. I managed to make pyo3 work, but not rust-cpython yet.

[ali5h]

updating one of the dependencies (paste to 0.1.17) fixed the issue, thanks for your help