Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking crates with exotic build.rs #41

Open
mfarrugi opened this issue Mar 30, 2018 · 57 comments
Open

Tracking crates with exotic build.rs #41

mfarrugi opened this issue Mar 30, 2018 · 57 comments

Comments

@mfarrugi
Copy link
Collaborator

mfarrugi commented Mar 30, 2018

Moving acmcarther/cargo-raze#16 over here.

Bazel needs some extra help with build.rs scripts that:

  • link cc libs to *-sys crates
  • use exotic cargo environment variables
  • use compile time file includes
  • produce --cfg flags
  • and do other bespoke logic

Conveniently I haven't run into many additional problems, and our set of overrides hasn't grown.

@mfarrugi
Copy link
Collaborator Author

https://github.com/briansmith/ring/blob/master/build.rs is a doozy, it's upstream of actix-web :|

@mfarrugi mfarrugi changed the title Snowflake crate issues (aka. exotic build.rs) Tracking crates with exotic build.rs Jul 24, 2018
@acmcarther
Copy link
Member

Got an initial read of that thing? Does it look like we can run it if we provide the right env vars?

@mfarrugi
Copy link
Collaborator Author

It might, wanted to keep track before I got the chance to take a deeper look.

@vitalyd
Copy link
Contributor

vitalyd commented Jul 25, 2018

A few notes from my attempt at looking at ring that @mfarrugi brought up:

  • https://github.com/briansmith/ring/blob/master/build.rs#L270 looks for the CARGO_PKG_NAME envvar - this isn't set in the build script wrapper that cargo raze generates (but is set in bazel rust, for example: https://github.com/bazelbuild/rules_rust/blob/master/rust/toolchain.bzl#L23).
    I'm pretty sure the intended build path here is to end up calling the ring_build_rs_main() fn, rather than enter the pregeneration step via pregenerate_asm_main(). I hand edited cargo raze's ring_build_script_executor genrule to get past this point.
  • Once I forced execution into ring_build_rs_main, we once again get blocked on missing envvars. In all, I had to add this to the same genrule:
 + " export CARGO_CFG_TARGET_ARCH=x86_64;"
        + " export CARGO_CFG_TARGET_OS=linux;"
        + " export CARGO_CFG_TARGET_ENV=;"
        + " export DEBUG=false;"
        + " export OPT_LEVEL=3;"
        + " export HOST=;"

The latter 3 are needed by the gcc invocation. Apparently, cargo sets this all up for build scripts, and I suspect cargo raze will eventually need to reimplement a whole bunch (if not all) of this.

HTH

@photex
Copy link

photex commented Aug 1, 2018

Howdy!

I didn't see any special handling for backtrace or backtrace-sys in the crater or other examples, but with my dependency on the failure crate I'm getting backtrace 0.3.9 and backtrace-sys 0.1.23 as transitive dependencies.

This version of backtrace-sys has been renovated compared to 0.1.16:
https://github.com/alexcrichton/backtrace-rs/blob/master/backtrace-sys/build.rs

I'm unable to link or build any crate (aka all of my crates at the moment) due to missing symbols:

<snip gobs of linker output/>
undefined reference to `__rbt_backtrace_create_state'
undefined reference to `__rbt_backtrace_pcinfo'
undefined reference to `__rbt_backtrace_syminfo'
          collect2: error: ld returned 1 exit status

Adding the following to my raze Cargo.toml has had no effect at all:

[raze.crates.backtrace-sys.'0.1.23']
gen_buildrs = true

@mfarrugi
Copy link
Collaborator Author

mfarrugi commented Aug 1, 2018 via email

@photex
Copy link

photex commented Aug 1, 2018

I see. I'd just be adding a new rule more or less replicating the effect of the build.rs then? And does that require fooling with output from cargo-raze?

@mfarrugi
Copy link
Collaborator Author

mfarrugi commented Aug 1, 2018

@photex

cargo/Cargo.toml

[dependencies]
backtrace = "0.3.9"

# ... 

[raze.crates.backtrace.'0.3.9']
# Overriden to hand-written BUILD file, because libloading is building its own c portion.
skipped_deps = [
    "backtrace-sys-0.1.23",
]
additional_deps = [
    "//cargo/overrides/backtrace-sys-0.1.23:backtrace_sys",
]

cargo/overrides/backtrace-sys-0.1.23/ contains a copy of the crate, with the following added

cargo/overrides/backtrace-sys-0.1.23/BUILD which is a modification of the generated file

package(default_visibility=["//ext/public/rust/cargo:__subpackages__"])

licenses([
    "notice",  # "MIT,Apache-2.0"
])

load(
    "@io_bazel_rules_rust//rust:rust.bzl",
    "rust_library",
    "rust_binary",
    "rust_test",
    "rust_bench_test",
)

genrule(
    name="touch_config_header",
    outs=[
        "config.h",
    ],
    cmd="touch $@",
)

genrule(
    name="touch_backtrace_supported_header",
    outs=[
        "backtrace-supported.h",
    ],
    cmd="touch $@",
)

cc_library(
    name="backtrace_native",
    srcs=[
        ":touch_config_header",
        ":touch_backtrace_supported_header",
        "src/libbacktrace/internal.h",
        "src/libbacktrace/backtrace.h",
        "src/libbacktrace/alloc.c",
        "src/libbacktrace/dwarf.c",
        "src/libbacktrace/fileline.c",
        "src/libbacktrace/posix.c",
        "src/libbacktrace/read.c",
        "src/libbacktrace/sort.c",
        "src/libbacktrace/state.c",
        "src/libbacktrace/elf.c",
    ],
    includes=["."],
    copts=[
        "-fvisibility=hidden",
        "-fPIC",
    ],
    defines=[
        "BACKTRACE_ELF_SIZE=64",
        "BACKTRACE_SUPPORTED=1",
        "BACKTRACE_USES_MALLOC=1",
        "BACKTRACE_SUPPORTS_THREADS=0",
        "BACKTRACE_SUPPORTS_DATA=0",
        "HAVE_DL_ITERATE_PHDR=1",
        "_GNU_SOURCE=1",
        "_LARGE_FILES=1",
        "backtrace_full=__rbt_backtrace_full",
        "backtrace_dwarf_add=__rbt_backtrace_dwarf_add",
        "backtrace_initialize=__rbt_backtrace_initialize",
        "backtrace_pcinfo=__rbt_backtrace_pcinfo",
        "backtrace_syminfo=__rbt_backtrace_syminfo",
        "backtrace_get_view=__rbt_backtrace_get_view",
        "backtrace_release_view=__rbt_backtrace_release_view",
        "backtrace_alloc=__rbt_backtrace_alloc",
        "backtrace_free=__rbt_backtrace_free",
        "backtrace_vector_finish=__rbt_backtrace_vector_finish",
        "backtrace_vector_grow=__rbt_backtrace_vector_grow",
        "backtrace_vector_release=__rbt_backtrace_vector_release",
        "backtrace_close=__rbt_backtrace_close",
        "backtrace_open=__rbt_backtrace_open",
        "backtrace_print=__rbt_backtrace_print",
        "backtrace_simple=__rbt_backtrace_simple",
        "backtrace_qsort=__rbt_backtrace_qsort",
        "backtrace_create_state=__rbt_backtrace_create_state",
        "backtrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug",
    ],
)

rust_library(
    name="backtrace_sys",
    crate_root="src/lib.rs",
    crate_type="lib",
    srcs=glob(["**/*.rs"]),
    deps=[
        "//cargo/vendor/libc-0.2.42:libc",
        ":backtrace_native",
    ],
    rustc_flags=[
        "--cap-lints allow",
        "--target=x86_64-unknown-linux-gnu",
    ],
    crate_features=[
    ],
)

This magical series of steps should be improved by #58

@photex
Copy link

photex commented Aug 1, 2018

Thanks for so quickly sharing your solution @mfarrugi. I would not have worked this out so fast on my own.

@mfarrugi
Copy link
Collaborator Author

mfarrugi commented Sep 15, 2018

Ran into new build scripts!

proc-macro2
lazy-static

Our solution was to hardcode the --cfg flags in the raze Cargo.toml that cargo would have come up with.

Edit:
More precisely, we added:

[raze.crates.lazy_static.'1.1.0']
additional_flags = [
    "--cfg=lazy_static_inline_impl",
]

[raze.crates.proc-macro2.'0.4.19']
additional_flags = [
    "--cfg use_proc_macro",
]

@acmcarther
Copy link
Member

acmcarther commented Sep 16, 2018

I needed to link libsqlite3-sys, and found that it uses some interesting bindgen options that I couldn't replicate via bindgen executable flags:

https://github.com/jgallagher/rusqlite/blob/bd1756adeff7a67a13f00b39325301f5935d8267/libsqlite3-sys/build.rs#L204-L226

Specifically, it overrides the integer type inference via a callback passed to the bindgen library to cause the #define decls in sqlite3.h to be i32 instead of u32. I'll be needing to write a bindgen rule that uses the bindgen library internally rather than the binary.

@mainrs
Copy link

mainrs commented Sep 30, 2018

@acmcarther I am currently trying to get diesel to compile through bazel but I always get weird errors with the following message: error: linking with /bin/gcc failed: exit code 1. Not sure what I am actually doing wrong. IIRC diesel does use libsqlite3-sys for the sqlite3 feature. Do I have to do any additional configuration? I have added the gen_buildrs flag but it still fails 😭

Did you somehow solve the problem?

@acmcarther
Copy link
Member

@sirwindfield
If your problem is the libsqlite3-sys, I can help with that, but you'll need to convert these instructions from rusqlite to diesel. This is also a pretty clunky process because I didn't actually write a bindgen rule, i just made this work manually:

  1. First, I vendored sqlite3, with the following additions:
    A "bindgen runner" that invokes bindgen via Rust over sqlite3.h. Note that this would probably be better placed into libsqlite3-sys and run over wrapper.h, but I was having a hard time making bindgen's clang include path work correctly. The build for sqlite3 also generates an out_dir_tar for libsqlite3-sys

  2. Next, I brought down libsqlite3-sys into a local directory so that I could override it's BUILD file.
    This overridden build file is identical to what raze produces, but it depends on my local sqlite3, and has sqlite3's out_tar as its out_dir_tar.

  1. Finally, I overrode the libsqlite3-sys dependency for rustqlite with my local libsqlite3-sys

@mfarrugi
Copy link
Collaborator Author

For completeness, here's our current raze Cargo.toml:

third_party/rust/cargo/Cargo.toml

[package]
name = "cargo_via_bazel"
version = "0.0.0"

[lib]
path = "fake_lib.rs"

[dependencies]
# Mostly libraries recommended by https://github.com/brson/stdx
bitflags            = "1.0.4"
backtrace           = "0.3.9"
base64              = "0.9.0"
bindgen             = "0.40.0"
byteorder           = "1.2.6"
bytes               = "0.4.10"
chrono              = "0.4.6"
clap                = "2.32.0"
crossbeam-channel   = "0.2.6"
csv                 = "1.0.1"
env_logger          = "0.5.13"
error-chain         = "0.12.0"
failure             = "0.1.2"
fnv                 = "1.0.6"
flate2              = "1.0.2"
futures             = "0.1.24"
hyper               = "0.11.22"
hyper-tls           = "0.1.2"
itertools           = "0.7.8"
lazy_static         = "1.0.1"
libc                = "0.2.43"
log                 = "0.4.5"
memmap              = "0.6.2"
native-tls          = "0.1.5"
ndarray             = "0.12.0"
num                 = "0.2.0"
num_cpus            = "1.8.0"
pretty_env_logger   = "0.2.4"
protobuf            = "1.4.4"
rand                = "0.5.5"
rayon               = "1.0.2"
regex               = "1.0.1"
reqwest             = "0.9.0"
semver              = "0.9.0"
serde               = "1.0.79"
serde_derive        = "1.0.79"
serde_json          = "1.0.28"
structopt           = "0.2.10"
structopt-derive    = "0.2.10"
tar                 = "0.4.13"
tempdir             = "0.3.5"
threadpool          = "1.7.1"
tokio               = "0.1.7"
tokio-codec         = "0.1.0"
tokio-core          = "0.1.17"
tokio-io            = "0.1.6"
toml                = "0.4.5"
tower-web           = "0.2.2"
url                 = "1.6.0"
walkdir             = "2.0.1"
zip                 = "0.3.1"

[raze]
workspace_path = "//third_party/rust/cargo"
target = "x86_64-unknown-linux-gnu"

[raze.crates.unicase.'1.4.2']
additional_flags = [
  "--cfg=__unicase__iter_cmp",
  "--cfg=__unicase__defauler_hasher",
]

[raze.crates.unicase.'2.1.0']
additional_flags = [
  "--cfg=__unicase__iter_cmp",
  "--cfg=__unicase__defauler_hasher",
]

[raze.crates.crc.'1.8.1']
gen_buildrs = true

[raze.crates.mime_guess.'2.0.0-alpha.6']
gen_buildrs = true

[raze.crates.openssl-sys.'0.9.36']
additional_deps = [
  "@openssl//:lib",
]
# openssl-sys is painful.  The magic incantation of flags was determined by doing the following:
#   1) git clone https://github.com/sfackler/rust-openssl
#   2) cd rust-openssl/openssl-sys
#   3) export OPENSSL_DIR=/path/to/openssl
#   4) cargo build -vv
# and observing the cfg's passed/used by its build.rs
additional_flags = [
    # ...
]

[raze.crates.openssl.'0.10.12']
additional_flags = [
    # ...
]

[raze.crates.openssl.'0.9.24']
additional_flags = [
    # ...
]


[raze.crates.bzip2-sys.'0.1.6']
additional_deps = [
  "//third_party/bz2/1/0/6:bz2",
]

[raze.crates.bindgen.'0.40.0']
gen_buildrs = true
extra_aliased_targets = [
  "cargo_bin_bindgen"
]

[raze.crates.clang-sys.'0.23.0']
gen_buildrs = false
# Overriden to hand-written BUILD file, because libloading is building its own c portion.
skipped_deps = [
    "libloading-0.5.0"
]
additional_deps = [
    "//third_party/rust/cargo/overrides/libloading-0.5.0:libloading",
]

[raze.crates.backtrace.'0.3.9']
# Similar to `clang-sys` above, we build backtrace-sys manually with Bazel overrides
# to compile libbacktrace from packaged sources
skipped_deps = [
    "backtrace-sys-0.1.24",
]
additional_deps = [
    "//third_party/rust/cargo/overrides/backtrace-sys-0.1.23:backtrace_sys",
]

[raze.crates.lazy_static.'1.1.0']
additional_flags = [
    "--cfg=lazy_static_inline_impl",
]

[raze.crates.proc-macro2.'0.4.19']
additional_flags = [
    "--cfg use_proc_macro",
]

third_party/rust/cargo/overrides/backtrace-sys-0.1.23/BUILD

"""
OVERRIDDEN:
cargo-raze crate build file.

- `backtrace-sys` comes with `libloading`, which we build via targets here rather
   than using backtrace-sys's build.rs
"""
package(default_visibility=["//third_party/rust/cargo:__subpackages__"])

licenses([
    "notice",  # "MIT,Apache-2.0"
])

load(
    "@io_bazel_rules_rust//rust:rust.bzl",
    "rust_library",
    "rust_binary",
    "rust_test",
    "rust_benchmark",
)

genrule(
    name="touch_config_header",
    outs=[
        "config.h",
    ],
    cmd="touch $@",
)

genrule(
    name="touch_backtrace_supported_header",
    outs=[
        "backtrace-supported.h",
    ],
    cmd="touch $@",
)

# This should roughly follow the Rust build script: https://github.com/alexcrichton/backtrace-rs/blob/master/backtrace-sys/build.rs
cc_library(
    name="backtrace_native",
    srcs=[
        ":touch_config_header",
        ":touch_backtrace_supported_header",
        "src/libbacktrace/internal.h",
        "src/libbacktrace/backtrace.h",
        "src/libbacktrace/alloc.c",
        "src/libbacktrace/dwarf.c",
        "src/libbacktrace/fileline.c",
        "src/libbacktrace/posix.c",
        "src/libbacktrace/read.c",
        "src/libbacktrace/sort.c",
        "src/libbacktrace/state.c",
        "src/libbacktrace/elf.c",
    ],
    includes=["."],
    copts=[
        "-fvisibility=hidden",
        "-fPIC",
    ],
    defines=[
        "BACKTRACE_ELF_SIZE=64",
        "BACKTRACE_SUPPORTED=1",
        "BACKTRACE_USES_MALLOC=1",
        "BACKTRACE_SUPPORTS_THREADS=0",
        "BACKTRACE_SUPPORTS_DATA=0",
        "HAVE_DL_ITERATE_PHDR=1",
        "_GNU_SOURCE=1",
        "_LARGE_FILES=1",
        "backtrace_full=__rbt_backtrace_full",
        "backtrace_dwarf_add=__rbt_backtrace_dwarf_add",
        "backtrace_initialize=__rbt_backtrace_initialize",
        "backtrace_pcinfo=__rbt_backtrace_pcinfo",
        "backtrace_syminfo=__rbt_backtrace_syminfo",
        "backtrace_get_view=__rbt_backtrace_get_view",
        "backtrace_release_view=__rbt_backtrace_release_view",
        "backtrace_alloc=__rbt_backtrace_alloc",
        "backtrace_free=__rbt_backtrace_free",
        "backtrace_vector_finish=__rbt_backtrace_vector_finish",
        "backtrace_vector_grow=__rbt_backtrace_vector_grow",
        "backtrace_vector_release=__rbt_backtrace_vector_release",
        "backtrace_close=__rbt_backtrace_close",
        "backtrace_open=__rbt_backtrace_open",
        "backtrace_print=__rbt_backtrace_print",
        "backtrace_simple=__rbt_backtrace_simple",
        "backtrace_qsort=__rbt_backtrace_qsort",
        "backtrace_create_state=__rbt_backtrace_create_state",
        "backtrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug",
    ],
)

rust_library(
    name="backtrace_sys",
    crate_root="src/lib.rs",
    crate_type="lib",
    srcs=glob(["**/*.rs"]),
    deps=[
        "//third_party/rust/cargo/vendor/libc-0.2.43:libc",
        ":backtrace_native",
    ],
    rustc_flags=[
        "--cap-lints allow",
        
    ],
    crate_features=[
    ],
)

third_party/rust/cargo/overrides/libloading-0.5.0/BUILD

"""
OVERRIDDEN:
cargo-raze crate build file.

- Libloading has a CC dep that needs to be brought in
"""

package(default_visibility = ["//visibility:public"])

licenses([
    "notice",  # "ISC"
])

load(
    "@io_bazel_rules_rust//rust:rust.bzl",
    "rust_library",
    "rust_binary",
    "rust_test",
    "rust_benchmark",
)

cc_library(
    name = "global_whatever",
    srcs = [
        "src/os/unix/global_static.c",
    ],
    copts = [ "-fPIC" ],
)

rust_library(
    name = "libloading",
    srcs = glob(["**/*.rs"]),
    crate_features = [
    ],
    crate_root = "src/lib.rs",
    crate_type = "lib",
    rustc_flags = [
        "--cap-lints allow",
        
    ],
    deps = [
        ":global_whatever",
    ],
)

this also serves as some additional context for #58

@akhilles
Copy link

akhilles commented May 11, 2019

A few notes from my attempt at looking at ring that @mfarrugi brought up:

  • https://github.com/briansmith/ring/blob/master/build.rs#L270 looks for the CARGO_PKG_NAME envvar - this isn't set in the build script wrapper that cargo raze generates (but is set in bazel rust, for example: https://github.com/bazelbuild/rules_rust/blob/master/rust/toolchain.bzl#L23).
    I'm pretty sure the intended build path here is to end up calling the ring_build_rs_main() fn, rather than enter the pregeneration step via pregenerate_asm_main(). I hand edited cargo raze's ring_build_script_executor genrule to get past this point.
  • Once I forced execution into ring_build_rs_main, we once again get blocked on missing envvars. In all, I had to add this to the same genrule:
 + " export CARGO_CFG_TARGET_ARCH=x86_64;"
        + " export CARGO_CFG_TARGET_OS=linux;"
        + " export CARGO_CFG_TARGET_ENV=;"
        + " export DEBUG=false;"
        + " export OPT_LEVEL=3;"
        + " export HOST=;"

The latter 3 are needed by the gcc invocation. Apparently, cargo sets this all up for build scripts, and I suspect cargo raze will eventually need to reimplement a whole bunch (if not all) of this.

HTH

@vitalyd covered almost all of the changes needed to get ring to compile. In order to address the include_bytes macro issue, some data directories need to be included in the sandbox. I was too lazy to find exactly what's needed, so I just added data_attr = "glob([\"src/**\"])" to Cargo.toml and this worked.

@prestonvanloon
Copy link

I tried the suggestions from #41 (comment) and #41 (comment) could not get ring to build.

ERROR: /home/preston/.cache/bazel/_bazel_preston/2bf51e8690558a83a0a0514ab422fd4b/external/raze__ring__0_14_6/BUILD.bazel:46:1: Executing genrule @raze__ring__0_14_6//:ring_build_script_executor failed (Exit 101)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 17, kind: AlreadyExists, message: "File exists" }', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   6: rust_begin_unwind
             at src/libstd/panicking.rs:308
   7: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   8: core::result::unwrap_failed
   9: ring_build_script::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:293
  12: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:87
  13: std::rt::lang_start_internal
             at src/libstd/panicking.rs:272
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  14: main
  15: __libc_start_main
  16: _start

@jonhoo
Copy link

jonhoo commented Nov 11, 2019

I think it'd be really useful to have an example in this repository for how to write a -sys crate that fits into an existing Bazel pipeline using cargo raze. Even one that has a relatively straightforward build.rs that just runs bindgen on the output of a cc_library would go a long way! This thread is the only real source of such examples at the moment, and they are all for relatively complicated builds.

@nhyne
Copy link

nhyne commented Dec 7, 2019

@prestonvanloon were you able to make any more progress on ring? I'm running into the same issue you have.

@prestonvanloon
Copy link

I don’t recall reaching a solution. Gave up on rust with bazel for now

@joprice
Copy link

joprice commented Dec 21, 2019

@mfarrugi Did you try getting openssl working with boringssl? It's especially convenient since they maintain a bazel branch.

@mfarrugi
Copy link
Collaborator Author

mfarrugi commented Dec 21, 2019 via email

@joprice
Copy link

joprice commented Dec 21, 2019

No problem. I'll keep smashing my head against it. I'm also hitting a glorious infinite loop when trying to compile unicase with the error "calls in statics are limited to constant functions, tuple structs and tuple variants".

@gluax
Copy link

gluax commented Dec 28, 2019

Has anyone gotten backtrace-0.3.4 to compile with bazel? I keep getting the below errors.

Loading: 
Loading: 0 packages loaded
Analyzing: target //rust/game:game (0 packages loaded, 0 targets configured)
INFO: Analyzed target //rust/game:game (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 7] [Prepa] BazelWorkspaceStatusAction stable-status.txt
ERROR: /home/gluax/docs/projects/wander-interactive-fiction-engine/cargo/vendor/backtrace-0.3.40/BUILD:29:1: error executing shell command: '/bin/bash -c CARGO_MANIFEST_DIR=$(pwd)/cargo/vendor/backtrace-0.3.40 external/rust_linux_x86_64/bin/rustc "$@" --remap-path-prefix="$(pwd)"=__bazel_redacted_pwd  cargo/vendor/backtrace-0.3.40/src/l...' failed (Exit 1)
error[E0308]: mismatched types
   --> cargo/vendor/backtrace-0.3.40/src/symbolize/libbacktrace.rs:100:45
    |
96  |     pub fn addr(&self) -> Option<*mut c_void> {
    |                           ------------------- expected `core::option::Option<*mut libc::unix::c_void>` because of return type
...
100 |             Symbol::Dladdr(ref s) => return s.addr(),
    |                                             ^^^^^^^^ expected enum `libc::unix::c_void`, found enum `core::ffi::c_void`
    |
    = note: expected type `core::option::Option<*mut libc::unix::c_void>`
               found type `core::option::Option<*mut core::ffi::c_void>`

error[E0308]: mismatched types
   --> cargo/vendor/backtrace-0.3.40/src/symbolize/libbacktrace.rs:433:32
    |
433 |         return dladdr_fallback(what.address_or_ip(), cb);
    |                                ^^^^^^^^^^^^^^^^^^^^ expected enum `libc::unix::c_void`, found enum `core::ffi::c_void`
    |
    = note: expected type `*mut libc::unix::c_void`
               found type `*mut core::ffi::c_void`

error[E0308]: mismatched types
   --> cargo/vendor/backtrace-0.3.40/src/symbolize/libbacktrace.rs:461:25
    |
461 |         dladdr_fallback(what.address_or_ip(), cb);
    |                         ^^^^^^^^^^^^^^^^^^^^ expected enum `libc::unix::c_void`, found enum `core::ffi::c_void`
    |
    = note: expected type `*mut libc::unix::c_void`
               found type `*mut core::ffi::c_void`

error[E0308]: mismatched types
   --> cargo/vendor/backtrace-0.3.40/src/symbolize/libbacktrace.rs:466:21
    |
466 |     dladdr::resolve(addr, &mut |sym| {
    |                     ^^^^ expected enum `core::ffi::c_void`, found enum `libc::unix::c_void`
    |
    = note: expected type `*mut core::ffi::c_void`
               found type `*mut libc::unix::c_void`

error: aborting due to 4 previous errors

For more information about this error, try `rustc --explain E0308`.
Target //rust/game:game failed to build
INFO: Elapsed time: 0.568s, Critical Path: 0.51s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully

and my cargo.toml is as follows.

package]
name = "interactive-fiction"
version = "0.0.0"

# Mandatory (or Cargo tooling is unhappy)
[lib]
path = "fake_lib.rs"

[raze]
# The WORKSPACE relative path to the Cargo.toml working directory.
workspace_path = "//cargo"

# The target to generate BUILD rules for.
target = "x86_64-unknown-linux-gnu"

[dependencies]
serde_json = "1.0.41"
diesel = { version = "1.4.3", features = ["sqlite"] }
serde = { version = "1.0.104", features = ["derive"] }
toml = "0.5.5"
gluon = "0.13.1"

[raze.crates.backtrace.'0.3.40']
skipped_deps = [
    "backtrace-sys-0.1.32",
]
additional_deps = [
    "//cargo/overrides/backtrace-sys-0.1.32:backtrace_sys",
]

[raze.crates.indexmap.'1.3.0']
additional_flags = [
"--cfg=has_std",
]

[raze.crates.'libsqlite3-sys'.'0.16.0']
gen_buildrs = true

[raze.crates.proc-macro2.'1.0.6'] 
additional_flags = [
"--cfg=use_proc_macro", 
]

@a1ph
Copy link
Contributor

a1ph commented Jun 12, 2020

@kornholi I tweaked mio dependency crates versions with additional_deps.

Then I stuck with openssl, but finally made it work today, although it required changes to rules_rust and cargo-raze

@a1ph
Copy link
Contributor

a1ph commented Jul 1, 2020

With the latest changes to rules_rust and cargo-raze I was able to remove all the tweaks and build with just default_gen_buildrs = true. 🎉

@hexdae
Copy link

hexdae commented Aug 6, 2020

The latest changes are amazing and solved most of the issues in my project. However, for some reason the default build script generated by cargo raze for libusb1-sys 0.3.7 hangs forever while building. (context, I am trying to build a project that depends on rusb 0.6

I tried adding the system libusb as a dependency using new_local_repository but I got the same result. Any suggestions would be very welcome.

Example `Cargo.toml` file to reproduce
[package]
name = "LibUSB_App"
version = "0.1.0"

[lib]
path = "fake_lib.rs"

[dependencies]
rusb = "0.6"
byte_struct = "0.4.2"

[raze]
workspace_path = "//cargo"
output_buildfile_suffix = "BUILD.bazel"
default_gen_buildrs = true

@parisholley
Copy link

couple notes for people who have some of the issues I had:

  • don't use create vendoring, lots of headaches that just aren't supported well if your dependencies require building c' libs
  • if you use default_gen_buildrs = true and run into some builds that seemingly fail to compile C code due to arch related issues, try adding gen_buildrs = false to the override, like i had to do for ring based on the example above:
[raze.crates.ring.'0.16.14']
gen_buildrs = false
data_attr = 'glob(["**/src/data/*", "**/src/ec/**/*.der"])'
additional_deps = [
    # provided by the additional_build_file
    ":ring-core",
]
additional_build_file = "builds/ring-0.16.BUILD"

for anyone running into issues with native-tls (macos build), i had to manually import all of it's extern crate dependencies:

security-framework = "0.4.4" # bazel fix
security-framework-sys = "0.4.3"
tempfile = "3.1.0"
libc = "0.2.76"

and add them as additional deps

[raze.crates.native-tls.'0.2.4']
additional_deps = [
    "@raze__lazy_static__1_4_0//:lazy_static",
    "@raze__security_framework__0_4_4//:security_framework",
    "@raze__security_framework_sys__0_4_3//:security_framework_sys",
    "@raze__tempfile__3_1_0//:tempfile",
    "@raze__libc__0_2_76//:libc"
]

@epigramengineer
Copy link

An interesting example I've found amethyst which has a lot of sys dependencies. I thought it would be clever to pull them in rules_nixpkgs:

rules_nixpkgs_version="dc24090573d74adcf38730422941fd69b87682c7"
rules_nixpkgs_sha256="aca86baa64174478c57f74ed09d5c2313113abe94aa3af030486d1b14032d3ed"

http_archive(
    name = "io_tweag_rules_nixpkgs",
    strip_prefix = "rules_nixpkgs-{0}".format(rules_nixpkgs_version),
    url = "https://github.com/tweag/rules_nixpkgs/archive/{0}.tar.gz".format(rules_nixpkgs_version),
    sha256 = rules_nixpkgs_sha256,
)

load("@io_tweag_rules_nixpkgs//nixpkgs:repositories.bzl", "rules_nixpkgs_dependencies")
rules_nixpkgs_dependencies()

load("@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl", "nixpkgs_git_repository", "nixpkgs_package")
nixpkgs_git_repository(
    name = "nixpkgs",
    revision = nixpkgs_version,
    sha256 = nixpkgs_sha256,
)

nixpkgs_package(name = "alsa-lib", repository = "@nixpkgs")
nixpkgs_package(name = "cmake", repository = "@nixpkgs")
nixpkgs_package(name = "expat", repository = "@nixpkgs")
nixpkgs_package(name = "openssl", repository = "@nixpkgs")
nixpkgs_package(name = "pkg-config", repository = "@nixpkgs")
nixpkgs_package(name = "python3", repository = "@nixpkgs")
nixpkgs_package(name = "vulkan-validation-layers", repository = "@nixpkgs")
nixpkgs_package(name = "xorg.libX11", repository = "@nixpkgs")

However actually configuring them as dependencies on each of the sys crates has proven difficult. Are there any recommendations on how debugging / identifying configuration options and specific flags. So far I've been using `bazel query deps(//3rdparty/cargo:amethyst)' to find the quickly identify dependent crates, but I don't know how to go about actually finding all the necessary flags or options.

@mfarrugi
Copy link
Collaborator Author

mfarrugi commented Sep 11, 2020 via email

@cherryland
Copy link

We dropped google/cargo-raze and adapted the build-strategy of project-oak, by simply referencing the generated artifacts (cargo-build)

# These files are built via cargo outside of Bazel
exports_files(srcs = glob(["target/x86_64-unknown-linux-musl/release/*"]))

It just works.

Messing around Cargo, bazelbuild/rules_rust and google/cargo-raze is an absolute waste of time, energy, resources and money.

@mfarrugi
Copy link
Collaborator Author

mfarrugi commented Oct 10, 2020 via email

@nikclayton-dfinity
Copy link

I'm trying to get logos-derive building (https://github.com/maciejhirsz/logos/blob/master/logos-derive/Cargo.toml), it's a dependency's dependency.

But the build fails with:

% bazel build :all
INFO: Analyzed 2 targets (5 packages loaded, 127 targets configured).
INFO: Found 2 targets...
INFO: From Compiling Rust proc-macro logos_derive v0.11.5 (23 files):
error[E0432]: unresolved import `proc_macro`
  --> external/raze__logos_derive__0_11_5/src/lib.rs:25:5
   |
25 | use proc_macro::TokenStream;
   |     ^^^^^^^^^^ help: a similar path exists: `syn::proc_macro`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
ERROR: /private/var/tmp/_bazel_nik/b7eabcb2312aee3132f61301201aeb4a/external/raze__logos_derive__0_11_5/BUILD.bazel:31:13: output 'external/raze__logos_derive__0_11_5/liblogos_derive-1722838883.dylib' was not created
ERROR: /private/var/tmp/_bazel_nik/b7eabcb2312aee3132f61301201aeb4a/external/raze__logos_derive__0_11_5/BUILD.bazel:31:13: not all outputs were created or valid
INFO: Elapsed time: 2.108s, Critical Path: 1.64s
INFO: 11 processes: 10 internal, 1 darwin-sandbox.
FAILED: Build did NOT complete successfully

This is with

[raze.crates.logos-derive.'0.11.5']
additional_flags = [
    "--cfg=use_proc_macro",
    "--cfg=wrap_proc_macro",
]

in the Cargo.toml file. The generated BUILD file for it is:

"""
@generated
cargo-raze crate build file.

DO NOT EDIT! Replaced on runs of cargo-raze
"""

# buildifier: disable=load
load(
    "@io_bazel_rules_rust//rust:rust.bzl",
    "rust_binary",
    "rust_library",
    "rust_test",
)

package(default_visibility = [
    # Public for visibility by "@raze__crate__version//" targets.
    #
    # Prefer access through "//cargo", which limits external
    # visibility to explicit Cargo.toml dependencies.
    "//visibility:public",
])

licenses([
    "notice",  # MIT from expression "MIT OR Apache-2.0"
])

# Generated targets

# buildifier: leave-alone
rust_library(
    name = "logos_derive",
    crate_type = "proc-macro",
    deps = [
        "@raze__beef__0_4_4//:beef",
        "@raze__fnv__1_0_7//:fnv",
        "@raze__proc_macro2__1_0_24//:proc_macro2",
        "@raze__quote__1_0_7//:quote",
        "@raze__regex_syntax__0_6_20//:regex_syntax",
        "@raze__syn__1_0_44//:syn",
        "@raze__utf8_ranges__1_0_4//:utf8_ranges",
    ],
    srcs = glob(["**/*.rs"]),
    crate_root = "src/lib.rs",
    edition = "2018",
    rustc_flags = [
        "--cap-lints=allow",
        "--cfg=use_proc_macro",
        "--cfg=wrap_proc_macro",
    ],
    version = "0.11.5",
    tags = [
        "cargo-raze",
        "manual",
    ],
    crate_features = [
    ],
)

I assume I've missed something obvious, but for the life of me I can't see what it is. Anyone got any pointers?

@mfarrugi
Copy link
Collaborator Author

This was fixed earlier today by bazelbuild/rules_rust#439

@nikclayton-dfinity
Copy link

nikclayton-dfinity commented Oct 16, 2020

@mfarrugi That's great, thanks for the pointer.

Do you think it would be worthwhile adding a repository where people can submit the overrides that they've determined need to happen, and integrating that in to cargo-raze?

For example, I've just discovered that to build the json5 crate you need:

[raze.crates.json5.'0.2.8']
data_attr = "['src/json5.pest']"

To discover that I had (a) diagnose the build failure, (b) poke through the cargo-raze code to discover how to add a custom data attribute, (c) actually add the attribute.

The fix above is going to need to be done by anyone who wants to use that crate with cargo-raze (and I've posted it here so people searching for cargo-raze json5 in the future have a chance of finding it.


Note that the above works with a version of io_bazel_rules_rust from 2020-10-15, but not 2020-10-26.

I.e., with this in the WORKSPACE file it works:

http_archive(
    name = "io_bazel_rules_rust",
    sha256 = "74bcc568a2cb8e410967eae7b44b8303062257ec2a69f59737173be5eab48b65",
    strip_prefix = "rules_rust-8a8be5834e5f226e9f01ecc77929e12563f18cb5",
    urls = [
        # Master branch as of 2020-10-15
        # Includes #439, see https://github.com/google/cargo-raze/issues/41#issuecomment-709600426
        "https://github.com/bazelbuild/rules_rust/archive/8a8be5834e5f226e9f01ecc77929e12563f18cb5.tar.gz",
    ],
)

And with this it doesn't:

http_archive(
    name = "io_bazel_rules_rust",
    sha256 = "1211f57f6cbc5d68aff40d78d2548c8adee8f2921b54dca6ec9a9e7df60db13b",
    strip_prefix = "rules_rust-224fe6a81b8ffa9c585c75f49a41f9b9924d516d",
    urls = [
        # Master branch as of 2020-10-26
        # Includes #439, see https://github.com/google/cargo-raze/issues/41#issuecomment-709600426
        "https://github.com/bazelbuild/rules_rust/archive/224fe6a81b8ffa9c585c75f49a41f9b9924d516d.tar.gz",
    ],
)

224fe6a81b8ffa9c585c75f49a41f9b9924d516d over there is the commit that breaks it, I've tested other commits up to and including the one immediately preceding it, 9c889b057ddf4feddae7c5ae6913b7282154aa24, and they work.


More details in bazelbuild/rules_rust#467

@zoidyzoidzoid
Copy link

@nikclayton-dfinity That would be amazing.

Maybe in here? https://github.com/acmcarther/cargo-raze-crater/tree/master/cargo

@acmcarther
Copy link
Member

We should consider moving that half-baked thing into this repo (probably outside of impl/, unless we're really good and can integrate these into raze's generation step... which would be awesome)

@nikclayton-dfinity
Copy link

Another useful source of rules -- Fuschia uses GNaw, which is like cargo-raze, but for the GN build system.

https://fuchsia.googlesource.com/fuchsia/+/master/third_party/rust_crates/Cargo.toml is their equivalent of the workspace Cargo.toml file, and the bottom half of the file has overrides for a lot of crates.

@agentydragon
Copy link

Another snowflake package is derive_builder, see bazelbuild/rules_rust#594.

@sinkingpoint
Copy link

sinkingpoint commented Jun 25, 2021

khronos_api seems to be another one.

compile_data_attr = "glob([\"api*/**/*.xml\", \"api*/**/*.idl\"])"

works to some extent, but the build.rs generates lots of include_bytes with absolute paths which seems to break things, e.g.

&*include_bytes!("/home/colin/.cache/bazel/_bazel_colin/4f2575136be2697c60d69a1bb6960d5d/sandbox/linux-sandbox/678/execroot/__main__/bazel-out/k8-opt-exec-2B5CBBC6/bin/src/cargo/vendor/khronos_api-3.1.0/khronos_api_build_script_script_.runfiles/__main__/src/cargo/vendor/khronos_api-3.1.0/api_webgl/extensions/WEBGL_security_sensitive_resources/extension.xml

Found when trying to compile smithay

@kshcherban
Copy link

khronos_api seems to be another one.

compile_data_attr = "glob([\"api*/**/*.xml\", \"api*/**/*.idl\"])"

works to some extent, but the build.rs generates lots of include_bytes with absolute paths which seems to break things, e.g.

&*include_bytes!("/home/colin/.cache/bazel/_bazel_colin/4f2575136be2697c60d69a1bb6960d5d/sandbox/linux-sandbox/678/execroot/__main__/bazel-out/k8-opt-exec-2B5CBBC6/bin/src/cargo/vendor/khronos_api-3.1.0/khronos_api_build_script_script_.runfiles/__main__/src/cargo/vendor/khronos_api-3.1.0/api_webgl/extensions/WEBGL_security_sensitive_resources/extension.xml

Found when trying to compile smithay

Tried to build with a modified build.rs, still didn't find a way:

ERROR: /home/kshcherban/.cache/bazel/_bazel_kshcherban/7ed6937fc5de55037626fc8eb5ff0996/external/raze__khronos_api__3_2_0/BUILD.bazel:40:19: CargoBuildScriptRun external/raze__khronos_api__3_2_0/khronos_api_build_script.out_dir failed: (Exit 1): cargo_build_script_runner failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_rust/cargo/cargo_build_script_runner/cargo_build_script_runner ... (remaining 9 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Build script process failed with exit code 101
--stdout:

--stderr:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', external/raze__khronos_api__3_2_0/khronos_api/build.rs:44:10

@kshcherban
Copy link

@sinkingpoint managed to fix khronos_api build with these changes: kshcherban/gl-rs@1a3d96c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests