Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update libFuzzer to llvm/llvm-project@60e32a1 #89

Merged
merged 4 commits into from
Mar 3, 2022

Conversation

fitzgen
Copy link
Member

@fitzgen fitzgen commented Jan 24, 2022

Supercedes #88

@nagisa
Copy link
Member

nagisa commented Jan 24, 2022

bors r+

never remember if we have this set up here or not... (we also will need to update our test suite sometime down the line for new pass names I bet)

@bors
Copy link
Contributor

bors bot commented Jan 24, 2022

Configuration problem:
bors.toml: not found

Copy link
Member

@Manishearth Manishearth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ci fails

@fitzgen
Copy link
Member Author

fitzgen commented Jan 25, 2022

Need to switch -Cpasses=sancov to -Cpasses=sancov-module for newer rustc/LLVM. New commit should fix CI.

@fitzgen
Copy link
Member Author

fitzgen commented Jan 25, 2022

Linking issues in CI. Not totally sure what's going on here.

https://github.com/rust-fuzz/libfuzzer/runs/4930210184?check_suite_focus=true#step:5:74

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "-Wl,-Bstatic" "-Wl,--whole-archive" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc-nightly_rt.asan.a" "-Wl,--no-whole-archive" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.0.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.1.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.10.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.11.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.12.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.13.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.14.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.15.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.2.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.3.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.4.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.5.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.6.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.7.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.8.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.9.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.1w1yy0cu9bxvqmzw.rcgu.o" "-Wl,--as-needed" "-L" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps" "-L" "/home/runner/work/libfuzzer/libfuzzer/target/release/build/libfuzzer-sys-60bdc856fc564d33/out" "-L" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/liblibfuzzer_sys-9667746bd1260caf.rlib" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/libonce_cell-7957dbfbd59bdd9d.rlib" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/libarbitrary-6173795e953c05ad.rlib" "-Wl,--start-group" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-516357af627e1a7d.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9cb5ef8235cf10de.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c94e3d0069d19330.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-5a532766d1e80b5b.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-d08f0c83526179dc.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-d995c6cafdb3d253.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-02ebc04a72bd9433.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-5cbc32ae14534ca9.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-969ebc8623abfcc9.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-7f98f837d3579544.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-c79b77d9c2e73c7a.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-c7b2792f3bc0d0eb.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-7558dde8464d46ce.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-b3d84848ab790cf7.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f48e57d6d73020ee.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-ffaac76756e3f83b.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-2a6a2797f7a73818.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-0e3656b1fda5fd7b.rlib" "-Wl,--end-group" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-f98cce041f593917.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: `.text.sancov.module_ctor_8bit_counters.9' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.9]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.3.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.9[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.3.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.6' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.6]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.1.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.6[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.1.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.4' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.4]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.10.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.4[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.10.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.8' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.8]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.15.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.8[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.15.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.11' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.11]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.2.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.11[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.2.rcgu.o
          collect2: error: ld returned 1 exit status

@Badel2
Copy link

Badel2 commented Feb 26, 2022

@fitzgen I get the same error when trying to compile master. A workaround is to add -Ccodegen-units=1 \ to every command in ci/script.sh. But note that this will probably force end users to also add this flag when compiling.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 28, 2022

Which means we would probably need to add it to cargo-fuzz as well.

And that is going to slow down build times too. Very much not ideal. Would prefer to figure out how to fix these linker errors without resorting to a single codegen unit, but I don't have time to dig into this myself right now.

@Badel2
Copy link

Badel2 commented Mar 1, 2022

I tried to take a look but linker errors are a bit outside my area of expertise. But I did found another strange bug, see #90.

This is not ideal, but it does provide two things:

1. It fixes bizarre linker errors about missing `sancov` symbols.

2. It allows LLVM to do inlining that it otherwise refuses to do. For some
reason, when sanitizers are enabled, LLVM refuses to inline across codegen
units. This is a problem because trivial methods like `Vec::len` won't be
inlined, resulting in 100x slowdowns.

`cargo fuzz` already restricts its builds to a single codegen unit, so we might
as well do the same thing in CI here.
@fitzgen
Copy link
Member Author

fitzgen commented Mar 3, 2022

Okay I pushed a commit to do -Ccodegen-units=1.

This is not ideal, but it does provide two things:

  1. It fixes bizarre linker errors about missing sancov symbols.

  2. It allows LLVM to do inlining that it otherwise refuses to do. For some
    reason, when sanitizers are enabled, LLVM refuses to inline across codegen
    units. This is a problem because trivial methods like Vec::len won't be
    inlined, resulting in 100x slowdowns.

cargo fuzz already restricts its builds to a single codegen unit, so we might
as well do the same thing in CI here.

@fitzgen
Copy link
Member Author

fitzgen commented Mar 3, 2022

Success!

@fitzgen fitzgen merged commit 76f10a7 into rust-fuzz:master Mar 3, 2022
@fitzgen fitzgen deleted the update-libfuzzer-60e32a1 branch March 3, 2022 17:36
alexcrichton added a commit to alexcrichton/libfuzzer that referenced this pull request Mar 3, 2022
This commit migrates CI to using the `cargo fuzz` binary with its
settings for compiling Rust code to avoid mismatches like the codegen
unit issue found in rust-fuzz#89
alexcrichton added a commit to alexcrichton/libfuzzer that referenced this pull request Mar 3, 2022
This commit migrates CI to using the `cargo fuzz` binary with its
settings for compiling Rust code to avoid mismatches like the codegen
unit issue found in rust-fuzz#89
@alexcrichton
Copy link
Contributor

FWIW there's some more info about the 1 cgu issue at rust-fuzz/cargo-fuzz#215

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants