Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fuzz: Add test/fuzz/test_runner.py and run it in travis #15295

Merged
merged 2 commits into from Feb 14, 2019

Conversation

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Jan 30, 2019

Can be run with ./test/fuzz/test_runner.py after building as described in doc/fuzzing.md

@practicalswift
Copy link
Contributor

@practicalswift practicalswift commented Jan 30, 2019

Concept ACK. Very nice!

Related: #10364 ("Feature request: Make Bitcoin libFuzzer-friendly and consider integration into the OSS-Fuzz project"). Feel free to collect the $20 000 USD bounty :-)

@MarcoFalke MarcoFalke force-pushed the Mf1901-qaFuzz branch 4 times, most recently from 032ac70 to 4da6a56 Jan 30, 2019
@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Jan 30, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #15134 (tests: Add a Travis ASan/LSan/UBSan job testing in a unsigned char environment (-funsigned-char) by practicalswift)
  • #15063 (GUI: If BIP70 is disabled, attempt to fall back to BIP21 parsing by luke-jr)
  • #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@MarcoFalke MarcoFalke force-pushed the Mf1901-qaFuzz branch 2 times, most recently from 1074dc1 to 0ca2db9 Jan 30, 2019
@fanquake fanquake added the Tests label Jan 30, 2019
@laanwj
Copy link
Member

@laanwj laanwj commented Jan 31, 2019

Can you please explain what you're adding here—
Will adding fuzzing to travis make the tests non-deterministic, by randomizing it? Or does it only verify the current corpus?

@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Jan 31, 2019

Indeed the script currently only supports running over all seeds exactly once

@Sjors
Copy link
Member

@Sjors Sjors commented Feb 1, 2019

Concept ACK. I can't get fuzzing to work on macOS (not that I tried hard), so having it on Travis sounds like a safe idea, if only to ensure people don't accidentally break the build.

test/fuzz/test_runner.py Outdated Show resolved Hide resolved
test/fuzz/test_runner.py Outdated Show resolved Hide resolved
test/fuzz/test_runner.py Outdated Show resolved Hide resolved
@MarcoFalke MarcoFalke force-pushed the Mf1901-qaFuzz branch 3 times, most recently from 0491dcb to 03ba30a Feb 11, 2019
test/fuzz/test_runner.py Outdated Show resolved Hide resolved
test/fuzz/test_runner.py Outdated Show resolved Hide resolved
doc/fuzzing.md Outdated Show resolved Hide resolved
@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Feb 13, 2019

Thx, done

@practicalswift
Copy link
Contributor

@practicalswift practicalswift commented Feb 14, 2019

utACK fa7ca8e

Let's get this merged :-)

@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Feb 14, 2019

Would be nice if at least some person other than myself and travis could test this. Just copy-paste the commands in the readme on a linux machine and tell me your computer didn't crash or something.

@jamesob
Copy link
Member

@jamesob jamesob commented Feb 14, 2019

Tested changes (fa7ca8e) to doc/fuzzing.md on Linux 4.15.0-43-generic #46-Ubuntu SMP x86_64 with afl-fuzz 2.52b. Installed afl from scratch, rebuilt Bitcoin with requisite flags, and confirmed that running afl-fuzz with the transaction_deserialize works as intended.

Used new test_runner script which looks like an easy way to delegate to calling src/test/fuzz/*. Have yet to get any meaningful output back, but maybe this is intended?

$ ./test/fuzz/test_runner.py -l DEBUG ~/src/qa-assets/fuzz_seed_corpus transaction_deserialize

Fuzz targets found: ['address_deserialize', 'addrman_deserialize', 'banentry_deserialize', 'block_deserialize', 'blockheader_deserialize', 'blocklocator_deserialize', 'blockmerkleroot', 'blocktransactions_deserialize', 'blocktransactionsrequest_deserialize', 'blockundo_deserialize', 'bloomfilter_deserialize', 'coins_deserialize', 'diskblockindex_deserialize', 'inv_deserialize', 'messageheader_deserialize', 'netaddr_deserialize', 'service_deserialize', 'transaction_deserialize', 'txoutcompressor_deserialize', 'txundo_deserialize']
Fuzz targets selected: ['transaction_deserialize']

logging.error("Must be built with libFuzzer")
sys.exit(1)
except subprocess.TimeoutExpired:
logging.error("subprocess timed out: Currently only libFuzzer is supported")
Copy link
Member

@jamesob jamesob Feb 14, 2019

Tested and works when bitcoin is built with afl.

Copy link
Member Author

@MarcoFalke MarcoFalke Feb 14, 2019

❤️

@jamesob
Copy link
Member

@jamesob jamesob commented Feb 14, 2019

Tested ACK fa535af. Rebuilt with libfuzzer and tested the command that previously failed with afl:

$ ./test/fuzz/test_runner.py -l DEBUG ~/src/qa-assets/fuzz_seed_corpus transaction_deserialize

Fuzz targets found: ['address_deserialize', 'addrman_deserialize', 'banentry_deserialize', 'block_deserialize', 'blockheader_deserialize', 'blocklocator_deserialize', 'blockmerkleroot', 'blocktransactions_deserialize', 'blocktransactionsrequest_deserialize', 'blockundo_deserialize', 'bloomfilter_deserialize', 'coins_deserialize', 'diskblockindex_deserialize', 'inv_deserialize', 'messageheader_deserialize', 'netaddr_deserialize', 'service_deserialize', 'transaction_deserialize', 'txoutcompressor_deserialize', 'txundo_deserialize']
Fuzz targets selected: ['transaction_deserialize']
Run transaction_deserialize with args ['/home/james/src/bitcoin/src/test/fuzz/transaction_deserialize', '-runs=1', '/home/james/src/qa-assets/fuzz_seed_corpus/transaction_deserialize']
Output: INFO: Seed: 819372946
INFO: Loaded 1 modules   (2002 inline 8-bit counters): 2002 [0x55b9cec15620, 0x55b9cec15df2),
INFO: Loaded 1 PC tables (2002 PCs): 2002 [0x55b9cec15df8,0x55b9cec1db18),
INFO:      295 files found in /home/james/src/qa-assets/fuzz_seed_corpus/transaction_deserialize
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 52575 bytes
INFO: seed corpus: files: 295 min: 1b max: 52575b total: 491499b rss: 45Mb
#296    INITED cov: 240 ft: 938 corp: 277/475Kb exec/s: 0 rss: 106Mb
#296    DONE   cov: 240 ft: 938 corp: 277/475Kb exec/s: 0 rss: 106Mb
Done 296 runs in 0 second(s)

Worth noting that I had to bump clang from 4. to 6. in order for the -fsanitize=fuzzer flags to work.

@MarcoFalke MarcoFalke merged commit fa535af into bitcoin:master Feb 14, 2019
1 check failed
MarcoFalke added a commit that referenced this issue Feb 14, 2019
fa535af fuzz: test_runner: Better error message when built with afl (MarcoFalke)
fa7ca8e qa: Add test/fuzz/test_runner.py (MarcoFalke)

Pull request description:

  Can be run with `./test/fuzz/test_runner.py` after building as described in `doc/fuzzing.md`

Tree-SHA512: f6a3cd8165ec2de4b363be4fd0a936b4a60829cce923f93fe5d6a046b1bbd64c959cdf790440bf70c0e13b0bb1b956a746a24c6fd92bddeab15b837ed50ffad2
@MarcoFalke MarcoFalke deleted the Mf1901-qaFuzz branch Feb 14, 2019
@@ -28,6 +27,8 @@ FUZZ_TARGETS = \

if ENABLE_FUZZ
noinst_PROGRAMS += $(FUZZ_TARGETS:=)
else
bin_PROGRAMS += test/test_bitcoin
Copy link
Contributor

@ryanofsky ryanofsky Feb 22, 2019

In commit "qa: Add test/fuzz/test_runner.py" (fa7ca8e)

@MarcoFalke Is this no longer building test_bitcoin if ENABLE_FUZZ is true? I wouldn't expect ENABLE_FUZZ to affect this, but if this is correct it'd be helpful to have a comment here explaining what this is doing.

Copy link
Member Author

@MarcoFalke MarcoFalke Feb 22, 2019

Yes, I think there is no reason to build test_bitcoin if you want to build the fuzz tests. Am I missing something?

Regardless of that, it would result in linker errors later in the build process, since you can't link libfuzzer to something that has a main function.

Copy link
Contributor

@ryanofsky ryanofsky Feb 22, 2019

Oh ok, I don't know really anything about fuzzing, and I assumed enabling a fuzzing option would just build some new binaries, not affect existing binaries in any way. But I guess due to autotools lack of support for side-by-side build configurations, ENABLE_FUZZ affects every existing binary we build, as well as building new binaries? Probably not worth adding a comment here just for test_bitcoin in this case.

Copy link
Member Author

@MarcoFalke MarcoFalke Feb 22, 2019

Yes, ideally it should disable all other binaries.

Currently we hack around that by specifying it manually:

BITCOIN_CONFIG="--disable-wallet --disable-bench --with-utils=no --with-daemon=no --with-libs=no --with-gui=no --enable-fuzz --with-sanitizers=fuzzer,address CC=clang CXX=clang++"

Copy link
Contributor

@apoelstra apoelstra Dec 9, 2020

It is surprising that --enable-fuzz silently stops building the test binary, especially as config.log continues to claim that test_bitcoin is being built.

Copy link
Member Author

@MarcoFalke MarcoFalke Dec 9, 2020

It is surprising that --enable-fuzz silently stops building the test binary

Might be fixed by #19388, which is blocked on #20560 . Though when building and linking with a fuzz engine, the test_bitcoin binary won't have any meaning. Not sure if it will even compile/link.

config.log continues to claim that test_bitcoin is being built.

I don't really understand build systems, so if this is an issue, I hope that someone will fix it

Copy link
Contributor

@apoelstra apoelstra Dec 9, 2020

I'll open a PR to change config to output something like

configure:27478: checking whether to build test_bitcoin-qt
configure:27481: result: no, because fuzzing is enabled
configure:27499: checking whether to build test_bitcoin
configure:27502: result: no, because fuzzing is enabled

It's not really surprising that enabling fuzzing would break all the other binaries, I was just misled by the config output.

Copy link
Contributor

@apoelstra apoelstra Dec 9, 2020

random-zebra added a commit to PIVX-Project/PIVX that referenced this issue May 28, 2021
d059544 [Build] fuzz target, change LIBBITCOIN_ZEROCOIN link order. (furszy)
2396e6b [fuzz] Add ContextualCheckTransaction call to transaction target. (furszy)
f0887a0 Fuzzing documentation "PIVX-fication" (furszy)
9631f46 [doc] add sanitizers documentation in developer-notes.md (furszy)
70a0ace tests: Test serialisation as part of deserialisation fuzzing. Test round-trip equality where possible. Avoid code repetition. (practicalswift)
e1b92b6 ignore new fuzz targets gitignore (furszy)
d058d8c tests: Add deserialization fuzzing harnesses (furszy)
e1f666c tests: Remove TRANSACTION_DESERIALIZE (replaced by transaction fuzzer) (practicalswift)
b5f291c tests: Add fuzzing harness for CheckTransaction(...), IsStandardTx(...) and other CTransaction related functions (furszy)
3205871 fuzz: Remove option --export_coverage from test_runner (MarcoFalke)
52693ee fuzz: Add option to merge input dir to test runner (MarcoFalke)
2b4f8aa doc: Remove --disable-ccache from docs (MarcoFalke)
b54b1d6 tests: Improve test runner output in case of target errors (practicalswift)
cd6134f test: Log output even if fuzzer failed (MarcoFalke)
48cd0c8 doc: Improve fuzzing docs for macOS users (Fabian Jahr)
d642b67 [Build] Do not disable wallet when fuzz is enabled. (furszy)
c3447b5 Update doc and CI config (qmma)
1266d3e Disable other targets when enable-fuzz is set (qmma)
f28ac9a build: Allow to configure --with-sanitizers=fuzzer (MarcoFalke)
425742c fuzz: test_runner: Better error message when built with afl (MarcoFalke)
541f442 qa: Add test/fuzz/test_runner.py (MarcoFalke)
89fe5b2 Add missing LIBBITCOIN_ZMQ to test target (furszy)
58dbe79 add fuzzing binaries to gitignore. (furszy)
393a126 fuzz: Move deserialize tests to test/fuzz/deserialize.cpp (MarcoFalke)
a568df5 test: Build fuzz targets into separate executables (furszy)
d5dddde [test] fuzz: make test_one_input return void (MarcoFalke)
2e4ec58 [fuzzing] initialize chain params by default. (furszy)
08d8ebe [tests] Add libFuzzer support. (practicalswift)
84f72da [test] Speed up fuzzing by ~200x when using afl-fuzz (practicalswift)
faf2be6 Init ECC context for test_bitcoin_fuzzy. (Gregory Maxwell)
11150df Make fuzzer actually test CTxOutCompressor (Pieter Wuille)
d6f6a85 doc: Add bare-bones documentation for fuzzing (Wladimir J. van der Laan)
5c3b550 Simple fuzzing framework (pstratem)

Pull request description:

  As the title says, adding fuzzing framework support so we can start getting serious on this area as well.

  Adapted the following PRs:

  * bitcoin#9172.
  * bitcoin#9354.
  * bitcoin#9691.
  * bitcoin#10415.
  * bitcoin#10440.
  * bitcoin#15043.
  * bitcoin#15047.
  * bitcoin#15295.
  * bitcoin#15399 (fabcfa5 only).
  * bitcoin#16338.
  * bitcoin#17051.
  * bitcoin#17076.
  * bitcoin#17225.
  * bitcoin#17942.
  * bitcoin#16236 (only fa35c42).
  * bitcoin#18166 (only f2472f6).
  * bitcoin#18300.
  * And.. probably will go further and continue adapting more PRs..

ACKs for top commit:
  random-zebra:
    utACK d059544 and merging...

Tree-SHA512: c0b05bca47bf99bafd8abf1453c5636fe05df75f16d0e9c750368ea2aed8142f0b28d28af1d23468b8829188412a80fd3b7bdbbda294b940d78aec80c1c7d03a
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 2, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 5, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 5, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 8, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

9 participants