Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FUZZ] Beaconfuzz_v2 crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c #61

Closed
7 tasks done
parithosh opened this issue Sep 1, 2020 · 9 comments
Closed
7 tasks done
Labels

Comments

@parithosh
Copy link

parithosh commented Sep 1, 2020

I've identified a fuzzer crash and am contributing to the security of Ethereum 2!

I've done and provided the following:

  • Checked to see if any other [FUZZ] issue already refers to that crasher
  • Attached the crashing input (either attached to the issue as a .zip or .gz, or as a link to a file sharing service)
  • Noted the beacon-fuzz version or commit used.
  • Provided crash output
  • Noted the command or fuzzer used to generate the crash:
  • Name of the original crash file.
  • (Optional but optimal) Checked if the crash can be consistently replicated by re-running the input.

Info to Reproduce

  • Command run: make fuzz_attester_slashing-struct
  • Crasher file name: crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c
    crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c.zip
  • Client exercised: Unsure, I ran it with the beaconfuzz_v2 differential fuzzer, I think it is Nim
  • Fuzzing engine used (if applicable): Structural fuzzing using libfuzzer

Crash output and stacktrace

Attached as a screenshot:
Screenshot 2020-08-31 at 1 40 01 PM

Re-ran the input on the nightly compiler (rustup default nightly, it refused to compile on default).
Command used to re-run input: RUST_BACKTRACE=1 cargo fuzz run struct_attester_slashing fuzz/artifacts/struct_attester_slashing/crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c

Stacktrace:

    Finished release [optimized] target(s) in 0.26s
     Running `fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing -artifact_prefix=/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/artifacts/struct_attester_slashing/ fuzz/artifacts/struct_attester_slashing/crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c`
INFO: Seed: 3589315948
INFO: Loaded 1 modules   (201440 inline 8-bit counters): 201440 [0x5555594799a1, 0x5555594aac81),
INFO: Loaded 1 PC tables (201440 PCs): 201440 [0x5555594aac88,0x5555597bda88),
fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing: Running 1 inputs 1 time(s) each.
Running: fuzz/artifacts/struct_attester_slashing/crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c
ERRO[0018] Could not get rough time result: no reply     prefix=roughtime
ERRO[0018] Could not get rough time result: no reply     prefix=roughtime
ERRO[0018] Could not get rough time result: no reply     prefix=roughtime
ERRO[0018] Could not get rough time result: no reply     prefix=roughtime
ERRO[0018] Could not get rough time result: no reply     prefix=roughtime
ERRO[0018] Could not get rough time result: no reply     prefix=roughtime
ERRO[0018] Failed to calculate roughtime offset          error="no valid responses" prefix=roughtime
INFO[0018] New calculated roughtime offset is 0 ns       prefix=roughtime
couldn't interpret ETH2FUZZ_BEACONSTATE: environment variable not found
thread '<unnamed>' panicked at 'No valid beaconstate in the seed folder', fuzz_targets/struct_attester_slashing.rs:122:13
stack backtrace:
   0: std::panicking::begin_panic
   1: core::ops::function::FnOnce::call_once
   2: std::sync::once::Once::call_once::{{closure}}
   3: std::sync::once::Once::call_inner
             at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/std/src/sync/once.rs:421
   4: rust_fuzzer_test_input
   5: __rust_try
   6: LLVMFuzzerTestOneInput
   7: _ZN6fuzzer6Fuzzer15ExecuteCallbackEPKhm
   8: _ZN6fuzzer10RunOneTestEPNS_6FuzzerEPKcm
   9: _ZN6fuzzer12FuzzerDriverEPiPPPcPFiPKhmE
  10: main
  11: __libc_start_main
  12: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last, using override)
/root/fuzzing/beaconfuzz_v2/nim-beacon-chain/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(614) signalHandler
SIGABRT: Abnormal termination.
==1211654== ERROR: libFuzzer: fuzz target exited
    #0 0x5555560d6901  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xb82901)
    #1 0x55555836c630  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2e18630)
    #2 0x55555835fb3b  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2e0bb3b)
    #3 0x7ffff7bf2a26  (/lib/x86_64-linux-gnu/libc.so.6+0x49a26)
    #4 0x7ffff7bf2bdf  (/lib/x86_64-linux-gnu/libc.so.6+0x49bdf)
    #5 0x55555635a2d7  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xe062d7)
    #6 0x7ffff7bef20f  (/lib/x86_64-linux-gnu/libc.so.6+0x4620f)
    #7 0x7ffff7bef18a  (/lib/x86_64-linux-gnu/libc.so.6+0x4618a)
    #8 0x7ffff7bce858  (/lib/x86_64-linux-gnu/libc.so.6+0x25858)
    #9 0x55555841d146  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2ec9146)
    #10 0x5555584061d5  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2eb21d5)
    #11 0x555558359176  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2e05176)
    #12 0x55555840d4e7  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2eb94e7)
    #13 0x555556120954  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xbcc954)
    #14 0x55555611d569  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xbc9569)
    #15 0x5555561207f4  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xbcc7f4)
    #16 0x55555612877c  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xbd477c)
    #17 0x55555611ff6b  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xbcbf6b)
    #18 0x555558407164  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2eb3164)
    #19 0x55555618605c  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xc3205c)
    #20 0x5555583591a0  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2e051a0)
    #21 0x555558358dff  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2e04dff)
    #22 0x55555835ff9c  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2e0bf9c)
    #23 0x55555833cbf9  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2de8bf9)
    #24 0x5555583469f2  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0x2df29f2)
    #25 0x5555560534b6  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xaff4b6)
    #26 0x7ffff7bd00b2  (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
    #27 0x55555605365d  (/root/fuzzing/beaconfuzz_v2/beacon-fuzz/beaconfuzz_v2/fuzz/target/x86_64-unknown-linux-gnu/release/struct_attester_slashing+0xaff65d)

SUMMARY: libFuzzer: fuzz target exited
────────────────────────────────────────────────────────────────────────────────

Error: Fuzz target exited with exit code: 77

Stack backtrace:
   0: cargo_fuzz::project::FuzzProject::exec_fuzz
   1: <cargo_fuzz::options::run::Run as cargo_fuzz::RunCommand>::run_command
   2: cargo_fuzz::main
   3: std::sys_common::backtrace::__rust_begin_short_backtrace
   4: std::rt::lang_start::{{closure}}
   5: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/core/src/ops/function.rs:259
      std::panicking::try::do_call
             at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/std/src/panicking.rs:381
      std::panicking::try
             at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/std/src/panicking.rs:345
      std::panic::catch_unwind
             at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/std/src/panic.rs:382
      std::rt::lang_start_internal
             at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/std/src/rt.rs:51
   6: main
   7: __libc_start_main
   8: _start

Your Environment

  • Fuzzer ran: Beaconfuzz_v2 using libfuzzer
  • Version/Commit used: b56bf452adc05ac5b0a73aa8844dac0465894ebc refs/heads/beaconfuzz_v2
  • Operating System and version: Ubuntu 20.04, Docker version 19.03.12, Linux 4.15.0 kernel, Running on a OpenVZ virtualized VPS
@parithosh
Copy link
Author

parithosh commented Sep 1, 2020

I just noticed that the stacktrace has a statement couldn't interpret ETH2FUZZ_BEACONSTATE: environment variable not found. I can't find any info on setting this variable in the README, please do let me know how one can set the variable. The issue might then just be a config related error and not a fuzzing crash.

@gnattishness
Copy link
Member

gnattishness commented Sep 1, 2020

Thanks for your efforts @parithosh!
Agreed there doesn't look to be any info on ETH2FUZZ_BEACONSTATE in the current README.
From the Makefile, we can see that it's set to point to eth2fuzz corpora's folder of known beaconstates:

fuzz_attestation:
ETH2FUZZ_BEACONSTATE=../eth2fuzz/workspace/corpora/beaconstate cargo hfuzz run diff_attestation

But this would only be a config issue during the attempt to reproduce, and not during the initial crash identification.
We'll look into it further!

@pventuzelo
Copy link
Collaborator

pventuzelo commented Sep 2, 2020

@gnattishness I confirm, when ETH2FUZZ_BEACONSTATE is not set, I make the program to panic ;)
Thanks @parithosh

@parithosh
Copy link
Author

parithosh commented Sep 2, 2020

@pventuzelo glad that it was a config error then :D
My other issue might have the same cause, so I think you can just add the same comment and close that as well.

@pventuzelo pventuzelo reopened this Sep 2, 2020
@pventuzelo
Copy link
Collaborator

pventuzelo commented Sep 2, 2020

I reopen the issue until we give a try to your testcase, look like the variable was not set only during the repro

@pventuzelo
Copy link
Collaborator

pventuzelo commented Sep 5, 2020

With the latest update of prysm and beaconfuzz_v2, i'm not able to repro the bug using this command:

ETH2FUZZ_BEACONSTATE=../eth2fuzz/workspace/corpora/beaconstate cargo +nightly fuzz run struct_attester_slashing crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c

or using:

./beaconfuzz_v2 debug

@pventuzelo
Copy link
Collaborator

pventuzelo commented Sep 10, 2020

Look like my environment was not correct during the previous test, i'm reopening this issue:

My command for testing:

ASAN_OPTIONS=detect_leaks=0 RUSTFLAGS='-L /home/scop/Documents/consulting/sigmaprime/prysm/pfuzz/ -L /home/scop/Documents/consulting/sigmaprime/nim-beacon-state/build/ ' ETH2FUZZ_BEACONSTATE=../eth2fuzz/workspace/corpora/beaconstate cargo +nightly fuzz run struct_attester_slashing crash-5f5801ee2ad7be3a2c9017f3c731ccdbd3b0e64c

other command:

./beaconfuzz_v2 debug nimbus_attester_slashing_issue_61/pre.ssz nimbus_attester_slashing_issue_61/attslash.ssz attesterslashing

Result:

thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `false`,
 right: `true`', /home/scop/Documents/consulting/sigmaprime/beacon-fuzz/beaconfuzz_v2/libs/eth2clientsfuzz/src/attester_slashing.rs:39:17

Meaning: nimbus is not agree with prysm and lighthouse

cc @zedt3ster

@zedt3ster
Copy link
Member

zedt3ster commented Sep 22, 2020

Thanks for reporting @parithosh - this is another great find by the structural fuzzer, highlighting a subtle difference between how Nimbus, Prysm and Lighthouse handle the AttesterSlashing processing.

As per the eth2 specification:

def process_attester_slashing(state: BeaconState, attester_slashing: AttesterSlashing) -> None:
    attestation_1 = attester_slashing.attestation_1
    attestation_2 = attester_slashing.attestation_2
    assert is_slashable_attestation_data(attestation_1.data, attestation_2.data)
    assert is_valid_indexed_attestation(state, attestation_1)
    assert is_valid_indexed_attestation(state, attestation_2)

    slashed_any = False
    indices = set(attestation_1.attesting_indices).intersection(attestation_2.attesting_indices)
    for index in sorted(indices):
        if is_slashable_validator(state.validators[index], get_current_epoch(state)):
            slash_validator(state, index)
            slashed_any = True
    assert slashed_any

The function is_valid_indexed_attestation is defined as follows:

def is_valid_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool:
    """
    Check if ``indexed_attestation`` is not empty, has sorted and unique indices and has a valid aggregate signature.
    """
    # Verify indices are sorted and unique
    indices = indexed_attestation.attesting_indices
    if len(indices) == 0 or not indices == sorted(set(indices)):
        return False
    # Verify aggregate signature
    pubkeys = [state.validators[i].pubkey for i in indices]
    domain = get_domain(state, DOMAIN_BEACON_ATTESTER, indexed_attestation.data.target.epoch)
    signing_root = compute_signing_root(indexed_attestation.data, domain)
    return bls.FastAggregateVerify(pubkeys, signing_root, indexed_attestation.signature)

It turns out that Lighthouse and Prysm are performing the indexed attestation validation as part of their signature verification:

In Lighthouse:

https://github.com/sigp/lighthouse/blob/c9596fcf0ee3a17bfb1df48e23ed6339c3791457/consensus/state_processing/src/per_block_processing/is_valid_indexed_attestation.rs#L39-L51

The closure |i| get_pubkey_from_state(state, i) would return an error for out-of-range attesting indices, however since fuzzing disables BLS verification, that particular check is not performed (the entire code block is skipped).

For Prysm:

When BLS signatures are disabled, PublicKeyFromBytes() returns an empty BLS signatures, regardless of the attesting indices provided (therefore not checking for out-of-range attesting indices).

As a result, the AttesterSlashing object produced by the structural fuzzer is considered valid by Lighthouse and Prysm, with the associated state transition resulting in a post-BeaconState.

For Nimbus:

In the snippet linked above, we can see that Nimbus performs an additional check on the attesting indices:

  let num_validators = state.validators.lenu64
  if anyIt(indexed_attestation.attesting_indices, it >= num_validators):
    return err("indexed attestation: not all indices valid validators")

As a result, the AttesterSlashing generated by the structural fuzzer is rejected by Nimbus (since the second attesting index in the second IndexedAttestation causes the check above to fail).

This explains the discrepancy raised by the fuzzer. Thanks again for reporting this @parithosh !

@parithosh
Copy link
Author

parithosh commented Sep 22, 2020

Thanks for the details response @zedt3ster ! Glad I could be of help :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants