-
Notifications
You must be signed in to change notification settings - Fork 9.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure that input validation between API and Apply is consistent #14561
Conversation
Looks great, would be good setup CI to run fuzz test. I think we should run:
What do you think? Do you need help with that? We doing the first one should be enough for now. |
Codecov Report
@@ Coverage Diff @@
## main #14561 +/- ##
==========================================
- Coverage 75.60% 75.44% -0.16%
==========================================
Files 457 457
Lines 37267 37267
==========================================
- Hits 28176 28117 -59
- Misses 7334 7383 +49
- Partials 1757 1767 +10
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
@serathius I have added the GitHub Action for fuzzing:
I'm proceeding by covering the others input validation methods. Any other feedback is obvs welcome |
scripts/fuzzing.sh
Outdated
|
||
log_callout -e "\\nExecuting v3rpc with target $FUZZ_TARGET \\n" | ||
|
||
cd $BASE_PATH |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This script requires providing multiple non-trivial environment variables for it to work. This script should only be called via github action, but also locally to reproduce issues. I think we should make the script independent so it can be called without to much hardship.
Can we have the script include a good defaults for running? Or if the default cannot be provided, could we add validation to give better error message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated the script:
- FUZZ_TIME now defaults to 300s
- TARGET_PATH display an error in the case that is unset
- FUZZ_TARGET display an error in the case that is unset
Let me know if you prefer another setup
scripts/fuzzing.sh
Outdated
# To fix according to newer version of go: | ||
# go get golang.org/dl/gotip | ||
# gotip download | ||
# GO_CMD="gotip" | ||
GO_CMD="go" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We no longer use gotip.
Note, you could clean up the commit history? Looks like you have been pulling other commits as PR has multiple borrowed commits from other PRs. I think it would be best if we squash everything into one commit. Please let me know if you need any help. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looks good. We should be good to expand list of methods covered in future PR.
Would be great if we could ensure that any new method added is required to have a fuzz test.
cc @AdamKorcz I recall that you added some fuzz test a couple of months back. Could you take a look at this PR as well? Thanks. |
@ahrtr Thanks for the tag.
|
Hey @AdamKorcz, this is independent fuzzing to address one of the bugs in the api we found. I don't think it was reported by OSS fuzz. I proposed to implement fuzz to prevent future regressions. Unfortunately the OSS-Fuzz was developed outside of the etcd repo, so they:
As so I would recommend that we migrate OSS-Fuzz code into the etcd repo. We can use this PR as the bases for the test. This should allow us to at test that the fuzz test build during CI. |
.github/workflows/fuzzing.yaml
Outdated
strategy: | ||
fail-fast: false | ||
matrix: | ||
Target: [FuzzRangeRequest, FuzzPutRequest, FuzzDeleteRangeRequest] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each time when any contributor add a new fuzz case, then they need to update this file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Unfortunately out-of-box fuzzing requires a specific target method to start the fuzzing. In case you try to use something like go test -fuzz Fuzz*
the returned error is:
testing: will not fuzz, -fuzz matches more than one fuzz test: [FuzzRangeRequest FuzzPutRequest FuzzDeleteRangeRequest]
Let me know if there is a better way of doing that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would suggest to get all targets defined in the scripts/fuzzing.sh
for now, so that we can manually easily run all fuzz tests locally.
We can update the script/fuzzing.sh something like below,
TARGETS="FuzzRangeRequest FuzzPutRequest FuzzDeleteRangeRequest"
for target in ${TARGETS}; do
run pushd ${target_path}
go test -fuzz ${target}
run popd
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I proceeded by updating the script as requested. Note that, the fuzzing for the different targets won't run in parallel. The fuzz process will start from FuzzRangeRequest
and finishing with FuzzDeleteRangeRequest
with the specified timeout
|
||
_, _, err := txn.Txn(ctx, zaptest.NewLogger(t), request, false, s, &lease.FakeLessor{}) | ||
if err != nil { | ||
t.Logf("Check: %s | Apply: %s", errCheck, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we fail the test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The goal of the test is to verify that the Tnx
function won't panic in case the validation check passes. it is okay in this context if the Tnx
handles the request with an error. In case of panic, the test should fail.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK.
Note: Golang supports fuzzing starting from 1.18, |
I'm already using the out-of-the-box fuzzing coming from golang |
I see lots of unrelated commits in this PR, please rebase this PR. |
Signed-off-by: Samuele Resca <sr7@ad.datcon.co.uk> Signed-off-by: Samuele Resca <samuele.resca@gmail.com>
Signed-off-by: Samuele Resca <sr7@ad.datcon.co.uk> Signed-off-by: Samuele Resca <samuele.resca@gmail.com>
Signed-off-by: Samuele Resca <samuele.resca@gmail.com>
Signed-off-by: Samuele Resca <samuele.resca@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thank you @samueleresca
for target in ${TARGETS}; do | ||
log_callout -e "\\nExecuting fuzzing with target ${target} in $target_path with a timeout of $fuzz_time\\n" | ||
run pushd "${target_path}" | ||
$GO_CMD test -fuzz "${target}" -fuzztime "${fuzz_time}" | ||
run popd | ||
log_success -e "\\COMPLETED: fuzzing with target $target in $target_path \\n" | ||
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor comment: It seems that we only need to execute pushd
and popd
once outside the for loop.
Of course, it isn't a big deal, feel free to fix it in a separate PR.
Fixes #13617
The fuzz input is passed to the input validation. In case the input validation passes - the application of that request shouldn't panic.
Adding a fuzzing script and a GitHub action for executing fuzzing with a 5 min time limit