Ensure that input validation between API and Apply is consistent by samueleresca · Pull Request #14561 · etcd-io/etcd

samueleresca · 2022-10-07T21:34:54Z

Fixes #13617
The fuzz input is passed to the input validation. In case the input validation passes - the application of that request shouldn't panic.

Adding a fuzzing script and a GitHub action for executing fuzzing with a 5 min time limit

serathius · 2022-10-08T13:18:23Z

Looks great, would be good setup CI to run fuzz test. I think we should run:

Short 5-10m run o every pull request to verify that fuzz works
Long 1-2h once a day to do a proper fuzzing

What do you think? Do you need help with that? We doing the first one should be enough for now.

codecov-commenter · 2022-10-09T16:17:53Z

Codecov Report

Merging #14561 (bbda804) into main (e5790d2) will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #14561      +/-   ##
==========================================
- Coverage   75.60%   75.44%   -0.16%     
==========================================
  Files         457      457              
  Lines       37267    37267              
==========================================
- Hits        28176    28117      -59     
- Misses       7334     7383      +49     
- Partials     1757     1767      +10

Flag	Coverage Δ
all	`75.44% <ø> (-0.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/storage/mvcc/watchable_store.go	`84.42% <0.00%> (-9.06%)`	⬇️
client/pkg/v3/tlsutil/tlsutil.go	`83.33% <0.00%> (-8.34%)`	⬇️
client/pkg/v3/testutil/leak.go	`62.83% <0.00%> (-7.08%)`	⬇️
client/v3/leasing/util.go	`91.66% <0.00%> (-6.67%)`	⬇️
raft/rafttest/node.go	`95.00% <0.00%> (-5.00%)`	⬇️
server/storage/mvcc/kvstore_compaction.go	`95.65% <0.00%> (-4.35%)`	⬇️
server/etcdserver/txn/util.go	`75.47% <0.00%> (-3.78%)`	⬇️
client/v3/concurrency/election.go	`79.68% <0.00%> (-2.35%)`	⬇️
server/etcdserver/api/v3rpc/interceptor.go	`75.52% <0.00%> (-2.09%)`	⬇️
server/storage/mvcc/kvstore.go	`89.00% <0.00%> (-1.42%)`	⬇️
... and 17 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

samueleresca · 2022-10-09T19:37:31Z

I think we should run:

Short 5-10m run o every pull request to verify that fuzz works

Long 1-2h once a day to do a proper fuzzing

What do you think? Do you need help with that? We doing the first one should be enough for now.

@serathius I have added the GitHub Action for fuzzing:

Tested that on my fork repo in case of success: https://github.com/samueleresca/etcd/actions/runs/3215190941
In case of failure: https://github.com/samueleresca/etcd/actions/runs/3215106275 (producing artifacts)

I'm proceeding by covering the others input validation methods.

Any other feedback is obvs welcome

serathius · 2022-10-10T07:54:18Z

+
+log_callout -e "\\nExecuting v3rpc with target $FUZZ_TARGET \\n"
+
+cd $BASE_PATH


This script requires providing multiple non-trivial environment variables for it to work. This script should only be called via github action, but also locally to reproduce issues. I think we should make the script independent so it can be called without to much hardship.

Can we have the script include a good defaults for running? Or if the default cannot be provided, could we add validation to give better error message?

I updated the script:

FUZZ_TIME now defaults to 300s

TARGET_PATH display an error in the case that is unset

FUZZ_TARGET display an error in the case that is unset

Let me know if you prefer another setup

serathius · 2022-10-13T07:02:54Z

+# To fix according to newer version of go:
+# go get golang.org/dl/gotip
+# gotip download
+# GO_CMD="gotip"
+GO_CMD="go"


We no longer use gotip.

serathius · 2022-10-13T07:09:43Z

Note, you could clean up the commit history? Looks like you have been pulling other commits as PR has multiple borrowed commits from other PRs. I think it would be best if we squash everything into one commit. Please let me know if you need any help.

serathius

Overall looks good. We should be good to expand list of methods covered in future PR.

Would be great if we could ensure that any new method added is required to have a fuzz test.

ahrtr · 2022-10-14T11:30:10Z

cc @AdamKorcz I recall that you added some fuzz test a couple of months back. Could you take a look at this PR as well? Thanks.

AdamKorcz · 2022-10-14T11:34:11Z

@ahrtr Thanks for the tag.

@samueleresca:

Do these fuzz tests replace any of the ones running on OSS-Fuzz? https://github.com/cncf/cncf-fuzzing/tree/main/projects/etcd
Could you also add these to the build script for OSS-Fuzz? https://github.com/cncf/cncf-fuzzing/blob/main/projects/etcd/build.sh

serathius · 2022-10-14T13:04:10Z

Hey @AdamKorcz, this is independent fuzzing to address one of the bugs in the api we found. I don't think it was reported by OSS fuzz. I proposed to implement fuzz to prevent future regressions.

Unfortunately the OSS-Fuzz was developed outside of the etcd repo, so they:

Were not reviewed by maintainers, we don't know the code, we don't know what is covered and what not.
Are apparently broken https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=51781
We cannot reliably maintain them as they are not integrated in the etcd CI and only maintainers have access to their reports.
Maintainers do not have capacity to maintain external project.

As so I would recommend that we migrate OSS-Fuzz code into the etcd repo. We can use this PR as the bases for the test. This should allow us to at test that the fuzz test build during CI.

ahrtr · 2022-10-14T21:30:47Z

+    strategy:
+      fail-fast: false
+      matrix:
+          Target: [FuzzRangeRequest, FuzzPutRequest, FuzzDeleteRangeRequest]


Each time when any contributor add a new fuzz case, then they need to update this file?

Yes. Unfortunately out-of-box fuzzing requires a specific target method to start the fuzzing. In case you try to use something like go test -fuzz Fuzz* the returned error is:

testing: will not fuzz, -fuzz matches more than one fuzz test: [FuzzRangeRequest FuzzPutRequest FuzzDeleteRangeRequest]

Let me know if there is a better way of doing that

I would suggest to get all targets defined in the scripts/fuzzing.sh for now, so that we can manually easily run all fuzz tests locally.

We can update the script/fuzzing.sh something like below,

TARGETS="FuzzRangeRequest FuzzPutRequest FuzzDeleteRangeRequest" for target in ${TARGETS}; do run pushd ${target_path} go test -fuzz ${target} run popd done

I proceeded by updating the script as requested. Note that, the fuzzing for the different targets won't run in parallel. The fuzz process will start from FuzzRangeRequest and finishing with FuzzDeleteRangeRequest with the specified timeout

ahrtr · 2022-10-14T21:32:14Z

+
+		_, _, err := txn.Txn(ctx, zaptest.NewLogger(t), request, false, s, &lease.FakeLessor{})
+		if err != nil {
+			t.Logf("Check: %s | Apply: %s", errCheck, err)


Shouldn't we fail the test?

The goal of the test is to verify that the Tnx function won't panic in case the validation check passes. it is okay in this context if the Tnx handles the request with an error. In case of panic, the test should fail.

ahrtr · 2022-10-14T21:39:54Z

Note: Golang supports fuzzing starting from 1.18,
https://go.dev/security/fuzz/

samueleresca · 2022-10-16T18:07:13Z

Note: Golang supports fuzzing starting from 1.18, https://go.dev/security/fuzz/

I'm already using the out-of-the-box fuzzing coming from golang

ahrtr · 2022-10-16T20:56:30Z

I see lots of unrelated commits in this PR, please rebase this PR.

Signed-off-by: Samuele Resca <sr7@ad.datcon.co.uk> Signed-off-by: Samuele Resca <samuele.resca@gmail.com>

Signed-off-by: Samuele Resca <samuele.resca@gmail.com>

ahrtr

LGTM

Thank you @samueleresca

ahrtr · 2022-10-24T02:21:06Z

+for target in ${TARGETS}; do
+    log_callout -e "\\nExecuting fuzzing with target ${target} in $target_path with a timeout of $fuzz_time\\n"
+    run pushd "${target_path}"
+        $GO_CMD test -fuzz "${target}" -fuzztime "${fuzz_time}"
+    run popd
+    log_success -e "\\COMPLETED: fuzzing with target $target in $target_path \\n"
+done


Minor comment: It seems that we only need to execute pushd and popd once outside the for loop.

Of course, it isn't a big deal, feel free to fix it in a separate PR.

samueleresca changed the title ~~Adding a draft of fuzzing on Range request.~~ Ensure that input validation between API and Apply is consistent Oct 7, 2022

serathius reviewed Oct 8, 2022

View reviewed changes

Comment thread server/etcdserver/api/v3rpc/validationfuzz_test.go Outdated

serathius reviewed Oct 10, 2022

View reviewed changes

Comment thread .github/workflows/fuzzing.yaml Outdated

serathius reviewed Oct 10, 2022

View reviewed changes

Comment thread .github/workflows/fuzzing.yaml Outdated

serathius reviewed Oct 13, 2022

View reviewed changes

Comment thread Makefile Outdated

serathius reviewed Oct 13, 2022

View reviewed changes

Comment thread Makefile Outdated

serathius marked this pull request as ready for review October 14, 2022 07:41

serathius approved these changes Oct 14, 2022

View reviewed changes

ahrtr reviewed Oct 14, 2022

View reviewed changes

Comment thread server/etcdserver/api/v3rpc/validationfuzz_test.go Outdated

Comment thread server/etcdserver/api/v3rpc/validationfuzz_test.go Outdated

Comment thread server/etcdserver/api/v3rpc/validationfuzz_test.go Outdated

ahrtr reviewed Oct 14, 2022

View reviewed changes

Samuele Resca and others added 4 commits October 23, 2022 13:46

Adding fuzz test on v3rpc interfaces.

3d9c5c6

Signed-off-by: Samuele Resca <sr7@ad.datcon.co.uk> Signed-off-by: Samuele Resca <samuele.resca@gmail.com>

Refactoring code to remove duplicate code test.

b58f9c2

Signed-off-by: Samuele Resca <sr7@ad.datcon.co.uk> Signed-off-by: Samuele Resca <samuele.resca@gmail.com>

Updating fuzzing script with list of target.

37d62ba

Signed-off-by: Samuele Resca <samuele.resca@gmail.com>

Updating script for static analysis readiness.

bbda804

Signed-off-by: Samuele Resca <samuele.resca@gmail.com>

ahrtr approved these changes Oct 24, 2022

View reviewed changes

ahrtr merged commit 0d46a6e into etcd-io:main Oct 24, 2022

ahrtr mentioned this pull request Nov 2, 2022

raft: add fuzzer #14674

Closed


		log_callout -e "\\nExecuting v3rpc with target $FUZZ_TARGET \\n"

		cd $BASE_PATH

Conversation

samueleresca commented Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

serathius commented Oct 8, 2022

Uh oh!

codecov-commenter commented Oct 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

samueleresca commented Oct 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

serathius commented Oct 13, 2022

Uh oh!

serathius left a comment

Choose a reason for hiding this comment

Uh oh!

ahrtr commented Oct 14, 2022

Uh oh!

AdamKorcz commented Oct 14, 2022

Uh oh!

serathius commented Oct 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahrtr commented Oct 14, 2022

Uh oh!

samueleresca commented Oct 16, 2022

Uh oh!

ahrtr commented Oct 16, 2022

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

samueleresca commented Oct 7, 2022 •

edited

Loading

codecov-commenter commented Oct 9, 2022 •

edited

Loading

serathius commented Oct 14, 2022 •

edited

Loading