Add Continuous Fuzzing via Fuzzit

[yevgenypats]

This PR adds a continuous fuzzing integration to prometheus's travis pipeline via Fuzzit service.

This means the following:

• Every time new code is pushed to master new fuzzers are built and push to fuzzit where they run continuously generate new test-cases and alerts if new crashes are found
• Every pull-request the fuzzers runs through a quick regression tests with the generated test-cases from the previous step. If something new/old bugs are introduced you will see this immediately in the Travis check.

To take ownership of the organisation, please login to https://app.fuzzit.dev and let me know your account.

Please review and feel free to comment/ask questions.

[yevgenypats]

@juliusv @kjk @msiebuhr CCing you here.

[yevgenypats]

@juliusv @kjk @msiebuhr This PR is ready for review.

[yevgenypats]

@juliusv @mseinhur ping. Can you review this?

[msiebuhr]

While I implemented the first round of fuzzing some time (years?) ago, I must admit not being up to speed with Prometheus nor familiar with fuzzit, so I've stuck to general observations.

[msiebuhr]

The first comment in fuzzit.sh says it should be called with either "fuzzing" or "local-regression".

[msiebuhr]

Ah. I see from the bottom of fuzzit.sh that it will auto-select "local-regression" for pull-requests and "fuzzing" for everything else. Perhaps this should be encoded in a special argument? ./fuzzit.sh auto-select-for-travis, so there's no doubt?

[yevgenypats]

Removed. This is old docs.

[yevgenypats]

Yes probably we will add something like this in the near future but for now I'm trying to avoid CI specific logic inside the CLI.

[msiebuhr]

Does it make better sense to write it in a single loop?

TARGETS=("FuzzParseMetric" "...")
for TARGET in "${TARGETS[@]}"
do
    go-fuzz-build -libfuzzer -func "$TARGET" -o "$TARGET.a" ./promql
    clang -fsanitize=fuzzer "$TARGET".a -o "$TARGET"
    rm "$TARGET".a
done

[yevgenypats]

Yes this would be better but the problem is that the name of targets on fuzzit and the function names are not the same. i.e -func $TARGET won't work. maybe I can do another array with FUZZ_FUNCTIONS?

[yevgenypats]

@msiebuhr I change it to one loop, let me know if you think this is better?

[msiebuhr]

Many places don't install wget by default. curl -s -o fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v2.4.23/fuzzit_Linux_x86_64 usually brings better luck.

[yevgenypats]

I can change this.

[yevgenypats]

done.

[yevgenypats]

The curl command doesnt work:( so I changed it back to wget it's available in travis so I hope it's not a problem.

[juliusv]

Could you add a comment what this env var is for (since you can't read its contents in the encrypted form here)?

[yevgenypats]

done.

[juliusv]

Could you add more comments here, and also for the go-fuzz-build steps above, to explain what their purpose is (for someone who isn't familiar with all the fuzz tools and steps of fuzzing)?

[yevgenypats]

done.

[juliusv]

Thanks! General Prometheus style comment: aim to make comments sentence-style (start capitalized, end with period).

Otherwise looks good, though I'm neither a fuzzing nor CI expert.

@simonpasquier you are the one with the most touched lines in .travis.yml so if you could just take a last look before merging if I missed anything serious, that'd be cool :)

[yevgenypats]

@juliusv and @msiebuhr Thanks for the quick review! Done regarding the comments style as well.

[yevgenypats]

All checks passes! Let's merge!:)

[simonpasquier]

I haven't reviewed the PR in details but in general I'm in favor of consolidating on Circle CI (nothing against Travis but Circle handles the build/release workflow in addition to test while Travis only runs tests). Would it be possible to apply this to .circleci/config.yml instead?

[yevgenypats]

Hi @simonpasquier I guess it would be possible. Though It was harder to do that in circle for some reason I don't recall now. Would it be possible to do that in another PR?

[simonpasquier]

I'd rather see it in Circle as eventually we don't want to maintain duplicate CI systems.

[yevgenypats]

Ok, got it. will try to migrate this to Circle now.

[yevgenypats]

@simonpasquier done. The fuzzing regression passes. This is ready for review. Also some of you need to sign up to https://app.fuzzit.dev and let me know what's your username so I can put as an admin of prometheus. once that done you will need to copy the api key from fuzzit to CircleCI secret as FUZZIT_API_KEY

[yevgenypats]

All checks passes!

[juliusv]

@yevgenypats Thanks for all the adjustments, especially the CircleCI port! Looks great to my naive eyes now - in principle ready to merge from my perspective, could you just add the DCO Signed-Off-By line to your last commit so that our DCO check passes?

I created a Fuzzit account for my GitHub user (juliusv - julius.volz@gmail.com).

[yevgenypats]

Done. Added you to Prometheus org. You need to copy the api key from here https://app.fuzzit.dev/orgs/prometheus/settings to CircleCi Environment as FUZZIT_API_KEY

[juliusv]

@yevgenypats Thanks! I added the environment variable on CircleCI. Gonna merge once the last check turns green :)

[yevgenypats]

Nice!

…

On Wed, Aug 21, 2019, 11:44 AM Julius Volz ***@***.***> wrote: Merged #5890 <#5890> into master. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5890?email_source=notifications&email_token=AD52CDWFTFXIFUOOKW2APJTQFT6ATA5CNFSM4ILRVR62YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTFIZNMA#event-2572261040>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD52CDXF3454HMWZM76SPSLQFT6ATANCNFSM4ILRVR6Q> .

[simonpasquier]

Thanks! Could you add me too (simonpasquier - pasquier.simon_at_gmail.com) please?

[yevgenypats]

@simonpasquier done. you should have access now to https://app.fuzzit.dev/orgs/prometheus/dashboard

[yevgenypats]

feel free to RT:)

https://twitter.com/fuzzitdev/status/1164122325798793217

[juliusv]

Done :) https://twitter.com/PrometheusIO/status/1164127977510199296?s=19