Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuzzer script #1534

Closed
veripoolbot opened this issue Sep 30, 2019 · 6 comments
Closed

Fuzzer script #1534

veripoolbot opened this issue Sep 30, 2019 · 6 comments

Comments

@veripoolbot
Copy link

@veripoolbot veripoolbot commented Sep 30, 2019


Author Name: Eric Rippey
Original Redmine Issue: 1534 from https://www.veripool.org

Original Assignee: Eric Rippey


In this bug: https://www.veripool.org/issues/1529-Verilator-Sefault-on-short-testcase

There was a request to file a bug with info about the fuzzing the produced issues 1529-1533 was run. This is that bug.

These were done with American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/)

The attached script should be do this when run on a Debian or Ubuntu based machine.

This script assumes that it is run as root. This is only needed for some of the steps. In general, running programs that are expected to have unusual behavior as root is probably not a great idea - you may want to edit this. This should get things started though.

Note that the tool's idea of what's an "interesting crash" is not perfect. There are both cases where something looks fine to the tool but it actually produced the "buffer overflow" error, and also cases where what it classifies as a crash do not have any obvious problem. So I also have a script which I was using to rerun the cases that it found. This is in "actual_fail.py". This assumes a couple of things about paths that will need to be edited.

@veripoolbot

This comment has been minimized.

Copy link
Author

@veripoolbot veripoolbot commented Oct 1, 2019


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-10-01T00:09:42Z


Sweet. Always wanted to try AFL but never got to it.

Ran it for 2 minutes (which know is nothing the the grand scheme). Perhaps you can set it up to use only ASCII to speed things up - as any control chars are unlikely to be interesting.

@veripoolbot

This comment has been minimized.

Copy link
Author

@veripoolbot veripoolbot commented Oct 1, 2019


Original Redmine Comment
Author Name: Eric Rippey
Original Date: 2019-10-01T00:50:54Z


As far as I know there isn't an option to keep the inputs ASCII.

There is an option to use a collection of interesting strings. That could be used to make the fuzzer know the whole set of valid keywords for Verilog and the set of valid symbols. I would expect that this would make things way more efficient. I've never used this option though and someone would have to write this list.

A different thing that can be done to speed up the fuzzer is to give it a larger corpus of examples to start with. In my example script, I created a single starting point, but to find the bugs that I filed I started with a set of around 1000 short files that I had around for other reasons. A good place to start might be to point it at the existing files in the test_regress/t directory.

You're right that two minutes is nowhere near how long things are expected to take. Assuming that you're seeing the number of paths found increase, it's not a bad idea to leave it to run overnight although I was seeing some results within the first half hour.

@veripoolbot

This comment has been minimized.

Copy link
Author

@veripoolbot veripoolbot commented Oct 1, 2019


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-10-01T01:53:46Z


This is much appreciated.

  1. First legal stuff, do you agree what you attach and future contributions are open source as described here? https://developercertificate.org/

How far do you want to take this? Not sure if you are willing to do more work, certainly would be helpful. Here's some suggestions based on your ideas:

  1. Make nodist/* files we can put in the distro that run with "cwd" in the root of the verilator kit (e.g. cd ; nodist/fuzzer). (Seems like just a few tweaks needed for this.)

  2. Add corpus from test_regress?

  3. Limit to ASCII if possible?

  4. Add words based on tokens from verilog.l, maybe with the letters a-z for symbol use?

@veripoolbot

This comment has been minimized.

Copy link
Author

@veripoolbot veripoolbot commented Oct 7, 2019


Original Redmine Comment
Author Name: Eric Rippey
Original Date: 2019-10-07T17:31:22Z


The contributor agreement is fine.

At this point, I don't have a real bird's eye view of the how the project works to be able to how exactly things should be integrated. For example, where would you want this in the tree?

And where would this be expected to be run? Is there a set of platforms that I should make sure things work on? From reading the bug about using Travis CI, it seems like there isn't a set of dedicated machines anywhere to run things on.

Is there a document somewhere that lays out some development guidelines? For example, one of the scripts that I attached is Python and that might be the first piece of Python in the tree.

@veripoolbot

This comment has been minimized.

Copy link
Author

@veripoolbot veripoolbot commented Oct 7, 2019


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-10-07T23:06:41Z


I'd suggest putting the files under the "nodist" directory, this is for things that are developer only and don't need to go into a tarball.

I personally ran the fuzzing from the $VERILATOR_ROOT directory (where the .git file is.)

I don't think we'd fuzz on Travis-CI, basically if if works on a standard Linux distro like Ubuntu 18.04 we're good.

There's some information in docs/CONTRIBUTING.adoc and docs/internals.adoc that might be of interest. (Make sure you pull first as these were recently cleaned up.)

Thanks for looking into this.

@veripoolbot

This comment has been minimized.

Copy link
Author

@veripoolbot veripoolbot commented Oct 17, 2019


Original Redmine Comment
Author Name: Wilson Snyder (@wsnyder)
Original Date: 2019-10-17T02:20:22Z


Scripts pushed to git.

Know there will be minor changes to e.g. ignore some internal errors, feel free to send as pull requests, or post a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.