Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quickchecking crate CLI #1177

Merged
merged 2 commits into from Dec 6, 2017

Conversation

Projects
None yet
4 participants
@shnewto
Copy link
Contributor

shnewto commented Dec 5, 2017

Prior to this commit the quickchecking crate used for generating proprty tests for bindgen was a [lib] target and had configurations that required commenting/uncommenting code to enable/disable. This meant it was inconvienent/prohibitive to configure the property tests on a per-run basis. This commit reorganizes the quickchecking crate to provide both [lib] and [[bin]] targets in order to expose those configurations through a CLI.

The configurations that are exposed through the [[bin]] target's CLI are:

  • Count/number of tests to run.
  • Directory to provide fuzzed headers
  • Generation range corresponding to the range quickcheck uses to
  • generate arbitrary.

Usage from the tests/quickchecking directory

quickchecking 0.2.0
Bindgen property tests with quickcheck. Generate random valid C code and pass it to the csmith/predicate.py script

USAGE:
    quickchecking [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --count <COUNT>    Count / number of tests to run. Running a fuzzed header through the predicate.py script can
                           take a long time, especially if the generation range is large. Increase this number if you're
                           willing to wait a while. [default: 2]
    -p, --path <PATH>      Optional. Preserve generated headers for inspection, provide directory path for header
                           output. [default: None] 
    -r, --range <RANGE>    Sets the range quickcheck uses during generation. Corresponds to things like arbitrary usize
                           and arbitrary vector length. This number doesn't have to grow much for that execution time to
                           increase significantly. [default: 32]

Because the actual work of running the property tests moved to the [[bin]]
target, rather than duplicate that code in the quickchecking crate's tests
directory, some actual (very basic) tests for the quickchecking crate were
added.

*Note: I'm not attached to any of the option flags, if there are better characters/words for any of the options I've exposed I'll be happy to revise!

Also, I'm not sure how palatable the "global singleton" is for managing context (output path) across tests in the lib.rs file. Very open to suggestions on how to manage that if it's not an acceptable approach.

Thanks for taking a look, looking forward to feedback!

Closes #1168

r? @fitzgen

@shnewto

This comment has been minimized.

Copy link
Contributor Author

shnewto commented Dec 5, 2017

For sanity here are a couple fuzzed headers generated with the state of the current code.
https://gist.github.com/snewt/8afbf552c8f40974a9fdd903e63beec6

Quickchecking crate CLI
Prior to this commit the quickchecking crate used for generating proprty
tests
for bindgen was a [lib] target and had configurations that required
commenting/uncommenting code to enable/disable. This meant it was
inconvienent/prohibitive to configure the property tests on a per-run
basis.
This commit reorganizes the `quickchecking` crate to provide both [lib]
and
[[bin]] targets in order to expose those configurations through a CLI.

The configurations that are exposed through the [[bin]] target's CLI
are:
* Count/number of tests to run.
* Directory to provide fuzzed headers
* Generation range corresponding to the range quickcheck uses to
* generate arbitrary.

__Usage from the__ `tests/quickchecking` __directory__
```bash
quickchecking 0.2.0
Bindgen property tests with quickcheck. Generate random valid C code and
pass it to the csmith/predicate.py script

USAGE:
    quickchecking [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --count <COUNT>    Count / number of tests to run. Running a
fuzzed header through the predicate.py script can
                           take a long time, especially if the
generation range is large. Increase this number if you're
                           willing to wait a while. [default: 2]
    -p, --path <PATH>      Optional. Preserve generated headers for
inspection, provide directory path for header
                           output. [default: None]
    -r, --range <RANGE>    Sets the range quickcheck uses during
generation. Corresponds to things like arbitrary usize
                           and arbitrary vector length. This number
doesn't have to grow much for that execution time to
                           increase significantly. [default: 32]
```

Because the actual work of running the property tests moved to the
[[bin]]
target, rather than duplicate that code in the `quickchecking` crate's
tests
directory, some actual (very basic) tests for the `quickchecking` crate
were
added.

*Note: I'm not attached to any of the option flags, if there are better
characters/words for any of the options I've exposed I'll be happy to
revise!

Thanks for taking a look, looking forward to feedback!

Closes #1168

r? @fitzgen

@shnewto shnewto force-pushed the shnewto:feat/quickcheck-as-bin branch from 9b38bd4 to b4ba092 Dec 5, 2017

@fitzgen

fitzgen approved these changes Dec 5, 2017

Copy link
Member

fitzgen left a comment

Looks great!

Now that the tests are about the quickchecking crate itself, rather than actually quickchecking bindgen, we should run cargo test in CI instead of cargo check.

Once you add that, you can comment @bors-servo r=fitzgen and the PR will get tested and merged.

Thanks!

@fitzgen

This comment has been minimized.

Copy link
Member

fitzgen commented Dec 5, 2017

@bors-servo delegate+

@bors-servo

This comment has been minimized.

Copy link
Contributor

bors-servo commented Dec 5, 2017

✌️ @snewt can now approve this pull request

@shnewto

This comment has been minimized.

Copy link
Contributor Author

shnewto commented Dec 5, 2017

@fitzgen ah awesome, can do (this evening).

quickchecking CI tests
The changes reflected in this PR include the logic to test the
`quickcheking` crate itself. Rather that just validate that the
`quickchecking` crate builds in CI with `cargo check`, we can run
now `cargo test`.
@shnewto

This comment has been minimized.

Copy link
Contributor Author

shnewto commented Dec 6, 2017

@bors-servo r=fitzgen

@bors-servo

This comment has been minimized.

Copy link
Contributor

bors-servo commented Dec 6, 2017

📌 Commit 2c77652 has been approved by fitzgen

@bors-servo

This comment has been minimized.

Copy link
Contributor

bors-servo commented Dec 6, 2017

⌛️ Testing commit 2c77652 with merge a59205e...

bors-servo added a commit that referenced this pull request Dec 6, 2017

Auto merge of #1177 - snewt:feat/quickcheck-as-bin, r=fitzgen
Quickchecking crate CLI

Prior to this commit the quickchecking crate used for generating proprty tests for bindgen was a [lib] target and had configurations that required commenting/uncommenting code to enable/disable. This meant it was inconvienent/prohibitive to configure the property tests on a per-run basis. This commit reorganizes the `quickchecking` crate to provide both [lib] and [[bin]] targets in order to expose those configurations through a CLI.

The configurations that are exposed through the [[bin]] target's CLI are:
* Count/number of tests to run.
* Directory to provide fuzzed headers
* Generation range corresponding to the range quickcheck uses to
* generate arbitrary.

__Usage from the__ `tests/quickchecking` __directory__
```bash
quickchecking 0.2.0
Bindgen property tests with quickcheck. Generate random valid C code and pass it to the csmith/predicate.py script

USAGE:
    quickchecking [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --count <COUNT>    Count / number of tests to run. Running a fuzzed header through the predicate.py script can
                           take a long time, especially if the generation range is large. Increase this number if you're
                           willing to wait a while. [default: 2]
    -p, --path <PATH>      Optional. Preserve generated headers for inspection, provide directory path for header
                           output. [default: None]
    -r, --range <RANGE>    Sets the range quickcheck uses during generation. Corresponds to things like arbitrary usize
                           and arbitrary vector length. This number doesn't have to grow much for that execution time to
                           increase significantly. [default: 32]
```

Because the actual work of running the property tests moved to the [[bin]]
target, rather than duplicate that code in the `quickchecking` crate's tests
directory, some actual (very basic) tests for the `quickchecking` crate were
added.

*Note: I'm not attached to any of the option flags, if there are better characters/words for any of the options I've exposed I'll be happy to revise!

Also, I'm not sure how palatable the "global singleton" is for managing context (output path) across tests in the `lib.rs` file. Very open to suggestions on how to manage that if it's not an acceptable approach.

Thanks for taking a look, looking forward to feedback!

Closes #1168

r? @fitzgen
@bors-servo

This comment has been minimized.

Copy link
Contributor

bors-servo commented Dec 6, 2017

☀️ Test successful - status-travis
Approved by: fitzgen
Pushing a59205e to master...

@bors-servo bors-servo merged commit 2c77652 into rust-lang:master Dec 6, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

@shnewto shnewto deleted the shnewto:feat/quickcheck-as-bin branch Dec 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.