Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve testing #41

Open
DavidKorczynski opened this issue Feb 16, 2022 · 4 comments
Open

Improve testing #41

DavidKorczynski opened this issue Feb 16, 2022 · 4 comments
Labels
core feature

Comments

@DavidKorczynski
Copy link
Contributor

We currently have some facilities to test fuzz-introspector, including:

There are room for improvements in terms of testing. This is an umbrella issue for testing improvements that commits can refer to.
@Navidem Navidem added the enhancement New feature or request label Feb 27, 2022
@DavidKorczynski DavidKorczynski mentioned this issue Mar 9, 2022
Open
@DavidKorczynski
Copy link
Contributor Author

Setting this to higher priority since we need to provide better guards for OSS-Fuzz bumps:
google/oss-fuzz#7635

@oliverchang
Copy link
Contributor

Thanks! Another idea for OSS-Fuzz integration may be to pin to a "stable" release for OSS-fuzz, and we can bump after manually verifying that a few reports on the latest revision looks OK.

May not be needed if the tests are comprehensive enough though.

DavidKorczynski added a commit to DavidKorczynski/fuzz-introspector-2 that referenced this issue Apr 30, 2022
@DavidKorczynski
oss-fuzz: add proper testing set up
836eaea 
Add testing set up for oss-fuzz that can automatically analyse
properties against a set of projects. This is achieved by making
fuzz-introspector write .json files with data about the run which is
then processed by the testing framework.

The testing framework has some hard-coded boundary checks on the data
for a given project. It has to be boundaries rather than fixed values
since the data may change from run to run depending on the fuzzing
results.

In order to make the check complete it needs to have hard-coded values
from a diverse set of projects, in particular both c and c++ projects.

Ref: ossf#41
@DavidKorczynski DavidKorczynski mentioned this issue Apr 30, 2022
Merged
DavidKorczynski added a commit that referenced this issue May 4, 2022
@DavidKorczynski
oss-fuzz: add proper testing set up (#251)
0722bbb 
* oss-fuzz: add proper testing set up

Add testing set up for oss-fuzz that can automatically analyse
properties against a set of projects. This is achieved by making
fuzz-introspector write .json files with data about the run which is
then processed by the testing framework.

The testing framework has some hard-coded boundary checks on the data
for a given project. It has to be boundaries rather than fixed values
since the data may change from run to run depending on the fuzzing
results.

In order to make the check complete it needs to have hard-coded values
from a diverse set of projects, in particular both c and c++ projects.

Ref: #41
DavidKorczynski added a commit to DavidKorczynski/fuzz-introspector-2 that referenced this issue May 13, 2022
@DavidKorczynski
test: oss-fuzz: add leveldb specs
8eb6223 
Ref: ossf#41
@DavidKorczynski DavidKorczynski mentioned this issue May 13, 2022
Merged
AdamKorcz pushed a commit that referenced this issue May 14, 2022
@DavidKorczynski
test: oss-fuzz: add leveldb specs (#265)
020191a 
Ref: #41
@DavidKorczynski
Copy link
Contributor Author

Ref #287

DavidKorczynski added a commit to DavidKorczynski/fuzz-introspector-2 that referenced this issue May 28, 2022
@DavidKorczynski
test: add wuffs to oss-fuzz testing
26326b7 
To prevent regression of:
ossf#288

Ref: ossf#41
@DavidKorczynski DavidKorczynski mentioned this issue May 28, 2022
Merged
AdamKorcz pushed a commit that referenced this issue May 28, 2022
@DavidKorczynski
test: add wuffs to oss-fuzz testing (#291)
f19aa7e 
To prevent regression of:
#288

Ref: #41
@DavidKorczynski DavidKorczynski mentioned this issue Jun 1, 2022
Merged
@DavidKorczynski DavidKorczynski removed enhancement New feature or request priority labels Jun 7, 2022
@DavidKorczynski
Copy link
Contributor Author

Removed priority and enhancement labels as the testing infrastructure is now at a much better place than when this issue was created.

Keeping the issue open so it can be referenced in commits.

@DavidKorczynski DavidKorczynski mentioned this issue Jul 20, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DavidKorczynski @oliverchang @Navidem