Skip to content

Commit

Permalink
✨ Detect fast-check PBT library for fuzz section (#3073)
Browse files Browse the repository at this point in the history 
* ✨ Detect fast-check PBT library for fuzz section

As suggested at #2792 (comment), we add support for the detection of fast-check as a possible fuzzing solution.

I also adapted the documentation related to fuzzing accordingly.

Signed-off-by: Nicolas DUBIEN <github@dubien.org>

* Typo

Signed-off-by: Nicolas DUBIEN <github@dubien.org>

* Update missing md files

Signed-off-by: Nicolas DUBIEN <github@dubien.org>

---------

Signed-off-by: Nicolas DUBIEN <github@dubien.org>
  • Loading branch information
@dubzzz
dubzzz committed Jun 4, 2023
1 parent fdfe2b9 commit d961dda
Show file tree
Hide file tree
Showing 5 changed files with 112 additions and 8 deletions.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -457,7 +457,7 @@ Name | Description | Risk Level | Token Req
[Contributors](docs/checks.md#contributors) | Does the project have contributors from at least two different organizations? | Low | PAT, GITHUB_TOKEN | Validating |
[Dangerous-Workflow](docs/checks.md#dangerous-workflow) | Does the project avoid dangerous coding patterns in GitHub Action workflows? | Critical | PAT, GITHUB_TOKEN | Unsupported |
[Dependency-Update-Tool](docs/checks.md#dependency-update-tool) | Does the project use tools to help update its dependencies? | High | PAT, GITHUB_TOKEN | Unsupported |
[Fuzzing](docs/checks.md#fuzzing) | Does the project use fuzzing tools, e.g. [OSS-Fuzz](https://github.com/google/oss-fuzz)? | Medium | PAT, GITHUB_TOKEN | Validating
[Fuzzing](docs/checks.md#fuzzing) | Does the project use fuzzing tools, e.g. [OSS-Fuzz](https://github.com/google/oss-fuzz), [QuickCheck](https://hackage.haskell.org/package/QuickCheck) or [fast-check](https://fast-check.dev/)? | Medium | PAT, GITHUB_TOKEN | Validating
[License](docs/checks.md#license) | Does the project declare a license? | Low | PAT, GITHUB_TOKEN | Validating |
[Maintained](docs/checks.md#maintained) | Is the project at least 90 days old, and maintained? | High | PAT, GITHUB_TOKEN | Validating |
[Pinned-Dependencies](docs/checks.md#pinned-dependencies) | Does the project declare and pin [dependencies](https://docs.github.com/en/free-pro-team@latest/github/visualizing-repository-data-with-graphs/about-the-dependency-graph#supported-package-ecosystems)? | Medium | PAT, GITHUB_TOKEN | Validating |
Expand Down
36 changes: 31 additions & 5 deletions checks/raw/fuzzing.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,13 @@ import (
)

const (
fuzzerOSSFuzz = "OSSFuzz"
fuzzerClusterFuzzLite = "ClusterFuzzLite"
oneFuzz = "OneFuzz"
fuzzerBuiltInGo = "GoBuiltInFuzzer"
fuzzerPropertyBasedHaskell = "HaskellPropertyBasedTesting"
fuzzerOSSFuzz = "OSSFuzz"
fuzzerClusterFuzzLite = "ClusterFuzzLite"
oneFuzz = "OneFuzz"
fuzzerBuiltInGo = "GoBuiltInFuzzer"
fuzzerPropertyBasedHaskell = "HaskellPropertyBasedTesting"
fuzzerPropertyBasedJavaScript = "JavaScriptPropertyBasedTesting"
fuzzerPropertyBasedTypeScript = "TypeScriptPropertyBasedTesting"
// TODO: add more fuzzing check supports.
)

Expand Down Expand Up @@ -87,6 +89,30 @@ var languageFuzzSpecs = map[clients.LanguageName]languageFuzzConfig{
"Property-based testing in Haskell generates test instances randomly or exhaustively " +
"and test that specific properties are satisfied."),
},
// Fuzz patterns for JavaScript and TypeScript based on property-based testing.
//
// Based on the import of one of these packages:
// * https://fast-check.dev/
//
// This is not an exhaustive list.
clients.JavaScript: {
filePattern: "*.js",
// Look for direct imports of fast-check.
funcPattern: `(from\s+['"]fast-check['"]|require\(\s*['"]fast-check['"]\s*\))`,
Name: fuzzerPropertyBasedJavaScript,
Desc: asPointer(
"Property-based testing in JavaScript generates test instances randomly or exhaustively " +
"and test that specific properties are satisfied."),
},
clients.TypeScript: {
filePattern: "*.ts",
// Look for direct imports of fast-check.
funcPattern: `(from\s+['"]fast-check['"]|require\(\s*['"]fast-check['"]\s*\))`,
Name: fuzzerPropertyBasedTypeScript,
Desc: asPointer(
"Property-based testing in TypeScript generates test instances randomly or exhaustively " +
"and test that specific properties are satisfied."),
},
// TODO: add more language-specific fuzz patterns & configs.
}

Expand Down
74 changes: 74 additions & 0 deletions checks/raw/fuzzing_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,80 @@ func Test_checkFuzzFunc(t *testing.T) {
},
fileContent: "import Test.Hspec",
},
{
name: "JavaScript fast-check via require",
want: true,
fileName: []string{"main.spec.js"},
langs: []clients.Language{
{
Name: clients.JavaScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-check');",
},
{
name: "JavaScript fast-check via import",
want: true,
fileName: []string{"main.spec.js"},
langs: []clients.Language{
{
Name: clients.JavaScript,
NumLines: 50,
},
},
fileContent: "import fc from \"fast-check\";",
},
{
name: "JavaScript with no property-based testing",
want: false,
fileName: []string{"main.spec.js"},
wantErr: true,
langs: []clients.Language{
{
Name: clients.JavaScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-other');",
},
{
name: "TypeScript fast-check via require",
want: true,
fileName: []string{"main.spec.ts"},
langs: []clients.Language{
{
Name: clients.TypeScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-check');",
},
{
name: "TypeScript fast-check via import",
want: true,
fileName: []string{"main.spec.ts"},
langs: []clients.Language{
{
Name: clients.TypeScript,
NumLines: 50,
},
},
fileContent: "import fc from \"fast-check\";",
},
{
name: "TypeScript with no property-based testing",
want: false,
fileName: []string{"main.spec.ts"},
wantErr: true,
langs: []clients.Language{
{
Name: clients.TypeScript,
NumLines: 50,
},
},
fileContent: "const fc = require('fast-other');",
},
}
for _, tt := range tests {
tt := tt
Expand Down
4 changes: 3 additions & 1 deletion docs/checks.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,9 @@ This check tries to determine if the project uses
1. if the repository name is included in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project list;
2. if [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) is deployed in the repository;
3. if there are user-defined language-specified fuzzing functions in the repository.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/) and a limited set of property-based testing libraries for Haskell.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/),
- a limited set of property-based testing libraries for Haskell including [QuickCheck](https://hackage.haskell.org/package/QuickCheck), [Hedgehog](https://hedgehog.qa/), [validity](https://hackage.haskell.org/package/validity) or [SmallCheck](https://hackage.haskell.org/package/smallcheck),
- a limited set of property-based testing libraries for JavaScript and TypeScript including [fast-check](https://fast-check.dev/).
4. if it contains a [OneFuzz](https://github.com/microsoft/onefuzz) integration [detection file](https://github.com/microsoft/onefuzz/blob/main/docs/getting-started.md#detecting-the-use-of-onefuzz);

Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data
Expand Down
4 changes: 3 additions & 1 deletion docs/checks/internal/checks.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,9 @@ checks:
1. if the repository name is included in the [OSS-Fuzz](https://github.com/google/oss-fuzz) project list;
2. if [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) is deployed in the repository;
3. if there are user-defined language-specified fuzzing functions in the repository.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/) and a limited set of property-based testing libraries for Haskell.
- currently only supports [Go fuzzing](https://go.dev/doc/fuzz/),
- a limited set of property-based testing libraries for Haskell including [QuickCheck](https://hackage.haskell.org/package/QuickCheck), [Hedgehog](https://hedgehog.qa/), [validity](https://hackage.haskell.org/package/validity) or [SmallCheck](https://hackage.haskell.org/package/smallcheck),
- a limited set of property-based testing libraries for JavaScript and TypeScript including [fast-check](https://fast-check.dev/).
4. if it contains a [OneFuzz](https://github.com/microsoft/onefuzz) integration [detection file](https://github.com/microsoft/onefuzz/blob/main/docs/getting-started.md#detecting-the-use-of-onefuzz);
Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data
Expand Down

0 comments on commit d961dda

Please sign in to comment.