Skip to content

Latest commit

 

History

History
167 lines (127 loc) · 4.93 KB

configuration.md

File metadata and controls

167 lines (127 loc) · 4.93 KB
Raw

Configuration

When you run processes with bin/cli, there are two ways to configure command options:

  • on the command line, using the appropriate flag.
  • in a configuration file.

Command line options for bin/cli

To view documentation on the command line, you can run bin/cli commands with the --help option.

For example, to view a summary of the commands available, run:

$ bin/cli --help

To view the documentation for the command-line options available for the bin/cli run command, run:

$ bin/cli run --help

Selecting CVEs of interest

Some bin/cli commands require you to select data for CVEs. There are three kinds of CVE selector you can use:

  • An explicit CVE identifier. For example, CVE-2018-3743.
  • Patterns that select a group of CVEs. For example, year:2020, CWE-88,
    and mitre-cwe-top:25:2020.
  • A wildcard operator '*' that selects all available benchmark CVEs.

To see a full list of the available selectors, run:

$ bin/cli list --help

For commands that require you to select CVEs, you can use any combination of selectors, and the command will use the union of the selected CVEs. For example, CWE-88 CWE-89 CVE-123-456 uses the single CVE CVE-123-456 and all CVEs with either CWE-88 or CWE-89.

For advanced scripting, the selectors can be provided as a line-separated input stream in stdin, rather than on the command line. For example:

$ echo CWE-88 > my-cves.txt
$ echo CWE-89 > my-cves.txt
$ echo CVE-123-456 > my-cves.txt
$ cat my-cves.txt | bin/cli list -

is equivalent to bin/cli list CWE-88 CWE-89 CVE-123-456.

Note that CVEs that are considered to be "incomplete" will not be selected by default. For more information, see Benchmark CVEs.

Configuration file options for bin/cli

Many commands accept options specified in a configuration file, config.json. By default, bin/cli looks in the current working directory for the configuration file. If you want to save your configuration file in an alternative location, you must specify the path to config.json using the --config option on the command line.

Most options that are set in config.json may be overridden individually on a per-command basis with identically named command-line arguments. For example, you can use sources to provide a location to download source code for CVEs in config.json. To override the sources value in config.json, you can specify --sources on the command line.

Configuring drivers for analysis tools

Configuration files are also used to configure drivers for analysis tools. For an example of configuring a driver for an analysis tool, see Configuring the driver for eslint, or browse the README.md files of each driver in contrib/tools.

Sample configuration files

This section contains various config.json examples.

For more information about the allowed contents of a configuration file, see the JSON schema for the Config type.

Minimal configuration file

The configuration file does not need to specify anything, but if config.json is present, it should at least be a valid JSON object.

{}

Configuration file with one configured driver

As mentioned above, the .tools property of the configuration file is used to configure drivers for analysis tools. The example below configures a driver for ESLint. You can then benchmark ESLint by running bin/cli run --tool eslint-default.

{
  "tools": {
    "eslint-default": {
      "bin": "node",
      "args": [
        "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
      ],
      "options": {
        "eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
      }
    }
  }
}

Configuration file with two configured drivers

It is rarely useful to only have one configured driver for benchmarking. The sample below configures drivers for two different versions of ESLint. You can benchmark both versions to compare their performance by running bin/cli run --tool eslint-default --tool eslint-2019.

{
  "tools": {
    "eslint-default": {
      "bin": "node",
      "args": [
        "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
      ],
      "options": {
        "eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
      }
    },
    "eslint-2019": {
      "bin": "node",
      "args": [
        "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
      ],
      "options": {
        "eslintDir": "/home/user-name/analysis-tools/eslint-2019-12-08"
      }
    }

  }
}

Configuration file with custom download and result locations

The .sources property of the configuration file controls the location that the relevant commits of CVEs are downloaded to.

{
  "sources": "/home/user-name/ossf-sources"
}