Skip to content

Latest commit

 

History

History
133 lines (109 loc) · 4.6 KB

benchmark-CVEs.md

File metadata and controls

133 lines (109 loc) · 4.6 KB
Raw

Benchmark CVEs

The CVEs directory contains benchmark CVE entries, encoded as a JSON file per CVE. A benchmark CVE (or BCVE) entry consists of a CVE identifier, in addition to some information that isn't normally available in an ordinary CVE.

For a CVE to be eligible as a benchmark CVE, the CVE must be for open-source software and be publicly accessible.

The basic content of a benchmark CVE is described by the following (informal) TypeScript type:

{
  CVE: CVEString;
  state: CVEState;
  repository: URLString;
  prePatch: {
    commit: CommitID;
    weaknesses: {
      location: SourceLocation;
      explanation: string;
    }[];
  };
  postPatch: {
    commit: CommitID;
  };
  CWEs: CWEString[]
}

With an an example instantiation being:

{
    "CVE": "CVE-2020-8203",
    "state": "PUBLISHED",
    "repository": "https://github.com/lodash/lodash.git",
    "prePatch": {
        "commit": "e7b28ea6cb17b4ca021e7c9d66218c8c89782f32",
        "weaknesses": [
            {
                "location": {
                    "file": "lodash.js",
                    "line": 2559
                },
                "explanation": "Prototype pollution in utility function"
            }
        ]
    },
    "postPatch": {
        "commit": "c84fe82760fb2d3e03a63379b297a1cc1a2fce12"
    },
    "CWEs": ["CWE-471"]
}

For more information, see the BCVE Schema. For the formal specification of the benchmark CVE format, see BCVE.schema.json.

Complete benchmark CVEs

A benchmark CVE is considered "complete" if all of the following hold:

  • state is PUBLISHED, meaning that no further work on the CVE is expected, and that the vulnerability has been acknowledged.
  • prePatch exists and contains at least one relevant weakness, meaning that the vulnerability was caused by a source code mistake.
  • postPatch exists and contains zero relevant weaknesses, meaning that the vulnerability has been fixed properly.

For each complete benchmark CVE, the ideal analysis tool produces at least one relevant alert on the prePatch commit, and no alerts on the postPatch commit. This means that the ideal analysis tool would have classified the prePatch commit as vulnerable, and that it recognizes the postPatch commit as a sufficient fix.

By default, incomplete benchmark CVEs are ignored by CVE benchmark tooling.

Commits related to a CVE

In most cases there are five different kinds of commit that are relevant for a CVE. This classification generalizes to the situation where multiple parallel versions of an application are vulnerable to the same CVE.

  • preIntroduction: the commit just before the vulnerability was introduced.
  • postIntroduction: the commit that introduces the vulnerability.
  • patchBase: the commit that a complete patch is applied to.
  • prePatch: the commit just before the vulnerability is fixed.
  • postPatch: the commit that fixes the vulnerability.

That is, the preIntroduction and postPatch commits are not expected to be affected by the vulnerability described in the CVE, while postIntroduction, patchBase, and prePatch are.

Identifying relevant commits

Ideally, security fixes are small, surgical commits. If a vulnerability is fixed in a single commit patchBase is equal to prePatch, and the commit is easy to identify. However, in reality, security fixes sometimes require several interdependent commits to completely address a vulnerability. For example:

  • A fix may require several commits between patchBase and prePatch that each partially address the vulnerability. In this case, it may be hard to find the exact preIntroduction and postIntroduction commits.

  • If preIntroduction or postPatch are discovered to be vulnerable to the CVE after the CVE data was published, extra commits are required to completely fix the vulnerability.

For simplicity, the project currently only considers a single pair of prePatch and postPatch commits for each CVE.

Permanent access of the relevant CVE commits

To ensure that the relevant commits for each benchmark CVE remains accessible, they will be uploaded to CVE-specific git repositories at https://github.com/ossf-cve-benchmark/ by the ossf-cve-benchmark project maintainers.

For git commits, the commit hashes will remain unchanged when pushed to these CVE-specific repositories, but that will not be the case for other version control systems. To remedy this, the branch names will encode information about the commit identifiers, and the benchmark tooling will find the expected commits using that information.