Proposed new criterion: Require software composition analysis (SCA)

[david-a-wheeler]

I think the time has come to more directly require certain kinds of software composition analysis (SCA) tools (aka origin analysis tools). More specifically, projects should at least use tools to monitor their dependencies to determine if any of them have publicly-known vulnerabilities, and if there's an upgrade available to counter them, do the upgrade in a timely way.

Vulnerable external dependencies are a common and well-known problem. This is OWASP Top 10 (2017) item A9, and it was also in the 2013 version of the OWASP top 10.

We were quite concerned about the issue when we set up the criteria. The silver requirements already have a section called "Externally-maintained components" that focus on components with publicly known vulnerabilities. This section has these silver criteria:

• The project MUST list external dependencies in a computer-processable way. [external_dependencies]. Details: Typically this is done using the conventions of package manager and/or build system. Note that this helps implement installation_development_quick.
• The project MUST either (1) make it easy to identify and update reused externally-maintained components or (2) use the standard components provided by the system or programming language. Then, if a vulnerability is found in a reused component, it will be easy to update that component. [updateable_reused_components] Details: A typical way to meet this criterion is to use system and programming language package management systems. Many FLOSS programs are distributed with "convenience libraries" that are local copies of standard libraries (possibly forked). By itself, that's fine. However, if the program must use these local (forked) copies, then updating the "standard" libraries as a security update will leave these additional copies still vulnerable. This is especially an issue for cloud-based systems; if the cloud provider updates their "standard" libaries but the program won't use them, then the updates don't actually help. See, e.g., "Chromium: Why it isn't in Fedora yet as a proper package" by Tom Callaway.
• The project SHOULD avoid using deprecated or obsolete functions and APIs where FLOSS alternatives are available in the set of technology it uses (its "technology stack") and to a supermajority of the users the project supports (so that users have ready access to the alternative). [interfaces_current]

However, we did not require the use of tools or services to do SCA. At the time such tools were less common, didn't handle as many languages, often were expensive, and often were proprietary (which we're trying to not mandate). The criteria must always be practical for projects without infinite funding.

I'm delighted to say that things have changed. The SCA tooling today has become far simpler & easier:

• Many language package managers today build in or have simple plug-ins to detect dependencies with known vulnerabilities, for example, npm (JavaScript) has npm audit and bundler (Ruby) has bundler audit.
• OWASP DependencyCheck provides an external analysis engine and continues to gain capabilities.
• Many collaborative software development services (aka forges) now include SCA analysis as part of their services. This includes GitHub (via its GitHub security alerts) and GitLab (Gemnasium).

I think we can now require tools or services for identifying vulnerable external dependencies, and not require identifying vendored components. Vendored components are local copies of external dependencies, which over time often become variants of their official releases. Vendored components can also be vulnerable, and of course they are a real concern. However, identifying vendored components is much harder than examining a few dependency files, so there are fewer tools that support analysis of vendored components; they are also much costlier and tend to be proprietary. Also, now that package managers provide so much functionality and so many people use them, I think vendoring has generally been on a steep decline. Of course, nothing prevents anyone from using a tool to look for vulnerable vendored components. The issue here is whether or not to require them at some level. I think it's better to focus on the much more common case that is easier to address (vulnerable external components that are not vendored), and in the future if good tools become OSS we can deal with the vendored case.

Here's a first cut at a criterion, which I propose to start in the "silver" level in the "Externally-maintained components" section:

• The project MUST use at least one tool or service to alert the project when a vulnerability becomes publicly known in an external component that the project's software depends on (directly or indirectly), as long as there is at least one FLOSS tool or service that can do so. External components in the underlying operating system and programming language run-time MAY be excluded (since these are often handled separately). [monitor_external_dependencies]

Details: "Publicly known" here means that it has been announced through a freely and publicly available machine-processable database of vulnerabilities such as the National Vulnerability Database; a public posting in a mailing list, commit, or component-specific changelog does not count for this purpose (since it is impractical to read all mailing lists, commits, and changelogs worldwide). Tools and services that do this are sometimes called software composition analysis (SCA) tools or origin analysis tools. Note that this criterion focuses only on external dependencies; we do not require this for vendored components, since tools and services to identify vendored components tend to be less common, more expensive, and proprietary. Instead, this merely requires the use of tools that can analyze package management data, which is much easier to automate and handle.

• The project MUST update or mitigate within 60 days all vulnerable external component dependencies to non-vulnerable versions, once the vulnerabilities become publicly known as discussed in [monitor_external_dependencies], if the vulnerable is exploitable in the project's default configuration and updates are available for the project's set of dependencies. Vulnerabilities in the project's underlying operating system or programming language run-time need not be handled this way. [upgrade_vulnerable_dependencies]

Details: Note that we only require updates or mitigations if the vulnerability is exploitable and an update is available. The phrase "project's set of dependencies" is used because in some cases upgrades will be incompatible with other project components, and this can make it impractical to upgrade. Ideally projects would always update or mitigate in a timely way, but in some cases this is impractical if the upstream is unable to fix it. It can be harder to determine if something is exploitable than to upgrade; in such cases we strongly recommend upgrading. Projects may want to investigate the dependency upgrade first to reduce the risk that the upgrade causes some other vulnerability (including malicious code).