External dependency release maturity

[htuch]

I have recently been looking at our external dependencies and noticed that a number of external dependencies, e.g. https://github.com/grpc-ecosystem/grpc-httpjson-transcoding, https://github.com/google/jwt_verify_lib/, https://github.com/circonus-labs/libcircllhist, do not provide versioned releases.

I think this is a concern from a security perspective, since it is challenging to systematically monitor for new releases when security issues creep into our external dependencies. You essentially have to watch every commit. There are other problems, for example being unclear when it's safe to take a master snapshot and call it stable.

I would like to propose that we revise our security policy to disallow any new external repositories that might impact the data plane unless they show some release discipline. Basically, they should have some history of cutting releases and providing release notes. We should also downgrade the security posture of extensions relying on external repositories directly that do not have release discipline.

@envoyproxy/security-team

[htuch]

Other problematic repositories:

• https://github.com/google/cel-cpp
• https://github.com/protocolbuffers/upb

[htuch]

Also CC @qiwzhang @kyessenov, do you folks know if you could add versioned releases and release notes to the repositories you own and pin Envoy to them? Thanks.

[kyessenov]

google/cel-cpp#51

[moderation]

My highly manual and not scalable solution is to track all dependencies in RSS. I usually run latest commit versions of all dependencies without releases. I'll occasionally include the updates in the regular dependency update PRs I make but there has been push back in the past on updating the non-release deps.

Full list of deps that don't have releases:

• https://github.com/envoyproxy/envoy-build-tools
• https://github.com/abseil/abseil-cpp *
• https://github.com/circonus-labs/libcircllhist
• https://github.com/envoyproxy/sql-parser *
• https://github.com/google/libprotobuf-mutator
• https://github.com/lightstep/lightstep-tracer-cpp *
• https://github.com/google/jwt_verify_lib
• https://github.com/Tencent/rapidjson *
• https://github.com/grpc-ecosystem/grpc-httpjson-transcoding
• https://github.com/bazelbuild/rules_foreign_cc
• https://github.com/census-instrumentation/opencensus-cpp *
• https://quiche.googlesource.com/quiche
• https://github.com/google/cel-cpp
• https://github.com/protocolbuffers/upb
• BoringSSL (I've never worked out how @PiotrSikora tracks these to Chrome releases)
• https://storage.googleapis.com/envoyproxy-wee8
• https://storage.googleapis.com/quiche-envoy-integration

So the issue is quite large.

In addition to bazel/repository_locations.bzl there are the following in api/bazel/repository_locations.bzl. OpenCensus and PGV do have releases but they are not kept current.

• https://github.com/envoyproxy/protoc-gen-validate *
• https://github.com/googleapis/googleapis
• https://github.com/cncf/udpa
• https://github.com/census-instrumentation/opencensus-proto *
• https://github.com/bazelbuild/rules_proto

* Either has releases we don't use or releases not updated to required commit

[antoniovicente]

How do we reconcile support for older Envoy releases with the need to pick up security fixes for base dependencies? We can't count on base libraries releasing fixes on a particular schedule. I think we wanted to avoid frequent updates to older releases.

Also, what can we do about existing dependencies that don't have regular releases?

[mattklein123]

FYI I'm going to raise this issue with CNCF SIG-security as dependency tracking is an issue that has come up multiple times for many projects.

Also, what can we do about existing dependencies that don't have regular releases?

We probably need to start by enumerating all of our dependencies as we have started to do above, trying to get rid of as many dependencies as possible that we don't actually need, and then looking at the remainder on a case by case basis. The obvious issues here is that just because a dependency has released doesn't actually mean anything security wise so per @htuch we probably need to get some idea of what we want to see from dependencies in addition to arbitrarily snapped releases.

[htuch]

The obvious issues here is that just because a dependency has released doesn't actually mean anything security wise so per @htuch we probably need to get some idea of what we want to see from dependencies in addition to arbitrarily snapped releases.

Yep. I think snapped releases is a necessary, not sufficient condition, and a smell if missing. A more complete list of criteria for what constitutes a dependency we're willing to take would make sense.

Other things I'd like to see:

• A security release policy (ideally in GH tab).
• A security bug report contact address or bug system.
• A history of having issued specific security releases.

Some of this might be hard to apply to smaller dependencies, but it's a sign of project maturity for the larger ones.

[twghu]

Wondering what is the vetting process for new dependencies.

Is there a gating procedure (apart from code reviews) that is currently in play?

• cpe assignment
• cna (CVE Numbering Authority, who can assign CVEs for specific dependencies)
• transitive dependencies
• recommendation of potential duplicity. (a new X lib provides functionality already provided by a current dependency Y)
• security status (aka has security analysis been performed on said dependencies.

[htuch]

@twghu I think these are all good points, I will put together a strawman policy document taking this and other points above into consideration.

[moderation]

I made the dependency spreadsheet public at https://docs.google.com/spreadsheets/d/1OYS9nU55ibLr4Z-io1esgKPO4FIoWsuOzKG60RxePO8/edit#gid=0

[keith]

Note that bazel has some upcoming license aggregation features that might help normalize some of this bazelbuild/bazel#10687

[htuch]

Fun fact, we now have over 60 dependencies total and over 30 on the data plane (if you include Thrift/Kafka/Wasm/LuaJIT etc.). We (Google) will be doing some work in the next few months in this area and want to help drive the policy forward. The first step is to understand what we have today, which is what #12639 is aimed at.

[htuch]

While reading https://www.linuxfoundation.org/wp-content/uploads/2020/02/oss_supply_chain_security.pdf, one thing came up that should feed into our maturity definition. While Envoy repository requires GitHub 2FA/MFA for all developers, it's unclear if this also applies to our dependencies. It seems reasonable to ask for this basic security practice in repositories we rely on given the history of developer account compromises discussed in the paper. I don't know if GH provides a mechanism to determine this programatically.

[htuch]

https://bestpractices.coreinfrastructure.org/en/criteria/0 is a pretty decent list of best practices we could build on.

[htuch]

OSSC also has candidate metrics at https://docs.google.com/spreadsheets/d/119YwYxf5Fy09CjPX6h-yuKZOY4XjCCuDSD_Ryj14s64/edit#gid=0

[htuch]

The OSSF Scorecards project just released some automatable checks, see https://github.com/ossf/scorecard#checks. This seems like a good baseline to adopt.

[htuch]

I've put together a proposal for discussion at https://docs.google.com/document/d/1HbREo7pv7rgeIIjQn6mNpySzQE5rx2Yv9dXm5NqR2N8/edit#, feedback and further input welcome.

[htuch]

@envoyproxy/dependency-shepherds have reviewed #10471 (comment), it might be also a good idea to get feedback from @envoyproxy/maintainers on this proposal as it has implications for future dependencies as well as existing ones.

[htuch]

I've run OSSF Scorecard criteria that we might adopt against all of the eligible deps, results at https://docs.google.com/spreadsheets/d/1caO4qMmG8o5i2nGoEof1qMpD5_WicfiC5WcxA_5isTY/edit#gid=0.

[htuch]

Feedback has been incorporated in https://docs.google.com/document/d/1HbREo7pv7rgeIIjQn6mNpySzQE5rx2Yv9dXm5NqR2N8/edit#. I'm wondering if there is anything else left before we can turn this into plan-of-record and start building out metadata/tooling for enforcement?