Question: what assurance do you offer the Root CA store is 'trustworthy'

[chrisdlangton]

So I've started using Rust more, in 2020 I found and utilised Rustls (thank you)

I listen to Deirdre Connolly who is a massive fan of Rustls (and I am a fan of her so by extension..) and it eventually occurred to me that I should at least verify 'my trust' in 'my' usage.

Inherently a Root CA store must be trusted when used, and to be trustworthy the expectation is the trust we place can be validated.

Being open source is a good start, but it doesn't answer the specific question; The certificates in this Root CA store are verifiable trustworthy

And this repo does not actually address assurance directly either

I can verify what the code does, bare with me as I'm the maintainer of tlstrust on pypi which aims to verify trustworthiness of all Root CA stores, there is no bias, you have not been targeted

---

Assertion 1. The crate contains curated root certificates for use with the webpki and anywhere rustls itself is used. You apply filtering on the subject and produce the resultant Root CA Trust Store.

Assertion 2. Before you process the bundle of certs you fetch over HTTP from mkcert.org, mistakenly referred to as the "Mozilla tooling", which it is not

Assertion 3. Mozilla's original source of these root certificates is the Common CA Database (CCADB) (also the source for Google, Microsoft, Apple, and many more) but the not-Mozilla mkcert tooling consumes the Network Security Services (NSS) bundle instead because it's already curated for their own products

---

Conclusion 1. The assurance of NSS is not yours to provide, it is gained from Mozilla

Conclusion 2. You cannot verify anything about the environment mkcert.org - what is served over HTTP could have been MitM anywhere along the line. It would have been verifiable had you run the mkcert tooling locally in an environment you control and/or validated the resultant fingerprints from CCADB/NSS

Conclusion 3. The subject is not secure so filtering on it is not a trustworthy mechanism. If additional filtering by you is necessary at all, the most appropriate filtering mechanism would be the Subject Key Identifier (SKI), or the serial number, or a SHA256 fingerprint.

---

At this stage, using well defined and accepted security characteristics, I can't see how the Rustls Root CA store is verifiable.
Without verification it really is not trustworthy.
And the single most important inherent characteristic of a Root CA store is that is must be trusted.

[ctz]

Assertion 1. The crate

I think some clarification is needed on the source of these assertions.

contains curated root certificates

Please clarify what is meant by "curated"?

Assertion 2. Before you process the bundle of certs you fetch over HTTP from mkcert.org,

It is fetched over HTTPS.

mistakenly referred to as the "Mozilla tooling", which it is not

I don't see the quoted text anywhere in this repo - please clarify?

Assertion 3. Mozilla's original source of these root certificates is the Common CA Database (CCADB)

I believe CCADB post-dates the NSS bundle by decades, so it's not correct to say it is the original source.

what is served over HTTP could have been MitM anywhere along the line

As above, incorrect.

The subject is not secure so filtering on it is not a trustworthy mechanism.

This is the filtering API provided by mkcert.org, used here to respond quicker to public CA incidents before Mozilla fully distrust. Note that the current state (and intended steady-state) is that this filtering has no effect, because the offending CAs were removed by the Mozilla root program.

[chrisdlangton]

I think some clarification is needed on the source of these assertions.

Happy to clarify, the 'source' of the assertions is (it not obvious already in my summary) the personal use and code review of this repo by me.
I assume all your needed clarifications are requested in your response already, so I will stay on only the topics you raised.

Please clarify what is meant by "curated"?

I mean the meaning of the word; selected, organized, and presented

i.e. the code shows the selection of a source to obtain root certs, the code organises these with applied filtering on the subject, and the result is the presentation layer consumed by crates like webpki and anywhere rustls itself is used.

Apart from quoting the definition of curate I have not added anything not already explained in the OP so if your understanding of this assertion is still a problem perhaps you could be more specific about your rationale and purpose for questioning the assertion in more than asking 'what', thanks.

It is fetched over HTTPS

Yes, layman terms there is a distinction that might be important, but the protocol literally is HTTP in the rfc9110 even going back to the HTTP bible rfc2818 it's semantically HTTP over TLS. This is a digression away from the point, HTTP is a protocol making a distinction from something within your control.
The assertion is as a HTTP client you are not in control of what is sent by the HTTP server (or how it was produced), this is the distinction where HTTP is used

TLS is assumed, and I want to make it very clear that the TLS part is not even considered, it is entirely irrelevant in context of the question (but TLS is obviously important for the overall solution)

I don't see the quoted text anywhere in this repo - please clarify?

It's not a quote, it's emphasis.
try this; https://github.com/rustls/webpki-roots/search?q=Mozilla

I believe CCADB post-dates the NSS bundle by decades, so it's not correct to say it is the original source.

the 'post-date' semantic is a distinction without a difference. Sure CCADB post-dated apple products use of CCADB, but all apple products before did get updates and now every updated apple device will be using CCADB now..

Essentially NSS and CCADB are distinct things, one does not exclude the other; https://wiki.mozilla.org/CA
CCADB is where certificates are 'source of truth' and NSS is a usage sourcing from that and they used to be their own source of truth, long ago.

As above, incorrect.

Please reconsider, it's clear things are not as you believed them to be and much research is required to catch up to the state of things today.

I hope this helped to get this thread back to the question about assurance, if there is anything else I can clarify I am more than happy to provide so the facts are well established to make any assurances :- trustworthy

[chrisdlangton]

@ctz will there be any effort to provide assurances (perhaps on the README or in this thread) to the users, rust community, and extended cybersecurity professionals who look for assurance when doing normal verification of rustls

Or a statement to clarify that there will be no commitment to assurance and transparency?

A comment either way, with or without timeframes, would be appreciated.

[djc]

@chrisdlangton from my point of view we already have complete transparency, since all the code used to generate the webpki-roots crate is part of the repository. Feel free to propose concrete steps we could take on how to improve our "commitment to assurance and transparency" in your perception.

[chrisdlangton]

I outlined the assurance issues already, so I have provided the details you need to my best of my ability. Without knowing what exactly you need clarity of, specifically, we're not making progress

I wasn't as specific about transparency as i could have, Transparency and open source are not identical, and it's the first time ive used the word in this thread so it deserves elaboration. I used it in context to being transparent about your plans for rustls providing assurances or not. Implying silence is a tactic to allow a thread to die, be closed, and a hard problem just goes away. Which is fine that it closes, if you make a statement before it is closed of course.

The best outcome is the concerns are addressed;

1. include in the source code the steps involving mkcert, in a way that rustls can show the curation is being done in a controlled environment, Or;

2. if external API remains to be utilised, add a validation step to ensure the root certificate signatures match the signatures within CCADB (or NSS if you prefer to call it)

3. change the filter logic from subject to a verifiable attribute listed

[djc]

The steps involving mkcert are in https://github.com/rustls/webpki-roots/blob/main/build.py#L62, that seems to be pretty clear?

[chrisdlangton]

The steps involving mkcert are in https://github.com/rustls/webpki-roots/blob/main/build.py#L62, that seems to be pretty clear?

Yes, and my analysis of the above is clear also, reiterated even, what's your point in this comment?

[djc]

My point is that there is a clear chain of trust from this repo, through the source code in build.py to the mkcert.org website (accessed through HTTPS) and its source code, which retrieves certificates from the Mozilla repo (also accessed through HTTPS). Note that the repository points to the source code of the NSS library maintained by Mozilla and thus, from our point of view, has no relation to the CCADB.

There might be ways to shorten this chain of trust but so far no one has considered that valuable enough to do the work.

If you don't trust the contents of this repo, you can use the rustls-native-certs crate or the newer rustls-platform-verifier instead. So far I don't think you've made a credible case that there is anything obvious here that we should improve.

[chrisdlangton]

It's 'trust' in the sense that the 'act' of curation/filtering is verifiable to not be tampered with because the identifier used for filtering is verifiable, or the certificates we pulled from the public internet using a free service that a random person operates are checked to be certificates we know are issued by CAs we trust.

Not blind trust as you're suggesting, verifiable, so that we gain assurance, that way we have verified trust

There's some interesting elements of your response; you're discussing TLS again when it was never in question, reiterated that TLS was not in question, and TLS itself was only introduced into the conversation by you, so it's not constructive to change topic back to TLS again now when it's clear theres nothing about TLS at all that is needing to be discussed further, or even addressed in any way at all in this entire thread. It's either you've not read the thread before replying, or lacking knowledge of what was read. I can't say which.
The other interesting thing in your latest response, as you put it "nothing to do with CCADB", you're disagreeing with Mozilla here. Because the link that delineated NSS and CCADB for you. It clearly states how CCADB is the database of certificates, and that NSS uses it to produce a bundle (your link) distributed for their products. This shows readers another example of you either not reading the thread before replying, or lacking knowledge of what was read. Again, It's not clear to me which is the reason for your odd responses.

Essentially, there are clearly described gaps of verification in the chain of trust in the OP.

At the risk of sound repetitive, without some assurance provided by intentional verifiable steps added to this repo, there is no way for anyone, me, you, anyone - to gain trust. We only have disparate external elements that have trustworthiness, as you and i acknowledged, but this repo itself pulls them together without any applying any attention to providing intentional assurance, such as filtering using verifiable identifiers, or checking the results retrieved from public free and unverified sources, all of the signatures for the certificates so we are sure they are what we expected.

Which, as you seem to suggest, means that we can blind trust this certificate bundle, or not use it, because you feel it is fine to not do validation or use verifiable identifiers for filtering, right?

[djc]

From my previous comment:

There might be ways to shorten this chain of trust but so far no one has considered that valuable enough to do the work.

This is me recognizing that some of the things that you are suggesting could improve the trustworthiness of the webpki-roots crate produced from this repository. However, it is also me saying that neither I nor, apparently, any other users, consider the limitations you've raised here to be important enough to spend time to fix. And, to be fair, I've also seen no sign of you making the effort to improve the situation, beyond merely commenting that the current state is not to your satisfaction.

Because the link that delineated NSS and CCADB for you.

I'm not sure what link you are talking about. I linked to the mkcert repo which, according to GitHub search, contains zero mentions of CCADB. I also linked to the certdata.txt file on a server hosted on mozilla.org that has nss in its path, and that file also doesn't contain any mention of CCADB. As such, this project is not trusting CCADB, we're trusting the hg.mozilla.org server serving a certificate bundle as part of the NSS library.

This will be my last comment on this issue, because I feel we're not making progress here. I'd be happy to review a PR to make improvements along the lines you suggested, although that doesn't mean that we'll necessarily accept them -- that would depend on the amount of added complexity and burden of maintenance.

[chrisdlangton]

The only Mozilla link; https://wiki.mozilla.org/CA

Of course this issue is about discussion, right? after a discussion we might have an agreed path forward, are you inferring that has been determined? Could you be concise, as the maintainer, what an outsider might produce as a PR?, my profile speaks for itself, if the changes are within my skill ill happily contribute, it's open source and this is an issue to come together, not an issue to complain.

So what's the plan? Would you like a PR to introduce assurance steps?
Would you like to apply filtering on SKI instead of the subject?
Would you like a build pipeline (GitHub action) to run mkcert directly to avoid blind trust in a random public API? Or refactor to rely on a different chain of trust OR source altogether?

[djc]

Filtering on SKI instead of subject sounds like it might be reasonable. I'm not sure running mkcert in CI makes sense to me. My question there is whether we gain that much trust if we, for example, get the certificates over HTTPS directly from Mozilla's Mercurial repository over mkcert -- that cuts out a middle man but still depends on HTTPS for trust.

[chrisdlangton]

My first preference if I were to make efforts to contribute to rustls would be entirely different approach involving CCADB (like Google, Apple, Microsoft all chose over NSS,).

I'd not have a reason to disagree or claim another choice (NSS) is better than theirs (CCADB) for any new project today, including an existing project making intentional improvements (like we're discussing)

Assuming that rustls maintainer/s preference is to continue using mkcert, then avoiding a black box compute environment and instead run in an Action with full logs is all the difference in terms of verifiable trust. It means the difference when applying TLS we're just obtaining the certificates from a trusted source straight to our build server, not requesting a service to obtained the certificates from a source we trust and serve them to us we hope were unmodified, having no way to know what happened in that compute environment, we only have TLS when making a request to the API

The benefit is not TLS, the benefit is cutting out a middle man that we weren't verifying or validating what they sent us.
Having said this, just because there's TLS doesn't mean the contents are validated for us (the certificates are arbitrary body text in this TLS context), therefore even when they're retrieved from NSS or CCADB, the data is just data, it's not being verified. rustls should at least iterate each root certificate and check the signature is valid to catch any issues before users find them (just a random thought i didn't realise earlier)

[djc]

Maybe a good first step would be to dig into the difference between NSS and the CCADB to figure out which choice aligns better with our goals for this crate. Would you be interested in working on that? It seems likely to me that there would be a significant amount of overlap, so I think reviewing the differences would be instructive.

[cpu]

As of #41 we're building the trust anchor set directly from CCADB data.

Some improvements of note:

• We're no longer performing any filtering based on subject but instead filter records from the IncludedCACertificateReportPEMCSV report that have a value of "Websites" in the "Trust Bits" column, and either omit a "Distrust for TLS After Date" value, or have a value that is beyond the date of generation.

• We've additionally chosen to perform path-building when connecting to the CCADB report endpoint using a single vendored trust anchor. Pinning a leaf certificate or intermediate wouldn't be appropriate for a service operated independently from this software.

• There is no longer any hand curated data in the crate (e.g. excluded roots, imposed name constraints) maintained outside of the upstream data source.

• The code that processes the CSV report is run in CI and fails on any produced deltas.

rustls should at least iterate each root certificate and check the signature is valid to catch any issues before users find them (just a random thought i didn't realise earlier)

I don't think this is a valuable addition. A trust anchor is defacto trusted and the signature on the certificate is not useful. From the perspective of the webpki crate the X.509 certificate is only a container for SPKI and subject. The signature is irrelevant and discarded when representing the trust anchor in webpki. This is the same reason that when the SHA1 deprecation occurred it did not necessitate regeneration of root certificates that were self-signed using a SHA1 algorithm. Similarly, many trust anchor stores (e.g. Android, webpki) don't maintain the not-before/not-after of roots, or enforce validity.

I believe we can consider this issue resolved in the absence of any additional concrete suggestions for improvements.

Thanks!