Adding a scenario for revoking signatures

[sudo-bmitch]

Signing means different things to different people. To some, it only means this artifact was produced by an entity. But to others it conveys that the signer has attested some qualities to the artifact, typically that the artifact is secure. Since the security of an artifact is mutable with the discovery of new vulnerabilities, we should also have a way to modify our trust in that artifact over time.

As a simple example, if a major CVE is discovered in an image, it should be possible for the signer to revoke that previous trust and force notary users to upgrade to a secure version of that image.

Signed-off-by: Brandon Mitchell git@bmitch.net

[SteveLasker]

I think what your suggesting is this:
An author of content identifies they have identified what they perceive as an issue with that content. net-monitor:v1@sha256aaa After the fact, as they woudln't knowingly publish something bad.
They publish net-monitor:v1.1@sha256bbb, which they may also update tag :v1 to point to the new digest.
While Wabbit Networks decided :v1@sha256aaa is invalid, ACME Rockets found they can't run :1.1 as it has some incompatibility in their environment, and won't run.
To ACME Rockets, digest aaa works, and digest bbb doesn't, so they don't actually want aaa to be considered invalid.

But, what I do think could solve your problem is a forwarding update.
First, let's assume wabbit networks publishes the update to :v1 - net-monitor:v1@sha256bbb
If ACME is pulling on tag, they'll see the update, and can determine if they want to move forward, as they hopefully have a gated-mirror in their environment to cache the artifacts they depend upon in their environment.

But, for those that may deploy on digests: net-monitor@sha256:aaa what if a registry could identify a forwarding:
Let's say you could add a linked artifact (signed of course), that says net-monitor@sha256:aaa has been updated to net-monitor@sha256:bbb. Kind of like a parts replacement in the manufacturing industry.

What I'm mostly getting at is:

• revocation should be when a signing key was truly stollen and used for evil purposes. Anything ever signed with that key should be considered invalid.
• we should support a forwarding concept, where the content is still valid (not hacked), just has something that may not be desirable, and thus, there's a new version, that doesn't have that problem, but likely has a new problem we'll discover next month.

[sudo-bmitch]

@SteveLasker Capturing some of the comments from our weekly meeting here:

To ACME Rockets, digest aaa works, and digest bbb doesn't, so they don't actually want aaa to be considered invalid.

When companies pull ingest external artifacts into their own registry, they would check the upstream signature at the time of ingesting, and then resign the artifact with their company signature. From that point forward, internal verification would happen against the company signature rather than the upstream, allowing the company to have their own policy of when they consider revoking the signature on an artifact, separate from the upstream policy.

But, for those that may deploy on digests: net-monitor@sha256:aaa what if a registry could identify a forwarding:
Let's say you could add a linked artifact (signed of course), that says net-monitor@sha256:aaa has been updated to net-monitor@sha256:bbb. Kind of like a parts replacement in the manufacturing industry.

I think that would expand the scope of notary more than we want. I'd like to keep this one simple, just at revoking a single signature without impacting everything else the signer has also signed.

What I'm mostly getting at is:

* revocation should be when a signing key was truly stollen and used for evil purposes. Anything ever signed with that key should be considered invalid.

This is a bit of a nuclear option. I think there's a value in being able to take back an accidental signature on an artifact when you sign the wrong thing without invoking a full key revocation and resigning of every other asset you previously signed.

* we should support a forwarding concept, where the content is still valid (not hacked), just has something that may not be desirable, and thus, there's a new version, that doesn't have that problem, but likely has a new problem we'll discover next month.

Allowing upstream to impose their opinion on what downstream users should use in place of another digest can be very problematic, especially if those users expect an immutable response to pulling by digest. The best option to me is telling the user the digest they are pulling is not trusted by the signer, even if it used to be, and leaving the decision on what to do (run the image anyway, rollback, upgrade) up to the downstream user.

[SteveLasker]

what if a registry could identify a forwarding:
I think that would expand the scope of notary more than we want. I'd like to keep this one simple,

I'm all for KISS. Just trying to figure out the problem/requirement, then focus a design that matches. While we should absolutely support deleting anything in a registry, including signatures, I wonder if we really need a revocation. If a clients can "sync" with another registry, and they pulled the wabbit networks signature, which was then deleted, they should be able to track the deletes, and cascade the deletes to the mirrored replica. Although, I fully recognize we don't have a tombstoning solution in registries today.

This is a bit of a nuclear option.

If we're tying revocation of the key for the deletion of a signature, I agree. Just not sure how to think about revoking a signature, vs. delete.

I like to model solutions to problems around other solutions to similar problems, even if in a completely different industry. Building occasionally connected software, I modeled around store and forward water and food supply systems. You always have a buffer, when (not if) something fails.

For this revocation scenario, which may not really be a revocation, rather a hint that there's a newer version, I'm just thinking there's something about the replacement parts concept. You can use this part # if you really want it, and have it. But, here's the newer part # that you may really want.

Allowing upstream to impose their opinion on what downstream users should use in place of another digest can be very problematic, especially if those users expect an immutable response to pulling by digest. The best option to me is telling the user the digest they are pulling is not trusted by the signer, even if it used to be, and leaving the decision on what to do (run the image anyway, rollback, upgrade) up to the downstream user.

So, just riffing on this a bit. If you revoke the signature, isn't that an even stronger opinion? It's basically saying, I built this thing, at a point and time. And now I want to revoke that (oops). Rather than revoke, what if there was a forwarding, with a possible severity? You can either say (critical forward, informational forward, or some other grammar). It then expresses the intent with more granularity. The issue is I don't know how to imply "trust". We must assume that all software has some amount of vulnerabilities. Some known, some unknown until a later period of time. Since the only non-vulnerable software is an empty container, the level of vulnerabilities must be left to the owner to determine how it impacts them.
Of the long list of vulnerabilities, how many of them really fall in that critical category? It's critical we support the critical vulnerability issues, but how do we support the majority of issues that are more opinionated for the consumer?

[sudo-bmitch]

So, just riffing on this a bit. If you revoke the signature, isn't that an even stronger opinion? It's basically saying, I built this thing, at a point and time. And now I want to revoke that (oops). Rather than revoke, what if there was a forwarding, with a possible severity? You can either say (critical forward, informational forward, or some other grammar). It then expresses the intent with more granularity.

Revoking a signature just says that the signer no longer trusts a specific image. We're leaving it up to the end user where they go from there. Adding a forwarding to another image breaks the trust that when I pull by digest that the object pulled will never change. It also feels like an additional attack vector to me, an attacker finds a way to create a forwarding policy to a vulnerable image that goes unnoticed since the original digest is unchanged and all the image references look the same.

The issue is I don't know how to imply "trust". We must assume that all software has some amount of vulnerabilities. Some known, some unknown until a later period of time. Since the only non-vulnerable software is an empty container, the level of vulnerabilities must be left to the owner to determine how it impacts them.
Of the long list of vulnerabilities, how many of them really fall in that critical category? It's critical we support the critical vulnerability issues, but how do we support the majority of issues that are more opinionated for the consumer?

Trust is always going to be between the signer and the verifier. Notary v2 doesn't need to define "trust" but it should provide the signer a way to provide something that can be trusted. Once we provide the tools, it's up to the signer to decide what situation would warrant revoking a signature, if ever.

To add a recent situation, I heard that Solar Winds still hasn't revoked their signing keys for malicious code they signed and shipped. Because if they do so, all of the other good code that they've shipped and that's currently running would also end up being revoked. So they have to get all of the old releases resigned, and get clients to update things on their side, to be able to revoke those keys without causing a disruption.

Why not just delete the signature on the registry? First, that implies the registry can be trusted, and if we started with that assumption, Notary wouldn't be needed. With the ability to clone repos, I could see deletes not getting cloned. Why not? What's to say the signature you don't see upstream wasn't locally created in the clone to add a local signature, if ACME clones Wabbit's images locally, the ACME signature won't be seen upstream. What if there's a DoS attack or poor network connectivity to upstream that causes updates to fail? But my biggest concern is what if there's a poorly secured local mirror? With Notary, if we can trust the signature, we have a cryptographic assurance the content can be trusted even if the registry is hacked, we don't even need HTTPS. But that only works if attackers can't easily replay stale data, and it would be useful if Notary could prevent that, similar to what we want with tag signing, but applied to not just tags but also digests.

[dlorenc]

Here's the reference to the post on the SolarWinds revocation issue: https://www.solarwinds.com/sa-overview/new-digital-certificate. Looks like it might have been revoked as of March 8th, I don't see any updates in that post though.

[mnm678]

This is a great addition, I like the separation between revoking a key and revoking a signature.

[SteveLasker]

Some good conversation came out of this for potential new ideas, but these look like good scenarios to add.
LGTM

[SteveLasker]

Adding a forwarding to another image breaks the trust that when I pull by digest that the object pulled will never change.

The forwarding info is just info. It says, this artifact has been replaced. Whether the original is considered vulernable is a different dimension a scanning solution would determine. There's nothing stating it can't be used, just like finding the original part. If the original part is still considered safe and is found, it can be used. If you're looking for thew newer version of "the part", here's a pointer to it.

It also feels like an additional attack vector to me, an attacker finds a way to create a forwarding policy to a vulnerable image that goes unnoticed since the original digest is unchanged and all the image references look the same.

The forwarding info would need to be signed, to be just as trusted as all other content.

BTW, these are just ideas for how we might cope with some of the "versioning" scenarios as I'm suggesting the majority of content isn't versioned in a serial stream.

[mnm678]

We revisited this in the 6/11 meeting, and it looks like it should be ready to merge, especially in light of the implementation discussions around revoking signatures. @justincormack or @SteveLasker would you be willing to merge?

[SteveLasker]

LGTM
Would like to see LGTM from @gokarnm, @shizhMSFT, @NiazFK as well

[sudo-bmitch]

I've rebased and squashed so this should be ready to merge again. @gokarnm, @shizhMSFT, and @NiazFK, PTAL.

[gokarnm]

Adding in feedback based in discussions we had in Rescinding signatures

The user may receive an error or warning based on their configuration for handling revocations. How about - Consumers that pulled the vulnerable artifact that verify the signature will receive an error or warning indicating the signer no longer trusts the artifact, based on the policy the Consumer defines.

[sudo-bmitch]

Good point, but I guess it depends on where the boundaries are in the scenario. Notary itself should report an error and only give a pass/fail. Tooling that checks Notary for the admission controller, container runtime, etc, may interpret that error as a warning or block the action.

[hallyn]

Would it suffice to say that whatever API lists the current valid tags for an image not list the revoked ones? But not try to 'reject' using an image which has been revoked? (Sorry, I'm not sure where the current detailed API list lives)

[sudo-bmitch]

@hallyn since the tag could have been previously pulled, I don't think a registry side API to remove it from the list would suffice. That would also require the tag list to be signed to avoid a malicious actor pushing a bad tag list.

@gokarnm I've updated the PR to define this as a "notification" which the client could decide different ways to handle.

[gokarnm]

I implication is not clear to me, I think the revocation status is associated with the artifact and not a signature. We discussed it here(second point).

[sudo-bmitch]

I don't think the hackmd is part of our scenarios or design yet, that's still a WIP. I'd recommend revoking the signature rather than the full artifact since you may have multiple signatures that mean different things. Revoking a signature that indicates the artifact was approved by sub-team-X for deployment in their cluster shouldn't also revoke the signature that says dev-team-y made this.

[gokarnm]

I agree, it makes sense to associate revocations to signature as the scenario you mentioned and also being a potential attack vector in #49 (comment).

[sudo-bmitch]

@gokarnm I've updated with feedback from our discussion on Friday. PTAL.

The only change I didn't make was to revoke trust in an entire artifact rather than only the signature on the artifact. Thinking more about it, I could see that as being a potential DoS attack, where a malicious actor gets access to register new signatures using an arbitrary (likely untrusted) key, and then revokes that trust. If that revoked the entire artifact's trust, it would deny clients the ability to use a signed artifact verified by a key they do trust.

[SteveLasker]

@gokarnm, @NiazFK, @sudo-bmitch, @mnm678, can you provide your review if this is ready for merge?

[sudo-bmitch]

@gokarnm, @NiazFK, @sudo-bmitch, @mnm678, can you provide your review if this is ready for merge?

LGTM :)

[shizhMSFT]

LGTM

[mnm678]

LGTM

[gokarnm]

LGTM!