Add verification algorithm to specification.

[msporny]

This PR attempts to address issue #1337 by adding a verification algorithm to the specification. The PR attempts to do this in a way that defers to the securing specifications on verifying the securing mechanism and defers to the rest of the normative statements in the specification (instead of re-stating them in the algorithm).

/cc @jyasskin (since he raised this issue as a major item of concern in his review)

---

Preview | Diff

[TallTed]

Editorial tweaks

[jyasskin]

This should call a particular algorithm in the Securing section and/or specifications. Since you have multiple ways to secure a VC, I suspect that [[#securing-verifiable-credentials]] will wind up defining an interface, which the various Securing specs will implement.

[msporny]

Yes, we can call out specific algorithms, like this (generalized) one:

https://w3c.github.io/vc-data-integrity/#verify-proof

or this (specific) one:

https://w3c.github.io/vc-di-eddsa/#verify-proof-eddsa-rdfc-2022

What I'm trying to avoid is limit future cryptosuites from being used by being too specific here. Your suggestion to define an interface that various securing specs implement sounds like a good direction. I'll add that to this PR.

[jyasskin]

Yep. A couple things I notice when comparing this call to those definitions are that:

• this call doesn't provide the proof options that https://www.w3.org/TR/vc-data-integrity/#verify-proof expects,
• https://w3c.github.io/vc-di-eddsa/#verify-proof-eddsa-rdfc-2022 doesn't pass them through,
• neither of those expects the media type that this call does pass, and
• https://www.w3.org/TR/vc-data-integrity/#verify-proof expects its input to have been already parsed to something that contains object fields, but this algorithm doesn't do that.

So there may be two dispatch layers here, possibly expressed as registries: first based on the media type to parse, and then based on the cryptosuite to verify. But I'm sure that omits some important details, and I'll look forward to seeing what you write.

[jyasskin]

We have a central expression of this sentiment at https://infra.spec.whatwg.org/#algorithm-conformance. If you add a dependency on [[Infra]], you won't have to say it for every algorithm.

[msporny]

We used [[Infra]] in the DID specification and while it worked, some in the WG might not want to pull it in just for this statement. I note you have other suggestions below that depend on Infra. Would it be acceptable if I don't pull in Infra yet (but use the same language for now). We'll pull in Infra if it becomes clear that we need to do so (and I don't think we've hit that point just yet).

[jyasskin]

Infra's definitely optional.

[msporny]

Fixed in 2cd54c1.

[jyasskin]

Does this need to be a byte stream rather than a byte sequence? Streams are sometimes important so you can react while bytes are still coming off the wire, but they're more difficult to specify. If you really do need to use streams, https://encoding.spec.whatwg.org/#concept-stream has a plausible interface.

[msporny]

Yes, this should probably say "byte sequence" instead, I'll update the language to that.

[msporny]

Fixed in 810c757.

[jyasskin]

You can use https://infra.spec.whatwg.org/#structs to formalize this a bit, but the "object" terminology is clear enough.

[msporny]

Got it, thanks (the case for using Infra is building) :)

[msporny]

I decided against using struct for now as the term "object" is probably more familiar to most developers that might read the spec (note: not browser engine developers).

[jyasskin]

Could you describe the type of each field?

[msporny]

Yes, will do.

[msporny]

Done in 03a5215.

[jyasskin]

This is worryingly imprecise. It raises questions like:

• What does it mean to "express context information"? (https://w3c.github.io/vc-data-model/#dfn-context)
• What does it mean to "express an identifier that others are expected to use when expressing statements about a specific thing identified by that identifier"? (https://w3c.github.io/vc-data-model/#identifiers)
• How can a credential or presentation "be associated with" additional more narrow types if it doesn't specify them? (https://w3c.github.io/vc-data-model/#types)
• credentialSubject doesn't say how to identify the "subjects" of the verifiable credential. #1341
• In a Presentation, how do you check if the verifiableCredential property holds "verifiable credential graphs in a cryptographically verifiable format"? Is the implementation running this algorithm supposed to verify them, and declare the document non-conforming if it can't? (https://w3c.github.io/vc-data-model/#defn-verifiableCredential)
• How can a verifier check that a credentialSchema "provide[s] verifiers with enough information to determine if the provided data conforms to the provided schema(s)."? Does it declare the document non-conforming if that particular verifier can't verify it? (https://w3c.github.io/vc-data-model/#defn-credentialSchema)

This might not be the whole set of questions; I didn't get through all of the uses of "MUST", and there might be other "relevant normative statements" that don't include "MUST".

[msporny]

We've written the spec such that all relevant normative statements include "MUST".

Your concern here seems to be less about the algorithm and more about normative language elsewhere in the specification, correct? So, what we could do is raise a new issue with your list above (to start). Would that work for you?

IOW, would you be fine w/ us dealing w/ the other normative statements in another issue and commenting on the changes you'd like to be made to the text above (presuming we'll address that other issue in time)?

[jyasskin]

Yes, and to be clear, I'm always happy to break up work into multiple PRs and let some of it land before all of it's done.

I'm worried about using the declarative style of "A conforming document is any concrete expression of the data model that complies with the normative statements in this specification," rather than the algorithmic style of "follow these steps to check if a document is conforming." Based on the web platform's experience with HTML, I think the declarative style is more likely to lead to differences in implementation and that it'll be harder to check for privacy or security problems. However, since this algorithm makes it optional to take any action based on non-conformance, I'm not firmly opposed to the declarative style in this particular call.

[msporny]

Yes, the first draft of this PR (on my local machine, which wasn't uploaded) used the algorithmic style, which duplicated the normative statements throughout the document. As you know, that leads to people complaining about duplicating normative statements AND often leads to sync problems between the two duplicated normative statements throughout the specification. We could split the difference and say something like: "Ensure that the foo field conforms to the normative statements in Section 2.5 Foo, and if not, raise an error." -- it points out more clearly what fields are expected to be checked during verification w/o duplicating normative text.

If you're ok w/ that (I know it's not exactly what you were asking for), I can take a shot at some language in that direction.

[jyasskin]

To be clear, you don't need to make me happy with the declarative style in places like this where enforcement is optional. (So, yes, I'm ok with your suggestion.) But I do think it doesn't serve you well, for 2 reasons, both based on the fact that you're fundamentally trying to constrain two different pieces of software with this language.

1. Spec authors tend to only think about one audience at a time, so language with two audiences (here, the issuer and verifier) tends to lose its applicability to one of the audiences over time. Here, since the text is mostly directed to the issuers, it has lost the detail that verifiers would need. Some specs, like https://httpwg.org/specs/rfc9110.html#rfc.section.4.2.1, have successfully interleaved their two audiences in the same text, but it's unusual.
2. Sometimes you really want to put different constraints on issuers vs verifiers. For example, in https://github.com/w3c/vc-data-model/pull/1349/files#r1391380302, I think it's very reasonable to say that the issuer "MUST identify the issuer in a globally unambiguous way" even though it's not reasonable for a verifier to check that. It'd be good to use a specification style that allows you to separate the two. (That said, I think it's also ok to only constrain the consumer, as web specs usually do.)

[msporny]

I have made a first attempt at expanding the algorithm to be more precise here:

6064efa#diff-0eb547304658805aad788d320f10bf1f292797b5e6d745a3bf617584da017051R4437-R4516

Going further than this would require a significant restructuring of the specification that I'm not sure would go over well with the WG (given that we're trying to get into CR by the end of the year).

I hear you wrt. declarative vs. imperative style. At present, we don't seem to be getting signals from the current crop of implementers (20+) that this has been an issue for them... but, perhaps we won't see that issue until the implementation pool gets larger than that.

Most people are just using a JSON Schema to do the object validation, and it seems to be working just fine for them. These algorithms are directed solely at verifiers, so my expectation is that most of them will be able to figure it out any shaky language (as it relates to them). That said, I'm not arguing strongly one way or the other. For example, in the future, we could replace each section with algorithms for verifying values for each property (but that might be overkill given that we're not getting loud complaints from implementers not knowing what a particular field is and isn't supposed to contain).

In any case, I'd like to try to land this PR (to get the logic down, roughly) and then see if there are things that we can do better in future revisions.

[jyasskin]

"the information that was secured" should be explicitly returned from the Securing method. With the wording that's here now, it's not clear if this is a copy of the input document or a subset that the Securing method actually checked.

[msporny]

Your first sentence is the intent, I'll update the PR to make that more clear.

[msporny]

Fixed in d45bf08.

[andresuribe87]

Where are appropriate warning and errors defined? In the sub-specs?

[msporny]

Yes, excellent question. I was trying to avoid defining these warnings/errors (and leave it up to implementations), but I expect that there are some common ones (such as MALFORMED_VALUE) that we could define in this specification. As @iherman mentioned, we do that sort of thing already in the Data Integrity specification:

https://w3c.github.io/vc-data-integrity/#processing-errors

I'll try to add a section for some basic ones in the next pass. This is probably going to trigger larger conversations around where the error definitions should go (the initial proposal will be: In the Verifiable Credential vocabulary).

[msporny]

Added in dc94b92.

@iherman, I chose to use the same terminology as in RFC9457 "Problem Details", and will align the DI specs w/ this new terminology when those documents enter CR.

[iherman]

The issue was discussed in a meeting on 2023-11-14

• no resolutions were taken

View the transcript

1.8. Add verification algorithm to specification. (pr vc-data-model#1338)

See github pull request vc-data-model#1338.

Brent Zundel: another before CR pr related to Jeffrey's review.
… I think the request for changes has been addressed already.

Manu Sporny: I would like the group to pay particular attention to this PR. It is the first time we are adding an algorithm with a set of steps into the spec.
… that is also dependent on the external securing specification.
… I believe we have a bead on the right levels of abstraction.
… this is also forcing us to describe what a warning or error object looks like (I've called these anomalies).
… looking for bikeshedding in the PR on this naming.

Joe Andrieu: +1, I'll take a look.

Manu Sporny: a W3C member has indicated formal objection if this was not defined this time around.

Brent Zundel: agreed that this addresses what was requested, and that folks should review. This has been open for six days, so review period is close to being up.

Ivan Herman: in the PR, there are a number of places where there are discussions going on and it is marked as outdated - not sure what I need to be looking at.
… can that be improved?

Manu Sporny: I did not want to resolve statements until WG members had a chance to look through them.
… whenever reviewing, always look at "files changed" to do your review, not the conversation tab.

Brent Zundel: just to note, the "outdated" label is specific to the code snippet, which has been changed. The conversation should still be available, and current code.
… is in the "Files changed" tab.

[jyasskin]

I'm still waiting to review the securing algorithm interface from #1338 (comment), but the rest looks good.

[jyasskin]

This "as" looks leftover from some change:

Suggested change

	mechanism as, <var>verified</var>.<var>mediaType</var> to the corresponding
	mechanism, <var>verified</var>.<var>mediaType</var> to the corresponding

[msporny]

This got fixed in a commit that @TallTed provided.

[jyasskin]

I noticed that code isn't in https://datatracker.ietf.org/doc/html/rfc9457#name-table-of-contents. You should probably identify it as an extension member.

[msporny]

Right, good catch, will do. Does the following change work for you?

Suggested change

	<dt>code</dt>
	<dd>
	The `code` <a>property</a> is OPTIONAL. If present, its value MUST be an integer
	that identifies the type of the anomaly. Integer codes are useful in systems
	that only provide integer return values.
	<dt>code</dt>
	<dd>
	The `code` <a>property</a> is an OPTIONAL
	<a data-cite="RFC9457#name-extension-members">
	RFC9457 extension member</a> . If present, its value MUST be
	an integer that identifies the type of the anomaly. Integer codes
	are useful in systems that only provide integer return values.

[jyasskin]

Yep, looks good.

[msporny]

Applied in dd7c36f.

[jyasskin]

Oh, based on the processing model described in #1349 (comment), some of these conformance checks need to happen before the securing mechanism gets called.

For an RDF-based securing mechanism, what's described here has the securing mechanism parse the document and then re-serialize part of it, and that serialized version is what has "use id and no other shorthand for @id" enforced. But that tests the securing mechanism's serialization code, not the input VC.

No need to fix this before merging this initial algorithm.

[msporny]

Oh, based on the processing model described in #1349 (comment), some of these conformance checks need to happen before the securing mechanism gets called.

Yes, there is debate on when the appropriate time to do this is (before or after you've checked the securing mechanism). AFAIK, most implementations do the conforming document checks before checking the signature (via the use of JSON Schema or hard-coded checks). It's conceivable that doing the check after (based only on the secured information) is another valid way to do this as well. There are good arguments for/against each approach; I'm on the fence on which one is the better approach, so strong opinions on the right thing to do would be welcome here.

For an RDF-based securing mechanism, what's described here has the securing mechanism parse the document and then re-serialize part of it, and that serialized version is what has "use id and no other shorthand for @id" enforced. But that tests the securing mechanism's serialization code, not the input VC.

Yes, but that's not the case of JOSE-based mechanisms. The thing that matters here is that the application is working w/ only the secured information after the securing mechanism has been checked, and the information it's working with is a conforming document. The order in which those operations happen is, arguably, not as important as the end result.

No need to fix this before merging this initial algorithm.

Thanks for the flexibility. We will, however, need to have a very good basis for why we do one before the other, IMHO... or, come to the conclusion that the order that the checks are done is up to the implementation, as long as the end result is that you're working with the information that was successfully secured and verified, and that information is a conforming document.

[iherman]

Something should be said, in my opinion, on who defines these URLs and codes. This may have to be specified in the section §4.10 on Securing Mechanisms, and may say that securing mechanisms are expected to define the possible type URL-s and codes for the anomaly descriptions.

I can see this is done in section §4.8 of the data integrity spec, so that is fine. But I have not found anything about this in the jose cose spec.

---

There is a terminology bike-shedding, though. This part of the VCDM spec refers to "anomalies" and their descriptions, while the DI spec (in the aforementioned section) talks about processing errors. This should be harmonized at some point...

[msporny]

Yes, agree that we should align the language/sections.

[iherman]

I must admit this order looks a bit unnatural to me. If I consider a JSON-LD + DI combination, as an implementor I would start by checking whether the document I receive is conforming; if it isn't, I would probably choose to stop processing it further. After all, it is only if the document is conforming that I can safely look for the other properties like proof, verifiableCredential, etc., that are necessary to determine whether the signature is o.k. Having this conformance check done in §3.1 seems to be in the wrong order.

I realize that the situation is different if the incoming document is, say, using JOSE. I wonder whether this does not mean that there should be actually two slightly different algorithms, in terms of logical order, depending on whether the VC being processed has an external proof or an embedded proof. The current description reflects the external case, whereas my description above the embedded one.

[msporny]

Yes, see commentary above: https://github.com/w3c/vc-data-model/pull/1338/files#r1395735708

External signatures force you to process the digital signature before you process the data. At the very least, base-decoding the payload. An attacker can force you to retrieve keys and do all sorts of cryptographic operations before you check to see if the document is conforming.

Embedded signatures allow you to check for a conforming document first, and then follow up with the cryptographic verification (which might fail).

In both cases, systems need to be able to detect these sorts of attacks and mitigate them, you can perform resource exhaustion attacks no matter which approach you take.

So, while we can have two algorithms to do this, the end result is (arguably) more or less the same. We can have some branching logic in here, but honestly, I'm not sure it matters. Can we just tell implementers that they can do this in whatever order they wish because both checks have to pass for verification to succeed?

[iherman]

This comment is purely editorial, and may be postponed for later.

I wonder whether we should not explicitly rely on the infra standard by putting in references to terms like "byte sequence", "string", use the concept of "list" instead of arrays, and "maps" instead of objects. This may be a bit of a pain because probably we should then use someting like verified["warnings"] instead of verified.warnings, but it would avoid starting discussions about, say, Unicode code points for the incoming documents (because that is amply detailed in the infra standard for strings).

Just a thought.

[msporny]

Yeah, I'm delaying pulling infra in for as long as possible, it caused a lot of problems (people had to learn Infra) when we started to use it in DID Core that I'd like to avoid. If people can't read "array" and "object" and know what we're talking about, then I don't think Infra is going to help those implementers. :)

In any case, I hear you, we can pull Infra in when somebody starts demanding that our algorithms have that level of precision, but honestly, these algorithms are not that complex and don't need to be typed so strictly.

If we do decide to pull in infra, I'd like to do it in a dedicated PR.

[msporny]

Normative, multiple reviews, changes requested and made, remaining concerns tracked as issues, no remaining objections, merging.