Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
We should decide (as soon as possible) whether we will be supporting exactly one or more than one key expression format in the DID specification.
Options include: JWK, COSE Key, LD Keys, PEM, or expression of key-like data structures (for things like
For us to have a well-informed discussion on this topic, it would be useful to have an inventory of the different kinds of public keys that we will want to be able represent. I'm aware of (1) RSA public keys, which need a mantissa and an exponent, (2) elliptic curve keys that need an X and Y pair, and (3) elliptic curve keys that only require an X value (all of which have standard JWK representations).
Are there different kinds of public keys we care about that aren't of any of the kinds 1-3 above? If so, what values do their representations contain? (For extra credit, please provide specification references so that we can all get up to speed them :-) .)
No, X and Y is not correct. What is needed is the finite field form (in the case of bitcoin's secp256k1 it is Galois y² = x³ + 7), the finite field's modulus (usually called p, which in secp25k1 is p = 2^256 – 2^32 – 977 which makes this a Kobliz Galois field), and a generator base point (usually called g, which in secp256k1 is TWO numbers, 0279BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798, or in “compressed form” 040x79BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798483ADA7726A3C4655DA4FBFC0E1108A8FD17B448A68554199C47D08FFB10D4B8).
25519 is quite a bit more complicated than that — it is a Montgomery curve in one form EdDSA and a twisted Edwards in other form Ed25519. Not all public keys are even allowed — there are 11 parameters that define 25519! https://medium.com/@LongHashDot/the-parameters-of-ed25519-fd7da1c0e8d4 And this is an IETF standard. Unfortunately, 25519 has problems with multisig, so there is a variation called Ristretto that isn't compatible with the standard though technically most of the parameters are the same. https://ristretto.group/why_ristretto.html
It gets much worse with signatures, where there can be multiple signature approaches for the same public/private key pair.
I don't consider myself a cryptographer any longer — talk to an active practitioner, but saying they can be defined easily with an x/y pair worries me in its simplification.
— Christopher Allen
You're of course right, Chris, that the underlying representations and parameter sets can be wickedly complicated. And that the two values used for secp256k1 do not represent a curve point, etc. (As another example of underlying complexity, efficient representations of RSA private keys can use 6 or more values, per https://tools.ietf.org/html/rfc7518#section-6.3.2 ).
And you're of course right that multiple signature algorithms can be used with a given key representation - which is why we also want to carry the algorithm identifier with the key.
Anyway, as a practical matter, I'm trying to have us inventory the kinds of public key representations that we're likely to need. As I'd written to Manu privately, if new kinds of representations are needed, I personally commit to quickly creating specifications to standardize their JWK representations and shepherding them through the IETF process - just as I created https://tools.ietf.org/html/draft-ietf-cose-webauthn-algorithms-01 to standardize identifiers needed for W3C WebAuthn (and for DIDs!). In my experience, if a W3C working group needs something from the IETF, the IETF is more than happy to oblige.
I have an odd emerging one for my wishlist (@selfissued, don't take this as a request for your IETF list yet, as it is only emerging), which is colloquially known as secQ. It basically is special mirror curve to secp256k1 that allows for certain kinds of advanced proofs.
See BlockchainCommons/secp256k1-schnorrsig#1 (comment) for an overview, and for the kinds of advance proofs it can do: https://electriccoin.co/wp-content/uploads/2019/09/Halo.pdf, the authors of which say that secQ may be a better approach for the technique in the curve for their paper at https://twitter.com/ebfull/status/1171531807105282049?s=20
It is emerging things like Halo, zk-snarks, etc. that is part of why I have been urging this community to use the term "proofs" rather than "signatures" as a classic digital signature is only one kind of proof.
-- Christopher Allen
Yes it would be nice to resolve this long-standing topic. Some additional things to consider:
I do think that's useful. But I think we'll find (and this thread thus far has highlighted) that the real answer here is "We don't know".
Whatever we pick today may change tomorrow. We don't want to throw up any undesirable roadblocks that require everyone to go through a heavy-weight standardization process in order to put a key into their DID Document. While it is admirable to see an individual commit to helping ensure any newly desired key format will become a standard via IETF, we should not rest our entire strategy for key extensibility on a single individual or group; rather we want to support innovation and organic adoption and consensus at the edges.
This is precisely one of the reasons we've chosen a generalized approach for the data model that supports decentralized extensibility. We have a graph-based data model that can be understood in the abstract without a specific syntax -- and it can be independently extended using standards such as JSON-LD, without asking for permission. We must, therefore, ensure that whatever key representation mechanism we use does not break from that model.
Having "one way" to do things is important, but not as important as extensibility. Taken together, it is most important to have one way of doing extensibility -- which we already have today in the spec through Linked Data. We want to give people space to innovate and continuously evolve -- and let further standardization happen when there is popular uptake.
So let's be clear that our goal (here with keys and elsewhere) is not to make a perfect, beautiful spec that has everything in it anyone might ever need, and if not, they can just ask a kind person who has promised to help. The goal is to get interop where we have consensus today and enable people to continue to innovate independently, within an extensibility framework that can lead to future standardization if desirable. This necessarily means adopting mechanisms that enable decentralized innovation. This is also in the spirit behind DIDs: recognizing independent existence.
So, if we need "one way" to represent keys, let's fit it into our existing model. This is why there is currently support and examples in the spec for LD keys: they fit into this model. While there is an LD key registry, a key needn't be on that registry to be used or expressed. You can independently define your own key terms and
We should also keep in mind that this extensibility framework helps when interop on specific key formats largely happens at the application level in a way we can't necessarily easily influence (where CBOR/COSE, etc. formats may be used and we want to include these keys in a DID Document). This fact also hints that it may be the case that it's not worth the trouble to convert specific key types to "one true format". It may be less trouble to just represent them in their common forms, using this extensibility framework, as best we can in a way that naturally fits with the particular application/platform that they are likely bound to (or should be bound to) anyway. This is the approach that was taken with
My preference is to have official support for
Here is an example DID that contains a bunch of different key types:
On a related note, I have used tools like https://github.com/EternalDeiwos/keyto#usage
To automatically convert keys of formats I can't use (with JOSE for example) to formats I can use.
One of my least favorite parts of converting keys before using them is that the publicKey
or GPG thumbprint or openssl fingerprint...
When handling a credential or ciphertext, its very helpful if the key identifier is well defined according to an RFC... key formats that don't have a standard thumbprint should be discouraged from inclusion in the standard context IMO.
I'd love to see some SHOULD language around key identifiers that encourages something like this:
Note the reuse of the standard
I would hope any key where a thumbprint/fingerprint is common would leverage that identifier to make lookups in the DID Document easier.
Even if we successfully put together an exhaustive list all key formats being used in all DID methods today (and that would be very hard), this list would still be rigid to the use of future key formats, and supporting those new formats would involve creating a new version of the spec every time.
Using something like
We need a more flexible way of describing the key format. I second @dlongley's proposition to leverage the JSON-LD structure of the DID Document, and embed the format of the key in the key object itself using a
Thanks @dlongley etc. for the useful thoughts on the kinds of extensibility that we want to have. I 100% support the goal of enabling new key representations and algorithms to be added by anyone when needed.
JWKs do enable extension by anyone through use of the IANA registries "JSON Web Key Types", "JSON Web Key Elliptic Curve", and "JSON Web Key Parameters" - all at https://www.iana.org/assignments/jose/jose.xhtml . Per the registration instructions at https://tools.ietf.org/html/rfc7517#section-8, any specification can register values. It doesn't have to be an IETF specification, or indeed, even a specification in a standards body. (For example, the W3C Web Crypto API took advantage of this flexibility by registering algorithm and key values at https://www.w3.org/TR/WebCryptoAPI/#iana-section ).
So please don't take my volunteering to be a proactive spec writer as we need new representations and algorithms as a sign that we would be dependent upon one individual or one organization. Anyone can do this at any time - by design.
As a practical engineering matter, our implementations will be simpler and more interoperable if we choose a standard key representation for key types that we already know about. There are dozens of JWK implementations in every conceivable language. For a very partial list, see https://openid.net/developers/jwt/ . And as I discussed with Manu offline, JWK (and JWS, JWE, and JWT) were intentionally designed to only require JSON representations, bringing us into the modern programming world. That way there would be no requirement for implementations to also support other formats such as ASN.1, CBOR, etc.
My goal here is to keep simple things simple. Developers will thank us for years if we do.
As stated above, I'm onboard with the argument that having a single way to do things where it will help with interop. But when we're talking about key formats, the main point of interop is at the application level. What formats are applications expecting? Will choosing "one true format" actually help with interop? Note that, over in our corner here, we do not have control over the rest of the crypto ecosystem. We shouldn't pretend that if we adopt JWK as the "one true format" for keys in DID Documents that it will mean that all of the various applications and platforms out that don't support JWK will suddenly add support. In fact, as we've mentioned above, some new applications/platforms are adopting COSE/CBOR style key formats despite JWK already being available as an option.
Sadly, what this means is that if DID Documents were to adopt JWKs as "the one true format" then we'd all be having to write key format conversion code everywhere! It would be much better to define some simple (and lossless) encoding rules that takes key formats in their popular forms and minimally encodes them for storage in Linked Data JSON. This is what the community has done thus far with things like
I agree that it is unfortunate that there are so many ways to represent the same keys. But we must admit that to a large extent, this is beyond our control. Trying to pick a winner here will only result in a loss. We should do something much simpler -- which I believe the community has already been doing. That doesn't mean that we shouldn't try to unify encoding formats where we can by using, for example, multibase. And we certainly should support JWK keys as some applications do accept these. I'd rather not have to convert there either. But adding an extra layer to convert all of the various formats to/from JWK is undesirable -- and I think that's exactly what we'd end up with because we don't control and can't easily influence the entire crypto application/platform space.
Note that a DID Document may express many different keys for many different intended purposes; only some subset may be used to establish control over modifying the DID Document itself (and would therefore be consumed in some fashion by the DID network/ledger according to the DID method). Others are often application/platform specific. This requirement may not have been understood before by everyone participating in the issue and could help explain why "one true format" isn't the best fit here.
Re: Application Level Support
There may be a number of different kinds of keys that should reside in a DID Document that are not used to validate the DID Document itself. In particular, keys in a number blockchain use cases may be shares of keys or partial keys to be used in multisig scenarios, or are proofs of key possession, such as a hash of a key (for instance a bitcoin address is a base58 hash of the public key). I worry that specifying JWK which does not support these kinds of keys will mean that those applications will not be able to use DID Document to share key information.
-- Christopher Allen
We had a useful clarifying discussion on this topic on today's call. I asked why people were talking about keys for applications when the keys being discussed are used by central resolvers to validate control of the DID. @dlongley clarified that some of the keys in a DID are application keys and not used for DID authentication, which is a useful distinction to have in mind.
As a result of today's discussion, I think that this issue is really two issues:
Having a standard representation for (1) public keys used to prove control of a DID would give us efficiencies both in implementation size and spec size. Standards are about making choices to improve interop and this is place we should choose.
Applications (2) can represent data however they want. That's not the business of the DID standard. What is critical for the standard to do, however it to make it syntactically clear which keys are of type (1) and which data structures are application data structures, including applicaiton data structures that may or may not represent keys.
I actually think this belongs in the domain of specific DID methods. A particular DID method may certainly say that JWK must be used for that purpose (and should if it makes sense). But I think it would be a bit odd to say all DID methods must use JWK to express the keys that are used internally to determine control over modification of a DID Document. These rules are entirely wrapped up in a particular method and the method should have full leeway to express keys in whatever way is most appropriate for the network/ledger/mechanism used to implement the method.
This issue was discussed in a meeting.
View the transcriptIvan Herman: See Issue #67
Michael Jones: As one of the assignees, I have taken the view that key types that the json web key is already a standard.
… There are other fundamental issues being raised. Keys in the DIDs are being used to check signatures.
… These are used by applications, so we should cater to the appliations. It seems that having only one way to do it would increase interop.
Manu Sporny: I think we should put in a couple of examples.
… We have requirements that are hinted at, but are not being written down.
… Other views need to be clearly articulated in the issue discussions.
… Please add your views to the conversation.
Michael Lodder: agreed
Manu Sporny: We need a full hour to discuss this.
… In DID resolution, an idea that an option could be passed to the resolver to request keys in a particular format.
Markus Sabadello: There needs to be more discussion about this. There is the linked data signature. There is multibase.
Michael Lodder: I’m concerned we are trying to force an encoding when we should allow the encoding in value
Michael Lodder: +1 markus_sabadello
Manu Sporny: I agree with mike-lodder
Manu Sporny: (based on the requirements we’ve heard)
Michael Jones: Dave wrote in the issue, that the keys are used by the DID resolver not an application level thing.
Dave Longley: The keys may not be verified by the resolver. Some keys can be bound to an application.
… Some keys can be used by the resolver, others by applications.
Dave Longley: selfissued: in short, a DID Document may express many different keys for many different intended purposes, only some subset may be used to establish control over modifying the DID Document itself
Dave Longley: (and would therefore be consumed in some fashion by the DID network/ledger according to the DID method)
Markus Sabadello: +1 to dlongley, keys in a DID Document may or may not be used by a method and the resolver.
To the extent that DID methods check signatures with public keys to validate control of the DID, I sure hope that we architect the data structures so that the signature checking can occur in a way that is independent of the DID method. Otherwise, interoperable implementations would have to implement the ever-increasing union of everything that developers might invent over time - a moving target that would never be hit in practice.
Making choices has clear benefits. (That's why we make standards!)
Regarding did document signing or self certifying cryptographic dids, that is where I feel we should rely on method specific extensions, blockchain specific signatures or publicKey like properties such as
If I want to check an ethereum signature for
This comment attempts to gather feedback across various communities and:
The following is a list of use cases for expressing public keys (and other verification methods) in DID Documents.
Note that the last two use cases, while not specifically public key use cases, are related such that the mechanism for expressing verification methods should ideally be able to support public keys and other verification methods.
The following requirements are derived from the use cases above.
I'm going to list all of the options that we are seriously considering so that people can see exactly what some of these decisions might look like. Some of the people in this thread are new to the topic, so seeing concrete examples might help more people contribute to the discussion. I'm not listing every option because I started to, got to 15 different representations, and felt that would further muddy the waters. If folks feel that there are other serious contenders, please speak up and I'll add them to the list below.
Option A: Pure JWK
Option B: Linked Data + various "native" key formats
Option C: Linked Data + various "native" key formats + Multikey (Multibase + Multicodec)
Rather than focusing on the requirements that are met, the list below focuses on the requirements that each option fails to achieve.
The use cases, requirements, and options are debatable and incomplete. This is a first attempt at providing a holistic analysis of the problem space. I'll update as more input rolls in.
Thanks, @msporny for this useful analysis. In a few ways, I believe that it is inaccurate and incomplete. The way that it is most incomplete, as I see it, is that it missing the following requirement.
Future-proofing of this kind is the whole point of standardization.
The Failed Requirements analysis in @msporny 's characterization is inaccurate in a few respects.
Option A (JWK representation) does meet these requirements:
I think the point about every DID consumer already being forced to use JSON is valuable. JWK is the best way to represent keys in JSON IMO, if we had to pick only one representation, I would choose JWK. I also think that harmonizing JSON-LD and JOSE is already underway in a number of other areas, and deciding to support
Just a few observations, because this is useful to tease out a shared understanding of the options on the table.
While this is true, "new fields can be defined by applications" and "can be used by any specification" typically involves a non-trivial centralized standardization process vs. a more decentralized extensibility model (such as that employed by the VC and DID specs). For example, I expect that a request to add
If you're doing RSA and you don't do PEM, then you need two data formats... PEM /and/ JWK... and conversion libraries between the two. If you're doing ECDSA (Bitcoin / Ethereum) secp256k1, you need JWK and hex encoding or Base58Check encoding. If you're doing Ed25519 you need JWK and conversion to native format (which is almost always just a raw byte string).
One could argue that JWK increases the encoding/decoding burden based on what native libraries have implemented today.
In theory, this should be true... but in reality, it's questionable... openssl takes PEM format (and still doesn't support JWK, AFAIK), Bitcoin/Ethereum libs don't regularly take JWK -- preferring their native format, same for a variety of Ed25519 implementations...
If JWK was a format widely recognized by libraries such as openssl, bitcoin libs, ethereum libs, ed25519 libs, etc... then there would be a strong argument to standardize on only it immediately. If JWK becomes that format in the future, there would be a strong argument to standardize on it then. That WebAuthn didn't pick JWK and went with COSE instead should be an indicator that, while it would be wonderful for the world to standardize on one key expression format, in reality, different applications make different encoding choices and the spec should be built in a way that matches that reality.
To put it another way, if we could get the openssl, Bitcoin, Ethereum, and WebAuthn communities to support JWK natively, we'd be in a much better place.
DID Documents do have a requirements 3, 4, and 6... which are requirements to support binary, because that's what the applications expose and we don't want to go through an unnecessary translation layer. If the WebAuthn spec kicks out a binary blob for a public key, we should be able to base-encode it and stuff it right into a DID Document, zero translation necessary.
Item 7 is more about proof of work, proof of elapsed time, invocation proofs and a variety of things that have little or nothing to do with public keys. I think it would be a stretch to state that a proof of work should be encoded as a JWK.
This is a fair point.
Interesting, don't know why this is being asserted.
An implementation checks the
Example of a resolver middleware key conversion code @tplooker mentioned on the call: https://github.com/decentralized-identity/interoperability/blob/982a09d4ad62a2be59b0f0da7d7d2ad8da1f9d9b/projects/cli-demo/src/resolver.js#L23
I expect we will see a lot of this kind of thing on clients, or an abundance of trusted resolvers who will do it for you....
the most common conversion code will likely be from X to jwk or pem.
In regards to size discussion, I was doing a character comparison between JWK and Base58PublicKey specifically.
In the example I mentioned on the call, I was considering the comparison between a pure JWK
and the raw key data (which @tplooker pointed out to me afterward is incorrect):
Rather the comparison should be between the two objects which give context about the keys. For example, the JWK should be compared to the object below because the type, as well as the
For comparison, the JWK representation (take notes these are different keys, but both ed25519 key types) is 211 characters. The second format is 130 characters. The encoding scheme used in the second format also plays into this to a degree. (e.g. using a multikey format adds more characters than base58)
However, the point I wanted us to be aware of is that the difference in 70 characters becomes quite significant for a DID registry at scale. For example, let's assume there are 100,000,000 DIDs in a DID registry and each DID holds only 1 key. The difference in size between nodes to store all DID Docs (e.g. it's a full node storing all DID Doc data), this one implication would be a 7GB difference.
7GB doesn't sound like much of a concern right now considering storage is quite cheap. At larger scales and with more keys being used, this bloat can grow which is why I point it out. For example, 10 billion DIDs that contain only 1 key would be a difference of 700GB of data. If we say each identity has 3 keys and there are 10 billion DIDs then it becomes 2.1 terabytes of excess data had we selected a different representation.
With that in mind, my ask is that no matter the representation we choose, we're considerate of the size of the key representation. It's not a huge point now, but at an internet-scale, it could become a problem in specific implementations where nodes must replicate the entire state of the DID registry (e.g. a blockchain without pruning).
I was just trolling through the JWE spec and saw this example of an RSA key being used in RSA-OAEP in JWK format:
RSA is definitely an extreme case of this problem.
This issue was discussed in a meeting.
View the transcriptKey representation
Daniel Burnett: #67
Manu Sporny: #85
Manu Sporny: #85 gets into whether data is application data or DID data , or …
Michael Jones: logged #85 based on discussion with dlongley, that application data may be formatted however the app likes, while DID data should be formatted according to the standard(s) we set
Manu Sporny: #85 is a bit of a meta-discussion, which is hard to resolve in a concall, and will make it harder to resolve #67 which is fairly focused
… #85 is still important, just less tangible and harder to grasp
Markus Sabadello: there’s a separate issue for “data about the DID subject” vs “data about the DID document”: #65 this is different from issue 85
Michael Jones: focusing on #67 for now is fine. hope is that folks will keep #85 in mind while doing so.
Manu Sporny: #67 (comment)
Manu Sporny: [ summarizes #67 ] …
Kyle Den Hartog: one concern is that data size will continue to grow, and that may pose a problem
Michael Jones: base* encoding does expand the size of things; that’s a valid concern.
… most important thing is raised as Requirement 11, about future proofing
… secondarily, JWK proponent (and author, so somewhat biased)
Dave Longley: regretful counter is that we have no control over formats used in the wild
… point of expressing your key is to let apps in the wild use your key, and many apps don’t consume JWK, so requiring that all keys are expressed in JWK imposes burden on apps now using openssl or nodejs or others
… would be fantastic if a single representation worked for all, but there are many tools already using multiple representations, so this is hard to force after-that-fact
… forcing JWK would slow adoption of DIDs and DID Methods
Drummond Reed: Mike, what’s your thoughts on Dave’s point?
Kyle Den Hartog: to clarify about size – it’s not about encoding; it’s about the characters required to express the base mathematics
Kyle Den Hartog: yeah, I can add them
Michael Jones: [ citation/example requested ]
… dlongley is right that any chosen format would require translation be implemented (once). idea is that once that translation is written, it’s done
Tobias Looker: DID resolver is going to be required in any case, let this handle the heavy lifting (the representation translation) for the app
Markus Sabadello: remember that we’re separating abstract content from concrete syntax
Manu Sporny: we’ll certainly talk about this again next time. the question is not “JWK or something-else”, it’s “something and JWK” – what is the something, that is capable of handling PEM or PEM in CBOR or ….?
Kyle Den Hartog:
Daniel Burnett: Thanks all. Will continue this discussion next week
Dave Longley: options: 1. support ONLY JWK, 2. support JWK and other popular formats
Manu Sporny: question is whether to support only JWK, or JWK-plus-others
Manu Sporny: +1
Manu Sporny: That’s what the group has to decide.
Ted Thibodeau Jr: fifth time’s the charm!
Responding to @kdenhartog 's comments about key representation size, what matters for DID documents is the size of the public key representation - not the private key. Yes, RSA private keys have lots of fields (to enable efficient Chinese Remainder Theorem computations), but RSA public keys have only two fields: "n", and "e". For a 2K key, "n" has 342 characters in JWK and "e" has 4.
Likewise, the private key field "d" shown in your example above won't be present in an EC public key representation and the "kid" is optional.
We should do apples-to-apples size comparisons by comparing unadorned public key representations.
@kdenhartog and @selfissued how a certain DID method/registry chooses to represent keys or other things internally is not too relevant for how keys are represented in the DID document. In many of the DID methods we have today there is some kind of transformation happening anyway; that's what DID resolution is for. Yes some DID methods use the DID document format internally (e.g. Veres One), but many of them don't.
[Update.. I changed "most" to "many".. in this comment, since several DID methods do use the DID document format internally]
@selfissued brought up the point at the end of the call that Multibase has many of the issues around not selecting a particular format leading to n number of conversions needing to be implemented. From my point of view, I see the point he's making around it not solving the conversion problem for implementors. Particularly, when suggesting multi-base (I'm by no means married to that solution nor have I implemented with it yet even) what I'm looking for is a concise representation of keys, so that what we decide at the DID Document layer becomes a common enough solution that it permeates quite commonly into the application layer. ( issue #85 )
From an implementation standpoint, if I could have my cake and eat it too, it would be the ability to have a concise representation of the key which MUST be self-describing (e.g. I don't have to know another developer is using ed25519 keys because I talked with them or read documents) and I would like it to be representable in a concise format. Right now, I would say that CWKs meet both capabilities. The difficulty then comes upstream because I either have to convert all messages to CBOR at the application layer or doing conversions from CBOR to JSON as a compromise. It also only helps for a JSON serialization format but would prove much less useful if someone wanted to use YAML or XML from what I can tell.
I'd also like to say that if I had to prioritize my asks, having to implement 1 or 2 formats would be far more preferable than to be concise. However, if we get conciseness, I believe this may help resolve issue 85 from my point of view.
This issue was discussed in a meeting.
View the transcriptKey representation (Issue #67)
Daniel Burnett: #67
Daniel Burnett: This is the primary issue; other issues are related to it.
Manu Sporny: We’ve had the discussion about key representation before and coming back to it with new information.
… I made an attempt to summarize what this is about.
… We have some use cases and requirements that hadn’t really been summarized before, so this is what my comment is trying to do.
… We’re trying to represent cryptographic information (together with other information) in DID documents, so it can be consumed by applications.
… Three categories of applications:
… 1. Three categories of applications: 1. Outside our scope (not our business to define key formats, but we still want to encourage interop)
… 2. Keys used by the DID registry (internally e.g. to control the DID document). This also includes things other than keys
… 3. Applications that are somehow in the middle. E.g. authentication. These same mechanisms may be used by the DID registry and also outside.
… I think there’s a lot of the back and forth with different objectives.
… One argument: Everybody standardizes on one key format, this simplifies implementations.
… Other argument: It would be nice to have one representation, but the reality is that we don’t. There are existing implementations that have considered this but decided not to pick a single format.
… So today we have multiple different representations of keys and we have to acknowledge that, and transformations between formats are happening
Michael Jones: A friend in the industry said: ‘Making choices that don’t matter. To get interoperability you have to make a choice. It doesn’t matter so much what you choose, but it’s important that you do choose.’
… I look at what’s been happening so far, and I think nobody makes any choices. Everything seems loose, the union of what everyone wants, but in the end this doesn’t serve anyone
Kyle Den Hartog: Question for markus_sabadello about your comment in the Github issue.
Markus Sabadello: The question you’re asking about the comment in IRC… I was trying to argue… size of different key representations, issue for certain did methods, registries, size of representation too large… whatever representation or format we use, is not necessarily what is stored in the registry. The way a lot of DID Methods work, they don’t store DID Document internally… it is a result of resolution process.
… Bitcoin, Ethereum, they don’t actually store DID Document on the ledger, whatever we agree on, doesn’t mean, doesn’t impact how DID Documents get implemented internally. DID Document may or may not use DID Document internally.
Kyle Den Hartog: Thanks Markus, that makes me feel like my point is more a moot point and is resolved from the registry perspective. At the application layer this has larger implications.
Manu Sporny: I wanted to focus on real-world repercussions of this discussion. I agree with selfissued that DID spec writers are not forcing one way of doing it
… There was a heated discussion on forcing one key format vs. allowing multiple formats
… Both choices could result in more complexity for implementors
… Many implementations don’t use JWK today and have chosen not to use it. They use e.g. binary encodings of keys. I think that’s terrible, but that’s how it is.
… E.g. openssl is not going to support JWK, therefore if DID document requires JWK, transformations have to be implemented.
… Example of Ed25519: Most people just use raw binary encoding
… In Bitcoin and Ethereum communities, people use different formats even though the key types are the same
… This is how the ecosystem has developed. Attempts for people to agree on a single format have failed
… Let’s support JWK and keep trying to get people to use the same format. But also recognize that this will increase, not decrease, implementation burden
Michael Jones: Clarification question: We’re only taking about public key representations, right?
Manu Sporny: Yes
Michael Jones: About your complexity argument: If you force a single format, translation has only ever to happen once (between other format, and the single support format)
… In the current ecosystem however, transformation is more complex since every format has to be transformed to every other. This gets worse with additional formats
… Therefore, making one choice, the overall translation problem gets easier
… We could say that for known key types (RSA, EC) we could choose JWK and one or two other common formats, but also state that we will accept no other
… I don’t like that, but it’s better than being completely open-ended
Daniel Burnett: I think standards is about writing down where you can get agreement. Let other things wait
… Even if you have to leave some things open (they may become clearer over time)
Manu Sporny: Agree with selfissued, trying to push JWK isn’t going to result in end state selfissued wants. I also want everyone to use the same key format. But in reality I don’t think we will be able to change that.
… Agree with selfissued to do the next-best things: E.g. accept 2 most popular formats but then don’t allow others. Aggressively narrow the list, stop proliferation
Kyle Den Hartog: I was confused by selfissued ‘s comments about RSA and EC key types. But I thought we’re talking about different serialization formats, not key types.
… +1 to manu (which was +1 to selfissued)
… I think we should restrict formats, not key types (e.g. quantum crypto will become important)
Daniel Burnett: Perhaps we schedule an additional dedicated call for this topic
Kyle Den Hartog: +1 to that
Michael Jones: My point was that for a given key type (e.g. RSA) we should choose 1 (or maybe 2) supported representations for this key type.
Manu Sporny: Agreeing more and more with selfissued. Another wrinkle is something markus_sabadello highlighted on the last call: Don’t assume that DID documents will always be encoded as JSON.
… Like in the VC spec, we separate the abstract data model from the concrete format. E.g. you could represent a DID document in a binary format like CBOR.
… Therefore we should keep this in mind when talking about public key formats.
… I think JWK can also be represented in CBOR, but let’s keep this in mind
Michael Jones: This is a fine point. I find multibase particularly offensive, since that format itself refuses to make a choice;
Daniel Burnett: We don’t have enough time to resolve this, but I feel we’re getting to something. It may be best to have a dedicated call with those who are interested.
Manu Sporny: You can do multibase, but force ONE encoding… you do that because you can’t predict 10 years in the future… who thought that base58 would be a thing… but it is
Kyle Den Hartog: I’d be interested to join this
Manu Sporny: We need more than selfissued and kdenhartog and the editors to resolve this. We have to make sure we get consensus of the whole WG.
Manu Sporny: Summary of where we are right now…
Summary: It sounds like we’re closing in on a path forward, which is to allow multiple types, but limit to perhaps 2 mechanisms for each key type… for example, for RSA, PEM encoding and JWK encoding… While this doesn’t get us down to one key format, it does limit an explosion of key formats. We will need to have further calls to ensure that we’re taking all use cases and requirements into account, and those calls will be scheduled by the Chairs and Staff. We need to make sure that we have more than a handful of people providing input (Manu Sporny)
I'm in favor of having a few different formats. I believe it is much easier to implement the conversion at the application level. If an application requires a specific format, then it will be less effort to just implement the conversion from a few mandatory formats to the format the application needs. I assume in many cases that won't be required at all. One application where this might be relevant is verifiable credentials/ presentations where a lot of different DID methods might be used across issuers and verifiers. Of course, that won't solve the problem as a whole as once the key was parsed, the application still needs to implement the dedicated handling for the different verification methods. So, I'm happy with the current direction of WG discussion.
I don't think it matters that JWK is potentially bigger as the DID Document often does not get stored as it is -- e.g., on a ledger. It could matter to IOT devices that have limited memory though.
@selfissued If we can find a way to use JWK to encode
Per #55 (comment), it sounds like an
I propose we open a PR which mandates that
Next, we should answer the question why do we require another key format?
Sure we could also support
Ed25519, secp256k1, both supported.
Anything that is consuming a DID Document is using some form of resolver... A resolver, can be wrapped to provide key conversion... transforms may eventually become standardized, see:
The question is how is key conversion related to the did-core data model?
"DID method specifications MAY define their own JSON-LD contexts. However it is NOT RECOMMENDED to define a new context unless necessary to properly implement the method. Method-specific contexts MUST NOT override the terms defined in the generic DID context."
If a DID Method wanted to support other key formats, or needed to, it can 100% go ahead and do that... today!
It might help us keep the "generic DID context" cleaner by establishing a clear distinction between MUST support, and optional support... its hard not to conflict with a generic did context that attempts to define lots of optional support for key formats, capabilities, authorization, etc...
Its much easier to update a context you control then it is to get
We can in theory have a single key format that is MUST support, I advocate for choosing JWK, and immediately demonstrating how we can use the other extensibility features of the did spec (contexts), to support publicKeyPem.
How much code needs to be changed to keep what we have today and do the following:
Why not define a context that just defines, PEM, Multibase, Ethereum, etc... and let a DID Method Implementer pick the contexts that they want... while requiring them to support interoperability with JWK... as we currently require them to conform to the DID Spec.
@dlongley am I misunderstanding the extensibility features of context here?
The whole reason I originally encountered this key representation issue was because of contexts, maybe they are the solution?
I want to echo what @OR13 said about having a JWK as a base format for descriptors, given I have not found any significant reason why we can't represent keys within the bounds of JWK, with the addition of new key types, if necessary. Method resolver code should be able to transform from whatever a Method uses internally to the base format, and such a simplification would help make this aspect of the spec easier to deal with.
My comment #100 (review) requests that we document the agreement to support the JWK representation for all public key types and that any additional representations would be an exception, as agreed to on the 6-Nov-19 special call. That will take us a long way towards the interop and simplicity goals.
I do believe that @OR13 is asking the right question when he asks "Next, we should answer the question why do we require another key format?". We should try to cover that during the next special call.
That is defintiely not what I thought we had consensus on. I agree that we were trying to support JWK across all key formats. I didn't think that the group agreed that other representations would be exceptions (as many key formats deployed today do not use JWK, even though some variation of JWK has been available for use for the past 8 years or so). More below...
I think that's what the current PRs attempt to do? What am I missing?
I think the answer is: there is no one supported key encoding format today. There have been attempts to get alignment (JWK), but those attempts have not resulted in what we'd like to see (which is one supported key encoding format across the entire public key expression ecosystem).
I think @kdenhartog's point here cuts to the heart of the matter:
In other words, by mandating one key format (that implementers had an opportunity to use, but didn't), we're not reflecting reality... and specs should reflect reality. If we don't do that, we have an academic ivory tower spec... not a pragmatic one (and the latter specs are the ones that get deployed). My understanding is that we were headed towards supporting multiple key formats. The latest arguments seem to be backpedaling to one key format (yes, that would be ideal... but reality states that this has not been happening over the past decade). Some data points to consider:
To mandate one key format will increase implementation burden, not reduce it.
If we are going to mandate the use of JWK, we're going to have to see if implementers are actually going to implement that solution. I suggest that, given the current trajectory of the conversation, that we get some real data from as many of the cryptographic library implementations as possible. Which libraries support which formats today, and use that to guide the discussion. Bonus points if we can get implementers of those libraries to solidly commit to supporting a single key format (like JWK).
We need a reality check, so let's do some science and get data from implementations and, if we can, implementers. I suggest that we:
You are understanding the extensibility features correctly. The issue we've had in the past is that there is a small subset of the community that doesn't want to use those features. The more
Our assumption is that you're going to have the base DID context and then a DID Method specific context, and that's it:
Yes, we could solve the issue like that, but what I heard @selfissued say on the call last week is that he would object to that approach. He would not like there to be other possibilities outside what the DID Core spec allows, even if a DID Method wants there to be. Did I misunderstand that requirement of yours, @selfissued?
I have started a spreadsheet here to gather the findings. I need help filling the spreadsheet out, everyone has edit privileges:
I have started a survey here to gather data from DID Method implementers:
@msporny wrote that "specs should reflect reality". That's certainly a good principle, but it's often the case before there's a standard in an area that the reality is that are many divergent implementations of essentially the same thing that differ in unnecessary ways.
A concrete example of this was signed JSON claims before the JWT standard. As you can read at https://self-issued.info/?p=1387, in 2010 there were at least four different formats for this in use by different systems. It seemed like time for a standard! We surveyed the features, created and refined a specification that would work for all their use cases, and created a standard. The reality now is all all these non-standard precursor formats are gone in favor of JWT.
The DID world is now in a similar place to where signed JSON was in 2010. There's been lots of useful prototyping and experimentation. People made different choices around things that needn't be different. There's no standard for them to coalesce around to create interoperability.
We have the opportunity now to create a DID standard that will foster interoperability, eliminating unnecessary differences. Taking the path of saying that "reality has a multiplicity of divergent representations - let's leave it that way" would be doing everyone a disservice, and would fail to utlitize the power of standards to create an interoperable reality for the future.
From the 2019-11-13 telecon:
Note: I have removed duplicate votes from the same organization to ensure that the votes weren't being packed by any particular organization, but only after confirming that all people from each organization voted the exact same way. Every proposal above was a straw poll and not a binding vote by the organization. The purpose of the exercise was to get a temperature of where the discussion was and not to create a binding decision by the group.