divergence between implementations serializing ed25519 public keys

[erickt]

There is some ambiguity in the spec regarding how to encode ed25519's public portion of the keyval:

The 'ed25519' format is:

  { "keytype" : "ed25519",
    "scheme" : "ed25519",
    "keyval" : {"public" : PUBLIC}
  }

where PUBLIC is a 32-byte string.

While the ed25519 public key is a 32-byte string, I could find no implementation that's serializing the public key to JSON as an array of integers. Instead, implementations are serializing it into a JSON string, but there is a fork in how this is implemented:

• python-tuf and go-tuf hex encode the public key
• rust-tuf encodes with base64url to encode the public key.
• notary, and hackage base64 standard encode the public key.

Can you provide any guidance on what serialization format we should be using?

edit: turns out rust-tuf uses base64url instead of base64standard for encoding.

[awwad]

That's a good point & question.

The document format section of the specification defines the format as PEM for RSA and ECDSA, but simply says that the ed25519 format should be a "32-byte string", which fits neither the hex string that we use nor the base64 encoding that others apparently use....

[awwad]

@JustinCappos:

1. Do you agree that it's wise for the spec to define the formats for key values for each key type? (as is done for RSA and ECDSA, but not quite for ed25519 currently)
2. To what extent should we alter the reference implementation's key serialization to accommodate rust-tuf, notary, and hackage?

The options, basically, are the 44-byte base64 they use, the 64-byte hex string we use, or a 32-byte int. The spec needs to be corrected in every case.

[awwad]

Adding @mnm678 as well, who's working on Uptane profiles/wire-line formats and so might have some input.

[JustinCappos]

@JustinCappos:

1. Do you agree that it's wise for the spec to define the formats for key values for each key type? (as is done for RSA and ECDSA, but not quite for ed25519 currently)
2. To what extent should we alter the reference implementation's key serialization to accommodate rust-tuf, notary, and hackage?

The options, basically, are the 44-byte base64 they use, the 64-byte hex string we use, or a 32-byte int. The spec needs to be corrected in every case.

If we're moving to profiles in TUF, then deciding that format does seem to fit there. It feels easier to me for us to change ours, unless @trishankkarthik disagrees.

[erickt]

The options, basically, are the 44-byte base64 they use, the 64-byte hex string we use, or a 32-byte int. The spec needs to be corrected in every case.

I would suggest against using a native JSON 32-byte integer. Many json parsers don't support integers larger than 64 bits.

[awwad]

I suppose it comes down to base64, then. It's a breaking change, so I'll pair this with the key ID calculation changes (that I expect we'll manage to confirm...) for the next major release.

Base64 standard (not base64url) makes the most sense to me. Eric, do you have any thoughts there? Does that switch make sense to you?

@JustinCappos: does this seem like a TAP? :( The existing text didn't really make sense ("32-byte string"), and nobody's using it, so perhaps it's just a TAP-less correction...?

[erickt]

@awwad: as best as I can tell, @heartsucker switched all encoding from hex to base64url in theupdateframework/rust-tuf#83. They used base64url over base64 due to these values showing up in URLs and filesystems for target files. I'm not sure if there was any particular reason for using base64url for things inside the metadata, that probably just happened out of using the same codepath for both encodings.

@heartsucker, is this correct?

[trishankatdatadog]

I agree with @JustinCappos we should be compatible with (go|rust)-tuf, notary, hackage, etc

[erickt]

It looks like there's also divergence on how implementations are encoding the snapshot/timestamp/ targets role hashes, both in the metadata and for the hash-prefixed files:

• python-tuf and go-tuf, hackage (as best as I can tell) uses hex encoding for metadata and files.
• rust-tuf uses base64url for metadata and files.
• notary uses base64standard for its metadata, and hex for target URLs.

Also, if we were changing ed25519 metadata to use base64, should the spec also change the keyid and path_hash_prefixes to be base64? The spec specifically calls out that they are hex encoded.

[JustinCappos]

I don't see this as TAP worthy. It's just a clarification / fix.

…

On Wed, Apr 10, 2019 at 8:49 PM Erick Tryzelaar ***@***.***> wrote: It looks like there's also divergence on how implementations are encoding the snapshot/timestamp/ targets role hashes, both in the metadata and for the hash-prefixed files: - python-tuf <https://github.com/secure-systems-lab/securesystemslib/blob/509a1860a54019cb82ec374dc1cd1487c34517ba/securesystemslib/util.py#L393> and go-tuf <https://github.com/flynn/go-tuf/blob/master/data/types.go#L108>, hackage <https://github.com/haskell/hackage-security/blob/21a385c1a3c6222d4a7b6b8ecaff23431999d130/hackage-security/src/Hackage/Security/TUF/FileInfo.hs#L68> (as best as I can tell) uses hex encoding for metadata and files. - rust-tuf <https://github.com/heartsucker/rust-tuf/blob/develop/src/crypto.rs#L763> uses base64url for metadata and files. - notary uses base64standard for its metadata <https://github.com/theupdateframework/notary/blob/ce99e3cc568549b10ced3eb49cbb7e2941924eb4/tuf/data/types.go#L174>, and hex for target URLs <https://github.com/theupdateframework/notary/blob/ce99e3cc568549b10ced3eb49cbb7e2941924eb4/tuf/utils/utils.go#L115> . Also, if we were changing ed25519 metadata to use base64, should the spec also change the keyid and path_hash_prefixes to be base64? The spec specifically calls out that they are hex encoded. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0XD1kwxi-jI5bUgV_xjIIzhmqwyWUjks5vfoa2gaJpZM4ck62t> .

[awwad]

@erickt

Also, if we were changing ed25519 metadata to use base64, should the spec also change the keyid and path_hash_prefixes to be base64? The spec specifically calls out that they are hex encoded.

Unlike the spec's instructions for ed25519 key values, where the spec was clearly wrong, the keyid and path_hash_prefixes being hex weren't mistakes, I don't think. Why switch? Is that what the other implementations do?

[erickt]

@awwad: Regarding keyid, almost everyone is using the spec:

• python-tuf, hackage, and notary is using hex(sha256(cjson(key))).
• rust-tuf uses sha256(cjson(encoded(pub_key_bytes))). I'm in the process to porting it to use hex(sha256(cjson(key))) though.

Regarding path_hash_prefixes, the only implementation I could find that uses it is python-tuf, and I couldn't find where it actually generates that value, only where it consumes it.

Personally I'm fine keeping the current behavior. We could make the "bytes to string" conversion more consistent for these fields once profiles are fleshed out if we really wanted to.

[heartsucker]

The reason I made these changes was because in past discussion it sounded like there was said that there was no expected compatibility between implementations. With that in mind, I made opinionated choices based on what I thought made more sense than the original python impl.

[erickt]

Hello everyone, just a bump of this ticket since I'm working on this aspect of rust-tuf. Is the plan to update the spec to state that the ed25519 public key and role hashes should use hex encoding, and possibly switch to using base64 encoding later on with a new profile?

[trishankatdatadog]

I think for now let's go along with hex encoding (python-tuf, hackage, go-tuf, notary) instead of base64. (Notary uses base64 elsewhere, like sigs and hashes, but we'll figure this out later.) If someone really complains about this, we'll figure it out later.

[erickt]

I just submitted #47 to update the spec to reflect hex encoding the ed25519 public key, and target file hashes.