Revise public key metadata format

[lukpueh]

Description of issue or feature request:

#251 discusses various consistency concerns related to securesystemslib (TUF/in-toto) cryptographic keys.

This is an attempt to carve (and flesh) out only those concerns related to the format of public keys as they appear in TUF/in-toto metadata, with the goal of a clear specification likely as part (or appendix) of the new signing-spec and its reference implementation in securesystemslib.

Below in 'Current behavior' is an exhaustive list of available values for public keys as currently implemented in securesystemslib. This includes rsa, ecdsa, ed25519 and spx (WIP #169) keys generated with securesystemslib, plus keys exported from a GnuPG keyring using securesystemslib.gpg or from a cryptographic token (e.g. HSM) using securesystemslib.hsm (WIP #229).

Related concerns and calls for action are listed further below in 'Expected behavior'.

Current behavior:

Public key container (mandatory fields)

{
  'keyid' <...>, # Optional in the spec, but included in the in-toto reference implementation
  'keytype': <...>,
  'scheme': <...>,
  'keyval': {
    'private': <...>, # Must be omitted or empty for public keys
    'public': <...>
  }
}

'keyid'

• ascii hex of canonical JSON (OLPC) of parts of the key dictionary
• RFC4880 OpenPGP Issuer Fingerprint (gpg keys only)
• custom value (see WIP sphincs+ support, for post-quantum crypto #169 , TAP 12 and Implement TAP 12 theupdateframework/python-tuf#1084)

Note that 'keyid' is not part of the public key metadata format according to in-toto/TUF specifications and tuf reference implementation, but it is included in the in-toto reference implementation (see #251 PR description).

'keytype'

• 'rsa'
• 'ed25519'
• 'ecdsa'
• 'ecdsa-sha2-nistp(256|384)' (legacy, new keys use 'ecdsa', see Better distinguish between keytype and scheme for ECDSA keys #239 and Better distinguish between keytype and scheme for ECDSA keys #267)
• 'spx' (WIP sphincs+ support, for post-quantum crypto #169)

'scheme'

• 'rsassa-pss-(md5|sha1|sha224|sha256|sha384|sha512)'
• 'rsa-pkcs1v15-(md5|sha1|sha224|sha256|sha384|sha512)'
• 'ecdsa-sha2-nistp(256|384)'
• 'ed25519'
• 'spx' (WIP sphincs+ support, for post-quantum crypto #169)

'type'

(gpg keys only, instead of 'keytype')

• 'rsa'
• 'dsa'
• 'eddsa'

'method'

(gpg keys only, instead of 'scheme')

• 'pgp+rsa-pkcsv1.5'
• 'pgp+dsa-fips-180-2'
• 'pgp+eddsa-ed25519'

'hashes'

(gpg keys only, instead of 'scheme')

• 'pgp+SHA2'

'keyval'

'public'

• rsa, ecdsa, ecdsa from HSM (WIP Add PKCS11-based HSM interface #229)
  X.509 SubjectPublicKeyInfo PEM string (RFC5280)
• ed25519
  32 bytes ascii hex key data (RFC8032)
• spx (WIP sphincs+ support, for post-quantum crypto #169)
  64 bytes key data (SPHINCS+-SHAKE256)
• rsa gpg (RFC4880)

    {
      "e": <public encryption exponent (ascii hex)>,
      "n": <public modulus (ascii hex)>
    }
  

• dsa gpg (RFC4880)

  {
    "y": <public value (ascii hex)>,
    "p": <public modulus (ascii hex)>,
    "q": <group order (ascii hex)>,
    "g": <group generator (ascii hex)>
  }
  

• eddsa gpg (RFC4880)

  {
    "q": <MPI of an EC point representing a public key (ascii hex)>
  }
  

'private'

(NOTE: The 'private' portion is not directly relevant for this issue, as it must not be included in TUF/in-toto metadata. For the sake of completeness we still list the private key values that are currently kept internally in securesystemslib).

• rsa, ecdsa
  PKCS#1/TraditionalOpenSSL PEM string (RFC8017)
• ed25519
  32 bytes ascii hex key data (RFC8032)
• spx (WIP sphincs+ support, for post-quantum crypto #169)
  128 bytes key data (SPHINCS+-SHAKE256)**

Optional fields

• 'creation_time'
• 'validity_period'
• 'expires'
• 'keyid_hash_algorithms' (Implement TAP 12 theupdateframework/python-tuf#1084)

Expected behavior:

• 'keyid'

  • better decouple default keyids from keys, to allow the use of custom keyids, but with a canonical json computed keyid as fallback
  • clarify if keyid is optional in the key object

• Consolidate 'keytype', 'type', 'scheme', 'method', 'hashes' fields and values

  • 'scheme' usually includes key type, signing scheme and hash algorithm, could one field be enough?
  • revise correctness/appropriateness of values (see e.g. 'eddsa' vs. 'ed25519' as different '(key)type's for the same type of key).
  • is there a standard for 'scheme' strings that we could use?

• 'keyval'

  • consolidate values (see e.g. serialized rsa PEM vs. non-serialized rsa numbers)
  • clarify if 'private' field may be omitted or should be just empty for public keys

• optional fields

  • deprecate 'keyid_hash_algorithms'
  • maybe consolidate 'validity_period' and 'expires'?

[lukpueh]

related: #587

[lukpueh]

It's hard to tell when to mark this issue as resolved, but I would say that the public key metadata format has been revised. Realistically, there won't be many changes, given that past changes have been quite disruptive for users.

The good news is that the implementation is way better decoupled from the public key metadata format these days, which makes all of this easier to address.

I suggest to fix remaining concerns above all in docs, for which we have more reasonably scoped tickets. See inline...

Expected behavior:

• 'keyid'

  • better decouple default keyids from keys, to allow the use of custom keyids, but with a canonical json computed keyid as fallback

done

• clarify if keyid is optional in the key object

The Key interfaces requires a keyid, the default serialization methods (to/from_dict), however, do not include a keyid. Users can easily override this behavior.

Related issues: #416, #670

• Consolidate 'keytype', 'type', 'scheme', 'method', 'hashes' fields and values

The concerns about redundancy and semantic correctness of keytype/scheme pairs are still valid, however, changing the existing values is not worth the effort (see all the issues related to the ecdsa keytype change).

method and hashes are specific to the securesystemslib.gpg subpackage, which is not part of the documented API. GPGKey provides a compatibility layer, which uses keytype/scheme instead.

• 'keyval'

  • consolidate values (see e.g. serialized rsa PEM vs. non-serialized rsa numbers)

The exact keyval format is a detail of a specific Key implementation. Despite inconsistency, the current formats for existing keytype/scheme pairs are unlikely to change for the reasons described above.

• clarify if 'private' field may be omitted or should be just empty for public keys

Private keys are now separated (Signer) from public keys (Key). This means that securesystemlsib no longer populates or uses a private field.

• optional fields

The Key interface supports de/serialization of unrecognised fields.

Related issues: #467, #587

• deprecate 'keyid_hash_algorithms'

done

• maybe consolidate 'validity_period' and 'expires'?

Removed from GPGKey, see GPGSigner.import_ docstring for background