Skip to content
This repository has been archived by the owner on Mar 29, 2022. It is now read-only.

Debugging ASN1 is very unpleasant #154

Open
awwad opened this issue Dec 20, 2017 · 2 comments
Open

Debugging ASN1 is very unpleasant #154

awwad opened this issue Dec 20, 2017 · 2 comments

Comments

@awwad
Copy link
Contributor

awwad commented Dec 20, 2017 •
edited
Loading

There are two issues here:

  1. Parsing the pyasn1 objects generated by the converter modules in Uptane's TUF fork and Uptane itself is a nightmare.
  2. The structure of the ASN1 data doesn't quite match the structure of the JSON data.

This can be simplified substantially.

Part of this complexity is because of the fact that the ASN.1 definition for signed metadata employs a choice of four relatively complex objects (timestamp, snapshot, targets, root, delegated targets). As a result, the pyasn1 objects for metadata include quite a bit of unnecessary information about the structure of other metadata types.

Let's consider an example. The below is a single Timestamp role translated into ASN.1 with our
current translators (metadata_asn1_definitions.py, asn1_codec.py, and timestamp_asn1_coder.py).

Note that this piece of Timestamp metadata TWICE defines what Root metadata looks like, what Snapshot metadata looks like, and what Targets metadata looks like....

x = Signed(
  componentType=NamedTypes(
    NamedType('type',
      RoleType(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
    NamedType('expires',
      UTCDateTime(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)))),
    NamedType('version',
      Natural(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
    NamedType('body',
      SignedBody(
        componentType=NamedTypes(
          NamedType('rootMetadata',
            RootMetadata(
              componentType=NamedTypes(
                NamedType('numberOfKeys',
                  Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                NamedType('keys',
                  PublicKeys(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
                NamedType('numberOfRoles',
                  Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
                NamedType('roles',
                  TopLevelRoles(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))))
                ),
              tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=0)))),
          NamedType('targetsMetadata',
            TargetsMetadata(
              componentType=NamedTypes(
                NamedType('numberOfTargets',
                  Natural(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                NamedType('targets',
                  Targets(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
                OptionalNamedType('delegations',
                  TargetsDelegations(
                    componentType=NamedTypes(
                      NamedType('numberOfKeys',
                        Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                      NamedType('keys',
                        PublicKeys(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
                      NamedType('numberOfDelegations',
                        Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
                      NamedType('delegations',
                        PrioritizedPathsToRoles(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))))),
                    tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=2))))),
              tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
          NamedType('snapshotMetadata',
            SnapshotMetadata(
              componentType=NamedTypes(
                NamedType('numberOfTargetRoleFiles',
                  Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                NamedType('targetRoleFileInfos',
                  TargetRoleFileInfos(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
                NamedType('rootRoleFileInfo',
                  RootRoleFileInfo(
                    componentType=NamedTypes(
                      NamedType('filename',
                        StrictFilename(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                      NamedType('version',
                        Version(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)))),
                      NamedType('length',
                        Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
                      NamedType('numberOfHashes',
                        Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=3)))),
                      NamedType('hashes',
                        Hashes(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=4))))),
                    tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=2))))),
              tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=2)))),
          NamedType('timestampMetadata',
            TimestampMetadata(
              componentType=NamedTypes(
                NamedType('filename',
                  Filename(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                NamedType('version',
                  Version(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)))),
                NamedType('length',
                  Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
                NamedType('numberOfHashes',
                  Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=3)))),
                NamedType('hashes',
                  Hashes(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=4))))),
              tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))))
          ), # ends SignedBody's NamedTypes
        tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))
        ))
    ), # Ends Signed's NamedTypes
  tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=0))
  ).\
setComponents(   # ends Signed()

  # Actual data starts here.

  RoleType('timestamp',
    tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0))),
  UTCDateTime(2137754778, tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1))),
  Natural(1, tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2))),
  SignedBody(
    componentType=NamedTypes(
      NamedType('rootMetadata',
        RootMetadata(
          componentType=NamedTypes(
            NamedType('numberOfKeys',
              Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
            NamedType('keys',
              PublicKeys(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
            NamedType('numberOfRoles',
              Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
            NamedType('roles',
              TopLevelRoles(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))))),
          tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=0)))),

      NamedType('targetsMetadata',
        TargetsMetadata(
          componentType=NamedTypes(
            NamedType('numberOfTargets',
              Natural(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
            NamedType('targets',
              Targets(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
            OptionalNamedType('delegations',
              TargetsDelegations(
                componentType=NamedTypes(
                  NamedType('numberOfKeys',
                    Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                  NamedType('keys',
                    PublicKeys(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
                  NamedType('numberOfDelegations',
                    Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
                  NamedType('delegations',
                    PrioritizedPathsToRoles(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))))),
                tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=2))))),
          tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),

      NamedType('snapshotMetadata',
        SnapshotMetadata(
          componentType=NamedTypes(
            NamedType('numberOfTargetRoleFiles',
              Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
            NamedType('targetRoleFileInfos',
              TargetRoleFileInfos(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))),
            NamedType('rootRoleFileInfo',
              RootRoleFileInfo(
                componentType=NamedTypes(
                  NamedType('filename',
                    StrictFilename(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                  NamedType('version',
                    Version(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)))),
                  NamedType('length',
                    Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
                  NamedType('numberOfHashes',
                    Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=3)))),
                  NamedType('hashes',
                    Hashes(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=4))))),
                tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=2))))),
          tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=2)))),

      NamedType('timestampMetadata',
        TimestampMetadata(
          componentType=NamedTypes(
            NamedType('filename',
              Filename(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
            NamedType('version',
              Version(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)))),
            NamedType('length',
              Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
            NamedType('numberOfHashes',
              Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=3)))),
            NamedType('hashes',
              Hashes(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=4))))),
          tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))))),
    tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))
    ).\
  setComponents( # end of SignedBody()
    None,
    None,
    None,
    TimestampMetadata(
      componentType=NamedTypes(
        NamedType('filename',
          Filename(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
        NamedType('version',
          Version(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)))),
        NamedType('length',
          Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2)))),
        NamedType('numberOfHashes',
          Length(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=3)))),
        NamedType('hashes',
          Hashes(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=4))))),
      tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=3))).\
        setComponents( # end of TimestampMetadata()
          Filename(b'snapshot.json', tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0))),
          Version(1, tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1))),
          Length(594, tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=2))),
          Length(1, tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=3))),
          Hashes(tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=4))).\
          setComponents(
            Hash(
              componentType=NamedTypes(
                NamedType('function',
                  HashFunction(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                NamedType('digest',
                  BinaryData(
                    componentType=NamedTypes(
                      NamedType('bitString',
                        BitString(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                      NamedType('octetString',
                        OctetString(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1))))),
                    tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1)))))
              ).\
            setComponents( # end of Hash()
              HashFunction('sha256',
                tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0))),
              BinaryData(
                componentType=NamedTypes(
                  NamedType('bitString',
                    BitString(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=0)))),
                  NamedType('octetString',
                    OctetString(tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1))))),
                tagSet=TagSet((), Tag(tagClass=128, tagFormat=32, tagId=1))
                ).\
              setComponents( # end of BinaryData()
                None,
                OctetString(
                  tagSet=TagSet((), Tag(tagClass=128, tagFormat=0, tagId=1)),
                  hexValue='d37a3f9b41bd7ad4c87bc978f0341cbeb081bdee0891f087f33d34963c565e16'))))))
)

Encoded in DER, the above pyasn1 object yields this:

b'''
\xa0T\x80\x01\x03\x81\x04\x7fk\x8c\x9a\x82\x01\x01\xa3F\xa3D
\x80\rsnapshot.json\x81\x01\x01\x82\x02\x02R\x83\x01\x01\xa4)0
\'\x80\x01\x01\xa1"\x81 \xd3z?\x9bA\xbdz\xd4\xc8{\xc9x\xf04
\x1c\xbe\xb0\x81\xbd\xee\x08\x91\xf0\x87\xf3=4\x96<V^\x16
'''
@awwad
Copy link
Contributor Author

awwad commented Dec 20, 2017 •
edited
Loading

I think I should dispose of structure SignedBody (see tuf/encoding/metadata_asn1_definitions.py, uptane/encoding/asn1_definitions.py, uptane/encoding/asn1_definitions.asn1).

Current model:

Metadata:
  sigs    list(Signature)
  signed  Signed

Signed:
  type    ...
  version ...
  expires ...
  body    SignedBody

SignedBody:
  Choice of:
    TimestampMetadata
    TargetsMetadata
    SnapshotMetadata
    RootMetadata

TimestampMetadata:
  filename ...
  version  ...
  length   ...
  number of hashes ...
  hashes   ...

SnapshotMetadata:
  numberOfTargetRoleFiles  ...
  targetRoleFileInfos list(TargetRoleFileInfo)
  rootRoleFileInfo RootRoleFileInfo

TargetRoleFileInfo: 
  filename  ...
  version   ...

RootRoleFileInfo:
  filename  ...
  version   ...
  length    ...
  numberOfHashes ...
  hashes    ...

TargetsMetadata:
  ...

RootMetadata:
  ...

Better model (matches the structure of JSON metadata and doesn't include a CHOICE()):

TimestampSignable:
  sigs  list(Signatures)
  signed  Timestamp

Timestamp:
  type    ...
  version ...
  expires ...
  #data about Snapshot:
    filename ...
    version  ...
    length   ...
    number of hashes ...
    hashes   ...

SnapshotSignable:
  sigs   list(Signatures)
  signed Snapshot

Snapshot:
  type    ...   # <- not necessary. You need to already know it to get this far.
  version ...
  expires ...
  numberOfTargetRoleFiles  ...
  targetRoleFileInfos list(TargetRoleFileInfo)
  rootRoleFileInfo RootRoleFileInfo

TargetRoleFileInfo: 
  filename  ...
  version   ...

RootRoleFileInfo:
  filename  ...
  version   ...
  length    ...
  numberOfHashes ...
  hashes    ...

TargetsSignable:
  ...

Targets:
  ...

RootSignable:
  ...

Root:
  ...

@awwad
Copy link
Contributor Author

awwad commented Jan 28, 2019

This will be resolved by the switch to asn1crypto upcoming for the main TUF repo (theupdateframework/tuf), which Uptane will move to using instead of the forked awwad/tuf.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@awwad