ngclient: Test against other implementations

[jku]

It would be good to have some client testing against repositories produced with other tuf implementations.

I think there are two options in general (could choose one or both):

• copy a snapshot of a public production repository, store contents in python-tuf git with the tests
• test against a public production repository

The former would be an additional regression test -- get it working once and then it might prevent code regressions in future. Latter would be more interesting in that the metadata may change over time (as the metadata just has different variations but also as the other implementation changes), but ultimately it is an unreliable test for multiple reasons so couldn't be a blocking test in CI. I'm not sure yet which makes most sense.

Some possible repos to test against (I have not spoken to the maintainers of any of these repositories about this):

• bottlerocket repository (I assume built with AWSLabs Tough): https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.16/x86_64/timestamp.json
• sigstore repository (I assume built with go-tuf): https://github.com/sigstore/root-signing/tree/main/repository/repository
• Datadog repository (built with ?): https://dd-integrations-core-wheels-build-stable.datadoghq.com/metadata.staged/timestamp.json

As for what to test, a client refresh from initial root.json certainly makes sense. But if we test against a live repo we could also:

• test target download
• test incremental refresh (as in, local metadata is cached and used during next test)

[jku]

copy a snapshot of a public production repository, store contents in python-tuf git with the tests

I forgot to mention: testing like this would require mocking the current time to roughly the time the snapshot was made. Otherwise the metadata will start expiring sooner or later.

[jku]

Current state based on manually running python-tuf ngclient against some repositories:

• datadog works
• sigstore (go-tuf)
  • microseconds in expiry
  • timezone in expiry (fixed in 2.root.json)
  • ecdsa key format is bad
• bottlerocket (awslabs/tough)
  • microseconds in expiry

both might have more issues, this is just what they fail on now.