Building up notation E2E testing

[JeyJeyGao]

What is Needed

End-to-end (E2E) tests that continuously validate the functional correctness of notation features.

Why It's Needed

As the number of supported commands grows, it's impossible to manually validate all the features before releasing them.
Although we have unit tests for CLI, things might still go wrong after integration. Having E2E tests would at least cover high-value use cases and reduce regression.

Requirement

For testing framework:

1. testing notation binary with a local registry
2. may need to support multiple registries in the future
3. a GitHub action to trigger the E2E testing before merging the code

For test cases:

1. basic scenario for Quickstart (P0)
2. sign and verify commands and flags combinations (P0)
3. verify commands with different trustpolicy.json files, including a variety of arguments combinations (P0)
4. other commands with flags (P1)
5. sign and verify with a plugin (P2)

Details

E2E Testing Framework

Basic Framwork

ORAS E2E testing is a good example for us. Both the notation and oras need a local registry for the E2E test. However, there are some differences:

1. notation has complicated configurations needed to be tested, such as config.json, trustpolicy.json, localkeys.json, and truststore directory.
2. notation's sign and verify processes contain encryption and decryption operations which are CPU-intensive tasks, so the CPU consumption will be much higher than oras.

We have choices to solve those two issues.

• We do a sequential test, and the different configurations will be deployed for each test case.
• We can containerize our test cases and the configuration files will be isolated, so they can be parallel.

Everything else can follow oras E2E test design.

My idea is to do a sequential test for now because we only have about 50 test cases, which should be done in a few minutes. If we have too many test cases in the future, then we can optimize it to run in parallel.

Registry

We can use Zot registry to do the notation testing because it supports OCI Artifact and can be deployed locally.

E2E test cases

basic scenario for Quickstart (P0)

sign and verify commands and flags combinations (about 16 test cases)

1. Sign an OCI artifact using the default signing key, with the default JWS envelope:
   notation sign <registry>/<repository>@<digest>
2. Sign an OCI artifact using the default signing key, with the COSE envelope:
   notation sign --signature-format cose <registry>/<repository>@<digest>
3. Sign an OCI artifact using a specified key
   notation sign --key <key_name> <registry>/<repository>@<digest>
4. Sign an OCI artifact identified by a tag (Notation will resolve tag to digest)
   notation sign <registry>/<repository>:<tag>
5. Sign an OCI artifact stored in a registry and specify the signature expiry duration, for example, 24 hours
   notation sign --expiry 24h <registry>/<repository>@<digest>
6. Verify a signature on an OCI artifact identified by a digest:
   notation verify <registry>/<repository>@<digest>
7. Verify a signature on an OCI artifact identified by a tag (Notation will resolve tag to digest):
   notation verify <registry>/<repository>:<tag>
8. Check the keyword of the output of the sign and verify commands with --debug and --verbose flags
9. Sign and verify with/without username and password

Each sign testing needs a verify command to check the result
Each verify testing needs to sign first

verify commands with different trustpolicy.json files, including a variety of arguments combinations (about 10 test cases)

For each argument in trustpolicy, including name, registryScopes, signatureVerification, trustStores, trustedIdentities, generate valid and invalid values combinations and assert the result with predicted output.

other commands with flags (about 22 test cases)

1. certificate add valid and invalid cases
2. certificate delete valid and invalid cases
3. certificate list lists empty results and multiple results
4. certificate show valid and invalid cases
5. list lists empty results and multiple results
6. login $REGISTRY_ADDRESS -u -p
7. login $REGISTRY_ADDRESS
8. logout $REGISTRY_ADDRESS
9. key add valid and invalid cases
10. key delete valid and invalid cases
11. key list empty result and multiple results
12. key update valid and invalid cases
13. version

sign and verify with a plugin (P2)

We need a toy plugin for testing and it should support get-plugin-metadata, describe-key, generate-signature, generate-envelope, verify-signature commands. Also, support trust identity and revocation check for verification.

[yizha1]

@vaninrao10 @priteshbandi Please help to review it.

[yizha1]

@JeyJeyGao Very good summary.

• For the registry section, can we add support for multiple registries? Zot is the default one.
• How many cases do you think we can complete in rc.1?

[JeyJeyGao]

• Yes, we can add more registries but currently we don't have another registry supporting OCI Artifact.
• It will take a week to finish the framework part, another 2 weeks to finish P0 test cases. Also we need time for reviewing and buffering, so the framework and P0 test cases can be completed in rc.2.

[FeynmanZhou]

Registry
We can use Zot registry to do the notation testing because it supports OCI Artifact and can be deployed locally.

Just clarify the "multi-registry" consideration of E2E testing, Notary is being resigned to support multi-registry registry natively, so it would be portable and convenient that design the "registry" as a variable that we can easily replace it with others. It will be easy to test the Notation conformance with other registries not only the default Zot registry.

[rchincha]

oras-project/oras#709
^ could be along these lines

[JeyJeyGao]

Thanks! Got it.

[yizha1]

Thanks @JeyJeyGao Overall, it is a good e2e testing plan. I don't have any comments now.

[priteshbandi]

• I would make the sign and verify with a plugin as P1 and other commands with flags as P2. The reason being other commands won't be used in production and since notation only supports signing with plugin(not local key signing) for production usecase having plugin test case has a higher priority.
• We are missing cross platform(linux, windows, MacOS) tests (P3). Probably write tests in platform independent fashion and then in next iteration we can try to run test on windows and MacOS
• Parameterize registry references in tests for easy portability.
• Can you please summarize, whats done differently in ORAS E2E testing ?

We can containerize our test cases and the configuration files will be isolated, so they can be parallel.

Do we know what's required to achieve this? We can surely start with sequential and then move to parallel but we should write tests in such a way that it should take minimum effort to parallelize them

Each verify testing needs to sign first

nit: We can sign once and then use verify multiple times

verify commands with different trustpolicy.json files, including a variety of arguments combinations (about 10 test cases)

Can you please list this testcases? I can think of more than 10 scenarios.
name - valid / invalid
registryScopes - scoped / star / invalid
signatureVerification -
trustStores - single cert trust store / multi cert trust store / multiple trust store / no trust store / siningAuthority ts / ca trust store
trustedIdentities - vaid / invalid / valid but not trusted

[JeyJeyGao]

Thanks for your valuable suggestions.

1. Yes, we can make the Plugin test as P1;
2. Good idea, I will consider it.
3. We may only use one specific registry because we need to prepare image/artifact and signatures for some test cases and they should be the same for each time the test cases run. That is to say, those underlying registry storage structures should be saved as files and kept in our repo, so the registry will have the same data each time it starts. Zot supports OCI layout, which can be used to save the registry data as files. If we want to make sure the best test effectiveness, we should do it in this way and give up portability.
4. In the ORAS E2E test, there is no file system isolation for each test case. However, notation needs configuration files, so many test cases need different config files., so we need to do file system isolation if we want the test cases running in parallel.
5. We can sign once and then use verify multiple times. Yes. And some signatures may need to be saved in the repo and keep it the same each time.
6. I will provide details trustpolicy test cases later.