Building up E2E Tests for ORAS

[shizhMSFT]

Discussed in #485

^{Originally posted by qweeah August 9, 2022}

What is Needed

End-to-end tests that continuously validate the functional correctness of oras features.

Why it's Needed

• As the number of supported commands grows, it's impossible to manually validate all the features before releasing.
• Although we have unit tests for CLI, things might still go wrong after integration. Having E2E tests would at least cover high-value use cases and reduce regression.

How to

Setup Test Environment

The E2E tests should run against oras CLI binary with oras-distribution as the backend.

Choose Test Framework

Technically, there are two different directions of tools we use to build E2E test suite:

1) Use golang-native test framework to run test based on Cobra command execution calls

ginkgo + gomega

• Pros:
  • Based on golang standard testing package, which matches tech-stack of most active contributors.
  • No change in github actions.
  • Applied by many state-of-the-art CLI projects like kubectl and frp so we have a lot of existing good practices to learn from.
• Cons:
  • Use Cobra command execution call to mock user input, not 100% E2E.
  • Brings new dependencies in only for testing.
  • Won't be able to be reused by oras-py or other registry CLI tools sharing same UX as oras.

2) Use common Behaviour-Driven Development (BDD) tools to test the built out binary

Gherkin + godog

• Pros:
  • Testing oras as an CLI application, 100% E2E.
  • Idomatic BDD framework.
  • The latest cucumber binary can be downloaded during test time, which brings no depedency into oras.
  • Test cases(specs) can be reused by oras-py or other registry CLI tools sharing same UX as oras.
• Cons:
  • Extra one-time setup effort in Github actions.
  • Not Golang-native. (As far as I know) No expert in the current active contributors.

Both options provides expressive assertion that can meet oras' testing need.

Future Works
Below items are valuable but not in scope of this discussion:

• Benchmark Test & Profiling.
• E2E tests based on registry services from different cloud providers.

If you have any other ideas or suggestions on building the E2E test suite for oras, feel free to response below. Any comments are highly appreciated 🍻

[qweeah]

@shizhMSFT Can you assign this to me, thanks.

[qweeah]

The work items of E2E tests are listed out as below:

Infrastructure

• e2e: run distribution in github actions #552
• Sample Image/Artifacts data files
• e2e: prepare registry mount files for command specs #553
• support runnning e2e specs based on multiple distribution #646
• support configurable secret in e2e tests #680

Utility package for writing E2E tests

• e2e: add execution helper #554
• e2e: add matcher to validate output of a spec run #555
• e2e: utility for matching file content #700

Sample Tests (oras artifact tests will be after OCI artifact is supported)

• e2e: test oras commands before 0.15.0 #563
• e2e: test oras commands before 0.16.0 #564
• e2e: collect common use cases for scenario-based e2e test  #562
• Add sample tests for collected scenarios

Documentation

• e2e: add documentation for E2E testing #556