Building up E2E Tests for ORAS

[qweeah]

What is Needed

End-to-end tests that continuously validate the functional correctness of oras features.

Why it's Needed

• As the number of supported commands grows, it's impossible to manually validate all the features before releasing.
• Although we have unit tests for CLI, things might still go wrong after integration. Having E2E tests would at least cover high-value use cases and reduce regression.

How to

Setup Test Environment

The E2E tests should run against oras CLI binary with oras-distribution as the backend.

Choose Test Framework

Technically, there are two different directions of tools we use to build E2E test suite:

1) Use golang-native test framework to run test based on Cobra command execution calls

ginkgo + gomega

• Pros:
  • Based on golang standard testing package, which matches tech-stack of most active contributors.
  • No change in github actions.
  • Applied by many state-of-the-art CLI projects like kubectl and frp so we have a lot of existing good practices to learn from.
• Cons:
  • Use Cobra command execution call to mock user input, not 100% E2E. (update: Gomega supports starting external binaries)
  • Brings new dependencies in only for testing.
  • Won't be able to be reused by oras-py or other registry CLI tools sharing same UX as oras.

2) Use common Behaviour-Driven Development (BDD) tools to test the built out binary

Gherkin + godog

• Pros:
  • Idiomatic BDD framework.
  • The latest cucumber binary can be downloaded during test time, which brings no depedency into oras.
  • Test cases(specs) can be reused by oras-py or other registry CLI tools sharing same UX as oras.
• Cons:
  • Extra one-time setup effort in Github actions.
  • Not Golang-native. (As far as I know) No expert in the current active contributors.

Both options provides expressive assertion that can meet oras' testing need.

Future Works
Below items are valuable but not in scope of this discussion:

• Benchmark Test & Profiling.
• E2E tests based on registry services from different cloud providers.

If you have any other ideas or suggestions on building the E2E test suite for oras, feel free to response below. Any comments are highly appreciated 🍻

[qweeah]

Existing examples on using ginkgo+gomega for E2E testing.

K8S
Doc: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md
Code: https://github.com/kubernetes/kubernetes/tree/master/test/e2e

FRP
Code: https://github.com/fatedier/frp/tree/dev/test/e2e
cc @dtzar

[dtzar]

I put my vote in for going with option 1 using Ginkgo to enable BDD as seen by examples such as the following:

var _ = lifecycle.SIGDescribe("[Feature:BootstrapTokens]", func() {
  /* ... */
  ginkgo.It("should sign the new added bootstrap tokens", func() {
    /* ... */
  })
  /* etc */
})

I would argue that option 2 should be what you listed as an alternative godog which implements cucumber with golang.. Roll-your-own cucumber.io with golang I don't think makes sense.

I like the idea of having Gherkin language to define BDD versus the Gherkin-like lingo from Ginkgo. The risk with this is that you're taking a dep on godog and there likely will be gaps/updates and breaking changes (it's still not 1.0).

[enraiha0307]

Hi @qweeah
I am facing this error when I try to run e2e tests with makefile :

~ make teste2e                                      
./test/e2e/scripts/e2e.sh /Users/akku/Documents/Opensource/oras --clean 
 === installing ginkgo  === 
realpath: illegal option -- -
usage: realpath [-q] [path ...]
make: *** [teste2e] Error 1

I tried installing Ginkgo also then also the same error.
Do we have steps on how to setup/run e2e-tests?
I'm using mac air m2.