Registry-support repository testing

[GeekArthur]

Is your feature request related to a problem? Please describe.
As the initial registry library PR has been merged, we need to consider how to test the registry on the OpenShift CI. Currently at least we need to test the registry library, but we may need to test more scenarios for registry testing.

Describe the solution you'd like
The registry testing should includes the following steps:

1. Login to a cluster (k8s/OpenShift) and create a new namespace for testing
2. Deploy the registry, currently we have three ways to deploy the registry and we need to test all of them
   • Install registry operator then use the registry operator to deploy the registry (instruction)
   • Use helm chart to deploy the registry (instruction will be updated after this PR is merged)
   • Use OpenShift template to deploy the registry (instruction will be updated after this PR is merged)
3. Test registry library against the registry
4. Delete registry, registry operator and the namespace for testing (Note: registry operator will be installed on registry-operator-system namespace, we may need to delete the registry-operator-system namespace as well)

[johnmcollier]

I recommend using Ginkgo for the tests. It was pretty straightforward writing them and getting them running on OpenShift CI.

See https://github.com/devfile/registry-operator/blob/main/tests/integration/pkg/tests/devfileregistry_tests.go and https://github.com/devfile/registry-operator/blob/main/.ci/openshift_integration.sh

CC @devfile/qe-team

[johnmcollier]

Any updates on this item?

[GeekArthur]

The new test cases need to test the bootstrapping processes of index server.

[johnmcollier]

Let's focus on Minikube / generic Kube first. Getting both the index server base image and resultant devflie registry index image built on the OpenShift CI seems overly complicated at this moment.

I think a decent flow would be the following in a Github action

1. Build the index server base image

2. Have test registry data that includes the following

   • Dockerfile (to build of course)
   • Simple stacks (each with literally just a devfile.yaml for now)
   • An extraDevfileEntries.yaml file with a couple sample entries

3. Build test registry image with the index server base image built in step 1. This will be the image we use to test both the index server and registry library.

4. Run a script that does the following

   a. Deploy registry using the Helm chart, with the index image built in step 3

   • Minimal overhead compared to operator.
   • Operator already has its own integration tests anyways

   b. Run index server and registry library tests against this registry.

   • Index server tests should validate that bootstrapping occurs and each endpoint returns correct data
   • Registry library tests should use the library to validate each of the core functions (index.json retrieval, stack pulling, etc)

[elsony]

What is the goal of this test? Do we want to use this test to make sure our production/staging registries is function and the registry library works well with them? I wonder if we need steps 1 to 3 to build with mock-up or selected simple stacks from scratch or just grabbing the output of the existing registry build and run #4 against it.

If we want to restrict this registry testing to purely testing the library functions using mock-up data, we'll still need to create another set of tests to test the actual production/staging registries. Although keeping them separate will help us to easily identify where the problem is in case of test failure. I wonder if it will reduce our maintenance costs on multiple testing frameworks and just have one to test them all.

[johnmcollier]

What is the goal of this test? Do we want to use this test to make sure our production/staging registries is function and the registry library works well with them? I wonder if we need steps 1 to 3 to build with mock-up or selected simple stacks from scratch or just grabbing the output of the existing registry build and run #4 against it.

So, I think we could use the staging/production registries to just test the devfile registry library, but I don't think that's sufficient here. For example, if a change was made to the index server that broke something on the devfile registry library (e.g. an endpoint was changed or removed, or returns different data now), that wouldn't be caught until after the change was delivered and promoted into staging (rather than during PR review).

So my thinking was that we build/test both the index server and registry library in the "registry tests" (maybe a better name for this issue is registry-support tests? 🤔 ). We'd test the various endpoints of the index server and also test the registry library, using the test index server we have set up. Then any changes/issues that affect both, would be caught.

Regarding the mock data, having the mock data also makes it easier to test new functions that haven't made it into the devfile/registry repository yet (like if we add new functionality to the registry build, or add a new file like extraDevfileEntry.yaml again).

[GeekArthur]

Agreed with John, I think the goal of this test is to test if the registry library in the PR can work with the registry in the PR properly (both registry library and registry source code are in the same registry-support repo), if we test against staging/production then the goal would be to test if the registry library in the PR can work with the public registry properly.

[johnmcollier]

Renaming this issue to clarify that it's for testing code that's in the registry-support repo