Skip to content

Commit 8200935

Browse files
camilamacedo86tmshort
camilamacedo86
authored and
tmshort
committed
UPSTREAM: <carry>: [OTE] Add Readme
1 parent 31bd723 commit 8200935

File tree

1 file changed

+155
-0
lines changed

1 file changed

+155
-0
lines changed

‎openshift/tests-extension/README.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# OLMv1 Tests Extension
2+
========================
3+
4+
This repository contains the OLMv1 tests for OpenShift.
5+
These tests run against OpenShift clusters and are meant to be used in the OpenShift CI/CD pipeline.
6+
They use the framework: https://github.com/openshift-eng/openshift-tests-extension
7+
8+
## How to Run the Tests Locally
9+
10+
| Command | Description |
11+
|-------------------------------------------------|--------------------------------------------------------------------------|
12+
| `make build` | Builds the OLMv1 test binary. |
13+
| `./bin/olmv1-tests-ext info` | Shows info about the test binary and registered test suites. |
14+
| `./bin/olmv1-tests-ext list` | Lists all available test cases. |
15+
| `./bin/olmv1-tests-ext run-suite olmv1/all` | Runs the full OLMv1 test suite. |
16+
| `./bin/olmv1-tests-ext run-test -n <test-name>` | Runs one specific test. Replace <test-name> with the test's full name. |
17+
18+
## Development Workflow
19+
20+
- Add or update tests in: `openshift/tests-extension/tests/`
21+
- Run `make build` to build the test binary
22+
- You can run the full suite or one test using the commands in the table above
23+
- Before committing your changes:
24+
- Run `make update-metadata` or `make build-update`
25+
- Run `make verify` to check formatting, linting, and validation
26+
27+
## How to Rename a Test
28+
29+
1. Run `make list-test-names` to see the current test names
30+
2. Find the name of the test you want to rename
31+
3. Add a Ginkgo label with the original name, like this:
32+
33+
```go
34+
It("should pass a renamed sanity check",
35+
Label("original-name:[sig-olmv1] OLMv1 should pass a trivial sanity check"),
36+
func(ctx context.Context) {
37+
Expect(len("test")).To(BeNumerically(">", 0))
38+
})
39+
```
40+
41+
4. Run `make build-update` to update the metadata
42+
43+
**Note:** Only add the label once. Do not update it again after future renames.
44+
45+
## How to Delete a Test
46+
47+
1. Run `make list-test-names` to find the test name
48+
2. Add the test name to the `IgnoreObsoleteTests` block in `main.go`, like this:
49+
50+
```go
51+
ext.IgnoreObsoleteTests(
52+
"[sig-olmv1] My removed test name",
53+
)
54+
```
55+
56+
3. Delete the test code from your suite (like `olmv1.go`)
57+
4. Run `make build-update` to clean the metadata
58+
59+
**WARNING**: Deleting a test may cause issues with Sippy https://sippy.dptools.openshift.org/sippy-ng/
60+
or other tools that expected the Unique TestID tracked outside of this repository. [More info](https://github.com/openshift-eng/ci-test-mapping)
61+
Check the status of https://issues.redhat.com/browse/TRT-2208 before proceeding with test deletions.
62+
63+
## E2E Test Configuration
64+
65+
Tests are configured in: [ci-operator/config/openshift/operator-framework-operator-controller](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/operator-framework-operator-controller/)
66+
67+
Here is a CI job example:
68+
69+
```yaml
70+
- as: e2e-aws-techpreview-olmv1-ext
71+
steps:
72+
cluster_profile: aws
73+
env:
74+
FEATURE_SET: TechPreviewNoUpgrade
75+
76+
# Only enable 'watch-namespaces' monitor to avoid job failures from other default monitors
77+
# in openshift-tests (like apiserver checks, alert summaries, etc). In this job, the selected
78+
# OLMv1 test passed, but the job failed because a default monitor failed.
79+
#
80+
# 'watch-namespaces' is very lightweight and rarely fails, so it's a safe choice.
81+
# There is no way to fully disable all monitors, but we can use this option to reduce noise.
82+
#
83+
# See: ./openshift-tests run --help (option: --monitor)
84+
TEST_ARGS: --monitor=watch-namespaces
85+
86+
TEST_SUITE: olmv1/all
87+
test:
88+
- ref: openshift-e2e-test
89+
workflow: openshift-e2e-aws
90+
```
91+
92+
This uses the `openshift-tests` binary to run OLMv1 tests against a test OpenShift release.
93+
94+
It works for pull request testing because of this:
95+
96+
```yaml
97+
releases:
98+
latest:
99+
integration:
100+
include_built_images: true
101+
```
102+
103+
More info: https://docs.ci.openshift.org/docs/architecture/ci-operator/#testing-with-an-ephemeral-openshift-release
104+
105+
## Makefile Commands
106+
107+
| Target | Description |
108+
|--------------------------|------------------------------------------------------------------------------|
109+
| `make build` | Builds the test binary. |
110+
| `make update-metadata` | Updates the metadata JSON file. |
111+
| `make build-update` | Runs build + update-metadata + cleans codeLocations. |
112+
| `make verify` | Runs formatting, vet, and linter. |
113+
| `make list-test-names` | Shows all test names in the binary. |
114+
| `make clean-metadata` | Removes machine-specific codeLocations from the JSON metadata. [More info](https://issues.redhat.com/browse/TRT-2186) |
115+
116+
**Note:** Metadata is stored in: `.openshift-tests-extension/openshift_payload_olmv1.json`
117+
118+
## FAQ
119+
120+
### Why don't we have a Dockerfile for `olmv1-tests-ext`?
121+
122+
We do not provide a Dockerfile for `olmv1-tests-ext` because building and shipping a
123+
standalone image for this test binary would introduce unnecessary complexity.
124+
125+
Technically, it is possible to create a new OpenShift component just for the
126+
OLMv1 tests and add a corresponding test image to the payload. However, doing so requires
127+
onboarding a new component, setting up build pipelines, and maintaining image promotion
128+
and test configuration — all of which adds overhead.
129+
130+
From the OpenShift architecture point of view:
131+
132+
1. Tests for payload components are part of the product. Many users (such as storage vendors, or third-party CNIs)
133+
rely on these tests to validate that their solutions are compatible and conformant with OpenShift.
134+
135+
2. Adding new images to the payload comes with significant overhead and cost.
136+
It is generally preferred to include tests in the same image as the component
137+
being tested whenever possible.
138+
139+
### Why do we need to run `make update-metadata`?
140+
141+
Running `make update-metadata` ensures that each test gets a unique and stable **TestID** over time.
142+
143+
The TestID is used to identify tests across the OpenShift CI/CD pipeline and reporting tools like Sippy.
144+
It helps track test results, detect regressions, and ensures the correct tests are
145+
executed and reported.
146+
147+
This step is important whenever you add, rename, or delete a test.
148+
More information:
149+
- https://github.com/openshift/enhancements/blob/master/enhancements/testing/openshift-tests-extension.md#test-id
150+
- https://github.com/openshift-eng/ci-test-mapping
151+
152+
### How to get help with OTE?
153+
154+
For help with the OpenShift Tests Extension (OTE), you can:
155+
#wg-openshift-tests-extension

0 commit comments

Comments
 (0)