Skip to content

Commit

Permalink
OPCT-196: Update documentation removing certification references (#58)
Browse files Browse the repository at this point in the history
https://issues.redhat.com/browse/OPCT-196

Renaming the tool name to remove 'certification' references that match
it's purpose.

- Phase I: Documentation updates

---------

Co-authored-by: Richard Vanderpool <49568690+rvanderp3@users.noreply.github.com>
  • Loading branch information
mtulio and rvanderp3 committed Apr 11, 2023
1 parent e690181 commit dd1b688
Show file tree
Hide file tree
Showing 9 changed files with 114 additions and 117 deletions.
4 changes: 2 additions & 2 deletions docs/FAQ.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# FAQ for OpenShift Provider Certification Tool
# FAQ for OPCT

**Q: What is the OpenShift Provider Certification Tool (OPCT) ?**
**Q: What is the OpenShift Provider Compatibility Tool (OPCT) ?**

OPCT was designed to validate the custom OpenShift installations in non-integrated providers (a.k.a [agnostic installation](https://docs.openshift.com/container-platform/4.11/installing/installing_platform_agnostic/installing-platform-agnostic.html) with `platform.none:{}` option for `install-config.yaml`), **running existing suite of e2e tests** from Kubernetes and OpenShift ecosystem.

Expand Down
10 changes: 5 additions & 5 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# OpenShift Provider Certification Tool
# OpenShift Provider Compatibility Tool

Welcome to the documentation for the OpenShift Provider Certification Tool!
Welcome to the documentation for the OpenShift Provider Compatibility Tool (OPCT)!

OpenShift Provider Certification Tool is used to evaluate an OpenShift installation on a provider or hardware is in conformance.
OPCT is used to evaluate an OpenShift installation on a provider or hardware is in conformance.

Here you can find the initial steps to use the OpenShift Provider Certification Tool.
Here you can find the initial steps to use the tool:

- [User Guide](./user.md)
- [Installation Check List](./user-installation-checklist.md)
- [Installation Review](./user-installation-review.md)
- [Results Review](./user-results-review.md)
- [Support Guide](./support-guide.md)
- [Development Guide](./dev.md)
- [Development Guide](./dev.md)
39 changes: 20 additions & 19 deletions docs/dev.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Provider Certification Tool - Developer Guide
# Developer Guide

This document is a guide for developers detailing the Provider Certification Tool solution, design choices and the implementation references.
This document is a guide for developers detailing the OPCT solution, design choices and the implementation references.

Table of Contents:

Expand All @@ -11,13 +11,13 @@ Table of Contents:
- [Sonobuoy Plugins](#dev-sonobuoy-plugins)
- [Diagrams](#dev-diagrams)
- [CLI Result filters](#dev-diagram-filters)
- [Running Customized Certification Plugins](#dev-running-custom-plugins)
- [Running Customized Plugins](#dev-running-custom-plugins)
- [Project Documentation](#dev-project-docs)

## Release <a name="release"></a>

Releasing a new version of the provider certification tool is done automatically through [this GitHub Action](https://github.com/redhat-openshift-ecosystem/provider-certification-tool/blob/main/.github/workflows/release.yaml)
which is run on new tags. Tags should the SemVer format. Example: v0.1.0, v0.1.0-alpha1 (...)
Releasing a new version of the tool is done automatically through [this GitHub Action](https://github.com/redhat-openshift-ecosystem/provider-certification-tool/blob/main/.github/workflows/release.yaml)
which is run on new tags. Tags should follow the SemVer format. Example: v0.1.0, v0.1.0-alpha1 (...)

Tags should only be created from the `main` branch which only accepts pull-requests that pass through [this CI GitHub Action](https://github.com/redhat-openshift-ecosystem/provider-certification-tool/blob/main/.github/workflows/go.yaml).

Expand All @@ -32,7 +32,7 @@ This tool builds heavily on
some high level knowledge of Sonobuoy is needed to really understand this tool. A
good place to start with Sonobuoy is [its documentation](https://sonobuoy.io/docs).

The OpenShift provider certification tool extends Sonobuoy in two places:
OPCT extends Sonobuoy in two places:

- Command line interface (CLI)
- Plugins
Expand All @@ -41,18 +41,19 @@ The OpenShift provider certification tool extends Sonobuoy in two places:

Sonobuoy provides its own CLI but it has a considerable number of flags and options
which can be overwhelming. This isn't an issue with Sonobuoy, it's just the result
of being a very flexible tool. However, for simplicity sake, the OpenShift
certification tool extends the Sonobuoy CLI with some strong opinions specific
to the realm certifying OpenShift on new infrastructure.
of being a very flexible tool. However, for simplicity sake, the OPCT extends
Sonobuoy CLI with some strong opinions specific to the realm validating
OpenShift on new infrastructure.

#### Integration with Sonobuoy CLI <a name="dev-integration-cli"></a>
The OpenShift provider certification tool's CLI is written in Golang so that extending
Sonobuoy is easily done. Sonobuoy has two specific areas on which we build on:

The OPCT's CLI is written in Golang so that extending Sonobuoy is easily done.
Sonobuoy has two specific areas on which we build on:

- Cobra commands (e.g. [sonobuoy run](https://github.com/vmware-tanzu/sonobuoy/blob/87e26ab7d2113bd32832a7bd70c2553ec31b2c2e/cmd/sonobuoy/app/run.go#L47-L62))
- Sonobuoy Client ([source code](https://github.com/vmware-tanzu/sonobuoy/blob/87e26ab7d2113bd32832a7bd70c2553ec31b2c2e/pkg/client/interfaces.go#L246-L250))

Ideally, the OpenShift Provider Cert tool's commands will interact with the Sonobuoy Client API. There may be some
Ideally, the OPCT's commands will interact with the Sonobuoy Client API. There may be some
situations where this isn't possible and you will need to call a Sonobuoy's Cobra Command directly. Keep in mind,
executing a Cobra Command directly adds some odd interaction; this should be avoided since the ability to cleanly \
set Sonobuoy's flags may be unsafe in code like below. The code below won't fail at compile time if there's a change
Expand Down Expand Up @@ -91,7 +92,7 @@ The CLI currently implements a few filters to help the reviewers (Partners, Supp
- A. `"Provider's Result"`: This is the original list of failures by the plugin available on the command `results`
- B. `"Suite List"`: This is the list of e2e tests available on the respective suite. For example: plugin `openshift-kubernetes-conformance` uses the suite `kubernetes/conformance`
- C. `"Baseline's Result"`: This is the list of e2e tests that failed in the baseline provider. That list is built from the same Certification Environment (OCP Agnostic Installation) in a known/supported platform (for example AWS and vSphere). Red Hat has many teams dedicated to reviewing and improving the thousands of e2e tests running in CI, that list is constantly reviewed for improvement to decrease the number of false negatives and help to look for the root cause.
- D. `"Sippy"`: Sippy is the system used to extract insights from the CI jobs. It can provide individual e2e test statistics of failures across the entire CI ecosystem, providing one picture of the failures happening in the provider's environment. The filter will check for each failed e2e if has an occurrence of failures in the version used to be certified.
- D. `"Sippy"`: Sippy is the system used to extract insights from the CI jobs. It can provide individual e2e test statistics of failures across the entire CI ecosystem, providing one picture of the failures happening in the provider's environment. The filter will check for each failed e2e if has an occurrence of failures in the version used to be validated.

Currently, this is the order of filters used to show the failures on the `process` command:

Expand All @@ -110,13 +111,13 @@ The diagram visualizing the filters is available on draw.io, stored on the share
- https://app.diagrams.net/#G1NOhcF3jJtE1MjWCtbVgLEeD24oKr3IGa


### Running Customized Certification Plugins <a name="dev-running-custom-plugins"></a>
### Running Customized Plugins <a name="dev-running-custom-plugins"></a>

In some situations, you may need to modify the certification plugins that are run by the certification tool.
Running the certification tool with customized plugin manifests cannot be used for final certification of an OpenShift cluster!
If you find issues or changes that are needed for certification to complete, please open a GitHub issue or reach out to your Red Hat contact assisting with certification.
In some situations, you may need to modify the plugins that are run by the OPCT.
Running the OPCT with customized plugin manifests cannot be used for final validation of an OpenShift cluster!
If you find issues or changes that are needed to complete, please open a GitHub issue or reach out to your Red Hat contact assisting with validation process.

1. Export default certification plugins to local filesystem:
1. Export default plugins to local filesystem:
```
openshift-provider-cert assets /tmp
INFO[2022-06-16T15:35:29-06:00] Asset openshift-conformance-validated.yaml saved to /tmp/openshift-conformance-validated.yaml
Expand All @@ -126,7 +127,7 @@ INFO[2022-06-16T15:35:29-06:00] Asset openshift-kube-conformance.yaml saved to /
```
vi /tmp/openshift-kube-conformance.yaml
```
3. Launch certification tool with customized plugin:
3. Launch the tool with customized plugin:
```
openshift-provider-cert run --plugin /tmp/openshift-kube-conformance.yaml --plugin /tmp/openshift-conformance-validated.yaml
```
Expand Down
26 changes: 13 additions & 13 deletions docs/support-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,20 +22,20 @@ Check-list to require when **new** support case has been opened:

- Documentation: Installing Steps containing the flavors/size of the Infrastructure and the steps to install OCP
- Documentation: Diagram of the Architecture including zonal deployment
- Archive with Certification results
- Archive with Conformance results
- Archive with must-gather
- [Installation Checklist (file `user-installation-checklist.md`)](./user-installation-checklist.md) with the partner's update to sign off post-instalation items

### New Executions <a name="check-list-new-executions"></a>

The following assets, certification assets, should be updated when certain conditions happen:
The assets below, conformance assets, should be updated when certain conditions happen:

- Certification Results
- Conformance Results
- Must Gather
- Install Documentation (when any item/flavor/configuration has been modified)


The following conditions require new certification assets:
The following conditions require new conformance assets:

- The version of the OpenShift Container Platform has been updated
- Any Infrastructure component(s) (e.g.: server size, disk category, ELB type/size/config) or cluster dependencies (e.g.: external storage backend for image registry) have been modified
Expand All @@ -45,17 +45,17 @@ The following conditions require new certification assets:

### Install Tools <a name="setup-install"></a>

- Download the [openshift-provider-cert](./user.md#install): OpenShift Provider Certification tool
- Download the [openshift-provider-cert](./user.md#install): OPCT
- Download the [`omg`](https://github.com/kxr/o-must-gather): tool to analyse Must-gather archive
```bash
pip3 install o-must-gather --user
```

### Download Baseline CI results <a name="setup-download-baseline"></a>

The Openshift provider certification tool is run periodically ([source code](https://github.com/openshift/release/blob/master/ci-operator/jobs/redhat-openshift-ecosystem/provider-certification-tool/redhat-openshift-ecosystem-provider-certification-tool-main-periodics.yaml)) in OpenShift CI using the latest stable release of OpenShift.
The OPCT run periodically ([source code](https://github.com/openshift/release/blob/master/ci-operator/jobs/redhat-openshift-ecosystem/provider-certification-tool/redhat-openshift-ecosystem-provider-certification-tool-main-periodics.yaml)) in OpenShift CI using the latest stable release of OpenShift.
These baseline results are stored long-term in an AWS S3 bucket (`s3://openshift-provider-certification/baseline-results`). An HTML listing can be found [here](https://openshift-provider-certification.s3.us-west-2.amazonaws.com/index.html).
These baseline results should be used as a reference when reviewing a partner's certification results.
These baseline results should be used as a reference when reviewing a partner's conformance results.

1. Identify cluster version in the partner's must gather:
```bash
Expand All @@ -73,7 +73,7 @@ $ file 4.11.13-20221125.tar.gz

### Download Partner Results <a name="setup-download-results"></a>

- Download the Provider certification archive from the Support Case. Example file name: `retrieved-archive.tar.gz`
- Download the conformance archive from the Support Case. Example file name: `retrieved-archive.tar.gz`
- Download the Must-gather from the Support Case. Example file name: `must-gather.tar.gz`

## Review guide: exploring the failed tests <a name="review-process"></a>
Expand All @@ -87,7 +87,7 @@ Required to use this section:
- OPCT CLI downloaded to the current directory
- OpenShift e2e test suite exported to the current directory
- Baseline results exported to the current directory
- The Certification Result is in the current directory
- The conformance result is in the current directory


### Exploring the failures <a name="review-process-exploring"></a>
Expand Down Expand Up @@ -276,7 +276,7 @@ This section is a guide of the initial files to review when start exploring the

Items to review:

- OCP version matches the certification request
- OCP version matches the ticket request
- Review the result file
- Check if the failures are 0, if not, need to check one by one
- To provide a better interaction between the review process, one spreadsheet named `failures-index.xlsx` is created inside the extracted directory (`./processed/` exemplified in the last section). It can be used as a tool to review failures and take notes about them.
Expand All @@ -290,7 +290,7 @@ Additional items to review:
## Review Guide: Manual Y-Stream Upgrade <a name="upgrade-review-process"></a>

Certification requires a successful y-stream upgrade (e.g. upgrade 4.11.17 to 4.12.0).
The validation process (when applyging for the Partner Support Case) requires a successful y-stream upgrade (e.g. upgrade 4.11.17 to 4.12.0).
Upgrade review should only proceed if there is reasonably high confidence in passing and not if there are still significant issues in passing the review process above.

> TODO: Rework this documentation after the automated upgrade procedure is merged in https://github.com/redhat-openshift-ecosystem/provider-certification-tool/pull/33
Expand All @@ -315,12 +315,12 @@ Is the manual intervention...
- Working around an issue specific to the cloud provider?

If the answer to any of the questions above is "Yes" then take the necessary steps to remediate the situation (if needed) through
documentation, bug reports, and escalations to meet the certification timeline.
documentation, bug reports, and escalations to meet the Validation requirements.

After a successful upgrade where any manual interventions aren't a blocker, review the Must Gather that was captured.
First, check the `ClusterVersion` resource to verify the upgrade was successful:

Using the `omg` tool...
Using the `omg` tool:

```
omg get clusterversion
Expand Down

0 comments on commit dd1b688

Please sign in to comment.