OPCT-5: support upgrade execution mode

[mtulio]

https://issues.redhat.com/browse/SPLAT-651

Support upgrade conformance.

• Introduces new flags to control whether the execution needs to run the upgrade cluster or not:
  • --mode=upgrade
  • --upgrade-to-image=<release_digest> ($(oc adm release info 4.Y+1.Z -o jsonpath={.image})
• Create config map with plugin variables (the sonobuoy native feature wipes all existing from podSpec which is undesired)
• add a new plugin instance of openshift-tests to run upgrades: 05-openshift-cluster-upgrade

Blocked by:

• SPLAT-869 - chore/manifests: rename plugins to be ordered by execution #31
• OPCT-6 - fix/rbac: manage custom ServiceAccount and roles #34

Blocked by Plugin release:

• OPCT-5 - plugin: add cluster-upgrade plugin provider-certification-plugins#24

Checklist:

• CLI changes to run in upgrade mode
• CLI changes to get the release image digest
• Plugin implementation: OPCT-5 - plugin: add cluster-upgrade plugin provider-certification-plugins#24
• Validate y-stream upgrades
• Fix RBAC OPCT-6 - fix/rbac: manage custom ServiceAccount and roles #34 for Cluster upgrade
• Fix SecurityContextMode for Sonobuoy aggregator stuck on 4.10->4.11 OPCT-7 - fix/securityContext: disable default SecurityContext created by Sonobuoy #39
• MachineConfigPool validation: 'opct' object is validated if present when running mode=upgrade on the runtime (plugin execution). Failures will be raised by the plugin when the MCP is not present (the User Docs should keep it very explicit): Tests described here: OPCT-5 - plugin: add cluster-upgrade plugin provider-certification-plugins#24 (comment)
• User Documentation

Tests checklist:

• upgrade 4.12-> 4.13

[mtulio]

Test results for upgrade 4.9->4.10, 4.10->4.11, 4.11->4.12.

Mode details / test matrix

Cluster upgrade finished in all tests. Sonobuoy got stuck on 4.10->4.11 - know issue OPCT-7

>> Results for test=/home/mtulio/opct/results/opct-49to410
Plugin: 05-openshift-cluster-upgrade
Status: failed
Total: 37
Passed: 36
Failed: 1
Skipped: 0
--
Plugin: 10-openshift-kube-conformance
Status: failed
Total: 428
Passed: 427
Failed: 1
Skipped: 0
--
Plugin: 20-openshift-conformance-validated
Status: failed
Total: 3149
Passed: 1217
Failed: 23
Skipped: 1909
--
Plugin: 99-openshift-artifacts-collector
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0
--
Run Details:
API Server version: v1.23.12+8a6bfe4
Node health: 7/7 (100%)

>> Results for test=opct-410to411
Plugin: 20-openshift-conformance-validated
Status: failed
Total: 1
Passed: 0
Failed: 1
Skipped: 0
--
Plugin: 99-openshift-artifacts-collector
Status: failed
Total: 1
Passed: 0
Failed: 1
Skipped: 0
--
Plugin: 05-openshift-cluster-upgrade
Status: failed
Total: 37
Passed: 36
Failed: 1
Skipped: 0
--
Plugin: 10-openshift-kube-conformance
Status: failed
Total: 1
Passed: 0
Failed: 1
Skipped: 0
--
Run Details:
API Server version: v1.24.6+5658434
Node health: 7/7 (100%)
Pods health: 212/212 (100%)

>> Results for test=opct-411to412
Plugin: 05-openshift-cluster-upgrade
Status: failed
Total: 38
Passed: 37
Failed: 1
Skipped: 0
--
Plugin: 10-openshift-kube-conformance
Status: failed
Total: 662
Passed: 657
Failed: 5
Skipped: 0
--
Plugin: 20-openshift-conformance-validated
Status: failed
Total: 3804
Passed: 1713
Failed: 41
Skipped: 2050
--
Plugin: 99-openshift-artifacts-collector
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0
--
Run Details:
API Server version: v1.25.4+77bec7a
Node health: 7/7 (100%)
Pods health: 219/220 (99%)

[mtulio]

Preview of total time installing, running upgrade 4.11->4.12 with OPCT, and destroying the cluster: 4:24:12

• Install cluster: 0:24:27
• Run (upgrade + conformance): 3:40:54
• Destroy cluster: 0:07:01
• Kube conformance failed ratio: 0.45%
• OCP Conformance failed ratio: 0.84%

CID=411to412
INSTALL_VERSION=4.11.19
UPGRADE_VERSION=4.12.0-rc.6
CLUSTER=opct-${CID}
WORKPATH=${PWD}/.opct-${CID}
UPGRADE_IMG="$(oc adm release info ${UPGRADE_VERSION} -o jsonpath={.image})"
$ podman run \
>     --env-file ${PWD}/.opct.env \
>     -v ${WORKPATH}/:/root/.ansible/okd-installer:Z \
>     -v ${HOME}/.ssh:/root/.ssh:Z \
>     -v ${HOME}/.openshift/pull-secret-latest.json:/pull-secret.json \
>     -v ${WORKPATH}-openshift-provider-cert:/openshift-provider-cert:Z \
>     --rm opct-runner:latest \
>         ansible-playbook opct-runner-all-aws.yaml \
>         -e cluster_name=$CLUSTER \
>         -e cluster_version=${INSTALL_VERSION} \
>         -e run_mode=upgrade \
>         -e opct_run_mode="--mode=upgrade" \
>         -e opct_run_args="--upgrade-to-image=\"${UPGRADE_IMG}\""

(...)
TASK [OKD Installer | Create all | Ending Summary] *****************************
ok: [localhost] => {
    "msg": [
        "start=[2023-01-10 05:45:08] end=[2023-01-10 06:09:35]",
        "total=[0:24:27]",
        "export KUBECONFIG=/root/.ansible/okd-installer/clusters/opct-411to412/auth/kubeconfig"
    ]
}
(...)
TASK [OPCT | Run Tool | Execution summary] *************************************
ok: [localhost] => {
    "msg": [
        "Job attempts=[]",
        "Job time delta=3:40:11.195498 start=2023-01-10 06:21:10.855608 end=2023-01-10 06:21:10.855608",
        "Job attempts=221",
        "Execution time: 3:40:54"
    ]
}
(...)
TASK [OKD Installer | Destroy | Timer end] *************************************
ok: [localhost] => {
    "msg": [
        "start=[2023-01-10 10:02:04] end=[2023-01-10 10:09:05]",
        "total=[0:07:01]"
    ]
}
(...)
TASK [OPCT | Runner All AWS | Ending Summary] **********************************
ok: [localhost] => {
    "msg": [
        "Execution time: 4:24:12"
    ]
}

PLAY RECAP *********************************************************************
localhost                  : ok=826  changed=101  unreachable=0    failed=0    skipped=233  rescued=0    ignored=2

$ ${WORKPATH}-openshift-provider-cert results ${WORKPATH}/clusters/$CLUSTER/opct/*.tar.gz |egrep ^'(Plugin:|Run)' -A 5
Plugin: 05-openshift-cluster-upgrade
Status: failed
Total: 38
Passed: 37
Failed: 1
Skipped: 0
--
Plugin: 10-openshift-kube-conformance
Status: failed
Total: 660
Passed: 657
Failed: 3
Skipped: 0
--
Plugin: 20-openshift-conformance-validated
Status: failed
Total: 3795
Passed: 1713
Failed: 32
Skipped: 2050
--
Plugin: 99-openshift-artifacts-collector
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0
--
Run Details:
API Server version: v1.25.4+77bec7a
Node health: 7/7 (100%)
Pods health: 229/229 (100%)

[mtulio]

@rvanderp3 this PR is ready for review. Ptal?
/assign @rvanderp3

[rvanderp3]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rvanderp3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rvanderp3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rvanderp3]

/hold

[mtulio]

Tested upgrades from 4.12 to 4.13 successfully (plugins v0.3.0-alpha0) :

$ $CLI_BIN_PATH version
OpenShift Provider Certification Tool: 0.0.0+40f5965
Sonobuoy Version: v0.56.10,

$ jq . .opct-4120/clusters/opct-4120-2023012601/opct/results/resources/ns/openshift-provider-certification/core_v1_pods.json  |grep 'quay.io/ocp-cert/openshift-tests-provider-cert:v' |uniq -c
     16             "image": "quay.io/ocp-cert/openshift-tests-provider-cert:v0.3.0-alpha0",

$ jq -r '.items[].status.history | map({"startTime": .startedTime , "completeTime": .completionTime, "version": .version}) ' .opct-4120/clusters/opct-4120-2023012601/opct/results/resources/cluster/config.openshift.io_v1_clusterversions.json 
[
  {
    "startTime": "2023-01-25T20:53:52Z",
    "completeTime": "2023-01-25T21:58:57Z",
    "version": "4.13.0-ec.2"
  },
  {
    "startTime": "2023-01-25T17:46:00Z",
    "completeTime": "2023-01-25T18:16:30Z",
    "version": "4.12.0"
  }
]

 $ ./openshift-provider-cert results .opct-4120/clusters/opct-4120-2023012601/opct/202301252051_sonobuoy_574e2b07-ed25-4e75-8a48-f910027afb04.tar.gz |grep ^Plugin -A 5
Plugin: 05-openshift-cluster-upgrade
Status: failed
Total: 38
Passed: 37
Failed: 1
Skipped: 0
--
Plugin: 10-openshift-kube-conformance
Status: failed
Total: 704
Passed: 684
Failed: 20
Skipped: 0
--
Plugin: 20-openshift-conformance-validated
Status: failed
Total: 3753
Passed: 1655
Failed: 45
Skipped: 2053
--
Plugin: 99-openshift-artifacts-collector
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0

[mtulio]

/unhold