Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop k8s <= v1.20.x #643

Merged
merged 6 commits into from
Jan 29, 2024
Merged

Drop k8s <= v1.20.x #643

merged 6 commits into from
Jan 29, 2024

Conversation

garloff
Copy link
Contributor

@garloff garloff commented Nov 11, 2023

k8s:

  • v1.28.1 -> v1.28.3
  • v1.27.5 -> v1.27.7
  • v1.26.8 -> v1.26.10
  • v1.25.13 -> v1.25.15
    Drop versions v1.19.x and v1.18.x

occm/csi:

  • v1.28.0 -> v1.28.1
  • v1.27.2 -> v1.27.3
  • v1.26.3 -> v1.26.4
    Also drop stuff prior to v1.20.

Edit:

  • No versions bump - it was done in Bump latest versions of k8s and occm: #664
  • Drop k8s <= v1.20 - pretty old + no available images from osism
  • Remove min_snapshot_master logic - no more need as the logic was for k8s < v1.21.0
kubernetes_version = "v1.27.x"

k8s:
* v1.28.1 -> v1.28.3
* v1.27.5 -> v1.27.7
* v1.26.8 -> v1.26.10
* v1.25.13 -> v1.25.15
Drop versions v1.19.x and v1.18.x

occm/csi:
* v1.28.0 -> v1.28.1
* v1.27.2 -> v1.27.3
* v1.26.3 -> v1.26.4
Also drop stuff prior to v1.20.

Signed-off-by: Kurt Garloff <kurt@garloff.de>
@garloff garloff added enhancement New feature or request Container Issues or pull requests relevant for Team 2: Container Infra and Tooling labels Nov 11, 2023
@garloff
Copy link
Contributor Author

garloff commented Nov 11, 2023

Testing remarks:

  • I have tested v1.27.7 after a rolling upgrade from 1.27.5. (Without upgrading occm.)
  • All others still need testing.
  • Does our zuul test infra somehow cover this?

Backporting remark:

  • This should go into the maintained branches v6.0.x and v6.x IMVHO (maybe without dropping 1.19 and 1.18 for 6.0.x?)

@berendt
Copy link
Member

berendt commented Nov 11, 2023

Doesn't it make sense to automate this using Renovate/CI?

In osism/k8s-capi-images with use https://github.com/osism/k8s-capi-images/blob/main/src/update-extra-vars.py for this task. Retrieves the current versions and then updates them in the corresponding files.

@chess-knight
Copy link
Member

chess-knight commented Nov 14, 2023

Does our zuul test infra somehow cover this?

I manually added kubernetes_version = "v1.26.x" (default is "v1.27.x" - so latest 1.27 can be tested) to the PR description according to this doc and added e2e-quick-test label to this PR. Then, when you check #643 (comment), you can see Custom ENV config section there. Now, it is possible to edit the PR description and run the e2e test again with the new custom config.

According to the #433, #577 should do it automatically.

@scszuulapp
Copy link

scszuulapp bot commented Nov 14, 2023

Build succeeded (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/54e244af4be94d708f609daf27fb60cd

✔️ k8s-cluster-api-provider-e2e-quick SUCCESS in 30m 22s

Warning:

Sonobouy results === Collecting results === time="2023-11-14T15:29:38Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy time="2023-11-14T15:29:38Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[sonobuoy-serviceaccount-sonobuoy]" time="2023-11-14T15:29:38Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[sonobuoy-serviceaccount-sonobuoy]" Plugin: systemd-logs Status: passed Total: 4 Passed: 4 Failed: 0 Skipped: 0

Plugin: e2e
Status: passed
Total: 7072
Passed: 4
Failed: 0
Skipped: 7068

Run Details:
API Server version: v1.26.10
Node health: 4/4 (100%)
Pods health: 33/33 (100%)
Errors detected in files:
Errors:
20 podlogs/kube-system/kube-controller-manager-pr643-3b0940-pr643-3b0940-control-plane-genc01-2qw22/logs/kube-controller-manager.txt
17 podlogs/kube-system/kube-apiserver-pr643-3b0940-pr643-3b0940-control-plane-genc01-2qw22/logs/kube-apiserver.txt
11 podlogs/kube-system/cilium-5ttfq/logs/cilium-agent.txt
11 podlogs/kube-system/cilium-b47w2/logs/cilium-agent.txt
11 podlogs/kube-system/snapshot-controller-54785b5645-vlgsk/logs/snapshot-controller.txt
9 podlogs/kube-system/cilium-6wh6w/logs/cilium-agent.txt
8 podlogs/kube-system/cilium-l6tzp/logs/cilium-agent.txt
6 podlogs/kube-system/openstack-cloud-controller-manager-x7pzm/logs/openstack-cloud-controller-manager.txt
3 podlogs/sonobuoy/sonobuoy-e2e-job-07ec1edfe4ab4d67/logs/e2e.txt
2 podlogs/kube-system/cilium-operator-5c4f875f75-xbcnx/logs/cilium-operator.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-jlxhs/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-66dk7/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-9d7tr/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-d8q4l/logs/node-driver-registrar.txt
Warnings:
483 podlogs/kube-system/etcd-pr643-3b0940-pr643-3b0940-control-plane-genc01-2qw22/logs/etcd.txt
34 podlogs/kube-system/kube-apiserver-pr643-3b0940-pr643-3b0940-control-plane-genc01-2qw22/logs/kube-apiserver.txt
18 podlogs/kube-system/kube-controller-manager-pr643-3b0940-pr643-3b0940-control-plane-genc01-2qw22/logs/kube-controller-manager.txt
8 podlogs/kube-system/openstack-cloud-controller-manager-x7pzm/logs/openstack-cloud-controller-manager.txt
7 podlogs/kube-system/cilium-5ttfq/logs/cilium-agent.txt
7 podlogs/kube-system/cilium-b47w2/logs/cilium-agent.txt
5 podlogs/kube-system/cilium-6wh6w/logs/cilium-agent.txt
4 podlogs/kube-system/cilium-l6tzp/logs/cilium-agent.txt
3 podlogs/kube-system/csi-cinder-nodeplugin-jlxhs/logs/node-driver-registrar.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-d8q4l/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-9d7tr/logs/liveness-probe.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-9d7tr/logs/node-driver-registrar.txt
1 podlogs/sonobuoy/sonobuoy-e2e-job-07ec1edfe4ab4d67/logs/e2e.txt
1 podlogs/sonobuoy/sonobuoy/logs/kube-sonobuoy.txt
1 podlogs/kube-system/csi-cinder-controllerplugin-664f87bf97-9d5jl/logs/csi-provisioner.txt
1 podlogs/kube-system/cilium-operator-5c4f875f75-xbcnx/logs/cilium-operator.txt
1 podlogs/kube-system/csi-cinder-controllerplugin-664f87bf97-9d5jl/logs/csi-attacher.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-66dk7/logs/liveness-probe.txt
1 podlogs/kube-system/metrics-server-d5f8d66cd-tz2fx/logs/metrics-server.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-66dk7/logs/node-driver-registrar.txt
time="2023-11-14T15:29:38Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy
time="2023-11-14T15:29:38Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[]"
time="2023-11-14T15:29:38Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[]"

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 5 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 4 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2023-11-14 15:29:43 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...
...
...

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.
=== Sonobuoy conformance tests passed in 74s ===
make[1]: Leaving directory '/home/ubuntu/src/github.com/SovereignCloudStack/k8s-cluster-api-provider/terraform'

Custom ENV config

kubernetes_version = "v1.26.x"

Copy link
Contributor

@NotTheEvilOne NotTheEvilOne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also remove support for k8s < 1.26? Based on "SCS K8S Version Policy" it is RECOMMENDED to not support versions after this period in order to not encourage usage of out-of-date versions. [1]

[1] https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0210-v2-k8s-version-policy.md

# Conflicts:
#	terraform/files/bin/openstack-kube-versions.inc
Signed-off-by: Roman Hros <roman.hros@dnation.cloud>
Always download 'master'

Signed-off-by: Roman Hros <roman.hros@dnation.cloud>
@chess-knight chess-knight changed the title Bump latest versions of k8s and occm: Drop k8s <= v1.20.x Jan 15, 2024
@chess-knight
Copy link
Member

Hello @garloff, @NotTheEvilOne
I reworked this PR as stated in #664 (comment).


Shouldn't we also remove support for k8s < 1.26? Based on "SCS K8S Version Policy" it is RECOMMENDED to not support versions after this period in order to not encourage usage of out-of-date versions. [1]

[1] https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0210-v2-k8s-version-policy.md

I think this is reasonable and can be done in the future PR for sure

@scszuulapp
Copy link

scszuulapp bot commented Jan 15, 2024

Build succeeded (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/a137afb296c5484e8884b30baf2a1b9a

✔️ k8s-cluster-api-provider-e2e-quick SUCCESS in 20m 11s

Warning:

Sonobouy results === Collecting results === time="2024-01-15T08:56:38Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy time="2024-01-15T08:56:38Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[sonobuoy-serviceaccount-sonobuoy]" time="2024-01-15T08:56:38Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[sonobuoy-serviceaccount-sonobuoy]" Plugin: e2e Status: passed Total: 7072 Passed: 4 Failed: 0 Skipped: 7068

Plugin: systemd-logs
Status: passed
Total: 4
Passed: 4
Failed: 0
Skipped: 0

Run Details:
API Server version: v1.26.11
Node health: 4/4 (100%)
Pods health: 33/33 (100%)
Errors detected in files:
Errors:
91 podlogs/kube-system/kube-scheduler-pr643-33ee32-control-plane-5tk9v-db82h/logs/kube-scheduler.txt
19 podlogs/kube-system/kube-controller-manager-pr643-33ee32-control-plane-5tk9v-db82h/logs/kube-controller-manager.txt
13 podlogs/kube-system/snapshot-controller-54785b5645-cx4ck/logs/snapshot-controller.txt
10 podlogs/kube-system/openstack-cloud-controller-manager-vlxb7/logs/openstack-cloud-controller-manager.txt
10 podlogs/kube-system/kube-apiserver-pr643-33ee32-control-plane-5tk9v-db82h/logs/kube-apiserver.txt
9 podlogs/kube-system/cilium-64mrb/logs/cilium-agent.txt
9 podlogs/kube-system/cilium-djz4p/logs/cilium-agent.txt
8 podlogs/kube-system/cilium-qn6jq/logs/cilium-agent.txt
7 podlogs/kube-system/cilium-2hldp/logs/cilium-agent.txt
6 podlogs/kube-system/etcd-pr643-33ee32-control-plane-5tk9v-db82h/logs/etcd.txt
4 podlogs/sonobuoy/sonobuoy-e2e-job-704ba145d46940d4/logs/e2e.txt
2 podlogs/kube-system/cilium-operator-5c4f875f75-rv5f5/logs/cilium-operator.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-scbck/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-5rtb6/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-4mtjv/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-j2pf8/logs/node-driver-registrar.txt
Warnings:
606 podlogs/kube-system/etcd-pr643-33ee32-control-plane-5tk9v-db82h/logs/etcd.txt
47 podlogs/kube-system/kube-scheduler-pr643-33ee32-control-plane-5tk9v-db82h/logs/kube-scheduler.txt
33 podlogs/kube-system/kube-apiserver-pr643-33ee32-control-plane-5tk9v-db82h/logs/kube-apiserver.txt
12 podlogs/kube-system/kube-controller-manager-pr643-33ee32-control-plane-5tk9v-db82h/logs/kube-controller-manager.txt
10 podlogs/kube-system/openstack-cloud-controller-manager-vlxb7/logs/openstack-cloud-controller-manager.txt
4 podlogs/kube-system/cilium-64mrb/logs/cilium-agent.txt
4 podlogs/kube-system/cilium-djz4p/logs/cilium-agent.txt
4 podlogs/kube-system/cilium-qn6jq/logs/cilium-agent.txt
3 podlogs/kube-system/csi-cinder-nodeplugin-j2pf8/logs/node-driver-registrar.txt
3 podlogs/kube-system/cilium-2hldp/logs/cilium-agent.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-scbck/logs/node-driver-registrar.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-4mtjv/logs/node-driver-registrar.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-5rtb6/logs/node-driver-registrar.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-4mtjv/logs/liveness-probe.txt
1 podlogs/sonobuoy/sonobuoy/logs/kube-sonobuoy.txt
1 podlogs/kube-system/csi-cinder-controllerplugin-59f546d548-rvm2p/logs/csi-attacher.txt
1 podlogs/kube-system/metrics-server-d5f8d66cd-94r6z/logs/metrics-server.txt
1 podlogs/sonobuoy/sonobuoy-e2e-job-704ba145d46940d4/logs/e2e.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-5rtb6/logs/liveness-probe.txt
1 podlogs/kube-system/cilium-operator-5c4f875f75-rv5f5/logs/cilium-operator.txt
1 podlogs/kube-system/csi-cinder-controllerplugin-59f546d548-rvm2p/logs/csi-provisioner.txt
time="2024-01-15T08:56:38Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy
time="2024-01-15T08:56:38Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[]"
time="2024-01-15T08:56:38Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[]"

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 5 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 4 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-15 08:56:44 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...
...
...

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.
=== Sonobuoy conformance tests passed in 102s ===
make[1]: Leaving directory '/home/ubuntu/src/github.com/SovereignCloudStack/k8s-cluster-api-provider/terraform'

Custom ENV config

kubernetes_version = "v1.26.x"

@scszuulapp
Copy link

scszuulapp bot commented Jan 23, 2024

Build failed (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/cd216ea439fa47829007c1329c349b80

k8s-cluster-api-provider-e2e-quick FAILURE in 5m 26s

Warning:

Custom ENV config

kubernetes_version = "v1.26.x"

@scszuulapp
Copy link

scszuulapp bot commented Jan 23, 2024

Build failed (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/2c9c8f9871b546c4a4d87dbc1630db0e

k8s-cluster-api-provider-e2e-quick FAILURE in 3m 57s

Warning:

Custom ENV config

kubernetes_version = "v1.26.x"

@scszuulapp
Copy link

scszuulapp bot commented Jan 23, 2024

Build succeeded (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/a034e752aff34880a48363d52ec81eee

✔️ k8s-cluster-api-provider-e2e-quick SUCCESS in 34m 35s

Warning:

Sonobouy results === Collecting results === time="2024-01-23T13:54:55Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy time="2024-01-23T13:54:55Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[sonobuoy-serviceaccount-sonobuoy]" time="2024-01-23T13:54:55Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[sonobuoy-serviceaccount-sonobuoy]" Plugin: e2e Status: passed Total: 7072 Passed: 4 Failed: 0 Skipped: 7068

Plugin: systemd-logs
Status: passed
Total: 4
Passed: 4
Failed: 0
Skipped: 0

Run Details:
API Server version: v1.26.11
Node health: 4/4 (100%)
Pods health: 33/33 (100%)
Errors detected in files:
Errors:
77 podlogs/kube-system/kube-scheduler-pr643-ba6949-control-plane-qgckv-m2p7k/logs/kube-scheduler.txt
67 podlogs/kube-system/kube-controller-manager-pr643-ba6949-control-plane-qgckv-m2p7k/logs/kube-controller-manager.txt
19 podlogs/kube-system/snapshot-controller-54785b5645-6v8dl/logs/snapshot-controller.txt
12 podlogs/kube-system/kube-apiserver-pr643-ba6949-control-plane-qgckv-m2p7k/logs/kube-apiserver.txt
10 podlogs/kube-system/openstack-cloud-controller-manager-67m6d/logs/openstack-cloud-controller-manager.txt
9 podlogs/kube-system/cilium-w8whw/logs/cilium-agent.txt
8 podlogs/kube-system/cilium-87rlp/logs/cilium-agent.txt
8 podlogs/kube-system/cilium-4zvv5/logs/cilium-agent.txt
8 podlogs/kube-system/cilium-xp9sp/logs/cilium-agent.txt
6 podlogs/sonobuoy/sonobuoy-e2e-job-683c0027488c4f79/logs/e2e.txt
2 podlogs/kube-system/etcd-pr643-ba6949-control-plane-qgckv-m2p7k/logs/etcd.txt
1 podlogs/kube-system/cilium-operator-5c4f875f75-lk8m2/logs/cilium-operator.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-gpjw5/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-x6q24/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-fcn46/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-jmxm6/logs/node-driver-registrar.txt
Warnings:
488 podlogs/kube-system/etcd-pr643-ba6949-control-plane-qgckv-m2p7k/logs/etcd.txt
35 podlogs/kube-system/kube-apiserver-pr643-ba6949-control-plane-qgckv-m2p7k/logs/kube-apiserver.txt
33 podlogs/kube-system/kube-scheduler-pr643-ba6949-control-plane-qgckv-m2p7k/logs/kube-scheduler.txt
17 podlogs/kube-system/kube-controller-manager-pr643-ba6949-control-plane-qgckv-m2p7k/logs/kube-controller-manager.txt
12 podlogs/kube-system/openstack-cloud-controller-manager-67m6d/logs/openstack-cloud-controller-manager.txt
6 podlogs/kube-system/csi-cinder-nodeplugin-gpjw5/logs/node-driver-registrar.txt
5 podlogs/kube-system/cilium-w8whw/logs/cilium-agent.txt
4 podlogs/kube-system/csi-cinder-nodeplugin-x6q24/logs/node-driver-registrar.txt
4 podlogs/kube-system/cilium-xp9sp/logs/cilium-agent.txt
4 podlogs/kube-system/cilium-4zvv5/logs/cilium-agent.txt
4 podlogs/kube-system/cilium-87rlp/logs/cilium-agent.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-fcn46/logs/node-driver-registrar.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-gpjw5/logs/liveness-probe.txt
2 podlogs/kube-system/csi-cinder-controllerplugin-6dcdfc998b-pzrnk/logs/csi-provisioner.txt
2 podlogs/kube-system/csi-cinder-controllerplugin-6dcdfc998b-pzrnk/logs/csi-attacher.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-jmxm6/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-x6q24/logs/liveness-probe.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-fcn46/logs/liveness-probe.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-jmxm6/logs/liveness-probe.txt
1 podlogs/kube-system/metrics-server-d5f8d66cd-qm7fp/logs/metrics-server.txt
1 podlogs/sonobuoy/sonobuoy-e2e-job-683c0027488c4f79/logs/e2e.txt
1 podlogs/sonobuoy/sonobuoy/logs/kube-sonobuoy.txt
time="2024-01-23T13:54:55Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy
time="2024-01-23T13:54:55Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[]"
time="2024-01-23T13:54:56Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[]"

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 5 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 4 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-23 13:55:00 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...
...
...

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.
=== Sonobuoy conformance tests passed in 103s ===
make[1]: Leaving directory '/home/ubuntu/src/github.com/SovereignCloudStack/k8s-cluster-api-provider/terraform'

Custom ENV config

kubernetes_version = "v1.26.x"

@scszuulapp
Copy link

scszuulapp bot commented Jan 24, 2024

Build failed (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/9812de0a678a44558a31271749bc1ebe

k8s-cluster-api-provider-e2e-quick FAILURE in 5m 02s

Warning:

Custom ENV config

kubernetes_version = "v1.26.x"

@scszuulapp
Copy link

scszuulapp bot commented Jan 24, 2024

Build succeeded (e2e-quick-test pipeline).
https://zuul.scs.community/t/SCS/buildset/73b719e6e9bd48f4be775f200a3c65a7

✔️ k8s-cluster-api-provider-e2e-quick SUCCESS in 38m 12s

Warning:

Sonobouy results === Collecting results === time="2024-01-24T21:06:15Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy time="2024-01-24T21:06:15Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[sonobuoy-serviceaccount-sonobuoy]" time="2024-01-24T21:06:15Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[sonobuoy-serviceaccount-sonobuoy]" Plugin: e2e Status: passed Total: 7072 Passed: 4 Failed: 0 Skipped: 7068

Plugin: systemd-logs
Status: passed
Total: 4
Passed: 4
Failed: 0
Skipped: 0

Run Details:
API Server version: v1.26.11
Node health: 4/4 (100%)
Pods health: 33/33 (100%)
Errors detected in files:
Errors:
77 podlogs/kube-system/kube-scheduler-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/kube-scheduler.txt
70 podlogs/kube-system/kube-controller-manager-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/kube-controller-manager.txt
28 podlogs/kube-system/snapshot-controller-54785b5645-9pwh6/logs/snapshot-controller.txt
15 podlogs/kube-system/kube-apiserver-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/kube-apiserver.txt
12 podlogs/kube-system/openstack-cloud-controller-manager-v67gn/logs/openstack-cloud-controller-manager.txt
10 podlogs/kube-system/cilium-rgwtn/logs/cilium-agent.txt
10 podlogs/kube-system/cilium-cfjln/logs/cilium-agent.txt
7 podlogs/kube-system/cilium-wdkxm/logs/cilium-agent.txt
7 podlogs/kube-system/cilium-wvgxj/logs/cilium-agent.txt
3 podlogs/kube-system/cilium-operator-5c4f875f75-jtr7h/logs/cilium-operator.txt
3 podlogs/sonobuoy/sonobuoy-e2e-job-eab778acc3934d30/logs/e2e.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-8x9bh/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-99q9k/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-hrrmn/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-pf8p5/logs/node-driver-registrar.txt
Warnings:
324 podlogs/kube-system/etcd-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/etcd.txt
36 podlogs/kube-system/kube-apiserver-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/kube-apiserver.txt
33 podlogs/kube-system/kube-scheduler-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/kube-scheduler.txt
20 podlogs/kube-system/kube-controller-manager-pr643-a2901a-control-plane-7r6sr-ss6s8/logs/kube-controller-manager.txt
14 podlogs/kube-system/openstack-cloud-controller-manager-v67gn/logs/openstack-cloud-controller-manager.txt
10 podlogs/kube-system/csi-cinder-controllerplugin-6895bbf8f5-7jbdk/logs/csi-attacher.txt
9 podlogs/kube-system/csi-cinder-nodeplugin-8x9bh/logs/node-driver-registrar.txt
7 podlogs/kube-system/csi-cinder-nodeplugin-8x9bh/logs/liveness-probe.txt
6 podlogs/kube-system/cilium-cfjln/logs/cilium-agent.txt
6 podlogs/kube-system/csi-cinder-controllerplugin-6895bbf8f5-7jbdk/logs/csi-provisioner.txt
6 podlogs/kube-system/cilium-rgwtn/logs/cilium-agent.txt
5 podlogs/kube-system/csi-cinder-nodeplugin-pf8p5/logs/node-driver-registrar.txt
4 podlogs/kube-system/csi-cinder-nodeplugin-99q9k/logs/node-driver-registrar.txt
3 podlogs/kube-system/cilium-wvgxj/logs/cilium-agent.txt
3 podlogs/kube-system/cilium-wdkxm/logs/cilium-agent.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-hrrmn/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-99q9k/logs/liveness-probe.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-pf8p5/logs/liveness-probe.txt
1 podlogs/sonobuoy/sonobuoy-e2e-job-eab778acc3934d30/logs/e2e.txt
1 podlogs/kube-system/csi-cinder-controllerplugin-6895bbf8f5-7jbdk/logs/csi-snapshotter.txt
1 podlogs/kube-system/cilium-operator-5c4f875f75-jtr7h/logs/cilium-operator.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-hrrmn/logs/liveness-probe.txt
1 podlogs/sonobuoy/sonobuoy/logs/kube-sonobuoy.txt
1 podlogs/kube-system/metrics-server-d5f8d66cd-g2wlm/logs/metrics-server.txt
time="2024-01-24T21:06:15Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy
time="2024-01-24T21:06:15Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[]"
time="2024-01-24T21:06:15Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[]"

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 5 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 4 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-24 21:06:21 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...
...
...

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.
=== Sonobuoy conformance tests passed in 124s ===
make[1]: Leaving directory '/home/ubuntu/src/github.com/SovereignCloudStack/k8s-cluster-api-provider/terraform'

Custom ENV config

kubernetes_version = "v1.26.x"

@scszuulapp
Copy link

scszuulapp bot commented Jan 26, 2024

Build succeeded (e2e-test pipeline).
https://zuul.scs.community/t/SCS/buildset/99baa3cf822749d5bf50adb8fcd8e4db

✔️ k8s-cluster-api-provider-e2e-conformance SUCCESS in 2h 34m 33s

Warning:

Sonobouy results === Collecting results === time="2024-01-26T11:45:26Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy time="2024-01-26T11:45:26Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[sonobuoy-serviceaccount-sonobuoy]" time="2024-01-26T11:45:26Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[sonobuoy-serviceaccount-sonobuoy]" time="2024-01-26T11:45:26Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=webhook-447 Plugin: e2e Status: passed Total: 7213 Passed: 382 Failed: 0 Skipped: 6831

Plugin: systemd-logs
Status: passed
Total: 4
Passed: 4
Failed: 0
Skipped: 0

Run Details:
API Server version: v1.27.8
Node health: 4/4 (100%)
Pods health: 33/34 (97%)
Details for failed pods:
webhook-447/webhook-to-be-mutated Ready:False: ContainersNotInitialized: containers with incomplete status: [webhook-added-init-container]
Errors detected in files:
Errors:
1123 podlogs/kube-system/kube-controller-manager-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/kube-controller-manager.txt
943 podlogs/kube-system/snapshot-controller-55687d7977-gcsts/logs/snapshot-controller.txt
384 podlogs/kube-system/cilium-jszrw/logs/cilium-agent.txt
278 podlogs/kube-system/cilium-f5xbh/logs/cilium-agent.txt
185 podlogs/kube-system/kube-apiserver-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/kube-apiserver.txt
149 podlogs/kube-system/cilium-9cswf/logs/cilium-agent.txt
58 podlogs/kube-system/kube-scheduler-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/kube-scheduler.txt
8 podlogs/kube-system/kube-proxy-r4dt2/logs/kube-proxy.txt
8 podlogs/kube-system/cilium-qmgt6/logs/cilium-agent.txt
8 podlogs/kube-system/kube-proxy-rbq5w/logs/kube-proxy.txt
8 podlogs/kube-system/kube-proxy-69cgq/logs/kube-proxy.txt
8 podlogs/kube-system/kube-proxy-sxw6f/logs/kube-proxy.txt
7 podlogs/kube-system/openstack-cloud-controller-manager-nptpc/logs/openstack-cloud-controller-manager.txt
5 podlogs/kube-system/cilium-operator-788c7d7585-6rsqd/logs/cilium-operator.txt
3 podlogs/kube-system/etcd-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/etcd.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-7fsng/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-n2jln/logs/node-driver-registrar.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-rnkgm/logs/node-driver-registrar.txt
1 podlogs/sonobuoy/sonobuoy-e2e-job-f7967b6d0e5b44ee/logs/e2e.txt
1 podlogs/kube-system/csi-cinder-nodeplugin-8sn8w/logs/node-driver-registrar.txt
Warnings:
7536 podlogs/kube-system/etcd-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/etcd.txt
516 podlogs/kube-system/kube-controller-manager-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/kube-controller-manager.txt
367 podlogs/kube-system/cilium-jszrw/logs/cilium-agent.txt
236 podlogs/kube-system/cilium-f5xbh/logs/cilium-agent.txt
114 podlogs/kube-system/cilium-9cswf/logs/cilium-agent.txt
103 podlogs/kube-system/kube-apiserver-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/kube-apiserver.txt
30 podlogs/kube-system/kube-scheduler-pr643-68b0f8-control-plane-kqkqg-cs48n/logs/kube-scheduler.txt
12 podlogs/kube-system/cilium-qmgt6/logs/cilium-agent.txt
7 podlogs/kube-system/csi-cinder-nodeplugin-7fsng/logs/node-driver-registrar.txt
7 podlogs/kube-system/openstack-cloud-controller-manager-nptpc/logs/openstack-cloud-controller-manager.txt
6 podlogs/kube-system/csi-cinder-nodeplugin-n2jln/logs/node-driver-registrar.txt
6 podlogs/kube-system/csi-cinder-controllerplugin-fbdc645cb-tlck9/logs/csi-attacher.txt
3 podlogs/kube-system/csi-cinder-nodeplugin-rnkgm/logs/node-driver-registrar.txt
3 podlogs/kube-system/csi-cinder-nodeplugin-8sn8w/logs/node-driver-registrar.txt
2 podlogs/kube-system/cilium-operator-788c7d7585-6rsqd/logs/cilium-operator.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-7fsng/logs/liveness-probe.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-n2jln/logs/liveness-probe.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-rnkgm/logs/liveness-probe.txt
2 podlogs/kube-system/csi-cinder-nodeplugin-8sn8w/logs/liveness-probe.txt
2 podlogs/kube-system/csi-cinder-controllerplugin-fbdc645cb-tlck9/logs/csi-provisioner.txt
1 podlogs/sonobuoy/sonobuoy-e2e-job-f7967b6d0e5b44ee/logs/e2e.txt
1 podlogs/kube-system/metrics-server-5fb5598cf8-2wnqx/logs/metrics-server.txt
1 podlogs/sonobuoy/sonobuoy/logs/kube-sonobuoy.txt
time="2024-01-26T11:45:27Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=sonobuoy
time="2024-01-26T11:45:27Z" level=info msg="delete request issued" dry-run=false kind=clusterrolebindings names="[]"
time="2024-01-26T11:45:27Z" level=info msg="delete request issued" dry-run=false kind=clusterroles names="[]"
time="2024-01-26T11:45:27Z" level=info msg="delete request issued" dry-run=false kind=namespace namespace=webhook-447

Namespace "sonobuoy" has status {Phase:Terminating Conditions:[{Type:NamespaceDeletionDiscoveryFailure Status:False LastTransitionTime:2024-01-26 11:45:32 +0000 UTC Reason:ResourcesDiscovered Message:All resources successfully discovered} {Type:NamespaceDeletionGroupVersionParsingFailure Status:False LastTransitionTime:2024-01-26 11:45:32 +0000 UTC Reason:ParsedGroupVersions Message:All legacy kube types successfully parsed} {Type:NamespaceDeletionContentFailure Status:False LastTransitionTime:2024-01-26 11:45:32 +0000 UTC Reason:ContentDeleted Message:All content successfully deleted, may be waiting on finalization} {Type:NamespaceContentRemaining Status:True LastTransitionTime:2024-01-26 11:45:32 +0000 UTC Reason:SomeResourcesRemain Message:Some resources are remaining: pods. has 1 resource instances} {Type:NamespaceFinalizersRemaining Status:False LastTransitionTime:2024-01-26 11:45:32 +0000 UTC Reason:ContentHasNoFinalizers Message:All content-preserving finalizers finished}]}
...

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.

Found 1 namespaces that still need to be deleted: [webhook-447]

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.

Found 1 namespaces that still need to be deleted: [webhook-447]

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.

Found 1 namespaces that still need to be deleted: [webhook-447]

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.

Found 1 namespaces that still need to be deleted: [webhook-447]

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.

Found 1 namespaces that still need to be deleted: [webhook-447]

Namespace "sonobuoy" has been deleted

Deleted all ClusterRoles and ClusterRoleBindings.

All E2E namespaces deleted
=== Sonobuoy conformance tests passed in 7530s ===
make[1]: Leaving directory '/home/ubuntu/src/github.com/SovereignCloudStack/k8s-cluster-api-provider/terraform'

Custom ENV config

kubernetes_version = "v1.27.x"

@garloff
Copy link
Contributor Author

garloff commented Jan 26, 2024

Nice.
We still need an approving review to merge this.

@chess-knight chess-knight merged commit eac1608 into main Jan 29, 2024
4 checks passed
@chess-knight chess-knight deleted the feat/k8s-1.28.3-occm-1.28.1 branch January 29, 2024 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Container Issues or pull requests relevant for Team 2: Container Infra and Tooling enhancement New feature or request
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

5 participants