bug 2042049: Delete feature gates removed from k8s.io/kubernetes/pkg/features/kube_features.go

[ingvagabund]

To free the KA logs from unrecognized feature gate warnings:

I0127 07:03:38.562893      17 plugins.go:84] "Registered admission plugin" plugin="security.openshift.io/ValidateSecurityContextConstraints"
I0127 07:03:38.562899      17 plugins.go:84] "Registered admission plugin" plugin="authorization.openshift.io/ValidateRoleBindingRestriction"
I0127 07:03:38.562910      17 plugins.go:84] "Registered admission plugin" plugin="config.openshift.io/ValidateNetwork"
I0127 07:03:38.562915      17 plugins.go:84] "Registered admission plugin" plugin="security.openshift.io/DefaultSecurityContextConstraints"
I0127 07:03:38.565735      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true]}
I0127 07:03:38.565813      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
W0127 07:03:38.565859      17 feature_gate.go:223] unrecognized feature gate: SupportPodPidsLimit
I0127 07:03:38.565866      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
W0127 07:03:38.565914      17 feature_gate.go:223] unrecognized feature gate: NodeDisruptionExclusion
I0127 07:03:38.565928      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
W0127 07:03:38.565973      17 feature_gate.go:223] unrecognized feature gate: ServiceNodeExclusion
I0127 07:03:38.565978      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
I0127 07:03:38.566028      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true RotateKubeletServerCertificate:true]}
I0127 07:03:38.566076      17 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
W0127 07:03:38.566123      17 feature_gate.go:223] unrecognized feature gate: LegacyNodeRoleBehavior

[openshift-ci]

@ingvagabund: This pull request references Bugzilla bug 2042049, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.10.0) matches configured target release for branch (4.10.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @kasturinarra

In response to this:

bug 2042049: Delete feature gates removed from k8s.io/kubernetes/pkg/features/kube_features.go

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ingvagabund]

/retest

[p0lyn0mial]

flags for KAS are updated dynamically from

cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/operator/configobserver/featuregates/observe_featuregates.go

Line 60 in 9e70f90

FeatureSet: configv1.Default,

(which reads from the API repo)

[p0lyn0mial]

/lgtm
/approve

for manual verification
/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, p0lyn0mial

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [p0lyn0mial]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ingvagabund]

Verifying the change through the KA logs:

I0127 10:16:04.297827      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true]}
I0127 10:16:04.297876      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.297912      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.297947      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.297978      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.298010      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.298043      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationGCE:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.298076      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationAzureDisk:false CSIMigrationGCE:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.298106      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationAzureDisk:false CSIMigrationAzureFile:false CSIMigrationGCE:false CSIMigrationOpenStack:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}
I0127 10:16:04.298142      18 feature_gate.go:246] feature gates: &{map[APIPriorityAndFairness:true CSIMigrationAWS:false CSIMigrationAzureDisk:false CSIMigrationAzureFile:false CSIMigrationGCE:false CSIMigrationOpenStack:false CSIMigrationvSphere:false DownwardAPIHugePages:true PodSecurity:true RotateKubeletServerCertificate:true]}

No more unrecognized feature gate logs there.

Also, checking the config CM in openshift-kube-apiserver NS:

{
            "apiVersion": "v1",
            "data": {
                "config.yaml": "{\"admission\":{\"pluginConfig\":{\"PodSecurity\":{\"configuration\":{\"apiVersion\":\"pod-security.admission.config.k8s.io/v1alpha1\",\"defaults\":{\"audit\":\"baseline\",\"audit-version\":\"latest\",\"enforce\":\"privileged\",\"enforce-version\":\"latest\",\"warn\":\"baseline\",\"warn-version\":\"latest\"},\"kind\":\"PodSecurityConfiguration\"}},\"network.openshift.io/ExternalIPRanger\":{\"configuration\":{\"allowIngressIP\":false,\"apiVersion\":\"network.openshift.io/v1\",\"externalIPNetworkCIDRs\":null,\"kind\":\"ExternalIPRangerAdmissionConfig\"},\"location\":\"\"},\"network.openshift.io/RestrictedEndpointsAdmission\":{\"configuration\":{\"apiVersion\":\"network.openshift.io/v1\",\"kind\":\"RestrictedEndpointsAdmissionConfig\",\"restrictedCIDRs\":[\"10.128.0.0/14\",\"172.30.0.0/16\"]}}}},\"apiServerArguments\":{\"allow-privileged\":[\"true\"],\"anonymous-auth\":[\"true\"],\"api-audiences\":[\"https://kubernetes.default.svc\"],\"audit-log-format\":[\"json\"],\"audit-log-maxbackup\":[\"10\"],\"audit-log-maxsize\":[\"100\"],\"audit-log-path\":[\"/var/log/kube-apiserver/audit.log\"],\"audit-policy-file\":[\"/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-audit-policies/policy.yaml\"],\"authentication-token-webhook-config-file\":[\"/etc/kubernetes/static-pod-resources/secrets/webhook-authenticator/kubeConfig\"],\"authentication-token-webhook-version\":[\"v1\"],\"authorization-mode\":[\"Scope\",\"SystemMasters\",\"RBAC\",\"Node\"],\"client-ca-file\":[\"/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt\"],\"cloud-provider\":[\"aws\"],\"disable-admission-plugins\":[\"PodSecurity\"],\"enable-admission-plugins\":[\"CertificateApproval\",\"CertificateSigning\",\"CertificateSubjectRestriction\",\"DefaultIngressClass\",\"DefaultStorageClass\",\"DefaultTolerationSeconds\",\"LimitRanger\",\"MutatingAdmissionWebhook\",\"NamespaceLifecycle\",\"NodeRestriction\",\"OwnerReferencesPermissionEnforcement\",\"PersistentVolumeClaimResize\",\"PersistentVolumeLabel\",\"PodNodeSelector\",\"PodTolerationRestriction\",\"Priority\",\"ResourceQuota\",\"RuntimeClass\",\"ServiceAccount\",\"StorageObjectInUseProtection\",\"TaintNodesByCondition\",\"ValidatingAdmissionWebhook\",\"authorization.openshift.io/RestrictSubjectBindings\",\"authorization.openshift.io/ValidateRoleBindingRestriction\",\"config.openshift.io/DenyDeleteClusterConfiguration\",\"config.openshift.io/ValidateAPIServer\",\"config.openshift.io/ValidateAuthentication\",\"config.openshift.io/ValidateConsole\",\"config.openshift.io/ValidateFeatureGate\",\"config.openshift.io/ValidateImage\",\"config.openshift.io/ValidateOAuth\",\"config.openshift.io/ValidateProject\",\"config.openshift.io/ValidateScheduler\",\"image.openshift.io/ImagePolicy\",\"network.openshift.io/ExternalIPRanger\",\"network.openshift.io/RestrictedEndpointsAdmission\",\"quota.openshift.io/ClusterResourceQuota\",\"quota.openshift.io/ValidateClusterResourceQuota\",\"route.openshift.io/IngressAdmission\",\"scheduling.openshift.io/OriginPodNodeEnvironment\",\"security.openshift.io/DefaultSecurityContextConstraints\",\"security.openshift.io/SCCExecRestrictions\",\"security.openshift.io/SecurityContextConstraint\",\"security.openshift.io/ValidateSecurityContextConstraints\"],\"enable-aggregator-routing\":[\"true\"],\"enable-logs-handler\":[\"false\"],\"enable-swagger-ui\":[\"true\"],\"endpoint-reconciler-type\":[\"lease\"],\"etcd-cafile\":[\"/etc/kubernetes/static-pod-resources/configmaps/etcd-serving-ca/ca-bundle.crt\"],\"etcd-certfile\":[\"/etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.crt\"],\"etcd-keyfile\":[\"/etc/kubernetes/static-pod-resources/secrets/etcd-client/tls.key\"],\"etcd-prefix\":[\"kubernetes.io\"],\"etcd-servers\":[\"https://10.0.179.126:2379\",\"https://10.0.186.221:2379\",\"https://10.0.255.89:2379\",\"https://localhost:2379\"],\"event-ttl\":[\"3h\"],\"feature-gates\":[\"APIPriorityAndFairness=true\",\"RotateKubeletServerCertificate=true\",\"DownwardAPIHugePages=true\",\"PodSecurity=true\",\"CSIMigrationAWS=false\",\"CSIMigrationOpenStack=false\",\"CSIMigrationGCE=false\",\"CSIMigrationAzureDisk=false\",\"CSIMigrationAzureFile=false\",\"CSIMigrationvSphere=false\"],\"goaway-chance\":[\"0\"],\"http2-max-streams-per-connection\":[\"2000\"],\"insecure-port\":[\"0\"],\"kubelet-certificate-authority\":[\"/etc/kubernetes/static-pod-resources/configmaps/kubelet-serving-ca/ca-bundle.crt\"],\"kubelet-client-certificate\":[\"/etc/kubernetes/static-pod-certs/secrets/kubelet-client/tls.crt\"],\"kubelet-client-key\":[\"/etc/kubernetes/static-pod-certs/secrets/kubelet-client/tls.key\"],\"kubelet-preferred-address-types\":[\"InternalIP\"],\"kubelet-read-only-port\":[\"0\"],\"kubernetes-service-node-port\":[\"0\"],\"max-mutating-requests-inflight\":[\"1000\"],\"max-requests-inflight\":[\"3000\"],\"min-request-timeout\":[\"3600\"],\"proxy-client-cert-file\":[\"/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt\"],\"proxy-client-key-file\":[\"/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key\"],\"requestheader-allowed-names\":[\"kube-apiserver-proxy\",\"system:kube-apiserver-proxy\",\"system:openshift-aggregator\"],\"requestheader-client-ca-file\":[\"/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt\"],\"requestheader-extra-headers-prefix\":[\"X-Remote-Extra-\"],\"requestheader-group-headers\":[\"X-Remote-Group\"],\"requestheader-username-headers\":[\"X-Remote-User\"],\"service-account-issuer\":[\"https://kubernetes.default.svc\"],\"service-account-jwks-uri\":[\"https://api-int.ci-op-hlnz1g5k-63f8c.origin-ci-int-aws.dev.rhcloud.com:6443/openid/v1/jwks\"],\"service-account-lookup\":[\"true\"],\"service-account-signing-key-file\":[\"/etc/kubernetes/static-pod-certs/secrets/bound-service-account-signing-key/service-account.key\"],\"service-node-port-range\":[\"30000-32767\"],\"shutdown-delay-duration\":[\"129s\"],\"shutdown-send-retry-after\":[\"true\"],\"storage-backend\":[\"etcd3\"],\"storage-media-type\":[\"application/vnd.kubernetes.protobuf\"],\"tls-cert-file\":[\"/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt\"],\"tls-private-key-file\":[\"/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key\"]},\"apiVersion\":\"kubecontrolplane.config.openshift.io/v1\",\"authConfig\":{\"oauthMetadataFile\":\"/etc/kubernetes/static-pod-resources/configmaps/oauth-metadata/oauthMetadata\"},\"consolePublicURL\":\"\",\"corsAllowedOrigins\":[\"//127\\\\.0\\\\.0\\\\.1(:|$)\",\"//localhost(:|$)\"],\"imagePolicyConfig\":{\"internalRegistryHostname\":\"image-registry.openshift-image-registry.svc:5000\"},\"kind\":\"KubeAPIServerConfig\",\"projectConfig\":{\"defaultNodeSelector\":\"\"},\"serviceAccountPublicKeyFiles\":[\"/etc/kubernetes/static-pod-resources/configmaps/sa-token-signing-certs\",\"/etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs\"],\"servicesSubnet\":\"172.30.0.0/16\",\"servingInfo\":{\"bindAddress\":\"0.0.0.0:6443\",\"bindNetwork\":\"tcp4\",\"cipherSuites\":[\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\",\"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\",\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\",\"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\",\"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\",\"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\"],\"minTLSVersion\":\"VersionTLS12\",\"namedCertificates\":[{\"certFile\":\"/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt\",\"keyFile\":\"/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key\"},{\"certFile\":\"/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt\",\"keyFile\":\"/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key\"},{\"certFile\":\"/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt\",\"keyFile\":\"/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key\"},{\"certFile\":\"/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt\",\"keyFile\":\"/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key\"},{\"certFile\":\"/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt\",\"keyFile\":\"/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key\"}]}}"
            },
            "kind": "ConfigMap",
            "metadata": {
                "creationTimestamp": "2022-01-27T10:09:08Z",
                "name": "config",
                "namespace": "openshift-kube-apiserver",
                "resourceVersion": "20750",
                "uid": "f263502c-3f2b-4ced-aa9d-e4b1cb1a1e02"
            }
        },

The feature-gates field is set to [\"APIPriorityAndFairness=true\",\"RotateKubeletServerCertificate=true\",\"DownwardAPIHugePages=true\",\"PodSecurity=true\",\"CSIMigrationAWS=false\",\"CSIMigrationOpenStack=false\",\"CSIMigrationGCE=false\",\"CSIMigrationAzureDisk=false\",\"CSIMigrationAzureFile=false\",\"CSIMigrationvSphere=false\"]. No mention of the removed features.

/hold cancel

[ingvagabund]

/retest-required

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[ingvagabund]

/retest-required

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@ingvagabund: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-operator-disruptive-single-node	121db0c	link	false	/test e2e-aws-operator-disruptive-single-node
ci/prow/e2e-aws-single-node	121db0c	link	false	/test e2e-aws-single-node
ci/prow/e2e-gcp-operator-single-node	121db0c	link	false	/test e2e-gcp-operator-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci]

@ingvagabund: All pull requests linked via external trackers have merged:

• openshift/api#1100
• openshift/cluster-kube-apiserver-operator#1298
• openshift/cluster-kube-controller-manager-operator#592
• openshift/cluster-kube-scheduler-operator#401

Bugzilla bug 2042049 has been moved to the MODIFIED state.

In response to this:

bug 2042049: Delete feature gates removed from k8s.io/kubernetes/pkg/features/kube_features.go

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.