[sig-api-machinery] webhook related test

[spyroot]

Which jobs are failing?

Hi Folks,

I'm checking the e2e test related to webhook. Notice that that webhook first failed now below. Notice
W0424 17:48:11.860818 1 dispatcher.go:187] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-2542.svc:8443/configmaps?timeout=10s": context deadline exceeded

Now, if I create a pod in the Kube System, I can reach this: https://e2e-test-webhook.webhook-2542.svc:8443.
I checked with Curl, so the webhook is active.

For example, this for patching/updating a validating webhook should work [Conformance]

First, I see this.

W0424 17:56:43.733083 1 dispatcher.go:195] rejected by webhook "validating-is-webhook-configuration-ready.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook "validating-is-webhook-configuration-ready.k8s.io" denied the request: this webhook denies all requests", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}

Then follow up

W0424 17:56:43.740017 1 dispatcher.go:195] rejected by webhook "deny-unwanted-configmap-data.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook "deny-unwanted-configmap-data.k8s.io" denied the request: the configmap contains unwanted key and value", Reason:"the configmap contains unwanted key and value", Details:(*v1.StatusDetails)(nil), Code:400}}

I0424 17:48:00.826632       1 alloc.go:327] "allocated clusterIPs" service="webhook-2542/e2e-test-webhook" clusterIPs=map[IPv4:100.64.163.203]
W0424 17:48:01.840232       1 dispatcher.go:195] rejected by webhook "validating-is-webhook-configuration-ready.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validating-is-webhook-configuration-ready.k8s.io\" denied the request: this webhook denies all requests", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0424 17:48:01.847172       1 dispatcher.go:195] rejected by webhook "deny-unwanted-configmap-data.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"deny-unwanted-configmap-data.k8s.io\" denied the request: the configmap contains unwanted key and value", Reason:"the configmap contains unwanted key and value", Details:(*v1.StatusDetails)(nil), Code:400}}
W0424 17:48:11.860818       1 dispatcher.go:187] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-2542.svc:8443/configmaps?timeout=10s": context deadline exceeded
I0424 17:48:11.860951       1 trace.go:219] Trace[259587671]: "Create" accept:application/vnd.kubernetes.protobuf, */*,audit-id:4f0b5848-6ce2-4b34-b656-34c8adc18a49,client:192.168.41.20,protocol:HTTP/2.0,resource:configmaps,scope:resource,url:/api/v1/namespaces/webhook-2542/configmaps,user-agent:e2e.test/v1.26.8 (linux/amd64) kubernetes/395f0a2 -- [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a validating webhook should work [Conformance],verb:POST (24-Apr-2024 17:48:01.860) (total time: 10000ms):
Trace[259587671]: ["Call validating webhook" configuration:webhook-2542,webhook:deny-unwanted-configmap-data.k8s.io,resource:/v1, Resource=configmaps,subresource:,operation:CREATE,UID:77f75f83-a189-47f4-ac50-435f52c2cbbe 10000ms (17:48:01.860)]
Trace[259587671]: [10.000736612s] [10.000736612s] END

Which tests are failing?

sig-api-machinery webhook-related patch / crd reject and a few other all related to web hooks

Since when has it been failing?

First, pass e2e successfully after re-running the test. Note that I checked all stale items, and there are none.

Testgrid link

No response

Reason for failure (if possible)

No response

Anything else we need to know?

Logs from e2e during a test run.

STEP: Deploying the webhook service 04/24/24 18:03:06.685
STEP: Verifying the service has paired with the endpoint 04/24/24 18:03:06.69
Apr 24 18:03:07.690: INFO: Waiting for amount of service:e2e-test-webhook endpoints to be 1
[It] patching/updating a validating webhook should work [Conformance]
test/e2e/apimachinery/webhook.go:413
STEP: Creating a validating webhook configuration 04/24/24 18:03:07.692
STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.704
STEP: Updating a validating webhook configuration's rules to not include the create operation 04/24/24 18:03:07.711
STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.715
STEP: Patching a validating webhook configuration's rules to include the create operation 04/24/24 18:03:07.719
STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.723
Apr 24 18:03:17.726: INFO: Unexpected error: Waiting for configMap in namespace webhook-4917 to be denied creation by validating webhook:
<*errors.StatusError | 0xc000352960>: {
ErrStatus:
code: 500
details:
causes:
- message: 'failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed
to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s":
context deadline exceeded'
message: 'Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io":
failed to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s":
context deadline exceeded'
metadata: {}
reason: InternalError
status: Failure,
}
Apr 24 18:03:17.726: FAIL: Waiting for configMap in namespace webhook-4917 to be denied creation by validating webhook: Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s": context deadline exceeded

Relevant SIG(s)

/sig-api-machinery

[k8s-ci-robot]

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spyroot]

I'm closing this case. For the rest, HTTP/2 uses a max segment size in the case of jumbo MTU, i.e., if you have some issue with MTU and a network configured with jumbo MTU. Still, you have slightly less than sufficient offset for overhead, i.e., in the case of some sort of bridge or tunnel, the frame size might exceed, so SYN-ACK never happened. Hence, kube-api webhooks never establish a TLS connection that uses HTTP/2.

I would say the recommended action is to create pods in the Kube system and check MTU at full mtu size to confirm that there is no issue on the network side with jumbo mtu, etc.

b) I still need confirmation, but it looked like PMTU does not work in case you have a bridge interface and uplink some sort of tunnel ( VXLAN, Geneve, etc). Hence, PMTU never kicks in, so SYN-ACK is never completed.

[spyroot]

I'm closing this case. For the rest, HTTP/2 uses a max segment size in the case of jumbo MTU, i.e., if you have some issue with MTU and a network configured with jumbo MTU. Still, you have slightly less than sufficient offset for overhead, i.e., in the case of some sort of bridge or tunnel, the frame size might exceed, so SYN-ACK never happened. Hence, kube-api webhooks never establish a TLS connection that uses HTTP/2.

I would say the recommended action is to create pods in the Kube system and check MTU at full mtu size to confirm that there is no issue on the network side with jumbo mtu, etc.

b) I still need confirmation, but it looks like PMTU does not work if you have a bridge interface and uplink some sort of tunnel ( VXLAN, Geneve, etc.). Hence, PMTU never kicks in, so SYN-ACK is never completed.