Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": Forbidden

[baluveermukkamala]

MetalLB Version

0.13.12

Deployment method

Charts

Main CNI

calico

Kubernetes Version

v1.27.7

Cluster Distribution

MicroK8s

Describe the bug

We are trying to setup metallb on our MicroK8s cluster but configuration of IPAddressPool failing at webhook validation with forbidden error. I tried updating webhook failures to be ignored but configuration of IPAddressPool still failing at webhook validation. Here are the errors that I can see in both Metallb Controller logs and Api Server logs.

MetallB Controller Errors:

{"level":"error","ts":"2023-11-15T06:03:40Z","logger":"cert-rotation","msg":"could not refresh CA and server certs","error":"Operation cannot be fulfilled on secrets "webhook-server-cert": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:309\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.28.3/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.28.3/pkg/util/wait/backoff.go:461\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:337\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:756\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2023-11-15T06:03:40Z","logger":"cert-rotation","msg":"no cert refresh needed"}
{"level":"error","ts":"2023-11-15T06:03:40Z","logger":"cert-rotation","msg":"secret is not well-formed, cannot update webhook configurations","error":"Cert secret is not well-formed, missing ca.crt","errorVerbose":"Cert secret is not well-formed, missing ca.crt\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.buildArtifactsFromSecret\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:488\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:768\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:770\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2023-11-15T06:03:40Z","logger":"cert-rotation","msg":"no cert refresh needed"}

Api Server Error:
I1115 01:55:01.689598 2638136 request.go:1212] Response Body: {"kind":"Namespace","apiVersion":"v1","metadata":{"name":"metallb-system","uid":"04bf9ff3-594d-4bdc-9c6a-41832aa5691a","resourceVersion":"1302835","creationTimestamp":"2023-11-15T06:03:38Z","labels":{"kubernetes.io/metadata.name":"metallb-system","name":"metallb-system"},"managedFields":[{"manager":"helm","operation":"Update","apiVersion":"v1","time":"2023-11-15T06:03:38Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:kubernetes.io/metadata.name":{},"f:name":{}}}}}]},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}
I1115 01:55:01.689707 2638136 request.go:1212] Request Body: {"apiVersion":"metallb.io/v1beta1","kind":"IPAddressPool","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"metallb.io/v1beta1","kind":"IPAddressPool","metadata":{"annotations":{},"name":"cheap","namespace":"metallb-system"},"spec":{"addresses":["10.250.146.254/32"]}}\n"},"name":"cheap","namespace":"metallb-system"},"spec":{"addresses":["10.250.146.254/32"]}}
I1115 01:55:01.689772 2638136 round_trippers.go:466] curl -v -XPOST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.28.3 (linux/amd64) kubernetes/a8a1abc" -H "Authorization: Bearer " 'https://10.250.146.181:16443/apis/metallb.io/v1beta1/namespaces/metallb-system/ipaddresspools?fieldManager=kubectl-client-side-apply&fieldValidation=Strict'
I1115 01:55:01.775340 2638136 round_trippers.go:553] POST https://10.250.146.181:16443/apis/metallb.io/v1beta1/namespaces/metallb-system/ipaddresspools?fieldManager=kubectl-client-side-apply&fieldValidation=Strict 500 Internal Server Error in 85 milliseconds
I1115 01:55:01.775478 2638136 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 85 ms Duration 85 ms
I1115 01:55:01.775530 2638136 round_trippers.go:577] Response Headers:
I1115 01:55:01.775578 2638136 round_trippers.go:580] Audit-Id: 2006f3a6-c823-42bf-8303-822e82df2441
I1115 01:55:01.775639 2638136 round_trippers.go:580] Cache-Control: no-cache, private
I1115 01:55:01.775689 2638136 round_trippers.go:580] Content-Type: application/json
I1115 01:55:01.775734 2638136 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 8c1b0388-5fad-4da2-95d7-e3ee277d34ce
I1115 01:55:01.775779 2638136 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 21a01a42-53d3-4d29-bcfe-9edf3a2d59d2
I1115 01:55:01.775826 2638136 round_trippers.go:580] Content-Length: 609
I1115 01:55:01.775872 2638136 round_trippers.go:580] Date: Wed, 15 Nov 2023 06:55:01 GMT
I1115 01:55:01.775942 2638136 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s\": Forbidden","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": Forbidden"}]},"code":500}
I1115 01:55:01.776135 2638136 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "error when creating "STDIN": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": Forbidden",
"reason": "InternalError",
"details": {
"causes": [
{
"message": "failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": Forbidden"
}
]
},
"code": 500
}]
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": Forbidden

To Reproduce

1. Install Metallb on MicroK8s using Helm Chart
2. Then apply below configuration

kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: cheap
namespace: metallb-system
spec:
addresses:

• 10.250.146.254/32
  EOF

Expected Behavior

Below configuration should be applied successfully with out any error

kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: cheap
namespace: metallb-system
spec:
addresses:

• 10.250.146.254/32
  EOF

kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2advertisement
namespace: metallb-system
spec:
ipAddressPools:

• cheap
  EOF

Additional Context

As mentioned above, we are using MicroK8s. We tried enabling Metallb as addon and also using helm chart but same issue.

We suspect this error as root cause but we are not sure:

{"level":"error","ts":"2023-11-15T06:03:40Z","logger":"cert-rotation","msg":"secret is not well-formed, cannot update webhook configurations","error":"Cert secret is not well-formed, missing ca.crt","errorVerbose":"Cert secret is not well-formed, missing ca.crt\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.buildArtifactsFromSecret\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.0/pkg/rotator/rotator.go:488\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.

I've read and agree with the following

• I've checked all open and closed issues and my request is not there.
• I've checked all open and closed pull requests and my request is not there.

I've read and agree with the following

• I've checked all open and closed issues and my issue is not there.
• This bug is reproducible when deploying MetalLB from the main branch
• I have read the troubleshooting guide and I am still not able to make it work
• I checked the logs and MetalLB is not discarding the configuration as not valid
• I enabled the debug logs, collected the information required from the cluster using the collect script and will attach them to the issue
• I will provide the definition of my service and the related endpoint slices and attach them to this issue

[kfox1111]

We hit the same issue. deleting the validatingwebhookconfiguration followed by helm upgrading again seemed to fix it.

[fedepaol]

Closing as duplicate of #1597