Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unable to update route error #128

Closed
tchellomello opened this issue Mar 28, 2022 · 3 comments
Closed

unable to update route error #128

tchellomello opened this issue Mar 28, 2022 · 3 comments

Comments

@tchellomello
Copy link

tchellomello commented Mar 28, 2022 •
edited
Loading

I'm testing the cert-utils-operator on Openshift 4.9.21 and I'm hitting the error below:

[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z	ERROR	controllers.route_certificate_controller	unable to update route	{"route-certificate": "default/wildcard-https", "route": {"apiVersion": "route.openshift.io/v1", "kind": "Route", "namespace": "default", "name": "wildcard-https"}, "error": "Operation cannot be fulfilled on routes.route.openshift.io \"wildcard-https\": the object has been modified; please apply your changes to the latest version and try again"} 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] github.com/go-logr/zapr.(*zapLogger).Error 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] github.com/redhat-cop/cert-utils-operator/controllers/route.(*RouteCertificateReconciler).Reconcile 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/work/cert-utils-operator/cert-utils-operator/controllers/route/route_controller.go:213 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntil 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.UntilWithContext 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z	DEBUG	util.api	object is not ConditionsAware, not setting status 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z	ERROR	controller-runtime.manager.controller.route	Reconciler error	{"reconciler group": "route.openshift.io", "reconciler kind": "Route", "name": "wildcard-https", "namespace": "default", "error": "Operation cannot be fulfilled on routes.route.openshift.io \"wildcard-https\": the object has been modified; please apply your changes to the latest version and try again"} 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] github.com/go-logr/zapr.(*zapLogger).Error 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:302 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.BackoffUntil 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntil 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] k8s.io/apimachinery/pkg/util/wait.UntilWithContext 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 	/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.465Z	DEBUG	controller-runtime.manager.events	Warning	{"object": {"kind":"Route","namespace":"default","name":"wildcard-https","uid":"8b51b5c7-7a20-4585-bac0-8bef898de398","apiVersion":"route.openshift.io/v1","resourceVersion":"1174098"}, "reason": "ProcessingError", "message": "Operation cannot be fulfilled on routes.route.openshift.io \"wildcard-https\": the object has been modified; please apply your changes to the latest version and try again"} 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.519Z	DEBUG	util.api	object is not ConditionsAware, not setting status 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-28T18:43:58.519Z	DEBUG	util.api	object is not ConditionsAware, not setting status

Some of the artifacts that I'm using:

custom domain

---
apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata:
  name: testing-local
spec:
  domain: testing.local
  certificate:
    name: wildcard-testing-local-tls-cert
    namespace: default

## patching
kubectl -n openshift-ingress-operator patch ingresscontroller testing-local -p '{"spec":{"routeAdmission":{"wildcardPolicy":"WildcardsAllowed"}}}' --type=merge
kubectl -n openshift-ingress-operator patch ingresscontroller default -p '{"spec":{"routeAdmission":{"wildcardPolicy":"WildcardsAllowed"}}}' --type=merge
kubectl -n openshift-ingress-operator patch ingresscontroller testing-local -p '{"spec":{"domain":"testing.local"}}' --type=merge
kubectl -n openshift-ingress-operator patch ingresscontroller testing-local -p '{"spec":{"routeSelector":{"matchLabels":{"type":"testing-local"}}}}' --type=merge

certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard
  namespace: default
spec:
  secretName: wildcard-testing-local-tls-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - Celonis
  commonName: wildcard.testing.local
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
    - wildcard.testing.local
    - "*.testing.local"
  issuerRef:
    name: testing-local-ca
    kind: ClusterIssuer
    group: cert-manager.io
  secretTemplate:
    annotations:
      # https://github.com/redhat-cop/cert-utils-operator#generating-kubernetes-events
      cert-utils-operator.redhat-cop.io/generate-cert-expiry-alert: "true"
      cert-utils-operator.redhat-cop.io/cert-expiry-check-frequency: "7d"           # (days)  with which frequency should the system check is a certificate is expiring
      cert-utils-operator.redhat-cop.io/cert-soon-to-expire-check-frequency: "1h"   # (hours) with which frequency should the system check is a certificate is expired, once it's close to expiring
      cert-utils-operator.redhat-cop.io/cert-soon-to-expire-threshold: "2160h"      # (90 days)  what is the interval of time below which we consider the certificate close to expiry

route

---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: hello-world
    type: testing-local
  name: wildcard-https
  annotations:
    cert-utils-operator.redhat-cop.io/inject-CA: "false"
    cert-utils-operator.redhat-cop.io/certs-from-secret: "wildcard-testing-local-tls-cert"
    route.openshift.io/termination: edge
spec:
  host: wildcard.testing.local
  port:
    targetPort: 8080-tcp
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  to:
    kind: Service
    name: hello-world
    weight: 100
  wildcardPolicy: Subdomain

Running a curl command I can see the certificate without problems, but why is the operator throwing this error?

Thanks
mmello
@raffaelespazzoli
Copy link
Contributor

usually that error is transient, and it happens when two operators update an object at the same time. It's the optimistic locking mechanism inside the go-client. Retrying usually fixes the error, so I was going to ask if eventually the operator was able to put the certs in the secret.
It looks like now from the manifest you shared, but can you confirm? anything else suspect in the log.
The reason why you see the secret is that you are setting it as the ingress default (so that every route will get it unless a route specific in defined) and you are setting it as a route specific secret.
You only need it once, being it a wildcard secret, it should probably be set as the default cert in the ingress.

@tchellomello
Copy link
Author

thanks for your quick response @raffaelespazzoli

yes, the cert-manager operator created and populate the wildcard-testing-local-tls-cert secret and then which was later consumed by the route. If I inspect the route object, I can see the cert-utils-operator

kubectl get routes wildcard-https -o yaml | kubectl neat
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
    cert-utils-operator.redhat-cop.io/certs-from-secret: wildcard-testing-local-tls-cert
    cert-utils-operator.redhat-cop.io/inject-CA: "false"
    route.openshift.io/termination: edge
  labels:
    app: hello-world
    type: testing-local
  name: wildcard-https
  namespace: default
spec:
  host: wildcard.testing.local
  port:
    targetPort: 8080-tcp
  tls:
    certificate: |
      -----BEGIN CERTIFICATE-----
      XXXX
      -----END CERTIFICATE----
    insecureEdgeTerminationPolicy: Redirect
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      xxxxx
      -----END RSA PRIVATE KEY-----
    termination: edge
  to:
    kind: Service
    name: hello-world
    weight: 100
  wildcardPolicy: Subdomain

I also can confirm that whenever I renew the certificate via cert-manager, the operator does it job updating the secret.

For example:

before

$ curl -k -L  -v https://b.testing.local 2>&1 >/dev/null   | grep 'date:' | head -n2
*  start date: Mar 28 18:46:39 2022 GMT
*  expire date: Jun 26 18:46:39 2022 GMT

refresh cert

$ kubectl cert-manager renew -n default wildcard
Manually triggered issuance of Certificate default/wildcard

after

$ curl -k -L  -v https://b.testing.local 2>&1 >/dev/null   | grep 'date:' | head -n2
*  start date: Mar 29 14:36:38 2022 GMT
*  expire date: Jun 27 14:36:38 2022 GMT

Upon refresh, the cert-utils-operator logs seem to be fine

[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:44:52.272Z	DEBUG	util.api	object is not ConditionsAware, not setting status 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:44:52.272Z	DEBUG	util.api	object is not ConditionsAware, not setting status 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:45:10.923Z	DEBUG	controller-runtime.manager.events	Warning	{"object": {"kind":"Secret","namespace":"default","name":"wildcard-testing-local-tls-cert","uid":"2c0cc2b1-f326-48e6-b0f2-9422857cd3cc","apiVersion":"v1","resourceVersion":"1553226"}, "reason": "Certs Soon to Expire", "message": "Certificate expiring in 89 days"} 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:45:10.989Z	DEBUG	util.api	object is not ConditionsAware, not setting status 
[cert-utils-operator-controller-manager-644575f449-ff67v manager] 2022-03-29T14:45:10.990Z	DEBUG	util.api	object is not ConditionsAware, not setting status

So @raffaelespazzoli is it safe and expected then to assume the race condition error?
Thanks

@tchellomello
Copy link
Author

Thanks, @raffaelespazzoli for your assistance.

I'm closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tchellomello @raffaelespazzoli