Disabling certs creation in v2.32.2 helm chart breaks the jaeger deployment

[AshutoshNirkhe]

Describe the bug
v2.32.x seems to be introducing admission webhooks and associated certificates. But if I want to disable them all, its not possible to do so as per https://github.com/jaegertracing/helm-charts/blob/jaeger-operator-2.32.2/charts/jaeger-operator/templates/deployment.yaml#L52

Are they mandatory going forward ?
Ref - https://www.jaegertracing.io/docs/1.35/operator/#prerequisite

To Reproduce
Steps to reproduce the behavior:

1. Disable webhooks and certificates via values.yaml

certs:
  certificate:
    create: false
  issuer:
    create: false
webhooks:
  mutatingWebhook:
    create: false
  service:
    create: false
  validatingWebhook:
    create: false

2. Try to upgrade jaeger helm chart from 2.27.0 to 2.32.2

3. Pod will stay in Waiting/ContainerCreating state forever and will show below message if we do kubectl describe on pod,

  Warning  FailedMount  50s (x10 over 5m)    kubelet            MountVolume.SetUp failed for volume "cert" : secret "jaeger-operator-service-cert" not found
  Warning  FailedMount  42s (x2 over 2m57s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-7bkf8]: timed out waiting for the condition

Expected behavior
Currently deployment.yaml tries to load volume from certificate secret even if we disable cert creation. It needs to have the condition to check for that flag before volume/volume mount creation. That way, webhooks/certificates creation won't be a mandatory configuration.

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

• OS: Linux Ubuntu
• Jaeger version: 1.34.1
• Deployment: Kubernetes v1.22

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.

Additional context

[rotarur]

is there an ETA for this bug fix? We're facing the same issue

[AshutoshNirkhe]

is there an ETA for this bug fix? We're facing the same issue

Seems like they don't want us to disable those. See #376 for details.

[avishefi]

There should be a way to disable creating the certificates and to provide the names of the relevant secrets created before running the jaeger-operator Helm chart.

[mjnagel]

There should be a way to disable creating the certificates and to provider the names of the relevant secrets created before running the jaeger-operator Helm chart.

@avishefi This is possible, reference the below values:

certs:
  issuer:
    create: false # defaults to true, but you can disable the chart creation of the issuer
    name: "" # leave empty
  certificate:
    create: false # defaults to true but you can disable the chart creation of the cert
    namespace: "jaeger" # namespace for the secret
    secretName: "my-secret" # secret with the cert

The other thing you may need depending on how you create the certs would be the clientConfig.caBundle value set in the webhook spec - which isn't currently exposed in values. I think if you make use of cert-manager this will all be handled for you via the annotations. Another option that we use is the webhook cert gen to dynamically generate these certs and then patch the webhooks (although this requires an extra job tied to the lifecycle of the chart).

You will need to have the cert no matter what, but you can absolutely disable the chart creating it.

[avishefi]

@mjnagel Thanks, I have raised it on the PR that introduced it.

As for the CA bundle - you are right, it is handled by the cert-manager annotation and the Helm chart currently doesn't support other implementations.

[razorsk8jz]

There should be a way to disable creating the certificates and to provider the names of the relevant secrets created before running the jaeger-operator Helm chart.

@avishefi This is possible, reference the below values:

certs:
  issuer:
    create: false # defaults to true, but you can disable the chart creation of the issuer
    name: "" # leave empty
  certificate:
    create: false # defaults to true but you can disable the chart creation of the cert
    namespace: "jaeger" # namespace for the secret
    secretName: "my-secret" # secret with the cert

The other thing you may need depending on how you create the certs would be the clientConfig.caBundle value set in the webhook spec - which isn't currently exposed in values. I think if you make use of cert-manager this will all be handled for you via the annotations. Another option that we use is the webhook cert gen to dynamically generate these certs and then patch the webhooks (although this requires an extra job tied to the lifecycle of the chart).

You will need to have the cert no matter what, but you can absolutely disable the chart creating it.

I attempted to do this but I appear to get an error when i try to use the cert - x509: certificate signed by unknown authority"

The cert/secret are created using these commands

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=jaeger-operator-webhook-service.monitoring.svc" -addext "subjectAltName = DNS:jaeger-operator-webhook-service.monitoring.svc"

kubectl -n monitoring create secret tls jaeger-operator-cert --key="tls.key" --cert="tls.crt" --dry-run=client -o yaml | kubectl apply -f -

Any ideas are appreciated

[avishefi]

@razorsk8jz this is a self-signed certificate so it won't be recognized. You can either use a known CA or use a self-signed CA certificate as a flag to Jaeger's skip-host-verify flags.

[sergiomcalzada]

@razorsk8jz I'm stuck too in that part. where should set the flag skip-host-verify? can't find the exact value in the values.yaml from the operator chart

I can provide also my custom certificates generated externally for my own managed domain [using lets-encrypt]. But the required host jaeger-operator-webhook-service.[namespace].svc prevent generating.

[sergiomcalzada]

I was investigating, an has created a certificate using the self kubernetes CA [https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/]

The issue here is that the operator, don't accept the kubernetes GA /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Will be great to have a flag to include that ca as valid. Or at least being able to customize the valid ca

[shicli]

Hi @mjnagel ，I am deploying the jaeger operator service through helm and would like to disable cert-magger as we have our own TLS service. But I disabled them in Value, but I still check Cert when starting the service. May I know how to handle this change?
certs:
issuer:
create: false
name: ""
certificate:
create: false
namespace: ""
secretName: ""
issuerKind: Issuer

webhooks:
mutatingWebhook:
create: false
validatingWebhook:
create: false
port: 9443
service:
annotations: {}
create: false
name: ""

[shicli]

@mjnagel I have submitted a question. Could you please help me take a look #492

[oxycoder]

The operator should allow us to use existing issuer.
Example I'm using cert-manager and had cluster issuer.
It's can be done easy in open telemetry

        --set admissionWebhooks.certManager.enabled=false \
        --set admissionWebhooks.certManager.issuerRef.kind=ClusterIssuer \
        --set admissionWebhooks.certManager.issuerRef.name=letsencrypt-prod \

But it's not possible in jaeger, is there anything relate to #376 ?