Query frontend should not be protected with mTLS when gateway is disabled

[vdurante]

Hello! Not sure if this is the right place to open this issue...

I recently followed the docs here on how to get started with tempo on k8s using the operator.

I followed some other articles from grafana (I can share the urls if necessary), and all guides show how to connect grafana to tempo through HTTP using the url on port 3200 (something like http://query-frontend.default.svc.cluster.local:3200).

I tried that, but the endpoint was failing with the message: Client sent an HTTP request to an HTTPS server. Checking the logs on tempo, I was getting the following error: 20:55:28 http: TLS handshake error from 127.0.0.1:40728: tls: client didn't provide a certificate

I then found on the Release changelog that mTLS is enabled by default now, which makes me believe it is related to the error.

To solve the issue, I had to manually disable httpEncryption and grpcEncryption to false.

Would be cool if the quick start guide had an explanation on how to set mTLS up, or how to disable it.

As a complete newbie to docker and kubernetes, it would have helped me a lot if the quick start simply worked out of the box with the default settings.

[frzifus]

cc @rubenvp8510 @andreasgerstmayr

[andreasgerstmayr]

@rubenvp8510 can we disable mTLS on the query-frontend?

[rubenvp8510]

Yes i think we can. Ill take a look at this to see how can we disable it

[pavolloffay]

@vdurante are you on OpenShift or Kubernetes. I think the mTLS should be disabled on Kubernetes

[pavolloffay]

It seems that internal mTLS is enabled on Kubernetes as well.

@rubenvp8510 the query frontend should not use mTLS if the gateway is not deployed. @vdurante could you please share your Tempo configuration?

[vdurante]

Hey, I can share whatever you need. Just let me know if I shared the wrong thing, since I am still a beginner on k8s.

I basically applied the tempo-operator using this command:

kubectl apply -f https://github.com/grafana/tempo-operator/releases/latest/download/tempo-operator.yaml

And then I booted it with the following command:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true

To fix it, I have a feeling I had to change something on this TempoStack yaml, but I wasn't able to do that, so what I did instead was download the tempo-operator.yaml and modify lines 1426 and 1427 to the following:

      httpEncryption: false
      grpcEncryption: false

[rubenvp8510]

I cannot find a way to only disable tls on the query-frontend, I'm looking to see if there is anything, any flag or something else.

There are a couple of solutions

• Use other configmap for the component with the server TLS configuration disabled. TBH I don't like this one.
• Disable all HTTP mTLS, looking at the HTTP endpoints, mostly of them are from the query-frontend, and there are others for health and debug. metrics, and status. Not sure how bad is to have those unprotected. (https://grafana.com/docs/tempo/latest/api_docs/)

[andreasgerstmayr]

Fixed for now by disabling mTLS by default. We'll look for a way to (partially) enable it again in the future (#561).