Unable to connect to the ES when using K8s secrets #471
Comments
|
Is the secret mounted into the cleaner job? |
|
Umm its not, i think that is the issue. Let me try |
|
So i mounted the secret on the cleaner Cron, and somehow its not picking it up. |
|
A couple of remarks. Changing k8s objects manually is not supported - all changes should be done via jaeger CR.
|
|
When using an already provisioned elasticsearch cluster with basic authentication and TLS enabled, what is the key/map format of the secret that we need to provide to Jaeger in the storage section? storage:
type: elasticsearch
options:
es:
server-urls: http://elasticsearch:9200
secretName: jaeger-secrets |
|
As shown here, using the keys |
|
@objectiser thank you! |
|
@kevinearls, @jkandasa: would one of you be able to test the use case mentioned by @dushyant03? It should be possible to use an external ES, with username and password coming from secrets. |
|
@jpkrohling Not in the short term, I have a few too many other things queued up. @jkandasa do you have time? |
|
@jpkrohling I am using and external ES (from the Elastic Cloud on Kubernets (ECK) Operator) with self-signed certificates and username and password from secrets. The generated resources by the operator seems wrong and don't works $ kubectl --namespace=observability get po -l job-name=local-jaeger-tracing-es-index-cleaner-1565567700 NAME READY STATUS RESTARTS AGE
local-jaeger-tracing-es-index-cleaner-1565567700-5cmzh 0/1 Error 0 12h
local-jaeger-tracing-es-index-cleaner-1565567700-5qjhl 0/1 Error 0 12h
local-jaeger-tracing-es-index-cleaner-1565567700-7cgms 0/1 Error 0 12h
local-jaeger-tracing-es-index-cleaner-1565567700-d2rqp 0/1 Error 0 12h
local-jaeger-tracing-es-index-cleaner-1565567700-df7vf 0/1 Error 0 12h
local-jaeger-tracing-es-index-cleaner-1565567700-r8hfw 0/1 Error 0 12h
local-jaeger-tracing-es-index-cleaner-1565567700-w92dw 0/1 Error 0 11hHere is the generated apiVersion: batch/v1beta1
kind: CronJob
metadata:
creationTimestamp: "2019-08-08T13:26:27Z"
labels:
app: jaeger
app.kubernetes.io/component: cronjob-es-index-cleaner
app.kubernetes.io/instance: local-jaeger-tracing
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: local-jaeger-tracing-es-index-cleaner
app.kubernetes.io/part-of: jaeger
name: local-jaeger-tracing-es-index-cleaner
namespace: observability
ownerReferences:
- apiVersion: jaegertracing.io/v1
controller: true
kind: Jaeger
name: local-jaeger-tracing
uid: fe49d753-b9df-11e9-9c3f-00155d25521e
resourceVersion: "1815932"
selfLink: /apis/batch/v1beta1/namespaces/observability/cronjobs/local-jaeger-tracing-es-index-cleaner
uid: 202af614-b9e0-11e9-9c3f-00155d25521e
spec:
concurrencyPolicy: Allow
failedJobsHistoryLimit: 1
jobTemplate:
metadata:
creationTimestamp: null
spec:
parallelism: 1
template:
metadata:
annotations:
prometheus.io/scrape: "false"
sidecar.istio.io/inject: "false"
creationTimestamp: null
spec:
containers:
- args:
- "7"
- https://invited-guppy-elasticsearch-local-es-http.observability.svc:9200
envFrom:
- secretRef:
name: tracingstack-storage-45aec255
image: jaegertracing/jaeger-es-index-cleaner
imagePullPolicy: Always
name: local-jaeger-tracing-es-index-cleaner
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
schedule: 55 23 * * *
successfulJobsHistoryLimit: 3
suspend: false
status:
lastScheduleTime: "2019-08-11T23:55:00Z"Here are the logs from a job: Here is the apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
creationTimestamp: "2019-08-08T13:25:30Z"
generation: 2
name: local-jaeger-tracing
namespace: observability
resourceVersion: "1263119"
selfLink: /apis/jaegertracing.io/v1/namespaces/observability/jaegers/local-jaeger-tracing
uid: fe49d753-b9df-11e9-9c3f-00155d25521e
spec:
agent:
image: jaegertracing/jaeger-agent:1.13
options: null
resources: {}
strategy: Sidecar
volumeMounts: null
volumes: null
allInOne:
image: docker.io/jaegertracing/all-in-one:1.13.1
options:
collector.zipkin.http-port: "9411"
resources: {}
volumeMounts: null
volumes: null
collector:
image: ""
options: null
replicas: null
resources: {}
size: 0
volumeMounts: null
volumes: null
ingester:
image: ""
options: null
replicas: null
resources: {}
size: 0
volumeMounts: null
volumes: null
ingress:
enabled: false
resources: {}
security: none
volumeMounts: null
volumes: null
query:
image: ""
options: null
replicas: null
resources: {}
size: 0
volumeMounts: null
volumes: null
resources: {}
sampling:
options: {}
storage:
cassandraCreateSchema:
datacenter: ""
enabled: null
image: ""
mode: ""
dependencies:
cassandraClientAuthEnabled: false
elasticsearchClientNodeOnly: false
elasticsearchNodesWanOnly: false
enabled: true
image: jaegertracing/spark-dependencies
javaOpts: ""
schedule: 55 23 * * *
sparkMaster: ""
elasticsearch:
image: ""
nodeCount: 1
redundancyPolicy: ZeroRedundancy
resources: {}
storage: {}
esIndexCleaner:
enabled: true
image: jaegertracing/jaeger-es-index-cleaner
numberOfDays: 7
schedule: 55 23 * * *
esRollover:
conditions: ""
image: jaegertracing/jaeger-es-rollover
readTTL: ""
schedule: '*/30 * * * *'
options:
es.server-urls: https://invited-guppy-elasticsearch-local-es-http.observability.svc:9200
es.tls.ca: /etc/ssl/certs/tls.crt
secretName: tracingstack-storage-45aec255
type: elasticsearch
strategy: allInOne
ui:
options: {}
volumeMounts:
- mountPath: /etc/ssl/certs
name: es-tls
volumes:
- name: es-tls
secret:
defaultMode: 420
secretName: invited-guppy-elasticsearch-local-es-http-certs-public
status: {} |
|
cc @pavolloffay |
I guess you will have to specify TLS options for the index cleaner https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/es/esCleaner.py#L22-L25 which is not supported at the moment. Index cleaner CR exposes only a few options https://godoc.org/github.com/jaegertracing/jaeger-operator/pkg/apis/jaegertracing/v1#JaegerEsIndexCleanerSpec. Other configuration is derived from storage flags e.g. |
|
@dushyant03 were you able to connect to ES using user/pass? |
|
I have crated #592 for the TLS configuration with ES jobs. |
|
I'm running into a similar issue also using the EKS operator. I've mounted the secret to the cronjob spec and supplied the arguments through the ENV setting but it still appears to be throwing cert errors. |
|
@malz remember that Jaeger operator will revert the changes manually done on the objects (you can undeploy operator to avoid it). I think you have to also use client cert and the key https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/es/esCleaner.py#L22-L25 |
Of course! I've tried deploying the pod using the Client Cert and Key however I'm getting issues with every combination of certs produced by the ES Operator. Not sure if this is necessarily a problem with Jaeger. |
|
@malz could you please provide a link to ES operator you are using? And if you make it work the ES CR and jaeger CR you used. It might be helpful for other folks. |
I'm using the official Elastic Operator: https://github.com/elastic/cloud-on-k8s version 0.0.9, running on GKE. Jaeger collector and query run perfectly but the index cleaning jobs aren't connecting. |
|
PR #614 adds TLS options to ES jobs. If the collector and query are able to connect then the cron jobs should be able too - if the configuration is correct. |
|
Hello, I am having exactly the same issue. I have tried many things, including recommendations in this post, but I´m not able of seeing what I´m doing wrong. Ingester and Query can connect to ES with no issues, however, the cronjobs fails because of the error @secat was having: "self signed certificate in certificate chain". I am using the Jaeger Operator as well and my storage definition is: es-cred contains elastic credentials: ES_PASSWORD: 24 bytes I am mounting es certificates through a secret on top of /es-certs. This is the cronjob created: Could you please help me understanding why the cron jobs don´t work with this configuration? Thank you very much. |
|
I will have a look at this shortly. Many people experience problems when using TLS with ES. In the meantime could you please paste here logs from cronjobs?
|
|
Hello @pavolloffay. Thank you very much for your quick reaction! Yes, I've seen many people having this issue when using the operator and, after reading some posts and threads, I haven't found any workaround or final solution, that´s why I'm calling for help :). I have tried pretty much every configuration and some of the changes proposed in the operator to make sure it mounts volumes and secrets required on the cronjob containers. Jaeger Ingester, Collector and Query connect to ES with no issue. Based on my understanding, the configuration should be the same as for the cronjobs (Operator gets it from storage options). I see username and password should be used and I'm passing these from a secret to the jobs via operator, together with the certs I've got from elasticsearch. However, the jobs seem to keep failing. Please find below the logs from the jobs, as per your request. Regarding the point you've made about disabling sparkjob when TLS is enabled, don´t we need spark jobs running every day, same as for the index cleaner? If we were to run spark and index cleaner cronjobs sepparately (flagging them as enable=false in the operator config), do you have any sample yaml I could use where TLS is used to connect to ES? Thank you very much! Thank you very much, @pavolloffay! |
|
To my understanding, all failures in this thread are caused when using user/pass with CA cert or The ES scripts/cronjobs do not support skip verify or using the CA without |
|
Hello @pavolloffay. Yes to enabling cronjobs to support skip-host-verify feature. Can you think of a workaround meanwhile? Regarding es.tls=true, in order to discard a problem with it, I tested creating a cronjob by myself instead of using the operator (I couldn´t pass this argument to the cronjobs via the operator because what we have already discussed). The job fails because of the same reason, with the same logs ("self signed certificate in certificate chain), what makes me think that it might not be the only issue. I´ve used same credentials and certificate as for the Query, Ingester and Collector, which connect properly. The cronjob definition I´ve used is the following one: |
|
The workaround is not to use insecure TLS and rather use mTLS. I am working on a fix to allow using insecure and CA cert in python scripts. |
|
This can be considered as a duplicate of #592 |
Alright, thank you very much @pavolloffay, that's great! From a configuration perspective, do we have to do any changes or cronjobs will pick up the username, password and ca cert from the storage configuration? |
|
There won't be any other configuration required. Here is a simple CR. I will also improve docs and probably write a blog post explaining how people can use Jaeger operator with Elastic CO operator. # setup an elasticsearch with `make es`
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
# Note: This assumes elasticsearch is running in the "default" namespace.
server-urls: https://quickstart-es-http.default.svc:9200
use-aliases: true
tls.ca: /es/secrets/ca.crt
# tls.skip-host-verify: true
# username: elastic
# password: ql7hbmqfzzkrtn6klcdsh8n5
secretName: jaeger-secret
volumeMounts:
- name: secrets
mountPath: /es/secrets/
readOnly: true
volumes:
- name: secrets
secret:
secretName: quickstart-es-http-certs-public |
HI,
I have setup the ES cleaner cronjob for an externally hosted Elastic search cluster, and i am trying to use secrets stored in kubernetes to connect to the ES cluster.
Below is my yaml
And it somehow does not work and i get the below error
PS - If i hardcode the username and password, it works fine.
Please let me know if i am doing something wrong
The text was updated successfully, but these errors were encountered: