Support TLS for spark dependencies

[pavolloffay]

Spark dependencies job uses Java keystore for certificates. The docker image allows to configure java opts with ssl configuration - JAVA_OPTS=-Djavax.net.ssl..

The certs can be mounted via volume and volume mounts which are part of JaegerCommonSpec. The issue that these certs have to be imported into java keystore/thruststore.

https://developers.redhat.com/blog/2017/11/22/dynamically-creating-java-keystores-openshift/ suggests using init container for the job.

Todos:

• allow configuring JAVA_OPTS to point to keystore/thruststore
• import certs into keystore/thruststore

[pavolloffay]

Dependent issue in spark-dependencies #294

[pavolloffay]

@jpkrohling this is not ES specific. The dependency job supports multiple backends.

[jorgex1]

Hi @pavolloffay are there any updates on this?

[pavolloffay]

There aren't ane news. If anybody has free cycles feel free to take it.

[jorgex1]

Spent a bit doing some digging through things and came across this page https://hub.helm.sh/charts/jaegertracing/jaeger

The page mentions that in order to have a version of the spark job working one would need to get the certificates from the elasticsearch cluster and do

$ keytool -import -trustcacerts -keystore trust.store -storepass <some pass> -alias es-root -file <es.pem>

$ kubectl create configmap jaeger-tls --from-file=trust.store --from-file=<es.pem>

And then use the jaeger-tls configmap within the spark job definition as such:

spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: jaeger-spark
            args:
              - --java.opts=-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=<some pass>
            volumeMounts:
              - name: jaeger-tls
                mountPath: /tls
                subPath: 
                readOnly: true
          volumes:
            - name: jaeger-tls
              configMap:
                name: jaeger-tls
          ...

Which all sounds like it is in accordance with the issue description.
Think it may be a bit better to define jaeger-tls as a secret though, but other than that, is all that more or less correct?

[pavolloffay]

Correct the certs have to be imported into java keystore in the spark image.

[pavolloffay]

My idea was to implement a script in the spark image that would do that if the certs are specified as env vars.

[jam01]

Hey guys, is there any work around for this atm? I'm trying to follow the thread of TLS related changes and options. Is there any skip-tls JAVA_OPT or something that can be used temporarily?

To maybe help others:
add the java options described above to the jaeger CR
manually create a JKS trust store from the elasticsearch-es-http-certs-public secret
downscale the operator replicaset to 0
edit the cronjob container template to mount a volume from the secret

[jpkrohling]

This would probably a question to ask in the repository that holds the code for the spark dependencies processor. https://github.com/jaegertracing/spark-dependencies

[aleksandrovpa]

Hi, maybe this is stupid question, but i couldn't run a spark job with following configuration:

spark:
  enabled: true
  cmdlineParams:
    java.opts: "-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit"
  extraConfigmapMounts:
    - name: jaeger-tls
      mountPath: /tls
      subPath: ""
      configMap: jaeger-tls
      readOnly: true

Here is error:

/entrypoint.sh: 37: exec: --java.opts=-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit: not found

Can someone show me, what i'm doing wrong? or maybe there are any workaround, how to get spark job works.
Thnx

[kevinearls]

@aleksandrovpa I don't have much time to look at this at the moment, but you may find what you need in these two items:

#1332
#1359

[jam01]

@aleksandrovpa we're just composing the cron job manually in order to bootstrap the trust store through an init container. You can check this out https://github.com/MS3Inc/tavros/blob/main/tests/integration/targets/playbooks/provision_playbook/example.com/platform/jaeger/default/cronjob-spark-dependencies.yaml#L39

[AnatomicJC]

Thanks @jorgex1 for your comment, it was very helpful
👍 🥇

[primeroz]

is it currently the only solution to write our own job ? i see that the dependencies definition now allow to mount volumes but i am not sure how to create the java trustStore without an init container like the one in @jam01 link

[zablvit]

I achieved the desired result (given you have pem certificate as a secret) by adding an alpine java init container to the spark dependencies pod, which will create me a truststore, which I can then mount to the spark container and provide the java opts means to it:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: jaeger-spark
spec:
  schedule: "30 */12 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          initContainers:
            - name: create-jks-truststore
              image: openjdk:11
              volumeMounts:
                - name: truststore-dir
                  mountPath: "/target"
                - name: es-cert
                  mountPath: "/src"
                  readOnly: true
              command:
                - "/bin/sh"
                - "-c"
                - "rm -rf /target/* && keytool -import -file /src/cert.pem -storetype JKS -keystore /target/truststore.jks -storepass password -noprompt"
          containers:
            - name: jaeger-spark
              image: jaegertracing/spark-dependencies:latest
              env:
                - name: STORAGE
                  value: "elasticsearch"
                - name: ES_NODES
                  value: "https://elasticsearch.default:9200"
                - name: ES_NODES_WAN_ONLY
                  value: "false"
                - name: ES_USERNAME
                  value: user
                - name: ES_PASSWORD
                  value: password
                - name: JAVA_OPTS
                  value: "-Djavax.net.ssl.trustStore=/elasticsearch/truststore.jks -Djavax.net.ssl.trustStorePassword=password"
              volumeMounts:
                - name: truststore-dir
                  mountPath: "/elasticsearch"
                - name: temp-dir
                  mountPath: "/tmp"
          restartPolicy: OnFailure
          volumes:
            - name: truststore-dir
              emptyDir: {}
            - name: temp-dir
              emptyDir: {}
            - name: es-cert
              secret:
                secretName: my_secret_with_elasticsearch_certificate_pem
                items:
                  - key: elasticsearch_certificate
                    path: "cert.pem"

Hope this helps someone.

[dankolt]

Hi, maybe this is stupid question, but i couldn't run a spark job with following configuration:

spark:
  enabled: true
  cmdlineParams:
    java.opts: "-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit"
  extraConfigmapMounts:
    - name: jaeger-tls
      mountPath: /tls
      subPath: ""
      configMap: jaeger-tls
      readOnly: true

Here is error:

/entrypoint.sh: 37: exec: --java.opts=-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit: not found

Can someone show me, what i'm doing wrong? or maybe there are any workaround, how to get spark job works. Thnx

It is much simple: entrypoint.sh from docker image is a shell script. So it used JAVA_OPTS env to pass options to java process. You need to pass env variable JAVA_OPTS by extraEnv param from values.yaml like this:

spark:
  extraEnv:
    - name: "JAVA_OPTS"
      value: "-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit"