Provide example for druid-kubernetes-overlord-extensions basePodSpec.yaml

[schmichri]

Hey all,

I'm struggling with the setup of druid-kubernetes-overlord-extensions with the customTemplateAdapter with the following setting snippet:

druid.indexer.runner.k8s.adapter.type=customTemplateAdapter
druid.indexer.runner.k8s.podTemplate.base=/druid/taskSpecTemplates/basePodSpec.yaml
druid.indexer.runner.k8s.podTemplate.index_parallel=/druid/taskSpecTemplates/basePodSpec.yaml

Basically I have no idea how the basePodSpec.yaml shall look like.

Is it just a "normal" Kubernetes PodSpec with kind: Pod or is it a kind: PodTemplate?

Can somebody maybe post a basePodSpec.yaml here as an first example?

[georgew5656]

@schmichri I got this working using the druid operator with the following template
pod-template.yaml

apiVersion: "v1"
kind: "PodTemplate"
template:
  metadata:
    annotations:
      custom-annotation: "hello"
    labels:
      custom-label: "hello"
  spec:
    affinity: {}
    containers:
    - command:
        - sh
        - -c
        - |
          /peon.sh /druid/data 1
          sleep 30
      env:
      - name: CUSTOM_ENV_VARIABLE
        value: "hello"
      image: apache/druid:26.0.0
      name: main
      ports:
      - containerPort: 8091
        name: druid-tls-port
        protocol: TCP
      - containerPort: 8100
        name: druid-port
        protocol: TCP
      resources:
        limits:
          cpu: "1"
          memory: 2400M
        requests:
          cpu: "1"
          memory: 2400M
      volumeMounts:
      - mountPath: /opt/druid/conf/druid/cluster/_common
        name: common-config-volume
        readOnly: true
      - mountPath: /opt/druid/conf/druid/cluster/master/coordinator-overlord # this is still mounted in this location because that's where peon.sh looks
        name: nodetype-config-volume
        readOnly: true
      - mountPath: /druid/data
        name: data-volume
      - mountPath: /druid/deepstorage
        name: deepstorage-volume
    restartPolicy: "Never"
    securityContext:
      fsGroup: 1000
      runAsGroup: 1000
      runAsUser: 1000
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - configMap:
        defaultMode: 420
        name: tiny-cluster-druid-common-config
      name: common-config-volume
    - configMap:
        defaultMode: 420
        name: druid-tiny-cluster-peons-config
      name: nodetype-config-volume
    - emptyDir: {}
      name: data-volume
    - emptyDir: {}
      name: deepstorage-volume

I created a configMap with the above template and mounted it to the overlord at

- mountPath: /opt/druid/conf/druid/cluster
      name: "peon-pod-template-volume"
      readOnly: true

I set this on the overlord

druid.indexer.runner.type=k8s
druid.indexer.runner.k8s.podTemplate.base=/opt/druid/conf/druid/cluster/pod-template.yaml
druid.indexer.runner.k8s.adapter.type=customTemplateAdapter
druid.indexer.runner.javaOptsArray=[]
druid.indexer.runner.namespace=default

I also had to add these two things to the peon runtime properties as well (otherwise the pod fails to start due to issues finding some files.

druid.indexer.task.baseTaskDir=/druid/data
druid.indexer.runner.type=k8s

I can try to put up a PR with more detailed startup instructions later.

[schmichri]

Thanks a lot!