Added multithreading support to thanos compact

[ahysing]

THIS IS WORK IN PROGRESS

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

• Added command line flag --compact.concurrency to thanos compact statefulset
• Added command line flag --downsample.concurrency to thanos compact statefulset

This change adds the flag --compact.concurrency to the thanos compact StatefulSet. This sets the number of goroutines used during compaction. Set this number to any value X greater than one, and thanos compact will be fixed to the given number of threads. this feature requires X times more CPU and memory to thanos compact as stated on Resources > Cpu. The default is one goroutine.

This change also adds the flag --downsample.concurrency to the thanos compact StatefulSet. This sets the number of goroutines used during downsampling. Resources > Flags this flag. The default is to run on one goroutine.

Verification

Then built with the regular routines

mkdir test-kube-thanos
cd test-kube-thanos
jb install github.com/ahysing/kube-thanos/jsonnet/kube-thanos@concurrent-thanos-compact

# Fetching build scripts and configuration
curl https://raw.githubusercontent.com/thanos-io/kube-thanos/main/build.sh > build.sh
curl https://raw.githubusercontent.com/thanos-io/kube-thanos/main/example.jsonnet > example.jsonnet

Added two new flags to example.json downsampleConcurrency: 2, and compactConcurrency: 2 right in line 58:

local c = t.compact(commonConfig {
  downsampleConcurrency: 2,
  compactConcurrency: 2,
  replicas: 1
});

At the end of the file add a thanos compactor by adding one line

{ ['thanos-compact-' + name]: c[name] for name in std.objectFields(c) } +

Performed a build with bash build.sh example.jsonnet. Inspect the output and verify that the file in /manifests looks OK.

Output from tests

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: database-compactor
    app.kubernetes.io/instance: thanos-compact
    app.kubernetes.io/name: thanos-compact
    app.kubernetes.io/version: v0.24.0
  name: thanos-compact
  namespace: thanos
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: database-compactor
      app.kubernetes.io/instance: thanos-compact
      app.kubernetes.io/name: thanos-compact
  serviceName: thanos-compact
  template:
    metadata:
      labels:
        app.kubernetes.io/component: database-compactor
        app.kubernetes.io/instance: thanos-compact
        app.kubernetes.io/name: thanos-compact
        app.kubernetes.io/version: v0.24.0
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - thanos-compact
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - thanos-compact
              namespaces:
              - thanos
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - compact
        - --wait
        - --log.level=info
        - --log.format=logfmt
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --retention.resolution-raw=0d
        - --retention.resolution-5m=0d
        - --retention.resolution-1h=0d
        - --delete-delay=48h
        - --compact.concurrency=2
        - --downsample.concurrency=2
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos.yaml
              name: thanos-objectstorage
        - name: HOST_IP_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: quay.io/thanos/thanos:v0.24.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-compact
        ports:
        - containerPort: 10902
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
        resources: {}
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/thanos/compact
          name: data
          readOnly: false
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        fsGroup: 65534
        runAsUser: 65534
      serviceAccountName: thanos-compact
      terminationGracePeriodSeconds: 120
      volumes: []
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/component: database-compactor
        app.kubernetes.io/instance: thanos-compact
        app.kubernetes.io/name: thanos-compact
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi

[yeya24]

What about skipping the equals 1 check and always explicitly setting the two flags?
This way users can always know the concurrency without knowing the default value is 1.

[ahysing]

I will get that done for you.

[yeya24]

Thank you! One small nit, can you please fix the lint issue?

[ahysing]

I have a hard time figuring out what that latest crash is all about... Job "build" crashed with exit code 1.
I can look into this tomorrow..

[yeya24]

@ahysing I guess you need to run generate to regenerate yamls.

[yeya24]

Thank you for the contribution! @ahysing