Kubernetes

Kubernetes

Helm chart: deploy/helm/niks3. Image: ghcr.io/mic92/niks3 (amd64/arm64, contains niks3-server and niks3).

niks3 is stateless. Multiple replicas are fine.

Install

helm install niks3 oci://ghcr.io/mic92/charts/niks3 -n niks3 --create-namespace -f values.yaml

database:
  existingSecret: niks3-pg-app   # key `uri`
s3:
  endpoint: s3.eu-central-1.amazonaws.com
  bucket: my-nix-cache
  useIAM: true
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/niks3
auth:
  existingSecret: niks3-token    # key `token`, >= 36 chars
signing:
  existingSecret: niks3-signing
  keys: [cache.example.com-1]
cacheURL: https://cache.example.com
ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  hosts:
    - host: niks3.example.com
      paths: [{ path: /, pathType: Prefix }]

helm show values oci://ghcr.io/mic92/charts/niks3 lists everything else. On OpenShift set podSecurityContext.runAsUser: null and runAsGroup: null so the SCC can assign a UID.

Postgres

One connection string, mounted from a Secret as NIKS3_DB_FILE. Migrations run on startup.

• CloudNativePG: database.existingSecret: <cluster>-app (key uri is the default).
• RDS / Cloud SQL: put the DSN in a Secret. For IAM auth run the vendor proxy as sidecar and point at localhost.
• database.url inline for quick tests.
• With neither set, the server uses PGHOST/PGUSER/... from extraEnv.

S3

s3.useIAM: true for IRSA / Pod Identity / GKE Workload Identity, otherwise a Secret with access-key / secret-key.

Clients upload straight to S3 via presigned URLs, so the S3 host must be reachable from wherever niks3 push runs. For in-cluster S3 (RustFS, Garage, MinIO), let the server use the Service and presign against the public route: endpoint: rustfs.rustfs.svc:9000, useSSL: false, bucketLookup: path, publicURL: https://s3.example.com.

Probes, metrics, GC

• /healthz liveness, /readyz readiness (503 while Postgres is down)
• /metrics for Prometheus. metrics.serviceMonitor.enabled adds a ServiceMonitor
• GC is a CronJob (gc.schedule, gc.olderThan, gc.failedUploadsOlderThan)

Pushing from pods

Service account tokens are OIDC JWTs, so in-cluster builders can push without the shared API token:

auth:
  workloadIdentity:
    enabled: true
    allowedServiceAccounts: ["ci:builder", "ci:nix-*"]
    scopes: [write]   # add admin for GC/pin deletion. Per-subject rules: see OIDC, via auth.oidcProviders

Builder pod:

spec:
  serviceAccountName: builder
  containers:
    - name: build
      env:
        - { name: NIKS3_SERVER_URL, value: http://niks3.niks3.svc }
        - { name: NIKS3_AUTH_TOKEN_FILE, value: /var/run/secrets/niks3/token }
      volumeMounts:
        - { name: niks3-token, mountPath: /var/run/secrets/niks3, readOnly: true }
  volumes:
    - name: niks3-token
      projected:
        sources:
          - serviceAccountToken: { audience: niks3, path: token }

niks3 push ./result then authenticates as system:serviceaccount:ci:builder. Outside a pod: kubectl -n ci create token builder --audience niks3.

Under the hood this is an OIDC provider with bound_subject, ca_file / bearer_token_file pointing at the pod's service account mount and jwks_url set to https://kubernetes.default.svc/openid/v1/jwks. The issuer is read from the iss claim of niks3's own service account token, so it matches on any distribution (kubeadm, EKS, GKE, AKS, ...). Set workloadIdentity.issuer only to override that. auth.oidcProviders adds further providers (GitHub, GitLab) to the same config.