Skip to content

deployment

Jump to bottom
Kadyapam edited this page May 24, 2026 · 2 revisions

Deployment

How noetl-gateway is built, packaged, and deployed.

The production target is GKE behind a Cloudflare Tunnel at https://gateway.mestumre.dev. The gateway is one component of the NoETL Helm release; the full GKE install lives at ops wiki / GKE Helm install.

Build

Cargo

cargo build --release

The release binary lands at target/release/noetl-gateway.

Container image

A multi-stage Dockerfile is at the repo root. Bundles the Rust release binary plus the Python Firestore listener sidecar (requirements-firestore.txt).

docker build -t noetl-gateway:<tag> .
docker push <registry>/noetl-gateway:<tag>

The image is the primary distribution channel for the gateway. GKE deployments pin to a tag; rolling forward is a tag bump.

Crate

cargo publish

Releases the noetl-gateway crate to crates.io. Useful for downstream consumers embedding the gateway or running it under their own supervision.

Kubernetes manifests

Reference manifests live in k8s/ in the gateway repo. Three concerns:

  1. Deployment — the gateway container, with an init/sidecar pattern for the Python Firestore listener.
  2. Service — typically ClusterIP; an Ingress or Cloudflare Tunnel forwards external traffic.
  3. Secret — the Firestore service-account JSON.

These manifests are reference snapshots; the actual production deployment for noetl/ops-managed clusters is templated by the Helm chart at noetl/ops/automation/helm/noetl/. That chart includes a gateway sub-deployment under the same release. The gateway's k8s/ manifests document the contract; the Helm chart implements it for the GKE profile.

Firestore credential provisioning

The gateway's subscription endpoint requires a Firestore service-account JSON mounted into the pod.

kubectl create secret generic gateway-firestore-credentials \
  --from-file=service-account.json=<path-to-your-sa-key.json> \
  -n gateway

Mount the secret in the gateway deployment at the path GATEWAY_FIRESTORE_CREDENTIALS_PATH (default /var/run/secrets/firestore/service-account.json).

If the gateway pod runs under a GKE workload identity that already has Firestore Viewer (or appropriate finer-grained roles) bound to it, the credential file may not be needed — the Python sidecar will pick up application-default credentials. See the secrets-and-credentials rule: already-in-place trust (workload identity) is preferred over re-mediating through a mounted credential.

Never commit the service-account JSON. The repo is public; the credential is per-deployment.

Health checks

The gateway exposes /health for liveness/readiness probes. Default GKE probe config:

livenessProbe:
  httpGet:
    path: /health
    port: 8090
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /health
    port: 8090
  initialDelaySeconds: 5
  periodSeconds: 5

The /health endpoint returns 200 when the gateway has:

  • a valid Postgres connection,
  • a valid NATS connection,
  • (if subscriptions are enabled) an open IPC handle to the Firestore sidecar.

Cloudflare Tunnel

External traffic for gateway.mestumre.dev enters via a Cloudflare Tunnel pod in the cluster's cloudflare namespace, which forwards to the in-cluster gateway service. The tunnel configuration sits in noetl/ops alongside the Helm chart; the gateway itself does not need TLS — TLS terminates at Cloudflare.

If you bring up your own gateway without Cloudflare, you can front the gateway with a Kubernetes Ingress + cert-manager (the same cert-manager the Helm chart already provisions).

Rolling forward

# 1. Bump version in Cargo.toml
# 2. Build + push the container
docker build -t <registry>/noetl-gateway:<new-tag> .
docker push <registry>/noetl-gateway:<new-tag>

# 3. Helm upgrade (from the ops repo)
helm upgrade --install noetl ./automation/helm/noetl \
  --namespace noetl \
  --reuse-values \
  --set gateway.image.tag=<new-tag>

# 4. Verify
kubectl rollout status deployment/gateway -n gateway
kubectl exec -n gateway deploy/gateway -- curl -s http://localhost:8090/health

The Helm chart handles the rolling update. Verify SSE clients reconnect cleanly after the new pod takes traffic.

Rolling back

helm history noetl -n noetl
helm rollback noetl <previous-revision> -n noetl

Atomic; restores the previous gateway image tag along with the rest of the chart.

Local development

cp env.example .env
# edit .env for your local NoETL + NATS + Postgres
cargo run

For a one-shot local NoETL stack, follow the kind install path in the ops repo.

Tests

cargo test

Runs the unit and integration tests. The v2.11.0 PR added mock- SSE coverage for the new subscription path; CI gates the build on these.

Related

Gateway

Surfaces

Operations

See also

Clone this wiki locally