Production Checklist

Production Checklist

Everything that must change between "it runs on my laptop" and "it runs in production".

Images now run non-root (UID 1654) on a chiselled base — no shell, no package manager, 11 OS packages instead of 112. Vulnerability scanning runs as a build gate before publication, and every image is signed and ships an SBOM and build provenance. This checklist covers what you control.

---

Must do

Secrets

• Signing key generated and set. openssl rand -base64 32. Never ship the placeholder — it is publicly known, and anyone with it can mint valid tokens for your instance.
• Admin password changed from the default password.
• Secrets injected from a secret store, not baked into an image or committed to git.
• No secrets on the command line — they land in shell history and docker inspect.
• Distinct keys per environment. Staging must not share a signing key with production.

Data

• A real database, not SQLite. → Persistence and Databases
• Both workflow and identity persistence configured.
• Backups running, and a restore actually tested.
• Connection pool sized for real concurrency.

Network

• TLS terminated at a reverse proxy or ingress. The containers speak plain HTTP.
• CORS restricted to specific origins. Never *.
• API not publicly exposed unless you need it to be. Consider Blazor Server Studio so the API stays internal → Choosing an Image
• Http:HttpActivityOptions:BaseUrl set to the externally reachable URL, if workflows expose HTTP endpoints.

Images

• Pinned to an immutable tag — <elsa-version>-build.<n>, or a digest. Never latest. → Pulling Images
• Nuplane packages pinned to exact versions, so a restart isn't an unplanned upgrade.
• AutomaticReconciliation disabled, so package changes follow your deploy schedule.
• Registry credentials stored as secrets. Your subscription token is pull-only by construction; keep it in a secret store, never in a repository.
• Image signature verified before deployment, and the signing identity checked. → Pulling Images

Operations

• Health probes wired: /health for readiness, /alive for liveness. Not the other way around — /health checks dependencies, so using it for liveness turns a brief database blip into a crash loop.
• Resource limits set. Unbounded memory plus a runaway workflow evicts its neighbours.
• Logs shipped somewhere that survives container replacement.
• Restart policy set (unless-stopped, or a Kubernetes controller).

---

Should do

Observability

The images emit OpenTelemetry traces, metrics and structured logs. Point them at a collector:

-e OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
-e OTEL_SERVICE_NAME=elsa-production

Worth alerting on: workflow execution failure rate, execution latency, /health failures, database connection pool exhaustion.

Scaling

Before running more than one replica:

• Shared database — never SQLite
• Message bus configured (RabbitMQ or Azure Service Bus)
• Distributed runtime features enabled
• Durable scheduling (Quartz), not in-memory

⚠️ Multiple replicas without distributed coordination execute scheduled and triggered workflows more than once. This is not a performance problem, it is a correctness one, and it will not be obvious in testing. → Extending with Nuplane

Access

• Admin permissions narrowed from ["*"] where practical
• Studio not publicly reachable unless it needs to be
• A plan for rotating the signing key (it invalidates active sessions)

---

Upgrades

• Back up the database first. Migrations run at startup and rolling the image back does not roll the schema back.
• Read the release notes.
• Test in a non-production environment on a copy of production data.
• Know your rollback path — including what to do about a migrated schema.
• Upgrade one minor version at a time.

---

Security posture

What Valence Works provides: non-root images on a minimal chiselled base, CVE scanning as a build gate that runs before publication, a published SBOM and build provenance, and cosign-signed images — all live now, on both the Community and paid lines. A committed security patch cadence is what the paid tiers add on top. → Pulling Images for how to verify a signature yourself.

Writable paths

Everything the application writes lives outside /app, on three paths with different natures — worth mounting differently:

Path	Holds	Mount as
/data	The default SQLite database	A real volume / PVC, if you use SQLite. Back it up
/var/lib/valence-runtime	Extracted packages, Nuplane store state and lock file — and HOME, so the Data Protection key ring	See below
/tmp	NuGet scratch space	tmpfs

On an external database, /data is unused — with the SQLite features disabled nothing opens /data/elsa.db, so you can skip that mount entirely.

⚠️ /var/lib/valence-runtime is not purely a cache. The packages and store state are reproducible, but HOME lives here too, which is where ASP.NET Data Protection keeps its key ring. On an emptyDir, a replacement pod gets a new key ring and every previously issued authentication cookie becomes unreadable — users are signed out on each rollout, and with more than one replica they are signed out unpredictably. Choose deliberately:

• Single replica, sign-outs on deploy acceptable → emptyDir is fine.
• Anything else → back it with a persistent volume, or configure Data Protection to persist its keys to a shared store and keep the emptyDir.

readOnlyRootFilesystem

Supported. Mount the three paths above writable — /data only if you use SQLite — and the root filesystem can be read-only:

docker run --read-only \
  -v elsa-data:/data -v elsa-runtime:/var/lib/valence-runtime --tmpfs /tmp \
  -v "$(pwd)/packages:/app/packages:ro" ...

A package directory may be mounted :ro, and needs no particular ownership.

Verified on the server and combined images: read-only root, --network none, :ro packages — full package set loaded, no filesystem violations, serving normally.

Studio needs no writable mounts at all — it hosts no Elsa runtime, so it has no package loader and no database. It does still register ASP.NET Data Protection, which wants to write a key ring under HOME; on a read-only root that falls back to in-memory keys, so plan for the same sign-out behaviour described above.

There is also no container HEALTHCHECK, because a chiselled image has no shell or curl to run one. Use an orchestrator HTTP probe against /health and /alive — Kubernetes httpGet needs nothing inside the image.

What remains yours: TLS, network boundaries, secret management, database security, access control, backups, and everything about the workflows you author.

Report suspected vulnerabilities privately — see Editions and Support. Please don't use the public issue tracker for these.

---

Quick audit

These run on the host. The images are chiselled — no shell, no printenv, no curl — so anything of the shape docker exec <container> printenv fails with executable file not found. Inspect from outside instead.

# What config actually landed, and is the signing key still the placeholder?
docker inspect elsa --format '{{json .Config.Env}}' | tr ',' '\n' | grep -i -E 'cshells|signingkey'

# Running as non-root? Expect 1654.
docker inspect <image> --format '{{.Config.User}}'

# Pinned to a digest?
docker inspect --format='{{index .RepoDigests 0}}' <image>

# Healthy?
curl -f https://elsa.example.com/health

---

Next: Troubleshooting · Editions and Support