Talks

Watch the SmallWorlds Lightning Talk.

Refer to the SmallWorlds Architecture Diagram for system topology and data flows.

SmallWorlds Setup Guide

The SmallWorlds dashboard — auto-discovered applications and infrastructure at a glance.

Warning

Prototype — not production-ready. SmallWorlds is still in an early prototyping state. It is intended for experimentation and evaluation only, and is not yet suitable for production environments. Expect breaking changes, incomplete hardening, and no stability or upgrade guarantees. Use at your own risk.

Important

Not Hetzner-only. smallworlds-init.sh supports two deployment targets: hetzner (a Hetzner Cloud VM provisioned by Terraform) and local (an existing Linux machine in your LAN, bootstrapped in place over SSH — no cloud account needed). Hetzner is still required for public DNS — either via Terraform on the hetzner target, or via a free Hetzner DNS zone + API token on an internet-exposed local deployment. A LAN-only local deployment needs no Hetzner account, no domain registration, and no cloud resources at all. See the "Deployment Instructions" section below and doc/local-deployment.md for details.

This document outlines the deployment process for a SmallWorlds GitOps cluster. The architecture relies on an upstream foundation repository and a private, user-controlled configuration repository.

System Components

This project is built upon several foundational open-source technologies, core infrastructure services (installed by default), and optional user applications (selectively installed during initialization):

Infrastructure and Cluster Management

Name	Source URL	Role in this Project
Terraform	terraform.io	Infrastructure as Code tool used on the hetzner target to provision the VM and DNS; on the local target only the (optional) DNS zone is Hetzner-managed, the node itself is bootstrapped over SSH without Terraform.
Kubernetes	kubernetes.io	Core container orchestration system that serves as the foundation for the cluster.
Argo CD	argoproj.github.io/cd	GitOps continuous delivery tool that synchronizes cluster state with the configuration repository (accessible at deploy.<domain>).
Velero	velero.io	Cluster backup and disaster recovery solution.
Grafana	grafana.com	Operational dashboard for cluster monitoring and observability.
CloudNativePG	cloudnative-pg.io	High-availability PostgreSQL database clustering.
Garage	garagehq.deuxfleurs.fr	S3-compatible object storage backend.
Homepage	gethomepage.dev	Application dashboard automatically configured and accessible at dashboard.<domain>.
Keycloak	keycloak.org	Identity Provider (IdP) for Single Sign-On (SSO) and WebAuthn/Passkey management.
Stalwart	stalw.art	Self-hosted mail server with OIDC directory integration.
Traefik	traefik.io	Ingress routing and reverse proxy for handling incoming requests.
Cert-Manager	cert-manager.io	Automated TLS certificate provisioning and management.
Operator Console	operator-console/	The privileged administration interface (console.<domain>). Authenticates through Keycloak with Observer/Operator/Owner roles and explains every capability from configuration, delivery, runtime, access and protection evidence. Reachable only from the Private Network. See doc/tenant-operator-console.md.

End User Applications

Name	Source URL	Role in this Project
Collabora Online	collaboraonline.com	Powerful online office suite for collaborative document editing. Integrated into Nextcloud.
Excalidraw	excalidraw.com	Virtual collaborative whiteboard tool.
Forgejo	forgejo.org	Git repository management and software collaboration.
Immich	immich.app	High performance photo and video backup.
Jitsi Meet	jitsi.org	Secure video conferencing and communication platform.
Nextcloud	nextcloud.com	File synchronization and collaboration.
Plane	plane.so	Open-source project management tool.
Bulwark	github.com/bulwarkmail/webmail	JMAP webmail client connected to Stalwart.

---

Deployment Instructions

SmallWorlds supports two deployment targets — the installer asks which one you want as its first question:

• hetzner — a Hetzner Cloud VM provisioned by Terraform, with public DNS and Let's Encrypt certificates. The internet-facing default.
• local — an existing Linux machine in your LAN (laptop, mini-PC, home server; 32 GB RAM recommended), bootstrapped in place over SSH without Terraform. By default LAN-only: no public DNS (you add router-DNS or /etc/hosts entries) and self-signed certificates. Optionally internet-exposed: with a registered domain, a Hetzner DNS token and router port forwards (80/tcp, 443/tcp, 10000/udp), the wizard sets up Let's Encrypt certificates and an in-cluster DDNS job that keeps the DNS records pointed at your home IP. See doc/local-deployment.md for requirements and limitations before you start.

Steps 1 and 3 below apply to the hetzner target only; steps 2 and 4 apply to both.

1. Domain Registration & DNS Configuration (hetzner target only)

Important

Domain Registration is manual: The SmallWorlds setup scripts do not automatically register or reserve the domain name for you. You must manually register the domain at a registrar of your choice (e.g., Hetzner Domain service, Namecheap, Cloudflare) and point the domain's Nameservers to Hetzner's DNS servers (e.g., helium.ns.hetzner.de, oxygen.ns.hetzner.com, hydrogen.ns.hetzner.com). Domain registration will incur costs at your registrar. For a LAN-only local deployment none of this applies — the domain never has to be registered, since name resolution happens inside your LAN. For an internet-exposed local deployment, domain registration and the Hetzner nameservers ARE required (records are then managed by the in-cluster DDNS job instead of Terraform).

The DNS zone and DNS records are automatically managed via the Hetzner API token provided during provisioning (which is free of charge). Subdomains are routed to the provisioned server IP.

A .dev cluster additionally gets its own scoped mail domain: addresses are user@dev.<domain>, with MX/SPF/DKIM/DMARC records placed under the dev subdomain so they can never conflict with production's mail records at the zone apex (see doc/tenant-stalwart.md).

2. Configuration Repository Initialization

A private Git repository is required to store application state and configuration overrides.

Execute the initialization script from the root of this repository:

./admin-tools/prepare-community-repo.sh

This script handles:

• Prompting for your target domain (e.g. smallworlds.network) and environment extension in subdomain syntax (e.g. .dev for a dev cluster, giving hostnames like identity.dev.smallworlds.network; leave empty for production).
• Automatically creating a private GitHub repository using the gh CLI (if installed), or allowing you to provide an empty Git URL manually.
• Interactive selection of optional applications.
• Generation of the corresponding kustomization.yaml overlays, injecting environment-specific patches for the chosen domain.
• Initialization of the local Git repository and automatic push to your remote repository.

3. Infrastructure Provider Setup (hetzner target only)

For the local target, skip this section — instead make sure the target machine meets the requirements in doc/local-deployment.md (systemd Linux, SSH + sudo access, firewalld disabled or configured for k3s, 100 GB+ free disk).

For the hetzner target, these steps are required:

1. Create a Hetzner Cloud account and a new project.
2. Generate an API Token with Read & Write permissions. Save this token.
3. In the Hetzner Cloud Console, navigate to Primary IPs and click Create Primary IP. Select IPv4, choose your target location (e.g., Helsinki/hel1), and name it exactly smallworlds-ip (or smallworlds-ip-dev if deploying a .dev environment — Hetzner resource names always use the dash form, regardless of the DNS syntax). Leave it unassigned. Terraform will attach it during provisioning.

4. Cluster Provisioning

Warning

Before your first run against a fresh hetzner node (or a first-time internet-exposed local node), make sure your overlay repo (my-community-config, from step 2) is pinned to the latest smallworlds release tag — see "Managing Updates" below. Some infrastructure-level changes (cloud-init) take effect immediately on a brand-new node, while their corresponding ArgoCD-managed component only appears once your overlay's pinned tag includes it. If those are out of sync at first boot, the result isn't a degraded app or two — it can be a total outage (e.g. every TLS certificate stuck, no app reachable) with nothing obviously pointing at the cause. A newly created overlay may not have caught up via the weekly Renovate PR yet, so check/bump it manually first.

Execute the bootstrap script to provision the server and install Kubernetes/ArgoCD. On the hetzner target it provisions the VM and DNS via Terraform; on the local target it bootstraps your LAN machine over SSH (asking for its SSH target, e.g. root@192.168.1.50, or localhost to install on the machine you are running the script on).

git clone https://github.com/stephan271/smallworlds.git
cd smallworlds
./smallworlds-init.sh

When prompted for Git credentials, provide:

• URL: The HTTPS URL of your private configuration repository (SSH URLs are unsupported).
• Username: Your Git platform username.
• Access Token: A Personal Access Token (PAT) with read-only access to repository contents.

5. Authentication Configuration

By default, registration is invitation-only. To enable self-registration, patch the Keycloak configuration via your kustomization.yaml:

patches:
  - target:
      kind: Job
      name: keycloak-realm-config
      namespace: keycloak
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/1/value
        value: "self-registration"

6. Custom Application Deployment

To deploy external applications, add standard Kubernetes manifests to your configuration repository and declare them in your kustomization.yaml. ArgoCD will synchronize the state.

---

Maintenance Operations

The rebuild procedures below are for the hetzner target (they drive Terraform). For local deployments, the equivalent lifecycle (uninstall/reinstall preserving data, full wipe) is described in doc/local-deployment.md.

Rebuild (Preserve Data)

This procedure replaces the VM while retaining the persistent volume containing cluster state and data.

cd infrastructure/terraform
terraform destroy -target=hcloud_server.smallworlds_pilot_node
terraform apply

Clean Rebuild (Wipe Data, Preserve TLS Certificates)

This procedure wipes all cluster data but first backs up the TLS certificates to your local machine (~/.smallworlds/cert-backups/<production|dev>/), then re-injects them into the new cluster to avoid Let's Encrypt rate limits. For the dev cluster, prefix each command with ENV_EXT=".dev".

./admin-tools/prepare-fresh-rebuild.sh
cd infrastructure/terraform
terraform destroy -target=hcloud_server.smallworlds_pilot_node
terraform apply
cd ../.. && ./admin-tools/restore-certs-from-laptop.sh

The restore script waits for the new cluster's API, then applies the saved certificate secrets before cert-manager re-issues them; if no local backup exists it exits cleanly and certificates are issued fresh. You can also snapshot certificates at any time without a rebuild via ./admin-tools/backup-certs-to-laptop.sh.

Backups & Storage

Storage layout, the backup chain, restore procedures and scaling paths are all documented in doc/storage-and-backup.md. The chain is Garage-first: databases (CloudNativePG/barman), Velero's cluster-state dumps and nightly PV copies all land in the in-cluster Garage S3, and a single replicator CronJob mirrors every bucket offsite at 04:00.

Important

The offsite leg requires one-time operator setup — an S3 target (recommended: a Backblaze B2 bucket with versioning) and the replicator-config-secret. Until then, backups never leave the node and the nightly replicator job fails (which the monitoring emails about). Follow infrastructure/kubernetes/tenants/backup-replicator/README.md.

Managing Updates — the two-repo model

SmallWorlds runs on two repositories, and understanding their interplay is the key to safe day-2 operations:

Repo	Role	Who changes it
smallworlds (this repo, public)	The upstream base: all app/infra manifests under infrastructure/kubernetes/. Released as semver tags (v1.0.0, v1.1.0, …).	The SmallWorlds project.
my-community-config (yours, private)	The overlay ArgoCD actually deploys from. Each app's kustomization.yaml remote-references the base at a pinned tag (?ref=v1.0.0) plus your local patches.	You, the operator.

Each cluster (production, .dev) has its own private overlay repo, generated the same way by prepare-community-repo.sh with the matching environment extension.

ArgoCD only watches your private overlay. It does not track the base's moving branch. Because the overlay pins the base to an immutable tag, upstream changes never reach your cluster on their own — adopting a new base version is always a deliberate, auditable action in your repo.

Note

This separates two independent concerns. Drift reconciliation (ArgoCD selfHeal) keeps the cluster matching whatever is declared and stays on — it's safe and low-risk. Version adoption (moving to newer upstream code) is the deliberate lever described below. Don't conflate them.

Adopting a new release manually

Bump the pinned tag everywhere in your overlay and commit — ArgoCD (which watches this repo) then syncs the change deterministically:

# in my-community-config, e.g. v1.0.0 -> v1.1.0
grep -rl 'v1.0.0' . | xargs sed -i 's#v1.0.0#v1.1.0#g'
git commit -am "Bump upstream smallworlds base to v1.1.0" && git push

Rollback is just as simple: revert that commit. Because the ref is immutable, what you tested is exactly what deploys.

Tip

prepare-community-repo.sh pins to a release tag by default (it prompts for the version). You can answer HEAD to always track the latest main, but avoid it in production: ArgoCD only re-pulls a floating HEAD non-deterministically (on cache expiry), so you lose reproducibility and can't tell what's actually running.

Automated weekly update proposals (Renovate)

An in-cluster Renovate CronJob is pre-wired to reduce the toil without giving up control. Every Monday it opens one pull request in your private overlay that bumps the pinned base tag to the newest smallworlds release (config in my-community-config/renovate.json). It does not auto-merge — you review the changelog and merge when ready; the merge is the commit that triggers ArgoCD. This gives you a low-effort cadence and a human gate and a full audit trail.

Requirements for the PR automation:

• The private overlay must be listed in the Renovate CronJob's RENOVATE_REPOSITORIES (added via an overlay patch in your kustomization.yaml, so operator-specific config stays out of the public base).
• The Git token Renovate uses (repo-git-creds) must have pull-request / write access to the private overlay repo, not just read.

---

Administrative Tools (admin-tools/)

The admin-tools/ directory contains helper scripts to automate common operations and maintenance tasks:

Cluster Lifecycle & Disaster Recovery

• backup-certs-to-laptop.sh: Downloads Let's Encrypt certificates from the cluster to your local machine to avoid rate limits during rebuilds.
• restore-certs-from-laptop.sh: Injects previously backed-up certificates into a newly built cluster before cert-manager re-issues them.
• prepare-fresh-rebuild.sh: Prepares a cluster for a clean reinstall by backing up certificates and wiping all persistent application data on the server.
• destroy-cluster.sh: Wraps the cluster teardown process. It automatically backs up your certificates and then safely executes terraform destroy to delete the cloud resources.
• build-golden-image.sh: Builds a pre-configured Hetzner snapshot (golden image) to speed up future VM provisioning.

Repository & Version Management

• prepare-community-repo.sh: Interactive script to initialize a new private configuration repository, select applications, and wire it up to the upstream base.
• bump-version.sh: Automatically detects the highest release tag, increments the patch version, creates a new Git tag, and pushes it to origin.
• update-community-version.sh: Updates all kustomization.yaml files in your private config repository to pin them to a specific upstream release, then commits and pushes the change.

Utilities

• test-pr-locally.sh: Deploys a local testing environment to validate pull requests.
• generate_domain_patches.py: Automatically generates Kustomize domain patches for all applications when a non-default domain or environment extension (e.g., .dev) is used.
• bulk-invite.py & update_realm_json.py: Python utilities for Keycloak realm manipulation and user invitations.

---

End-to-End Smoke Tests

Browser-based Playwright smoke tests simulate real users logging in via SSO and exercising each application. They live in e2e-tests/tests and run against a live SmallWorlds community.

./e2e-tests/run-smoke-tests.sh <domain> [keycloak-admin-password]

# e.g.
./e2e-tests/run-smoke-tests.sh smallworlds.network

If the Keycloak admin password is omitted, the runner reads it from the cluster via kubectl. The script checks service availability, provisions two test users (sw-test-alice, sw-test-bob), and runs the suite.

OIDC test depth (two-tier)

Full OIDC login roundtrips require the applications to trust the TLS certificate of identity.<domain> for their server-side discovery/token calls. That holds in production (Let's Encrypt) but is structurally impossible on ephemeral staging clusters. The suite therefore runs at one of two depths:

Mode	How to run	What it verifies
Shallow wiring (default)	./e2e-tests/run-smoke-tests.sh <domain>	Each app redirects into Keycloak's authorize endpoint — proving client config, secrets, issuer URL, in-cluster DNS, and OIDC wiring. The deeper login-roundtrip tests are skipped.
Full OIDC	FULL_OIDC=1 ./e2e-tests/run-smoke-tests.sh <domain>	The complete login roundtrips run — auto-login into each app, then asserting the app's authenticated UI loads (Files listing, inbox, timeline, dashboard, etc.). Requires app-trusted certificates, i.e. production.

In shallow mode you'll see the roundtrip tests reported as skipped (with the reason Full OIDC roundtrip needs app-trusted certificates — run with FULL_OIDC=1); this is expected, not a failure.

Other options

These environment variables override the CLI arguments:

Variable	Effect
DOMAIN	Target domain (alternative to the first positional argument).
KC_ADMIN_PASS	Keycloak admin password (alternative to the second positional argument).
FULL_OIDC=1	Run the full OIDC login roundtrips (see above).
HEADED=1	Run in headed browser mode instead of headless.
SLOWMO=500	Slow operations down by the given number of milliseconds — useful when watching a headed run.
SKIP_PROVISION=1	Skip test-user provisioning (reuse existing sw-test-* users).
KUBECONFIG	Path to the kubeconfig used to read the admin password (default: ~/.smallworlds/kubeconfigs/<production|dev>.yaml, matching the configured env_ext).

The HTML report is written to e2e-tests/reports/html; view it with cd e2e-tests && npx playwright show-report reports/html.

---

Developer Guide: Cutting a New Release

A SmallWorlds release is a Git tag plus two independently published families of signed attachments. Tagging alone does not produce a usable release — the tag only becomes installable once both publishing workflows have run against it.

What a complete release contains

Group	Files	Produced by	Consumed by
Bootstrap assets (4)	smallworlds-bootstrap-<tag>-linux-amd64.tar.gz + .sha256 + .sig, bootstrap-assets.manifest.json	Publish Bootstrap Assets	The launcher, which downloads and verifies them, then installs their contents onto the Cluster Node.
Launcher artifacts (10)	5 native archives (linux_amd64, linux_arm64, darwin_amd64, darwin_arm64, windows_amd64), SHA256SUMS + .sig + .pub, the SPDX SBOM, THIRD-PARTY-NOTICES.txt	Publish Bootstrap Launcher	Operators, who download one archive by hand to get the launcher in the first place.

Note

The GitHub UI shows two extra entries — "Source code (zip)" and "Source code (tar.gz)". Those are auto-generated by GitHub and are not real attachments, so a complete release reads as 16 in the browser but 14 via gh release view --json assets.

A tag carrying only the bootstrap assets is installable by an operator who already has a launcher, but offers no way to obtain one. Releases v1.2.25 through v1.2.27 are in exactly that state; v1.2.28 is the first complete release.

One-time setup

Both workflows sign with the repository secret SMALLWORLDS_RELEASE_ED25519_PRIVATE_KEY_B64. See docs/releases/github-release-signing-setup.md.

Important

The signing key must match defaultReleaseSigningPublicKey, compiled into the launcher under operator-console/internal/bootstrapassets/. That compiled key is the only trust anchor a launcher has: assets signed by any other key are rejected outright, with no override. Rotating the release key is therefore a code change plus a launcher rebuild, not a secrets change.

Step 1 — Review and commit the input lock

Every release needs a reviewed lock at docs/releases/bootstrap-inputs/<tag>.json pinning the exact K3s installer and Argo CD manifest — version, HTTPS URL, and SHA-256 — that the packager may download. Copy the previous release's lock, update release, and either keep the pins (re-verify the digests against their official sources first) or bump them.

cp docs/releases/bootstrap-inputs/v1.2.28.json docs/releases/bootstrap-inputs/v1.2.29.json
# edit "release", then independently re-verify each digest:
curl -sS https://get.k3s.io | sha256sum

Commit and push this before tagging. See docs/releases/bootstrap-inputs/README.md.

Step 2 — Tag the release

./admin-tools/bump-version.sh        # tags main with the next patch version and pushes

Warning

Both workflows build from the tagged tree, not from main. Anything committed after the tag is invisible to them, and a tag can never be packaged with tooling it does not contain. This is precisely why v1.2.25–v1.2.27 cannot be given launcher artifacts retroactively: admin-tools/package-bootstrap-launcher.sh was committed after all three were tagged. The fix is always a new tag, never a re-run.

bump-version.sh only tags — it does not create the input lock. Do step 1 first.

Step 3 — Validate before publishing

Run both workflows with publish=false first. Each builds and signs with a throwaway key and uploads the result as a downloadable workflow artifact, touching no release:

gh workflow run publish-bootstrap-assets.yml   -f release=v1.2.29 -f publish=false
gh workflow run publish-bootstrap-launcher.yml -f release=v1.2.29 -f publish=false
gh run list --workflow=publish-bootstrap-launcher.yml

Note

In a validation run the step "Publish GitHub Release attachments" reports skipped and "Upload validation artifact" reports success. That is the designed outcome, not a failure.

Step 4 — Publish

gh workflow run publish-bootstrap-assets.yml   -f release=v1.2.29 -f publish=true
gh workflow run publish-bootstrap-launcher.yml -f release=v1.2.29 -f publish=true

Run the assets workflow first: it creates the GitHub Release object if absent (gh release view || gh release create), whereas the launcher workflow only does gh release upload and needs the release to exist.

The launcher job takes roughly five minutes — npm ci, the client build, go test ./..., then five cross-compiles — and uploads all ten files in its final step. An empty-looking release mid-run is normal; check the run, not the release page. Avoid dispatching concurrent publish=true runs against one tag: they race on gh release upload --clobber over identical filenames.

Step 5 — Verify the published release

gh api repos/stephan271/smallworlds/releases/tags/v1.2.29 -q '.assets | length'   # expect 14

gh release download v1.2.29 -D /tmp/verify -p 'SHA256SUMS*' \
  -p 'smallworlds-bootstrap-launcher_v1.2.29_linux_amd64.tar.gz'
cd /tmp/verify
openssl pkeyutl -verify -rawin -pubin -inkey SHA256SUMS.pub -in SHA256SUMS -sigfile SHA256SUMS.sig
sha256sum -c --ignore-missing SHA256SUMS

# the published key must equal the launcher's compiled trust anchor
openssl pkey -pubin -in SHA256SUMS.pub -outform DER | tail -c 32 | base64 -w0
grep -o 'defaultReleaseSigningPublicKey = "[^"]*"' \
  operator-console/internal/bootstrapassets/*.go

The last check is the one that matters: if the published key and the compiled anchor diverge, every launcher will refuse the release.

Local dry run

To exercise packaging without touching GitHub — worth doing before a first release from a materially changed tree:

cd operator-console/web && npm ci && npm run generate:api && npm run check && npm run build
cd .. && go test ./...
cd .. && SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)" \
  admin-tools/package-bootstrap-launcher.sh --version v1.2.29 --output-directory /tmp/launcher-test
admin-tools/test-build-bootstrap-assets.sh      # structural + reproducibility check for the asset archive

Packaging is deterministic within a toolchain: CI pins Go 1.25.0, so archives built locally on a different Go version will not match CI's checksums. That is expected and not a reproducibility failure. A local GOEXPERIMENT (e.g. nodwarf5) can also cause spurious internal compiler errors during the cross-compiles; clear it with GOEXPERIMENT=none.

Tip

Adding a descriptor for the new release to DefaultCatalog() in operator-console/internal/bootstrapassets/manager.go is optional. A launcher that has never heard of a release falls back to fetching that release's published bootstrap-assets.manifest.json and accepting it solely on the compiled signing key's word — which is what lets an old launcher install a newer release, and what avoids the circularity of needing an archive's digest before the archive exists.

Step 6 — Let operators adopt it

Publishing does not deploy anything. Each operator's private overlay pins an immutable tag, so the new release only reaches a cluster when someone bumps that pin — either manually or by merging the weekly Renovate PR. See "Managing Updates — the two-repo model" above.

---

Developer Guide: Adding a New Application

When adding a new application (tenant) to the SmallWorlds cluster, please ensure you complete all the items on this integration checklist:

• Pin Specific Versions: Always use a specific, stable container image tag (e.g., v2.4.1 or 24.04) rather than latest in your Kubernetes manifests to ensure reproducible deployments.
• Add E2E Tests: Write end-to-end smoke tests (using Playwright) in the e2e-tests/tests directory to verify the application's core functionality and SSO integration.
• Integrate with Dashboard: Add Homepage annotations (e.g., gethomepage.dev/enabled: "true") to the application's Ingress resource so it automatically appears in the user dashboard.
• Make it Selectable: Add the application's identifier to the OPTIONAL_APPS array in the admin-tools/prepare-community-repo.sh script so users can easily toggle its installation.
• Update the README Table: Add the application to the "End User Applications" table in this README file, including a description and its source URL.
• Document the Implementation: Add a descriptive markdown file (or update an existing one) in the doc/ directory detailing the application's YAML manifests, configurations, and architecture.
• Configure DNS Records: Add the application's generic subdomain (e.g., whiteboard, meet, office) to the DNS records array in infrastructure/terraform/main.tf so Terraform provisions the A-record.
• Provision Web Certificates: Ensure the Ingress resource specifies the correct cert-manager.io/cluster-issuer: letsencrypt-prod annotation and the tls hosts block to automate Let's Encrypt SSL certificate generation.