SmallWorlds v1.2.43

Important

Stalwart is no longer installed by default. An overlay that relied on the base for it loses its mail server at this bump. To keep it, add to your overlay's root kustomization.yaml:

  - https://raw.githubusercontent.com/stephan271/smallworlds/v1.2.43/infrastructure/kubernetes/apps/stalwart.yaml

and a stalwart/kustomization.yaml referencing tenants/stalwart at this tag, as for any other optional app.

Mail is opt-in

Mail is an opt-in capability (docs/adr/0049), but the mail server was installed unconditionally — every community ran one whether or not it wanted mail. Nothing in the cluster requires it: onboarding hands out action-token links out of band, and alerting is moving to HTTP push. This is the first piece of that ADR to reach the manifests.

Stalwart leaves the master kustomization and joins OPTIONAL_APPS. Its namespace stays in namespaces.yaml, exactly as every other optional app already behaves. In the Operator Console it moves from a platform service (Required: true) to a community application, and out of the always-installed list its overlay renderer writes, so the renderer and prepare-community-repo.sh still agree file for file.

Bulwark is webmail, not a mail server, so it needs one. It can now be pointed at a mail server outside the cluster — typically another Stalwart on a host with a real IP and PTR, which is often the better deployment anyway. doc/tenant-other.md gains "Bulwark against a mail server outside the cluster" with the three values to patch and the real constraints (JMAP rather than IMAP; an auth scheme shared with this Keycloak; CORS on the external server). prepare-community-repo.sh warns when Bulwark is selected without Stalwart rather than silently shipping a webmail client nobody can log into.

Local installs default to Let's Encrypt

Certificates and internet exposure used to be one question, so a LAN cluster could only have self-signed certificates. They are independent: DNS-01 resolves the challenge through a TXT record, needing no inbound connection and no public A record, so a cluster nobody can reach still gets publicly trusted certificates. The wizard now asks separately — certificates default to letsencrypt, exposure still defaults to no.

This matters more than a browser warning suggests. Every app validates Keycloak's certificate against its own trust store with no override, so with self-signed certificates single sign-on does not work at all — Immich reports only Error in OAuth discovery: TypeError: fetch failed, naming neither TLS nor Keycloak. Devices that pin a certificate (the pod archive) must also be re-pinned at every renewal, on every member's hardware. Self-signed remains available for throwaway clusters, and the prompt now says what it costs.

Two related fixes: the Hetzner token was withheld from any LAN-only install, which with ACME as the default would leave cert-manager holding an empty token while the issuer reported Ready and every certificate sat pending with nothing saying why. It is now withheld only when the install does no DNS at all. And doc/local-deployment.md claimed HTTP-01 in two places when both bootstraps have always used DNS-01; its conversion recipe also claimed an issuer's type cannot be changed in place, when a merge patch nulling selfSigned while adding acme is accepted.

Removed

The golden-image rebuild workflow. It needed a Hetzner Cloud token as a repository secret — a token that cannot be scoped to a resource, so the trade was full account control in CI in exchange for staging boot speed. It never made that trade: no secret was ever set, and every scheduled run failed one second in. Nothing depended on it; test-pr-locally.sh falls back to a plain image, and admin-tools/build-golden-image.sh stays for manual runs.

Bootstrap inputs

Unchanged from v1.2.42: k3s v1.36.2+k3s1, Argo CD v3.4.5, both digests re-verified against their official sources.

SmallWorlds v1.2.42

A single fix, to the identity layer: member onboarding and recovery-code login could not work, on any cluster whose realm came from this repo's import.

What was broken

smallworlds-realm.json used three identifiers Keycloak does not have. The admin API accepts an unknown provider id without complaint and stores it as a realm row that reads back as enabled: true, so every static check passed and the failure surfaced only when a member clicked their onboarding link:

in the realm	what Keycloak calls it
recovery-auth-code-register	CONFIGURE_RECOVERY_AUTHN_CODES
recovery-auth-code-form	auth-recovery-authn-code-form
(absent)	UPDATE_PROFILE

UPDATE_PROFILE was missing outright: a realm import replaces Keycloak's default required-action set, and the import listed only four providers — while admin-tools/bulk-invite.py asks for that action on every invitation. Every link the script has ever produced failed with "Required actions included in the link are not valid."

The authenticator id sat in the bound login path (browser → forms → passkey-or-password → must-authenticate), so recovery-code login was dead too — the only fallback a passwordless realm has.

Also fixed: VERIFY_EMAIL no longer sets defaultAction, which had been attaching a pending action to every account created on a cluster with no mail (docs/adr/0049). It stays enabled, so self-registration mode still triggers it through realm-level verifyEmail.

What operators need to do

Bumping to this tag is not enough for an existing cluster. realm-config-job.yaml creates the realm only if it is absent and never updates an imported one, so a cluster already running keeps the broken identifiers after the bump. Those need the corrections applied to the live realm by hand — register UPDATE_PROFILE and CONFIGURE_RECOVERY_AUTHN_CODES as required actions, drop the phantom recovery-auth-code-register row, and replace the must-authenticate execution with auth-recovery-authn-code-form (a newly added execution lands at priority 0, ahead of the passkey, so it needs an explicit lower-priority call).

New clusters get all of it from the import.

Verified against a running Keycloak on a LAN cluster: every authenticator id in the realm now resolves against the live provider list, and an invitation link renders "Update Profile, WebAuthn Register Passwordless, Generate Recovery Codes" and completes through passkey registration.

Housekeeping

invite-links.csv and invite-qr/ are now git-ignored. bulk-invite.py writes them 0600 because each row logs its holder into that member's account, but nothing had stopped them being committed.

Bootstrap inputs are unchanged from v1.2.41: k3s v1.36.2+k3s1, Argo CD v3.4.5, both digests re-verified against their official sources.

SmallWorlds v1.2.41

Everything here was found by running the system end to end on a real LAN
deployment rather than by reading it. Each fix is on a path this project's own
documentation recommends, and none would have shown up in a code review.

Read before adopting

The database backup chain no longer uses CloudNativePG's in-tree Barman Cloud
support, which is deprecated in CNPG 1.30 and removed entirely in 1.31.0.
Before this release, a routine operator upgrade would have deleted both the
production of database Recovery Points and the ability to read the ones already
written — and because a cluster whose backups have stopped looks exactly like one
whose backups work, it would have surfaced during a restore.

Each database now declares an ObjectStore (barmancloud.cnpg.io/v1) and
references it from spec.plugins; the plugin itself installs at sync wave −10/−9
and needs cert-manager, which is already present. destinationPath and
serverName are unchanged, so archives written by the previous release remain
readable — verified by restoring from them through the plugin.

Two things to expect when you adopt it:

• A short WAL archiving gap at cutover. Between the in-tree archiver being
  removed and the plugin taking over, a few pg_stat_archiver failures on one
  segment are normal; the next segment archives cleanly. Four were observed here.
• serverName moves to the Cluster. The plugin's webhook forbids it on the
  ObjectStore — worth knowing if you carry local patches, because the upstream
  migration guide tells you to copy the in-tree block wholesale, and that puts
  the field in the one place it is not allowed.

See docs/adr/0050.

Fixes for things that could not work at all

• pod-gateway could never be selected. Its id contains a hyphen, so
  prepare-community-repo.sh aborted under set -e when it reached that app —
  after creating the GitHub repository and before writing a single overlay file.
  The one optional application the pod archive needs was unreachable through the
  supported path.
• POD_DATA_DIR never worked on a device. ReadWritePaths is a list, so the
  drop-in appended to it instead of replacing it and the unit kept a directory a
  custom-data-dir install never creates; systemd killed the service with
  226/NAMESPACE before the agent ran. This is the documented way to put a
  member's archive on an external disk, which is the normal case.
• First-boot certificate race. The bootstrap waited for cert-manager's CRD to
  be Established and then applied the ClusterIssuer immediately — but the
  validating webhook is a separate Deployment, and a single apply lost that race
  on a cold node. The result was not a missing object but a cluster where nothing
  would ever get a certificate, and an install that stopped before collecting the
  kubeconfig. Both bootstraps now retry until their existing deadline.

Restore procedures, corrected and drilled

doc/storage-and-backup.md §7.1 could not have worked as written: it pointed at
the operational Garage rather than garage-backup, omitted the signing region
without which Garage answers HTTP 400, and described the credential trap
backwards. All three are fixed, and §7.1, §7.2 (Velero) and §7.4 (Immich
originals from the pod archive) now carry the results of having actually been
run — including a database recovered with zero data loss across an hour of WAL
replay, and a deleted original restored with a byte-identical digest.

§7.2 also no longer assumes the velero CLI, which is precisely what a rebuilt
operator laptop will not have.

Full Changelog: v1.2.40...v1.2.41

SmallWorlds v1.2.40

A large release: the administrative surface is closed off, mail becomes opt-in,
and the shell installer's local (LAN) path works again.

Read before adopting

• Mail is now opt-in, and system notification no longer routes through it.
  Transactional mail is separated from the optional mail server. If you relied
  on alert email arriving via Stalwart, check where your notifications go now.
• Administrative interfaces are no longer publicly reachable. Argo CD,
  Grafana, SSH and the Kubernetes API are closed off, the Keycloak admin console
  is taken off the public internet, and the master realm is closed to it as well.
  If you administer your cluster through public URLs today, establish your
  private path before bumping to this release.
• Keycloak realm changes. Passwords are removed from the realm and the admin
  gains a second factor; recovery codes are reachable at login and issued at
  onboarding; invitation onboarding is fixed and gains a printable delivery path.
  The passkey policy is documented and the earlier "passwordless" claim corrected.
• The node patches itself automatically, and lateral movement to the
  databases is blocked by policy.

Installing on a LAN machine works again

smallworlds-init.sh with target local had stopped working entirely: the
bootstrap became an execution payload that refuses to run without release-pinned
verified assets and a profile that owns the node, and only the Bootstrap Launcher
ever supplied those. The shell path now builds the same payload from the same
declared inputs — verifying every download against
docs/releases/bootstrap-inputs/ — caches it, and rebuilds it whenever the
bootstrap script itself moves on.

It also pins the Argo CD root application to an exact overlay commit, which
it resolves before touching the machine. Consequence worth knowing: your overlay
repository must already exist and carry its first commit. If it does not, the
installer now says so immediately and names prepare-community-repo.sh, rather
than failing twenty minutes later inside Argo CD.

The pod archive is addressable

pod was missing from the in-cluster DNS override in both bootstraps and from
the generated hosts entries, and neither hostname generator knew the app — so a
selected pod gateway kept the project's own domain on its Ingress no matter what
domain the operator chose. All three now carry it.

The Immich exporter reaches the gateway through its Service rather than the
public hostname. A nightly job that never leaves the node should not depend on
public DNS resolving inside the cluster, nor on a certificate the pod's trust
store accepts — on a LAN deployment neither holds. https://pod.<domain>
remains what it always was: the door for members' devices.

Fixes for console-installed clusters

• The cluster secrets never included the backup Garage instance, whose chart
  runs with secret.create=false. Without it that instance never starts, and
  every backup producer — barman, Velero, pv-backup, the Nextcloud file copy,
  the pod archive — has nowhere to write. Such an installation looks healthy
  right up to the moment somebody needs a Recovery Point.
• The protection inventory still reported the Immich library as covered by a
  pv-backup rclone job removed in docs/adr/0047. It now reports the
  append-only pod archive, with no offsite copy, and Nextcloud's filesystem and
  bucket datasets are listed separately, since they have different producers and
  different schedules.
• The console's copy of the hostname map did not know about the Keycloak admin
  Ingress added by this release's hardening work, so a console-rendered overlay
  left the admin console on the project's domain.

Also

Staging: the runner can compare against a branch other than main, deploy a
chosen subset of apps, and configure the VM's location and server type; it
tolerates Traefik's CRD lag and deploys garage-backup, without which Keycloak
never finishes syncing. Realm import fixed — a subflow cannot have two parents.

Full Changelog: v1.2.39...v1.2.40

SmallWorlds v1.2.39

Backup data now lives on its own volume, separate from the data it protects.

Upgrade note — rebuild, do not bump

Do not adopt this release on an existing cluster by bumping your overlay's pinned tag. Nodes provisioned before it have no /mnt/smallworlds-backup, so garage-backup's volumes never bind, and every tenant's garage-init blocks waiting for an instance that can never start — a total sync stall with nothing obviously pointing at the cause. Rebuild the node from this release instead.

If you have ever configured the offsite leg, members' personal pods are at your offsite provider now. The previous replicator synced the whole Garage instance and the setup guide granted read on every bucket, so pod archives — unencrypted, including the PIN-protected immich-locked/ prefix — were copied offsite in contradiction of ADR 0047. This release stops that, but does not undo it. Delete those objects by hand, prior versions included.

Device enrolment is now mandatory. The nightly Immich export fails while any member holding assets has no enrolled device, and names them. Set REQUIRE_ENROLMENT=false on the export CronJob if your community has accepted the risk.

What changed

• Two volumes. A second block volume holds every Recovery Point, served by a new garage-backup instance with both its data and its metadata on it. Terraform provisions it; both bootstraps refuse to start if the two paths land on the same device. (ADR 0048)
• Database backups ran hourly, not daily. CloudNativePG's schedule takes a six-field cron with seconds first, so "0 2 * * *" meant "minute 2 of every hour" — roughly 144 base backups a day across six clusters, all retained seven days. Now "0 0 2 * * *".
• Garage never used its 120 Gi volume. apps/garage.yaml set persistence.size and persistence.storageClass, which chart 0.7.1 does not define, so Helm ignored them and Garage ran on a 1 Gi local-path claim.
• Nextcloud's user files had no server-side backup. pv-backup covered only /var/www/html. They are now copied nightly to garage-backup, with deleted and superseded objects retained under versions/<date>/.
• The offsite leg is scoped. An explicit bucket list, never pod-gateway, plus a refusal to run if the replicator key can even see that bucket. Destination buckets must be pre-created with versioning enabled — buckets rclone creates on first sync are un-versioned, which silently reduces the offsite copy to a mirror.
• Restoring Immich originals is now documented and tooled (admin-tools/restore-immich-originals.py, §7.4). It reads the pod bucket by default, verifies every object against its manifest digest, and refuses a mismatch rather than writing it. Note it requires a restored database first: only the database maps an archive key to a library path.
• Pod archive monitoring. The gateway's metrics were never scraped; there is now a ServiceMonitor and alerts for a device that has gone stale, one that has never checked in, and orphaned objects.

Verification

The chain was exercised end to end on a purpose-built cluster: barman, Velero and pod objects landing on the separate device, a CNPG restore from Garage, and Immich originals restored from the pod bucket and served back byte-identical to their export-time digests. The full rebuild-from-nothing sequence (§7.6) has still not been drilled and is documented as a plan.

Release inputs

The k3s installer digest is bumped: upstream replaced the script at https://get.k3s.io, so the previous pin no longer matches. The k3s version it installs is unchanged (v1.36.2+k3s1), as is the Argo CD manifest.

SmallWorlds v1.2.38

Operator-console image pinned to v1.2.38 (had lagged at v1.2.34); console now reports the current release.

SmallWorlds v1.2.37

Append-only personal data pods + Immich export bridge (ADR 0047). NOTE: Immich originals are no longer covered by pv-backup/offsite; pod-gateway is an optional app to enable in your overlay.

SmallWorlds v1.2.36

Early node-disk alerting and monitoring-stack resilience under node pressure.

SmallWorlds v1.2.35

S3 requests now signed with Garage's region, restoring WAL archiving. Root-cause fix for the 2026-08-09 disk-full outage.

SmallWorlds v1.2.34

Full Changelog: v1.2.33...v1.2.34