Kwerft v0.6.0-rc.14

What v0.6.0-rc.13 was meant to be (its release stopped on a test and
published nothing), plus:

Restores say why the backup's contents cannot be read ("Reading the
backup's contents failed, retrying: the storage answered 403 …") and fail
with that error after 2 minutes, instead of waiting without a word.

From rc.13: DNS can change and delete wildcard records again (adding a
second node's address to *.apps. failed with "not found"); the
plan's Rebalance action and App autoscaling; e2e runs wait for the
Hetzner project's resource limits.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.14/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.14 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:b18ab74ddc065e66b6711af86b88d6db32657689bc6bc6b0cc54962a09c2d317
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.14 · ghcr.io/ehilzinger/charts/kwerft@sha256:7c5e9a0fae3b1553f616909cbc23e098d071e05cbfa8c8c52f32cb6a459d4f24
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.12...v0.6.0-rc.14

Kwerft v0.6.0-rc.12

Disk health: wear alone is a warning, not a failing disk. An NVMe drive
past its rated endurance sets critical warning 0x04 and smartctl reports
it FAILED; such a drive stays usable while its spare blocks are above
their threshold. It now raises "Disk worn" and shows as a disk to watch;
"Disk failing" fires for the other warning bits, spare below threshold, a
FAILED status with another cause, and new media errors.

License: SPDX headers on every source file; the contributor license
agreement (CLA.md, a draft pending legal review) with a "cla" check on pull
requests; the trademark policy (TRADEMARKS.md).

e2e: the harness locks the output buffer it shares between stdout and
stderr, and asks the registry check again when its answer is empty.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.12/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.12 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:f37eb92b0cbfc4673a9c8f5a6fae4ec2783befae7d3157b54bdcec0ad6cc2896
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.12 · ghcr.io/ehilzinger/charts/kwerft@sha256:1f43ba455511187a1ceeb1d5b8c23e03f75f55c4e536193cd364321f2249ab50
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.11...v0.6.0-rc.12

Kwerft v0.6.0-rc.9

The same code as v0.6.0-rc.8, released again so that a console on rc.8
(the first with a working upgrade runner) has a newer version to upgrade
to from Settings › Updates: the first console upgrade without any step
by hand, on kwerft-dev-test and in this release's e2e run.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.9/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.9 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:139f1f0b26fb304eb711d00cd6fa557b7643d6e40b43a1a2d667d66e6c861ba2
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.9 · ghcr.io/ehilzinger/charts/kwerft@sha256:0852669190663b1055ede70963d749c3889cacc74203fbfb01af1e96e68be8b2
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.7...v0.6.0-rc.9

Kwerft v0.6.0-rc.8

Upgrades from the console start the installer: the runner starts the
installer's unit from a short host unit it waits for (a systemd-run that
neither waits nor pipes cannot reach systemd from the pod). The first
console upgrade on a real server (rc.6 → rc.7) succeeded with that step
done by hand. A console on rc.7 or earlier has the old runner: move it
to rc.8 with install.sh once; console upgrades work from rc.8 on.

Tasks and Apps with egress https reach their own cluster's hostnames
(TCP 443 to the host and remote nodes); Task network policies are
CiliumNetworkPolicies. A Task killed for memory says so: terminationReason
and memoryLimit in its status, Ready reason OutOfMemory, "out of memory
(limit 256Mi)" in the console.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.8/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.8 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:89f285fbdeec17a8aa22091e6f7bee72e7d3c0fb506eed064eaf99a5085260f3
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.8 · ghcr.io/ehilzinger/charts/kwerft@sha256:fcb8764bcaaac3cf82ce7270145c679b104cd4614f4b4d72bc090af5cb7d020e
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.7...v0.6.0-rc.8

Kwerft v0.6.0-rc.7

Database copies (before an upgrade, on a version's first start, and for
backups) are readable by their owner only (0600), like the database.

The first release a console on rc.6 can upgrade to from Settings ›
Updates; its e2e run upgrades from rc.6 through the console.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.7/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.7 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:672661c2959ceafded1d5066da949e95e8ec6b0934bd32966441900f138bc7e4
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.7 · ghcr.io/ehilzinger/charts/kwerft@sha256:088c7077686ab0eff68463344a89ff78e491170d614430b83f750ad38d29aff3
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.6...v0.6.0-rc.7

Kwerft v0.6.0-rc.6

What v0.6.0-rc.5 was meant to be (its release stopped on a flaky test
and published nothing):

Upgrades from the console get past their Backup step: the runner lists
Helm releases with flags Helm 4 still has (--all is gone). A console on
rc.4 or earlier still has the old runner: move it to rc.6 with install.sh
once; console upgrades work from rc.6 on.

Disk health on dedicated servers: md RAID and SMART readings (an exporter
on dedicated nodes), alerts for a degraded array, a failing or wearing
disk and missing readings, and disk health under Clusters › Nodes.

The e2e runs wait for a just-published installer; two backup tests no
longer race an earlier test's encryption key.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.6/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.6 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:85bc9a0eef0833de9fb4adced6ea5470ea216d4bd1c707cc6df4b5b99cac31a4
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.6 · ghcr.io/ehilzinger/charts/kwerft@sha256:f6f5cc1e89cf2efe08543604a2d14bb1b71215bfde90ec8deff26b47fc6c6a2c
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.4...v0.6.0-rc.6

Kwerft v0.6.0-rc.4

The console copies its database the first time a new version starts,
before migrating it: an upgrade by re-running install.sh now leaves a copy
to roll back to as well (backups/pre--from-.db, the
newest 3 kept).

Secret sets: App pod templates carry a keyed hash of their secrets.

The release's e2e run upgrades from v0.6.0-rc.3 through the console, the
first release run to do so.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.4/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.4 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:6a67c8d21bdb628cdd1184fcbc4548edeac26eaf64dc290d1dd563299be979b2
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.4 · ghcr.io/ehilzinger/charts/kwerft@sha256:d431ae6ee56f2303daa5e56005f37811a9820d1bd9ed8443e756e65381ec6479
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.3...v0.6.0-rc.4

Kwerft v0.6.0-rc.3

Upgrades started from the console work on Ubuntu: the upgrade runner
reaches the host's systemd (it was refused with "Access denied" in the
Backup step). A server on rc.2 or earlier moves to rc.3 with install.sh
once; console upgrades work from rc.3 on.

A failed upgrade stops alerting once its version runs, e.g. after
install.sh was run by hand.

Monitoring leaves out the controller manager, scheduler and etcd, which
k3s runs inside its own process: no more alerts that always fire
(KubeControllerManagerDown, KubeSchedulerDown, ScrapePoolHasNoTargets,
RecordingRulesNoData), and vmagent is no longer CPU-throttled.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.3/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.3 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:bb76b6897c962f1f8f9b5dbaa04ed9ae72b19c42ad2afab0bc20baff8d2549b2
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.3 · ghcr.io/ehilzinger/charts/kwerft@sha256:718e3d305f054d7069c0a29566ef506f842c5b394554b8a9ff193a791269a2e0
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.2...v0.6.0-rc.3

Kwerft v0.6.0-rc.11

Compose import and templates review again: a plan without problems sent
"problems": null, and the Review step crashed ("Cannot read properties of
null (reading 'length')"). Found deploying a template on kwerft-dev-test.

The Overview map: lanes above the servers, labels off the chips,
selections clear of the inspector.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.11/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.11 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:eea89e0c65470053ec7accc70e27b24edf622d3bc17ec5837a09b0cdc0e5f98d
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.11 · ghcr.io/ehilzinger/charts/kwerft@sha256:41b4e59d64690455d7bfde2bfd3bd03f7e178d4bd16fc7ce7b134ad580a3fa8a
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.10...v0.6.0-rc.11

Kwerft v0.6.0-rc.10

Registry credentials: every push to the in-cluster registry needs a
credential, and each project's builds push with their own, to their own
repositories only (a build in one project can no longer push into
another's). Kwerft keeps tags with its own credential; pulls stay
anonymous (nodes, console and build pods; App pods cannot reach the
registry). No console role can read the credentials. Existing installs:
the nodes and running Apps are untouched, the registry restarts once,
builds running across the upgrade finish as before.

install.sh --config with an owner: block creates that owner: the console
reads it from the installer, creates the owner and removes the password
from the cluster. Without owner:, or when the owner is rejected, the
install hands out a setup token as before.

Install

On a fresh Ubuntu server (Hetzner Cloud or dedicated); re-run it on an existing
Kwerft server to upgrade:

curl -fsSL https://kwerft.dev/v0.6.0-rc.10/install.sh | sudo bash -s -- --domain ops.example.com --yes

The install.sh attached here is the same file; SHA256SUMS lists both scripts.

Image	ghcr.io/ehilzinger/kwerft:0.6.0-rc.10 (linux/amd64, linux/arm64) · ghcr.io/ehilzinger/kwerft@sha256:f3c98c4c5e64559ccc29baebc6d77a0783a5bb4f9e59267fc7d819414e077613
Helm chart	oci://ghcr.io/ehilzinger/charts/kwerft version 0.6.0-rc.10 · ghcr.io/ehilzinger/charts/kwerft@sha256:1ee0eee6054d3f7c4a334d00d25beae5f39d3183cdd9af9f1d3147055c1ad4c6
Kubernetes	k3s v1.37.1+k3s1 for new installs
Upgrades from	0.4.0 and later

Full Changelog: v0.6.0-rc.9...v0.6.0-rc.10