Skip to content

cloudfall 0.8.0rc59

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 16 Nov 17:06
· 80 commits to main since this release

Changes since v0.7.0:

Breaking

  • deploy --yes and migrate --yes on an application that declares approval: manual now need a person's approval of the release: without a cloudfall release approve record that counts (the same archive SHA-256, exactly the component's declared servers, not expired), deploy --yes exits 4 PRECONDITION with deployment_approval_missing before any playbook runs, naming the release approve command, and lists each record that doesn't count with digest_mismatch, servers_changed or expired. This hits every project that declares manual and deploys today, the example crm.yaml and every Render import included. Two ways out: keep manual and approve each release first (build it, run cloudfall release approve at a terminal, commit the record under decisions/releases/ when CI deploys, then deploy), or switch the application to approval: automatic when another gate, such as GitHub's required reviewers, is the real one. The deploy plan and result gain approval (policy, and for manual the status, the counting record's path, approver and expires_at or approved_at, the records that don't count and the command to run), deploy gains --approvals (default decisions/releases) and the RECORD_INVALID exit for an approvals file that isn't a valid ReleaseApproval, and cloudfall mcp serve binds the approvals folder (its own --approvals) so a deploy call can't name its own; an automatic application's deploy never reads the folder (#163)
  • A LoggingStack now needs a storage pool on its backend's baseline: cloudfall config validate refuses a LoggingStack whose backend server's Baseline declares no storage.pools with CONFIG_INVALID and code volume_requirement_pool_missing, so a project whose LoggingStack validated on 0.8.0rc22 stops validating. Before upgrading, declare the volume group the Loki and Prometheus volumes go into as a pool on that baseline (storage.pools with id, volumeGroup and primary); with more than one pool, name it on each volume in spec.backend.storage.pool and spec.metrics.backend.storagePool, or validation refuses with volume_requirement_pool_ambiguous (S-014-2)
  • cloudfall config validate refuses one SSH key (same type and body, any comment) declared active and revoked in the same environment, with CONFIG_INVALID and code ssh_public_key_lifecycle_conflict, so a project that declares a key that way stops validating; before, the baseline installed it. Before upgrading, delete the revoked declaration if the key should keep access, or mark the active one revoked if it should not (#354)
  • Some source URLs that cloudfall data copy and cloudfall migrate --data read through --dbname before are now refused, because a libpq service file can't hold them as written: a # in the database name or a query parameter value, a password that ends in whitespace, and a value holding a line break. The copy stops before anything is staged, with lifecycle_execution_failed and a reason that names the URL part without quoting it. Before upgrading, percent-encode a # there as %23; a password that ends in whitespace or holds a line break can't be staged even percent-encoded, so change that password first (#355)

Added

  • A Component can declare spec.buildTools (tool name to minimum version), spec.build (ordered steps, each a workingDirectory inside the packaged root, an argv command and an optional timeoutSeconds from 1 to 3600, default 900) and spec.artifact.include (relative paths the build produces); cloudfall config validate refuses a step whose command[0] is not in buildTools (component_build_tool_undeclared), an absolute or .. path, and buildTools or artifact without build. Validation only: artifact build does not run the steps yet. Components without build validate as before (#113)
  • cloudfall-engine artifact build runs a Component's declared spec.build steps on the controller before packaging: it checks every spec.buildTools entry is on PATH at its minimum version before cloning (artifact_build_tool_missing, artifact_build_tool_unsupported), runs each step in the clean checkout with only PATH, HOME, LANG, TMPDIR and CI=true in its environment and its output on stderr, fails without writing an archive on a non-zero exit or a step outrunning its own timeoutSeconds (artifact_build_step_failed), a missing include path (artifact_include_missing), an include path that leads out of the checkout through a symlink (artifact_include_outside_checkout) or a tracked file a step deleted (artifact_tracked_file_missing), packages the tracked files plus every spec.artifact.include path (a gitignored web/dist, say), and records the steps and tool versions as the new optional spec.build of the Artifact metadata. The command's time limit is now the clone time (1800 seconds) plus every declared step timeout, on the CLI and the SDK/MCP path alike; an explicit --timeout replaces it, and --timeout 0 still disables it. A Component without build is packaged as before (#115)
  • The converge_domains MCP tool takes an optional domain: given, domains.yml runs only on that Domain's proxy server (--limit) and renders only that Domain, and an unknown id exits PRECONDITION (lifecycle_domain_missing) before the inventory render; omitted, it runs on every server as before. Each domain receipt now records previousConfigurationSha256, the sha256 of the vhost file the run replaced, or null when none was there; receipts without it still load (#116)
  • cloudfall domains apply <domain> [--issue-certificate] [--yes] converges one Domain: domains.yml runs on its proxy server alone and renders only it, so no baseline, firewall or service task runs. Without --yes it exits 0 with a plan. --issue-certificate refuses a Domain with tls.mode: disabled, and first checks on the controller that the primary name and every alias resolve to the proxy server (PRECONDITION, domains_dns_unverified, naming each failing name); a proxied Domain skips the check (dnsCheck: "skipped-proxied"). With --yes it answers the receipt path under --deployments (#117)
  • cloudfall domains observe <domain> reads one Domain on its proxy server alone, through the new read-only domains-observe.yml: the vhost file and its sha256, the certificate's expiry, certbot.timer, and whether the vhost still matches the last receipt (receipt: null before the first). It changes nothing on the server or in the project, and a missing vhost exits 0. cloudfall mcp serve lists domains_apply (with its yes gate) and domains_observe as tools (#118)
  • Components: a release.command (with optional workingDirectory and timeoutSeconds, 1 to 1800, default 300) runs once per deploy on the first server of deployment.servers, which now deploys first, after the release's dependencies are materialized and before the unit file changes or current moves; it runs as the application's user with the component's env file loaded under systemd's EnvironmentFile= rules. A non-zero exit or a timeout fails the deploy with ENGINE_STEP_FAILED before any server switches, and a passing run is recorded as releaseCommand (server, command, exit code, last 2000 characters of output) in every release receipt of that deploy (#119)
  • deploy without --yes shows the component's release command as release_command (its command, the working directory resolved under the new release directory, timeout_seconds and the release server), or null for a component without one, and says in would_run that it runs once on that server before any switch; the rollback plan of a component with a release command shows release_command_skipped: true, since a rollback never runs it (#120)
  • A PostgreSQL Service database may name an owner role and its passwordRef (environment, path, key in the secrets store) instead of an application, so a project of a server and its database alone validates; spec.postgresql.remoteAccess declares a listen address and the client networks allowed in. config validate refuses an entry with both or neither of application and owner, an owner without passwordRef or a passwordRef next to application, a reserved owner (service_database_owner_reserved), and a remoteAccess with no owner database, no firewall, the port already open, or a source of every address. Converging these declarations comes later: until then the PostgreSQL role refuses them before any task runs (#121)
  • services.yml converges a PostgreSQL database owned by a declared role: it creates the owner login role, sets its password from passwordRef as SCRAM-SHA-256 on every converge, and gives it the database; backups, the restore drill, and data copy (restoring as the owner, its receipt's spec.owner) cover it like any other database. Every path that runs services.yml (the converge_services MCP tool, the services step of cloudfall migrate, operator approve and operator run, and cloudfall-engine playbook run services.yml) reads the password from a --secrets-dir flag (default secrets) and stages it in a 0600 file in a private temp dir outside the project, removed after the run; the value is in no inventory, command line, Ansible output, answer, or receipt, and the answer's owned_roles names the reference, the key, and the value's SHA-256; operator approve records them as ownedRoles in the proposal receipt's outcome (an optional property) and operator run in each executed proposal's entry. A missing secrets file or key fails with PRECONDITION naming the expected path and key before any playbook runs (#122)
  • A PostgreSQL service with spec.postgresql.remoteAccess opens to its declared sources over TLS: services.yml sets listen_addresses = '127.0.0.1,<listenAddress>' and ssl = on with the Debian ssl-cert snakeoil pair (restarting the cluster), and writes one hostssl <database> <owner> <cidr> scram-sha-256 line per owned database and source between Cloudfall markers in pg_hba.conf, reloading only after pg_hba_file_rules reports no error; baseline.yml adds one ip saddr/ip6 saddr <cidr> tcp dport <port> accept rule per source, rendered as a cloudfall_firewall.allowedInbound entry with a new optional sources list. Removing remoteAccess resets the listener and removes the pg_hba.conf block and the firewall rule; databases named by application stay socket-only (#123)
  • cloudfall data copy takes a repeatable --source-schema NAME (and the data_copy MCP tool a source_schema list): the dump and restore copy only the named schemas, the empty-target check and the per-table row-count check read exactly those schemas, and the plan without --yes names them; a name that is empty, longer than 63 bytes, or holds a control character or a brace is refused as a usage error. Without it the dump is the whole database as in 0.7.0, and the checks now cover every non-system schema of the source instead of public only, so a row-count mismatch outside public now fails the copy. The DataMigrationReceipt gains spec.schemas (sorted) and a schema on each spec.tables[] entry beside table and rows (#124)
  • cloudfall data copy, and the data:<database> step of cloudfall migrate, run a preflight on the target host after the empty-target check and before pg_dump: it reads both catalogs and refuses a copy the target can't hold, with nothing dumped, the target untouched and no receipt, exiting PRECONDITION with code data_copy_preflight_refused and four lists in the error context, each item named: missingSchemas (a named schema the source lacks), extensions (one the selected schemas need, through pg_depend or by living in them, or every source extension but plpgsql without --source-schema, that the target's pg_available_extensions lacks), roles (a role a row-level security policy on a selected table names that the target's pg_roles lacks) and authReferences (a function or policy of the selected schemas that references auth. when auth isn't copied). The role writes the findings to <service>-<database>.preflight.json under --data-migrations, so the lists arrive whole. With --source-schema, pg_dump also gets one --extension per extension the schemas need (pg_dump 14 or later), and pg_restore now runs with --single-transaction, so a failed restore rolls back whole, writes no receipt, exits ENGINE_STEP_FAILED, and the next run passes the empty-target check (#125)
  • cloudfall migrate takes a repeatable --data-schema DATABASE=NAME beside --data DATABASE=PATH (and the migrate MCP tool a data_schema list): the data:<database> step copies only the named schemas, as data copy --source-schema does, and the plan names them. A --source-schema or --data-schema name must now match ^[a-z_][a-z0-9_]*$ and be at most 63 characters, and may be given once per copy; a name that doesn't, a repeated name, or a --data-schema for a database with no --data exits PRECONDITION (code data_copy_schema_invalid or migrate_data_schema_without_data) before any playbook runs, where a malformed name was a usage error and a repeat was silently dropped (#126)
  • A Domain can declare spec.routes, path routes that each either proxy to an upstream (address, port) or serve a component's release files (static: component, directory, optional spaFallback). cloudfall config validate refuses a Domain with both proxy.upstream and routes (domain_route_source_invalid), a path that doesn't start with / or uses characters outside letters, digits and ._~/-, or a static path without a trailing / (domain_route_path_invalid), a path declared twice (domain_route_path_duplicate), a path under /.well-known/acme-challenge (domain_route_path_reserved), a missing static component (resource_reference_missing) or one not deployed on the Domain's proxy.server (domain_route_component_not_on_proxy), and a directory or spaFallback that is absolute, climbs out with .., or, for directory, names the release root. Validation only: the domains playbook doesn't render routes yet and still refuses a Domain without proxy.upstream before any change. Domains with only proxy.upstream validate and render as before (#159)
  • cloudfall release approve <component> --release <release> records a person's approval of one built release as a ReleaseApproval (new release-approval.schema.json) under decisions/releases/: the application, component, release, archive SHA-256, git commit, servers, approver, approvedAt, expiresAt (--valid-for hours, 1 to 168, default 24) and via: terminal. It verifies the artifact first, refuses a component of an automatic application with release_approval_not_required, and writes the record only once a person typed the release id at a terminal; --yes never answers it, and it is no MCP tool. It deploys nothing (#160)
  • Components: a healthCheck of type: command (a required non-empty command argv, timeoutSeconds 1 to 300 and attempts 1 to 20, no other key) validates, and deploy, restart and rollback without --yes name the check and its command in their plan's wouldRun (#161)
  • cloudfall domains apply renders a Domain's spec.routes: a proxy route becomes a location that proxies to its upstream with the full request path, and a static route serves <installRoot>/current/<directory> of its component, answering spaFallback for an unknown path when it declares one and 404 when it doesn't. Without a / route, a path no route matches answers 404, not Nginx's default page. Locations render longest path first, whatever the declared order, where the single proxy location rendered before (the port 80 server without active TLS, the 443 server with it); the port 80 redirect and the ACME challenge location stay. The site role grants the Nginx worker (www-data) traverse-only access (POSIX ACL user:www-data:--x, installing acl) from the application root to the install root and its releases/, so it reads the release files but never shared/env/<component>.env, and a deploy or cloudfall rollback switches what a static route serves without re-running the domains playbook or reloading Nginx. The domains playbook's early check now refuses only a Domain with neither proxy.upstream nor routes. A Domain with only proxy.upstream renders a byte-identical vhost (#162)
  • Deploy: a component's healthCheck of type: command now gates deploy --yes. After the switch and the restart it runs on each server in a transient systemd unit cloudfall-health-<component>, as the application's user, in <installRoot>/current, with the service's env file and timeoutSeconds as its runtime limit, up to attempts times 3 seconds apart. When every attempt fails or times out, the previous release is restored (a first release is stopped), the deploy fails with ENGINE_STEP_FAILED naming the check and its exit code or timeout, and the release receipt is written with healthCheckPassed: false. Receipts of a command check carry healthCheck (command, attempts, exit code, timed out), never the command's output, which stays in the server's journal. The release receipt schema accepts healthCheckPassed: false only with a failed command healthCheck, and every receipt written before (#164)
  • restart, rollback and health run a component's type: command health check instead of refusing it: the same transient unit as deploy's gate, with its user, working directory, env file, timeout, attempts and redacted output. health reports unhealthy when the check exits non-zero or times out, a failed check fails restart or rollback without restoring anything, as an http check does, and none of them writes a release receipt. The check starts its unit as root, so health of such a component needs the inventory SSH account to become root, as restart and rollback already do; http and none checks are unchanged (#166)
  • cloudfall config upgrade: rewrites a project from the deprecated ServerType names to Baseline (moves server-types/ to baselines/, rewrites kind and spec.serverType), keeping comments, formatting and line endings; a second run changes nothing, and it refuses, changing nothing, a move onto an existing file or a file or folder reached through a symlink (upgrade_symlink) (#198)
  • audit checks the certificates a server's host requirements name (present, for the right names, more than 30 days left at the observation's time) from new certificates evidence inspect collects, and expects the logging gateway's firewall rule, so a deployed stack audits compliant (S-011)
  • cloudfall logging deploy provides the stack's mTLS material: each host makes its own key and a signing request (logging-requests.yml), the project CA signs the missing and due certificates, and they are delivered with the CA; certificates provided by hand are kept (S-011)
  • cloudfall logging deploy: deploy the declared LoggingStack after meeting its secret requirements; generated secrets are written to the store, an input secret that is missing stops the run (exit 4, INPUT_REQUIRED), and each stored secret is copied to its host path without overwriting a file already there; without --yes it shows the plan (S-011)
  • Logging collectors pin the gateway name to the backend's address in /etc/hosts (loopback on the backend itself), so the gateway needs no DNS record when the backend declares an IP (S-011)
  • The logging gateway's port opens on the backend's firewall to each remote collector's addresses only, rendered beside the remoteAccess rules and enforced by logging.yml; a remote collector with no IP behind a firewall fails config validate (logging_gateway_collector_unaddressed) (S-011)
  • LoggingStack: migration.mode: none for a host with no legacy agents; parallel still requires preserveLegacyAgents: true (S-011)
  • Logging roles check every package pin against the host's APT sources before installing or configuring anything, and stop naming the versions on offer when one is missing (S-011)
  • Certificate provider, controller side: a project CA per environment, kept sops-encrypted under secrets/<environment>/pki/, that signs host requests for exactly the names and usage a requirement grants (S-011)
  • cloudfall requirements show: the volumes, ports, secrets, certificates, DNS names and package pins each declared LoggingStack needs from its servers, by server (S-011)
  • cloudfall secrets generate and cloudfall secrets input: keep the secrets a LoggingStack requires sops-encrypted under secrets/, generated or typed at a terminal with echo off; requirements show gives secrets a state and lists the missing ones (S-011)
  • A decision record and a proposal receipt name the Cloudfall version that last wrote them in spec.writerVersion, an optional schema field: records written before it stay valid (#218)
  • deploy, rollback, restart and an artifact build hold their component's lock under tmp/locks/ on the management host until they end, passed, failed or interrupted, so a second run of the same component exits CONFLICT (component_locked) before any server is contacted, naming who holds the lock, since when, and whether that process still runs; cloudfall lock clear <component> shows a lock a killed run left, with its age, and removes it once a person types the component id at a terminal, never while its process still runs on this host (#221)
  • deploy.yml, rollback.yml and restart.yml hold the component's lock on each server, at /var/lib/cloudfall/locks/<component>.json, from before they change anything there until that server's turn ends, passed or failed, so runs from two controllers, cloudfall-engine playbook run and approved runs of those playbooks no longer race on current, the unit file or the release command's unit: a second run fails on that server, naming the action, user, controller, start time and age of the lock it found (#221)
  • A Component's optional deployment.retainReleases (an integer, at least 2, 5 when absent) says how many releases a deploy keeps on each server; the SDK inventory and the deploy role carry it, and retainUntilCleanup: true still validates and means the cleanup is on (#224)
  • The release receipt names, in each server entry whose gate passed, the releases the deploy removed there under removedReleases; release-receipt.schema.json accepts it and refuses it on a failed entry (#224)
  • The PostgreSQL backup writes globals-<timestamp>.sql from pg_dumpall --globals-only --no-role-passwords beside the per-database dumps, so a restore onto a fresh server has its roles; retention prunes it with the dumps (#225)
  • Baseline storage.pools: existing LVM volume groups that workload volumes go into; each LoggingStack volume names its pool, LV, LV type and physical volumes in requirements show, and config validate refuses a volume with no pool (S-014)
  • audit checks each required volume (volume.<name>: type, physical volumes, size, health, mount) from new volumes evidence inspect collects (S-014)
  • cloudfall logging deploy provides the stack's data volumes: a read-only report from each host, refusals decided on the controller before any host change (volume_refused), then lvcreate/mkfs for missing volumes, lvextend --resizefs for small ones, an fstab line by UUID, the mount and RequiresMountsFor drop-ins (S-014)
  • cloudfall decisions mark-interrupted <id> --confirm-dead and cloudfall operator mark-interrupted <id> --confirm-dead close a record left running by a run that died midway as the new status interrupted, once a person confirmed the run is dead; the record keeps its approval and gains interruption.confirmedBy (--by, default USER) and interruption.confirmedAt. Nothing runs on the fleet, and neither command is an MCP tool (#244)
  • AlertChannel (alert-channels/): a Telegram chat that a Baseline's hosts and a LoggingStack receiver name; its bot token is an input secret on every server that sends, set with cloudfall secrets input telegram-bot-token --owner <channel> (S-015)
  • A LoggingStack receiver may name an AlertChannel: Alertmanager delivers to it with its native telegram_configs and bot_token_file, the token delivered by logging deploy; an Alertmanager older than 0.26 is refused before its configuration is written (S-015)
  • cloudfall alerts test <server>: send one test message through the server's alert channel; audit checks each host's channel (alerting.channel: sender, token and chat id present, mdadm and smartd pointed at the sender) from new alertChannel evidence inspect collects, never the token (S-015)
  • cloudfall baseline converge: run baseline.yml with each host's alert channel; hosts whose baseline names an AlertChannel get /usr/local/sbin/cloudfall-alert, the channel's token and chat id, and mdadm and smartd pointed at the sender; a token not in the store stops it before any server (S-015)
  • A Service can be serviceKind: elasticsearch (S-012): spec.elasticsearch declares version (an exact X.Y.Z of a line Elastic still supports, 8.19 or 9.x), heapMb (512 to 31744), and optionally nodes, clusterName, existingCluster, users (each with indexPatterns and a passwordRef, never a password), remoteAccess, snapshotRepository and restoreCheckIndices. cloudfall config validate refuses an unsupported version (service_elasticsearch_version), bad nodes (service_elasticsearch_nodes), an existingCluster without clusterName, with a CA file that is missing, outside the project, not a CA or holding a key, or with a seed host that is a declared node (service_elasticsearch_existing_cluster), a server in two clusters (service_elasticsearch_server_shared), several nodes or an existing cluster without a snapshotRepository (service_elasticsearch_snapshot_repository_local), a reserved or repeated user name (service_elasticsearch_user_reserved, service_elasticsearch_user_duplicate), a pattern of every or a hidden index (service_elasticsearch_pattern), and remoteAccess under the PostgreSQL rules; it warns ELASTICSEARCH_VERSION_ENDING with the date when a line is within 90 days of its end of life. The rendered inventory carries the service, with nodes and clusterName resolved, on its server; nothing converges it yet (#309)
  • services.yml converges a one-node serviceKind: elasticsearch service on Debian 13 through the new cloudfall_elasticsearch role (S-012). Its first task refuses a heapMb above half of the host's memory before anything changes. It installs elasticsearch=X.Y.Z from Elastic's APT repository for the declared major and holds it, and writes elasticsearch.yml (cluster.name, node.name, HTTP on the loopback bind only, bootstrap.memory_lock: true, security on, never discovery.type), a jvm.options.d heap file and a unit drop-in with LimitMEMLOCK=infinity, all before the package, so its security auto-configuration skips the node. cluster.initial_master_nodes is written only while the data path holds no cluster state and leaves the file in the same converge once the cluster has formed, with no restart, so a second converge changes nothing; the converge stops when the node runs without its memory locked. Cloudfall's own cloudfall account is a superuser in the file realm, with only its bcrypt hash in users and its password in a 0600 file owned by elasticsearch; the elastic password is set once, as a bcrypt hash computed on the controller (D-9). Each node of the service yields the elasticsearch package and the generated elasticsearch-cloudfall-password as S-011 requirements, and cloudfall secrets generate also generates the elastic password (ELASTIC_PASSWORD beside CLOUDFALL_PASSWORD in secrets/<environment>/services/<id>.env); every path that runs services.yml decrypts both first and stops before any playbook when one is missing. More nodes, declared users, remoteAccess and existingCluster stop the converge until their slices land (#310)
  • An Elasticsearch service's declared users converge (S-012): from the service's home server, each user gets a role of its own name limited to its indexPatterns (read: read and view_index_metadata; write adds write, create_index and manage) and a native user holding that role alone. Every path that runs services.yml decrypts each user's passwordRef first, stops before any playbook when one is missing, and stages only its bcrypt hash, computed on the controller and sent as password_hash (D-9); the salt is derived from the service's cloudfall password, so an unchanged password puts nothing and a second converge changes nothing. Users and roles Cloudfall makes carry metadata.cloudfall_managed_by: <service id>; a converge deletes the marked ones no longer declared and never changes or deletes an unmarked one, and refuses a declared name an unmarked user or role already holds. inspect reads the cluster's users and roles on the home server (elasticsearchUsers in the observation), and audit adds elasticsearch.users[<id>]: drift when a declared user or role is missing or a marked one is left over, with users and roles not made by Cloudfall listed as unmanaged (#311)
  • An Elasticsearch service grows to two or more nodes and rolls a version one node at a time (S-012). Each node yields an elasticsearch data volume of 10 GiB at /var/lib/elasticsearch in its baseline's only storage pool, mirrored for a cluster of one and expendable for two or more; a node of a cluster of two or more also yields port 9300/tcp open to the other nodes only and an elasticsearch-transport certificate from the project CA for its hostname and addresses, of the usage server-and-client: the one certificate remoteAccess serves HTTP with, which the services converge now issues for a node with remoteAccess too. The role refuses remoteAccess on a cluster of two or more, since its one listenAddress exists on one server only. config validate refuses a node whose baseline has no pool (volume_requirement_pool_missing), several (volume_requirement_pool_ambiguous), or, for a cluster of one, no secondary physical volume (volume_requirement_pool_unmirrored), and a node of a cluster of two or more with no IPv4 (service_elasticsearch_node_unaddressed). The rendered inventory carries the service on every node with its transport address and seed hosts, adds the 9300 rule to each node's firewall, and lists each cluster's home first. Every path that runs services.yml through cloudfall first reads each node's volume facts and decides its volume, then has each node make its key and a signing request, signs the missing and due certificates with the project CA and delivers them, all before any host changes. The role creates and mounts the volume, enforces the firewall, binds the transport to the node's address with TLS verification_mode: full, bootstraps a cluster only from the home node and refuses when another node of it already runs, sets the elastic password from the home only, and restarts a running node only from a green cluster (yellow with one node), with replica allocation held and a flush (released again even when the restart, the rejoin or the version check fails), then waits for the node to rejoin on the declared version and the cluster to return to its status before the next server (#312)
  • An Elasticsearch service's S3 snapshotRepository is converged instead of refused (S-012). Its credentialsRef names one key of the secret store holding <access key id>:<secret access key>; every path that runs services.yml decrypts it on the controller, stages the two keys in 0600 files and passes only their paths, and stops before any playbook when the key is missing or malformed. On every node the role puts them in the keystore as s3.client.default.access_key and s3.client.default.secret_key through stdin, keeps only their SHA-256 beside it so a second converge changes nothing, names the endpoint as s3.client.default.endpoint in elasticsearch.yml, reloads a running node's secure settings, and registers the repository cloudfall as type: s3 with the bucket and base path, replacing the filesystem repository of a service that grew; the snapshot and restore-drill scripts use it unchanged (#312)
  • config validate refuses an Elasticsearch cluster of two or more nodes on a baseline without a firewall, since nothing would narrow 9300 to the other nodes (service_elasticsearch_cluster_firewall_missing); the inventory render refuses it too (#312)
  • cloudfall backup run and cloudfall backup verify work for a serviceKind: elasticsearch service (S-012). The converge adds backup.directory to path.repo, which restarts each existing node once on the first converge after the upgrade, registers it as the filesystem snapshot repository cloudfall, and installs cloudfall-elasticsearch-backup and cloudfall-elasticsearch-restore-check with sandboxed units and timers on backup.onCalendar and backup.restoreCheckOnCalendar. The backup counts the documents of every index restoreCheckIndices matches (or of the smallest index) and writes the counts as one document per snapshot into Cloudfall's own index cloudfall-snapshot-counts. It then takes the snapshot cloudfall-<timestamp> of every index, that one included, and the cluster state, waits for it, and fails unless its state is SUCCESS, so a PARTIAL or FAILED snapshot writes no receipt. Finally it deletes Cloudfall's snapshots of the service older than retentionDays with their counts, always keeping the newest and never touching a snapshot it did not take. The restore drill verifies the repository, restores the snapshot's counts and every recorded index as cloudfall-restore-check-<index> with no replicas and without aliases, checks the low disk watermark, waits for green, compares each document count, and deletes the restored copies even when a step failed; it refuses to start over an index of a copy's name, and fails when no snapshot exists. Both run as the elasticsearch user, reach the node as the role does (with remoteAccess, over HTTPS on the listen address, verified against the project CA) and write the usual BackupReceipt. An S3 snapshotRepository stops the converge before it changes the host, with a message that S3 repository registration is not built yet (#313)
  • An Elasticsearch service (S-012) is audited, opened by remoteAccess, and scraped for metrics. cloudfall audit reports drift on each node when the installed elasticsearch package differs from version, when elasticsearch.service is not running and enabled, and when the managed settings of elasticsearch.yml (network.host, http.port, http.host, xpack.security.enabled, xpack.security.http.ssl.enabled) differ from what the role writes, so a hand edit is drift before the node restarts with it; it reports the node as running an unsupported version once its line is past Elastic's End of Maintenance date at the observation's time (D-12). inspect reads elasticsearch.yml as text on a server with an Elasticsearch service (spec.elasticsearch of the observation). With remoteAccess, the firewall opens bind.port to allowedSources only, HTTP listens on loopback and listenAddress and is TLS-only on both, with the node's one certificate from the project CA (the elasticsearch-transport certificate requirement of the new usage server-and-client, for the server's hostname and every address, the listen address included, at /etc/elasticsearch/certs/transport.crt, transport.key and ca.crt), which the role checks before writing a configuration that names it, and stops on while nothing issues it yet; the role's own API calls then go over HTTPS to the listen address, verifying the project CA. With metrics.enabled, the collector renders one Alloy prometheus.exporter.elasticsearch for the local node, as the cloudfall account, labelled with the service id; Alloy reads the password, and with remoteAccess the project CA, as systemd credentials (LoadCredential), and logging deploy stops before changing the collector when the node's files are not there yet (#314)
  • An Elasticsearch service with existingCluster is converged instead of refused: its nodes join a cluster Cloudfall did not create (S-012). Each node's transport leaves loopback even when it is the only Cloudfall node: discovery.seed_hosts lists the existing cluster's seed hosts, then the Cloudfall nodes; its firewall admits 9300 from the other Cloudfall nodes and the seed hosts' addresses only; it gets its transport certificate from the project CA and trusts the existing cluster's caCertificateFile beside it, which the services converge delivers to /etc/elasticsearch/certs/existing-cluster-ca.crt, with verification_mode: certificate. A joining node never writes cluster.initial_master_nodes and never sets the elastic password, which stays the existing cluster's, and the existing nodes are only read. Before the data volume and the package, the role asks the seed hosts over HTTPS on port 9200, verified against that CA, for GET /_nodes, and stops, naming the seed host or node and its value, when the cluster's name differs from clusterName, when a node runs another major, when a node is newer than version, or when no seed host answers. Once existingCluster is removed, a converge drops the seed hosts and the CA from the configuration, restarts one node at a time, removes the CA file, and waits for the cluster to be green with two or more Cloudfall nodes, or yellow with a single one, whose replicas have nowhere to go; clusterName keeps the name. config validate refuses a seed host given by name rather than address, and a node that joins an existing cluster without a firewall or an IPv4, as for a node of a cluster of two or more (#315)
  • existingCluster.credentialsRef names one key of the project's store holding <username>:<password> of an account the team already has on the existing cluster with the monitor privilege; the join check logs in with it, so nobody creates a user on the old cluster by hand. Without it, the check logs in as the cloudfall account, which the team then grants monitor there. A missing key, or a value not of that form, stops the converge before any playbook, naming the file and the key; the login is staged in 0600 files on the controller only, removed after the run, and never reaches a node, the inventory, task output or an error (#315)
  • cloudfall operator mark-interrupted <id> --confirm-dead also closes a receipt left executed, whose verification never recorded an outcome, as interrupted, as it does a running one; it then stops blocking new proposals for the same alert or drift (#358)
  • A project states its format as [tool.cloudfall] format in its pyproject.toml, and cloudfall init writes the current one (format = 1). cloudfall config validate checks it before validating: a project without the key, or without a pyproject.toml, reads as format 0 and still validates, with one PROJECT_FORMAT_OLD warning naming cloudfall config upgrade; a format outside the ones this release reads (0 to 1), or one that is not a non-negative integer, stops with the new exit PROJECT_FORMAT_UNSUPPORTED (95) and error.context.code project_format_too_new, project_format_too_old or project_format_invalid (#375)
  • cloudfall changelog lists entries: each release now records its interface changes (commands, flags, output fields, exit codes) in schema-changelog.json and its manifest snapshot, both in the wheel, as the release bot's after_stamp step runs treaty changelog-add cloudfall.app:app. The first release seeds the snapshot, so its entry lists the whole interface as added; a release whose interface did not change adds no entry. The Releases section of CONTRIBUTING.md names no manual step any more (#378)
  • A Service can be serviceKind: mariadb (S-019): spec.mariadb declares version (an exact X.Y.Z of a long-term series MariaDB still maintains: 10.11, 11.4, 11.8 or 12.3), innodbBufferPoolMb (128 or more), and optionally databases (each with an application, or an owner with a passwordRef, never a password, and authentication: ed25519 if wanted), remoteAccess, and replication (role primary or replica, serverId, a primary's listenAddress, a replica's primary service or source with host, port, credentialsRef and caCertificateFile). cloudfall config validate refuses an unsupported version (service_mariadb_version), a server with two MariaDB services (service_mariadb_server_shared), a reserved owner or database name (service_database_owner_reserved), a repeated serverId (service_mariadb_server_id_duplicate), a primary without listenAddress or a replica with one (service_mariadb_replication_listen), a replica without exactly one primary or source, or whose primary is no primary MariaDB service of its environment on another server (service_mariadb_replication_primary), a replica with databases (service_mariadb_replica_databases), of another series than its primary or older (service_mariadb_replica_version), a pair without a firewall (service_mariadb_replication_unfirewalled), a replica server with no address its primary can admit (service_mariadb_replica_unaddressed), a source CA file that is missing, outside the project or not a CA (service_mariadb_replication_source), a server without one mirrored storage pool (the S-014 codes), and remoteAccess under the PostgreSQL rules; it warns MARIADB_VERSION_ENDING with the date when a series is within 90 days of its end of life, and MARIADB_REPLICATION_UNENCRYPTED for a source without a CA file. The kind yields its mariadb volume, package, replication port, certificate and generated replication password as host requirements, and the rendered inventory carries the service with its databases and references, never a password; nothing converges it yet (#419)
  • services converges a standalone MariaDB service (S-019) with the new cloudfall_mariadb role: it refuses, before any task changes the host, an innodbBufferPoolMb above three quarters of the server's memory, a series MariaDB publishes no package of for the server's Debian release, and a series other than the installed one; it mounts the data directory on the mirrored mariadb volume the services converge decides with the Elasticsearch volumes, installs mariadb-server at the exact declared release from MariaDB's archive and holds it, and writes the managed settings to /etc/mysql/mariadb.conf.d/90-cloudfall.cnf (listener on bind.address only, buffer pool, utf8mb4; no binary log). An application database gets a user named after the application's Linux user that logs in through the unix socket only; an owner database gets a user whose password reaches the host only as the hash its plugin stores (mysql_native_password, or ed25519 with authentication: ed25519), computed on the controller and set only when it differs from the stored one, so a second converge changes nothing. Each user has every privilege on its own database. A user Cloudfall made that is no longer declared is locked, its database left with its rows, and users and databases made by hand are left alone; cloudfall audit lists both kinds as undeclared and unmanaged. A passwordRef whose file or key is missing stops the converge with PRECONDITION before any playbook runs. remoteAccess and replication are still refused by the role until their own issues land (#420)
  • A MariaDB service converges its remoteAccess: the services converge has the server make its key and the project CA sign its certificate for its hostname and addresses, the role installs it (the key 0600 for the mysql account), listens on loopback and listenAddress with ssl_cert, ssl_key and ssl_ca, and gives each owner with a password one account per allowed source from the same controller-computed hash, requiring TLS, so a plain connection from a source is refused; the firewall admits the port from the allowed sources only, and audit expects that rule, the listener and those accounts. An account of a source no longer declared is locked, never dropped; a renewed certificate is loaded with FLUSH SSL, without a restart. replication is still refused by the role (#421)
  • A MariaDB service is backed up and its restore drilled (S-019): services installs cloudfall-mariadb-backup and cloudfall-mariadb-restore-check with their systemd timers (the drill's only with restoreCheckOnCalendar), run as root through the socket, and cloudfall backup run <service> and cloudfall backup verify <service> run them and write their BackupReceipt. A backup writes, for every declared database, one gzip dump taken with mariadb-dump --single-transaction (one consistent snapshot of the InnoDB tables, no table locks; tables of another engine are named in the output) and beside it a count file of every table and its rows, counted from the stored dump, both 0600 in a 0700 directory; then it deletes each database's dumps older than retentionDays, never the newest, and only files of the names it writes. The drill loads the newest dump of each database into the scratch database cloudfall-restore-check without writing to the binary log, compares its tables and row counts with the count file (never with the live source, so later writes cannot fail it), and drops it on every path; it fails, and says so, when a database has no dump or no count file, when the load fails, and on any difference, and leaves a database of the scratch name it did not make alone. The role checks onCalendar and restoreCheckOnCalendar before it changes the host. The dumps hold tables, views, routines and triggers but no scheduled events (--events is not passed, so no event can run in the drill's scratch database): a team that uses events recreates them after a restore (#422)
  • A CI deploy guide (docs/ci-deploy-guide.md): one GitHub Actions workflow that renders secrets, builds the artifact with cloudfall-engine artifact build, deploys with cloudfall deploy --yes and uploads the receipts, with one SSH key and one age key per environment and required reviewers standing in for approval: manual, which deploy does not enforce; not yet run against a real server (#89)
  • An open proposal whose basis hosts have been observed again since is now reported as stale: decisions list, decisions show and cloudfall why add stale and staleHosts to each proposed decision, --stale on decisions list and why keeps only those, and decisions approve names each stale host and its newer observation time before asking for the id. Worked out on read from the observation files; the record and its schema do not change (#26)
  • audit checks a logging backend's dashboards (logging.dashboards): inspect records the name and sha256 of each file in Grafana's Cloudfall dashboards directory (new loggingDashboards evidence, never anything else of Grafana's), and the check is compliant when they are exactly the dashboards the installed release bundles; drift names each missing, extra or changed file (S-016)
  • logging deploy provisions four dashboards into a read-only Cloudfall folder in the stack's Grafana: Fleet (every server's freshness, CPU, memory, fullest filesystem, load and the firing alerts), Server (one server's host metrics and log volume by unit), Logs (log volume, error lines and the lines themselves by server and unit, with a filter) and Services (PostgreSQL and Redis exporters by service). They read servers from the data and name the provisioned data sources by uid, so they work on any project; a dashboard dropped by a later release is removed on the next deploy (S-016)
  • cloudfall logging open opens the LoggingStack's Grafana: it forwards a local port over ssh to Grafana on the backend's loopback, built from the backend server's declared address, SSH user and port, answers with the URL of the Cloudfall dashboards once the forward is up, and holds it until stopped. --port picks the local port (one in use exits 4 before ssh starts), --stack picks the stack, and --print-command answers with the ssh command and runs nothing. The answer names where the admin password is kept, never the password (S-016)

Changed

  • Breaking: cloudfall config validate now refuses a Component with runtime.type node or pm2, or runtime.packageManager npm, pnpm or yarn, with CONFIG_INVALID and code component_runtime_unsupported naming the field: these configs validated before but could never deploy, since the deploy role runs only Python with uv. The schema keeps those values as reserved. The Render importer no longer writes a Component for a Node service; it lists the service as an unsupported gap instead (#114)
  • migrate --yes checks a deploy:<component> step's approval as deploy --yes does: for an application that declares approval: manual and no approval that counts, it pauses at that step with PARTIAL_FAILURE, context.code: deployment_approval_missing, the built release and the cloudfall release approve command, and once a person approved it, migrate --yes resumes there with the release the completed build:<component> step recorded, without building again and without needing --build again. The deploy step's detail gains approval (the record it went on), migrate gains --approvals (default decisions/releases, bound by cloudfall mcp serve) and the RECORD_INVALID exit for an approvals file that isn't a valid ReleaseApproval. rollback needs no approval: its plan gains approval, {"policy": "manual", "required": false, "reason": ...} for a manual application and {"policy": "automatic"} otherwise (#165)
  • The resource every server runs is now kind: Baseline, stored under baselines/, referenced as Server.spec.baseline and checked by baseline.schema.json (was ServerType, server-types/, spec.serverType, server-type.schema.json); cloudfall add baseline declares one and cloudfall add server --baseline names it (#198)
  • Outputs carry the new names: inventory show writes baselines and each server's baseline, config validate counts Baseline in by_kind, audit writes each server's baseline and checks baseline.id (was server_type.id; an operator proposal recorded with server_type.id verifies once baseline.id is compliant, and blocks a new proposal for the same drift, with no removal date, since a record's fields are never deprecated within cloudfall/v1), the inspect role records baseline in observed server snapshots, the rendered Ansible inventory has the host variable cloudfall_baseline, the brownfield catalog variable is cloudfall_baselines, and the error code server_type_entry_duplicate is baseline_entry_duplicate (#198)
  • Site: cloudfall.tech is the primary domain; cloudfall.dev and both www hosts redirect to it with a 301, and the PyPI homepage and the README cloudfall init writes point there (#199)
  • Receipts and operator proposals now land in the project's committed record/ directory instead of the git-ignored tmp/, so the audit trail survives losing the management host (D-16): the defaults of --deployments, --releases, --backups, --data-migrations, --env-receipts and --proposals (on every command and on cloudfall mcp serve) are record/deployments, record/releases, record/backups, record/data-migrations, record/env-receipts and record/operator/proposals. tmp/ keeps only what can be rebuilt: observations, rendered inventory, built artifacts, rendered environment files and the migration plan. A path you pass explicitly keeps working as before. cloudfall config upgrade now also moves an existing project's receipts and proposals from tmp/ into record/, listing each move in upgraded with its moved_from; it drops a file whose identical copy is already in record/, refuses with CONFLICT and code upgrade_target_exists, naming both paths and moving nothing, when a different file is, and a second run changes nothing. Until it runs, operator list, audit, services status and the operator's autonomy pass don't see the receipts left under tmp/, so run cloudfall config upgrade once after upgrading, then commit record/. cloudfall init writes the new layout into the project's README, AGENTS.md and .gitignore, and the operator guide says how to copy record/ off a management host with no git remote (#219)
  • A release receipt no longer holds its release command's output: the command is the application's and may print a secret, and the receipt is now committed. The last 2000 characters stay in tmp/output/releases/<component>.<release>.log (0600), and spec.releaseCommand names that file and its SHA-256 under output (path, sha256) instead of outputTail. The ReleaseReceipt schema stays cloudfall/v1: outputTail becomes optional, so receipts written before still validate, and a receipt may not hold both. cloudfall config upgrade moves the tail of an old receipt, under tmp/ or already in record/, out to that file the same way, refusing to replace a different file there. A reader of releaseCommand.outputTail reads the named file instead, which is gone on another host or once tmp/ is cleared (#219)
  • Contract: cloudfall lock clear <component> now also reads the component's lock on each of its servers through the new lock.yml, shows them with their ages in the same prompt, and once the component id is typed removes exactly the locks it showed; it takes --engine and --inventory-file, answers serverLocks and unreadServers (a server it could not reach), may exit ENGINE_STEP_FAILED, and is listed among the commands that change servers (#221)
  • The release receipt holds one entry per server under spec.servers (the release the server ended on, previousRelease, deployedAt, healthCheckPassed, rolledBack and the command healthCheck), with the deployed release, artifactSha256 and releaseCommand once per receipt; release-receipt.schema.json accepts it and still accepts a receipt written before, with one server, which reads as a one-entry list (#222)
  • Site: contact addresses moved to cloudfall.tech (pilot@, conduct@); mail to cloudfall.dev still arrives (#260)
  • A mirrored volume now meets an expendable requirement (D-18): when a one-node Elasticsearch service grows, its first node keeps its raid1 data volume, and the volume provider keeps and grows it, and audit checks it, as the mirror it is; a linear volume still never meets a mirrored requirement (#312)
  • When installing a new Elasticsearch version fails after the node was stopped for it, the role starts the node again on the package it had and lets its cluster move replicas again before the converge stops (#312)
  • Open proposals made before the upgrade carry no binding, so decisions approve refuses them with decision_binding_changed: propose them again (#352)
  • decisions approve gives the approved run and its verify step their own time limit of 6 hours (21600 seconds) instead of the check run's 1800 seconds, which check mode keeps. A run that reaches it is stopped with its whole process group and the record settles failed with the verdict the run timed out after 21600s and was stopped; the run's output now goes to its log as it comes, so a stopped run keeps what it printed. Before, a run past 1800 seconds was killed, its record stayed running with no outcome, and the CLI's log of it was lost (#359)
  • Every command that reads a project now checks its [tool.cloudfall] format first, as cloudfall config validate already did: cloudfall audit, deploy, add, health and the rest, cloudfall mcp serve once at start, and cloudfall-engine inventory render and artifact build. A project written by a newer Cloudfall, one below the formats this release reads, or one with a malformed format stops the command with exit PROJECT_FORMAT_UNSUPPORTED (95, now on cloudfall-engine too) before the project is validated, before anything is written under its tmp/ and before any Ansible run, and cloudfall mcp serve refuses to serve it. A format 0 project (no [tool.cloudfall] table, or no pyproject.toml) still runs every command, each now answering with one PROJECT_FORMAT_OLD warning naming cloudfall config upgrade. cloudfall init, changelog, --schema and cloudfall-engine manifest read no project and run whatever its format (#379)
  • The logging backend and collector roles declare Debian 13 (trixie) next to Debian 12, like every other bundled role, so a trixie host built from the bundled Debian 13 baseline can carry the logging stack. Loki, Grafana and Alloy come from Grafana's distribution-independent APT repository; Prometheus comes from the host's Debian release, so a software.prometheusPackageVersion pin must name a version that release ships

Security

  • An owner role's password no longer reaches the PostgreSQL host or server: services.yml and every path that runs it compute the password's SCRAM-SHA-256 verifier on the controller (a fresh 16-byte salt, 4096 iterations, the password prepared with SASLprep as PostgreSQL does) and stage that verifier in the 0600 file instead of the password, so ALTER ROLE ... PASSWORD sends a verifier PostgreSQL stores as it is and the plaintext can't show in the server log under log_statement = 'ddl' or 'all', in a failing statement, or in pg_stat_activity; answers and receipts still name the reference, the key, and the password's SHA-256 (#140)
  • A declared config value is never run as an Ansible template: cloudfall-engine inventory render and the overlay cloudfall observe writes mark every string unsafe ({"__ansible_unsafe": ...}), so a value holding {{ ... }} reaches the unit file, the release command and the health check as written. Before, a service.command, healthCheck.command or release.command argument such as {{ lookup('pipe', '...') }} ran on the management host during cloudfall health or cloudfall deploy, and an honest one such as docker inspect --format {{.State.Health.Status}} app failed with a template syntax error. A declared value that relied on Ansible evaluating {{ ... }} is now literal, and the rendered inventory's strings read as {"__ansible_unsafe": "..."} in its JSON and in ansible-inventory --list (#174)
  • A catalog operation's inputs are never run as an Ansible template: cloudfall operations propose, cloudfall decisions approve and cloudfall agent investigate hand them to the team's playbook in --extra-vars with every string marked unsafe ({"__ansible_unsafe": ...}), so an input holding {{ ... }} reaches the playbook as written, and integers and booleans keep their types. Before, an input such as {{ lookup('pipe', '...') }} ran on the management host with the operator's rights when the playbook started, and an honest input holding {{ failed with a template syntax error. A team playbook that relied on an input being evaluated as Jinja now gets the literal text. The decision record's inputs are unchanged, and recordings made before this change replay as before (#181)
  • cloudfall data copy and cloudfall migrate --data no longer put the source database URL, password included, on the command line of psql and pg_dump, where every local user could read it in /proc/<pid>/cmdline while the copy ran: the URL becomes a 0600 libpq service file on the target host, named to the clients through PGSERVICEFILE and PGSERVICE and removed when the copy ends, on failure too. A URL libpq can't read (another scheme, a bad percent-encoding, a parameter set twice) is refused before anything is staged, without quoting it (#355)
  • A declared operation's check diff and the logs of its approved run and verify step are no longer written to the committed decisions/ directory: they go to the project's ignored tmp/output/decisions/ (0600), and the decision record keeps each file's path, sha256 and size as before, so a secret a team's playbook prints with --diff (a templated password without no_log or diff: false) never reaches git. A --decisions folder elsewhere in the project keeps its output under the same path below tmp/output/; one already under tmp/ keeps it beside its records. decisions approve refuses with PRECONDITION and decision_diff_missing, naming the file and its sha256, when the raw diff is not on this machine, such as on another clone. Records written before keep citing their files in decisions/ and are read and approved in place; a file of theirs already committed stays in git's history, so one that may hold a secret is for you to purge (#357)

Fixed

  • cloudfall audit no longer reports drift on a server whose PostgreSQL service declares spec.postgresql.remoteAccess: firewall.allowedInbound expects one tcp/<port> from <cidr> entry per declared source and reads the ip saddr/ip6 saddr match of each accepted nftables rule (a single address printed bare by nft compares equal to its /32 or /128), and services.bind[<id>] requires a listener on the declared listenAddress as well as on the bind address, reporting loopbackOnly: false and a remoteAccess.listenAddress key for such a service; a missing listenAddress listener, any other non-loopback listener, an unrestricted rule on the port and an undeclared source are drift. Servers without remoteAccess audit exactly as before. The engine and the audit now derive the source-restricted rules from one SDK function, remote_access_rules_for_server (#143)
  • A pg_hba.conf reload that failed is no longer forgotten: services.yml leaves a marker file at /var/lib/cloudfall/postgresql/<service>.pg_hba-reload-pending when it closes remote access after a failed check or reload, and the next converge checks and reloads pg_hba.conf even though the file is unchanged, then removes the marker, so the old hostssl lines no longer stay live until a restart (#144)
  • cloudfall config validate refuses an owned database or owner named all, sameuser, samerole, samegroup or replication on a PostgreSQL service with remoteAccess, as CONFIG_INVALID with code service_remote_access_name_reserved, instead of leaving the refusal to the converge; without remoteAccess those names still validate (#144)
  • A first deploy over a hand-made systemd unit of the same name no longer loses it when the release fails its health gate: the deploy role keeps a unit without the # Managed by Cloudfall header at /etc/systemd/system/<name>.service.pre-cloudfall before installing the managed unit, and on a failed first release copies it back, restores its enabled state and starts it again when it was active; check mode reports the unit it would keep, and a hand-made unit that differs from an earlier kept copy refuses the deploy (#151)
  • The data copy preflight now names an extension that a selected table needs through an array or a domain of the extension's type, such as a citext[] column or a column of a domain over citext (chained domains included), under extensions and passes it to pg_dump --extension; before, only a direct use of the extension's type was seen, so the copy skipped the extension and the restore failed (#173)
  • cloudfall migrate --yes no longer crashes with HANDLER_CRASHED when it reaches its audit or verify-routes step on a project with no server observations: the step fails with migrate_observations_missing, naming the missing directory and cloudfall observe, and a rerun after observe resumes at that step (#183)
  • Decision records and operator proposal receipts are written through a temporary file and renamed into place, so a crash or a full disk mid-write no longer leaves a truncated record; a new record is created exclusively, so two writers of one id can no longer both succeed, and each read-modify-write of a record holds its directory's lock (#217)
  • An approval is recorded, with the new status running, before the first playbook or the operator's executor starts. A run killed midway stays running and is refused by decisions approve and operator approve with PRECONDITION (decision_run_interrupted, operator_proposal_run_interrupted) instead of being approvable and run a second time; running is added to both record schemas (#217)
  • One decision record or proposal receipt that cannot be read no longer stops every reader of the store: operations propose, decisions list, why and operator list go on with the records they can read and add a RECORD_UNREADABLE warning naming each skipped file and why; an operator pass reports the receipts it skipped under unreadable, and its autonomy pass grants nothing while a receipt cannot be read. decisions show and decisions approve still answer RECORD_INVALID for the one record they read (#218)
  • Each record is read with the schema its apiVersion names, so a later schema version can sit beside cloudfall/v1; a record of a version this Cloudfall does not read is reported, not fatal. A golden set of records in the shapes released versions wrote is loaded by the tests (#218)
  • deploy, rollback and restart contact only the component's own servers (its component_<id> inventory group, still in inventory order), and a backup contacts only the service's server, so an unreachable server the component or service is not on no longer fails them; a component with no servers in the inventory, or a service on no inventory server, now fails before any host is contacted (#220)
  • A component on several servers no longer ends a failed deploy on two releases: when a later server fails its health gate, the servers that switched to the release earlier in the run go back to the release and service unit they had (a first release is stopped), last one first, each gated again, and the failure names them (#222)
  • The release receipt <receipts>/<component>.json names every server of the deploy instead of only the last one, which overwrote the others (#222)
  • A failed deploy whose component has a release command says the command already ran and is not reverted; the engine README says a release command must work with the previous release too (#222)
  • Deploying the release that is already current unpacks into a fresh releases/<release>.redeploy-<UTC time> directory instead of writing into the live one, so a failed health gate rolls back to the untouched copy; rollback to that release id takes its newest copy, and receipts still name the release id (#223)
  • A failed health gate restores the service unit the previous release ran under, not only the current symlink, then gates the restored release on its health check; a restored release that fails it fails the deploy with its own message (#223)
  • deploy.yml --check runs to its end: the dry run reports the uv release it would install and skips the digest check, the unpack and uv sync of an archive it never transferred (#223)
  • Old releases are removed: once a deploy passes its health gate on a server, it keeps deployment.retainReleases releases under releases/ and removes the older ones, a release's redeploy copies with it, always keeping the deployed release, current and the release before it whatever their order; a failed gate removes nothing, check mode only reports what would go, and a directory that is not a release id is left alone (#224)
  • A rollback to a release that is no longer on the server says it may have been removed by a deploy's cleanup and names the releases still kept there (#224)
  • The PostgreSQL restore drill compares the row count of every table the dump holds with the restored table and fails on a difference or on a dump without a table; it counted pg_class rows before, which an empty database has too, so a restore that brought back nothing passed (#225)
  • The restore drill's scratch database keeps to PostgreSQL's 63-byte names (<database>_restore_check, the name cut to 49 characters), and cloudfall config validate and the converge refuse a declared database named as another's scratch database, as CONFIG_INVALID with code service_database_restore_check_name (#225)
  • The test suite passes on a checkout whose config/examples/tmp/ holds rendered inventory or observations (as task inventory and CI leave it): every test that copies the examples now goes through one helper, sdk/tests/examples_project.py, that leaves that folder behind, and a test refuses a direct copy (#226)
  • Docs that no longer matched the code: ARCHITECTURE.md now says a project pins cloudfall==<version>, that decision records are committed while runtime state stays under tmp/, and which Taskfile.yml tasks call Ansible directly; the AGENTS.md that cloudfall init writes names the cloudfall==<version> pin instead of a rev pin; CONTRIBUTING.md names config/ instead of a state/ directory; the CLI evaluation under evaluations/cloudfall/ uses repository-relative paths (#229)
  • An operator proposal left running no longer blocks the operator for good: once closed as interrupted it stops blocking new proposals for the same alert or drift, and the refusals of decisions approve and operator approve name the command that closes it (#244)
  • A decision record or proposal receipt write now flushes its directory after the rename, so a power loss cannot bring the old record back (#244)
  • A redeploy copy (releases/<release>.redeploy-<UTC time>) that fails its health gate is removed by the deploy's rollback once the previous copy runs again, so rollback <release> takes the newest copy that passed instead of switching to the failed one; the copy's time now has microseconds, so two redeploys in the same second get two directories (#245)
  • health contacts only the component's own servers and fails before any host when the component has no servers in the inventory, instead of reporting success with nothing probed; and deploy, rollback and restart stop at the first of the component's servers they could not reach, where before Ansible skipped the unreachable server and went on to the next one (#246)
  • The PostgreSQL restore drill counts each restored table without its inheritance children (count(*) FROM ONLY), as the dump's COPY data holds them, so a database with classic table inheritance no longer fails the drill every night after a correct restore (#247)
  • config validate refuses a storage pool device path with .., a volumeGroup of . or .., and a pool naming one device as both primary and secondary (storage_pool_same_physical_volume); before, the first two passed validation and crashed requirements show (#248)
  • baseline, bootstrap, services, data, domains, logging, time and backup stop at the first server they could not reach, as deploy, rollback and restart do since #246, where before Ansible skipped the unreachable server and changed the next one; the check lives in one shared task file, playbooks/tasks/rollout-start.yml, and inspect still goes on past a server it cannot reach, reporting it as missing (#254)
  • An operator proposal closed as interrupted now suspends autonomy for its operation kind the way a failed one does, until a human-approved run verifies, and an autonomous run closed as interrupted counts toward maxAutonomousPerHour from its interruption.confirmedAt; before, a killed unattended run left the next proposal for the same alert free to run unattended while what the first run changed was unknown (#258)
  • cloudfall lock clear <component> no longer deletes a server's lock that a run took again after it was shown: lock.yml compares the shown token and removes the lock in one command on the server, instead of reading it and deleting it in two tasks, so a lock taken since is kept and the command exits CONFLICT; and with no lock found while a server could not be reached it exits CONFLICT with component_lock_not_cleared and unreadServers, instead of answering noop as if no run held the lock (#271)
  • cloudfall baseline converge keeps a host's own smartd DEVICESCAN options (-s self-test schedules, -W temperature limits) and replaces only its -M exec, adding -m root when the line has no -m; a failed restart of mdadm's monitor or smartd now fails the run instead of reporting success, and only a unit the host does not have is skipped (#281)
  • logging deploy's plan shows each receiver channel's bot token: a new channels field lists {channel, tokenStored}, and a token not in the store is in missing as telegram-bot-token of <channel>, the name --yes refuses with, where the plan used to say nothing was missing (#282)
  • A command that runs a playbook on one server refuses a Server whose id is one of Ansible's implicit group names (all, ungrouped) with lifecycle_limit_reserved_group before anything runs: --limit all reached every host and --limit ungrouped none (#283)
  • A storage pool's primary and secondary may be stable paths such as /dev/disk/by-id/nvme-<model>_<serial>-part5: logging deploy and inspect resolve each one on the host with realpath, the volume provider and audit's volume.<name> compare the device it names with LVM's physical volumes, lvcreate and lvextend name the physical volume as LVM reports it, and audit shows each declared path next to its device (resolvedPhysicalVolumes); before, any path but the kernel name failed with "is not in vg_data", and NVMe kernel names can swap between boots. A pool path that does not exist on the host, and a pool whose primary and secondary resolve to one device, are refused naming the server, pool and path (#293)
  • logging deploy holds Loki, Grafana, Alloy, and Prometheus and Alertmanager when they have a declared version, at the LoggingStack's declared versions: each role writes /etc/apt/preferences.d/cloudfall-logging-<package> (Pin: version <declared>, Pin-Priority: 999) before its pinned install, so a later apt-get dist-upgrade no longer moves them, and removes the file of a package now installed without a declared version; cloudfall audit adds a logging.packages[<package>] check per pinned package on each of the stack's hosts, reporting drift when the installed version is not the declared one (#294)
  • logging deploy checks, before anything is installed, that each service user can reach the files it reads (Grafana's admin password, Alertmanager's SMTP password and Telegram token files, Alloy's client TLS files) through every existing parent directory, and stops naming the directory that blocks it, as in /etc/cloudfall is 0700 root:root; grafana can't reach /etc/cloudfall/logging/grafana-admin-password, where Grafana used to fail only when it started; the host's directory modes are left as they are (#295)
  • cloudfall-engine playbook run <playbook> --engine <dir> runs a playbook of an engine directory that has no services.yml, where it used to crash with HANDLER_CRASHED and playbook does not exist: .../services.yml whatever playbook was asked for; a services.yml run still decrypts and stages the owner passwords as before (#296)
  • logging deploy no longer leaves a host whose pinned logging package is already newer than declared (Grafana 13.3.1 installed, 13.2.6 declared) with a hold that makes every later apt-get dist-upgrade and unattended-upgrades abort on a downgrade: the holds are written at Pin-Priority: 999, which keeps a held package from moving up without ever forcing a downgrade, and before writing them each role compares the installed version with the declared one (dpkg --compare-versions, a dry run too) and stops on a newer one, naming the package and both versions and saying to declare the installed version or downgrade the package by hand, with no hold written on that host (#302)
  • A missing Elasticsearch exporter credential no longer stops Alloy on the whole host: the collector's systemd drop-in gives each LoadCredential= a SetCredential= default, so when the node's cloudfall password or project CA is gone (the service taken out of the project, its files cleaned up) Alloy still starts and only that exporter's scrapes fail; the next logging deploy drops the exporter and its credential lines (#326)
  • Adding or removing remoteAccess on an Elasticsearch service with metrics.enabled no longer leaves the collector scraping the node the old way until the next logging deploy: the services converge re-renders the Alloy configuration and its systemd drop-in on each logging collector where logging deploy already set Alloy up, validates it, and restarts Alloy only when a file changed, so the exporter switches between plain HTTP and HTTPS with the project CA in the same run; a host that is not a collector, or whose Alloy is not set up yet, is left untouched (#329)
  • The PostgreSQL restore drill (cloudfall backup verify and the nightly restore-check timer) passes --file=- to pg_restore when it reads the dump's row counts; without it every PostgreSQL since 12 refuses with one of -d/--dbname and -f/--file must be specified, so the drill failed on every run (#340)
  • A bootstrap or baseline after a deploy no longer reports changes on each component's releases/ and shared/: the deploy now creates them 0750, as bootstrap does, instead of flipping them to 0755, so a converge after a deploy is idempotent; the Nginx worker still reaches a static route's release through its own ACL entry (#341)
  • On a host running more than one service, such as h1 in config/examples with PostgreSQL and Redis, the Redis backup no longer fails with Permission denied: the PostgreSQL, Redis and Elasticsearch roles now create the shared backup root /var/backups/cloudfall root-owned 0755 before their own backup directory, instead of letting the first service's directory task create it with its own owner and 0700, and the next services converge repairs a root already owned by postgres; each service's own directory keeps its owner and mode (#342)
  • An approved operator proposal now runs on the one server it names: every playbook of converge-services and converge-baseline, the Elasticsearch node preparation included, runs with --limit <server>, a drift proposal's verify step inspects that server alone, and the receipt names only the owner passwords set there; before, a staging alert approved by the staging policy converged every declared server, production included (#350)
  • Operator autonomy is now earned per environment: the history an OperatorPolicy grant is judged on, and the rule that suspends it after a failed or interrupted receipt, count only the receipts of servers in the policy's environment, and a receipt of a server no longer declared counts for none; before, verified runs on staging earned autonomy in production, and a failed staging run suspended production (#351)
  • decisions approve runs only what was checked: a proposal's record now binds the sha256 of its playbook and verify playbook and, when the project is in a git work tree, its HEAD commit and a digest of its uncommitted changes (leaving out the decisions directory, record/ and tmp/), and approve recomputes them and the diff file's sha256 before anything runs; any difference is refused with PRECONDITION and decision_binding_changed, naming what changed and asking for a new proposal, and the record stays proposed. Before, an edited playbook, role or diff file ran unseen under the old diff's approval. Outside git only the two playbooks and the diff are bound (#352)
  • A failed deploy or a rollback brings the previous environment back with the previous release: each release now reads its own environment file, shared/env/<component>.<release>.env (a redeploy copy gets one of its own), named by its unit and read by its release command and command health check, so a deploy that fails only because of a bad environment value leaves the previous release running, and restarting, on the environment it had; rollback points the unit at the target release's file. A deploy without --env-file copies in the file the previous release reads, a component with no environment still names shared/env/<component>.env, which no deploy writes or removes any more, and retention removes a removed release's environment files (#353)
  • A baseline run now removes an SSH key declared lifecycle: revoked from root's authorized_keys, after the active keys are in, matching the key's type and body whatever its comment or options; keys the project doesn't declare stay (D-22). The rendered Ansible inventory's cloudfall_ssh_public_keys now carries a server's revoked keys too, each with its lifecycle, so the role can remove them (#354)
  • Two proposals of one operation in the same second no longer share a decision id: the id is reserved before the check run writes anything, and a second proposal in that second takes <operation>-<stamp>-2, then -3, so each record keeps its own diff and host stats, instead of the second overwriting the first one's diff and then failing with decision_exists; a proposal whose check run fails frees its id and leaves no record (#356)
  • An operator verify step that errors (the alert feed unreachable, the audit failing) now settles the receipt failed with the error as its detail, so autonomy for that operation kind is suspended, and the command still fails with the error; before, the receipt stayed executed for good and blocked its trigger (#358)
  • decisions approve no longer settles a decision verified when its verify step observed no host: every host the check run or the approved run left recap counts for must have counts from the run and a task the verify step ran on it, and a verify play that matched no host, skipped a targeted one or skipped every task on it settles the decision failed, with a verdict naming the hosts that were not observed. Before, a verify step that exited 0 and listed no host as changed proved the run, even when it looked at nothing. A verify playbook aimed at other hosts than the run's (such as localhost checking a URL) now settles failed too; aim it at the hosts the operation targets (#360)
  • Docs: D-10 in docs/decisions.md read as if every component already ran in the full systemd sandbox; a status line now says the unit sets only User, Group, NoNewPrivileges=true and PrivateTmp=true, that secrets reach it through the per-release EnvironmentFile (D-21) and deploy tasks run as root, and that the rest of the sandbox is not built yet (#362)
  • Docs gave the deploy order as health-check gate before symlink switch: ARCHITECTURE.md now says the switch and the restart come first, so the gate runs on the live release, and ARCHITECTURE.md, docs/ci-deploy-guide.md and the Render and Neon migration guides say a rollback, automatic or cloudfall rollback, does not undo the release command, so a migration must stay compatible with the release before it (#363)
  • A deploy no longer fails on one slow download of the pinned uv release: the deploy role retries the download five times, ten seconds apart, still checking its SHA-256, before failing, and a test over every role now requires each get_url download to retry a bounded number of times (#372)
  • A redeploy copy (releases/<release>.redeploy-<UTC time>) whose release command fails is removed with its environment file, as one that fails its gate is, so rollback <release> takes the newest copy that passed instead of starting one that never passed its release command or gate; the previous copy and its own environment file stay (#373)
  • A deploy run with Ansible's --diff no longer prints the component's environment file, secrets included: the tasks that install a release's environment file or carry the previous one over set diff: false (#374)
  • An operator proposal recorded with the audit check server_type.id keeps verifying once baseline.id is compliant, and keeps blocking a new proposal for the same drift, with no removal date instead of until 0.9.0, since a record's fields are never deprecated within cloudfall/v1 (#387)
  • A re-converge of a PostgreSQL, Elasticsearch or logging service no longer needs the network for its package repository's signing key: the PostgreSQL, Elastic and Grafana roles download the key only when it is missing (a get_url without force still asked the server whether the key had changed, so a slow www.postgresql.org failed a converge that should change nothing), keep the installed key's owner and mode, and retry a failed download five times, ten seconds apart, before failing (#388)
  • Docs promised a gate the agent does not control for every server-changing action: ARCHITECTURE.md, docs/brownfield-design.md and MANIFESTO.md now say it holds for declared operations (a person approves at a terminal) and the operator (a person at a terminal or an OperatorPolicy grant), while in project mode the cloudfall mcp serve confirmation (yes: true, confirm_destructive: true) is the agent's own, except for a release whose application declares approval: manual; policy-gated self-confirmation is planned (#391)
  • SECURITY.md says who the private-reporting rule is for: outside reporters use private vulnerability reporting; the maintainers track findings that need existing access as public issues, and keep findings a stranger could exploit remotely private until a fix is released (#392)
  • A PostgreSQL owner role whose database names a passwordRef no longer reports a change on every services converge: the role reads the owner's stored verifier, computes the password's SCRAM-SHA-256 verifier on the controller with that verifier's salt and iteration count (a fresh salt when the role has no SCRAM verifier), and runs ALTER ROLE only when the two differ or the role can't log in, so a converge with the same password changes nothing (D-25). The controller now stages the password itself in its private 0600 file, which never leaves the controller; only the verifier reaches the host, as before (D-9). A stored password that is neither a SCRAM-SHA-256 nor an md5 verifier stops the run, naming the role, never the value (#398)
  • audit now sees REQUIRE SSL removed by hand from a MariaDB owner account at an allowed source (ALTER USER ... REQUIRE NONE): inspect reads each remote account's ssl_type into a new optional sslTypes list of mariadbAccounts, and the mariadb.accounts check reports drift for a remote owner account whose ssl_type isn't ANY (what converge's REQUIRE SSL sets), naming the account and its type. A record an older inspect wrote, which has no sslTypes, now reports that check as unknown instead of compliant when the service has remote owner accounts, until the next inspect; local accounts are checked as before (#435)
  • domains.yml now fails before it changes any server when a Domain it would render has no proxy.upstream, and names every such Domain; before, the site role's assert stopped the run only on that Domain's server, after serial: 1 had already converged the servers before it. upstream stays optional in the schema, for a Domain the fleet routes with its own proxy (#90)
  • A deploy of a component with healthCheck: {type: none} no longer fails its health gate on the missing HTTP fields and stops the release: the gate now requires the systemd unit to be active right after the restart and still active, with no automatic restart in between, 10 seconds later, so a worker that crashes or crash-loops on start still fails and rolls back (#96)

Deprecated

  • kind: ServerType, the server-types/ folder, Server.spec.serverType, the cloudfall_server_types catalog variable and a host's server_type, cloudfall add server-type and cloudfall add server --type keep working until 0.9.0; cloudfall config validate warns DEPRECATED_NAME for each file using an old name, and a server setting both serverType and baseline fails with server_baseline_conflict (#198)
  • Until 0.9.0 the old output names stay beside the new ones with the same values: the rendered Ansible inventory's host variable cloudfall_server_type, inventory show's server_types and each server's server_type, config validate's by_kind.ServerType, and audit's per-server server_type; an observed server snapshot recorded with spec.serverType is still read as spec.baseline, and one with both fails with observation_baseline_conflict (#198)