Skip to content

cloudfall 0.8.0rc90

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 21 Nov 19:58
· 1 commit to main since this release

Changes since v0.7.0:

Breaking

  • deploy --yes and migrate --yes on an application that declares approval: manual now need a person's approval of the release: without a cloudfall release approve record that counts (the same archive SHA-256, exactly the component's declared servers, not expired), deploy --yes exits 4 PRECONDITION with deployment_approval_missing before any playbook runs, naming the release approve command, and lists each record that doesn't count with digest_mismatch, servers_changed or expired. This hits every project that declares manual and deploys today, the example crm.yaml and every Render import included. Two ways out: keep manual and approve each release first (build it, run cloudfall release approve at a terminal, commit the record under decisions/releases/ when CI deploys, then deploy), or switch the application to approval: automatic when another gate, such as GitHub's required reviewers, is the real one. The deploy plan and result gain approval (policy, and for manual the status, the counting record's path, approver and expires_at or approved_at, the records that don't count and the command to run), deploy gains --approvals (default decisions/releases) and the RECORD_INVALID exit for an approvals file that isn't a valid ReleaseApproval, and cloudfall mcp serve binds the approvals folder (its own --approvals) so a deploy call can't name its own; an automatic application's deploy never reads the folder (#163)
  • A LoggingStack now needs a storage pool on its backend's baseline: cloudfall config validate refuses a LoggingStack whose backend server's Baseline declares no storage.pools with CONFIG_INVALID and code volume_requirement_pool_missing, so a project whose LoggingStack validated on 0.8.0rc22 stops validating. Before upgrading, declare the volume group the Loki and Prometheus volumes go into as a pool on that baseline (storage.pools with id, volumeGroup and primary); with more than one pool, name it on each volume in spec.backend.storage.pool and spec.metrics.backend.storagePool, or validation refuses with volume_requirement_pool_ambiguous (S-014-2)
  • cloudfall config validate refuses one SSH key (same type and body, any comment) declared active and revoked in the same environment, with CONFIG_INVALID and code ssh_public_key_lifecycle_conflict, so a project that declares a key that way stops validating; before, the baseline installed it. Before upgrading, delete the revoked declaration if the key should keep access, or mark the active one revoked if it should not (#354)
  • Some source URLs that cloudfall data copy and cloudfall migrate --data read through --dbname before are now refused, because a libpq service file can't hold them as written: a # in the database name or a query parameter value, a password that ends in whitespace, and a value holding a line break. The copy stops before anything is staged, with lifecycle_execution_failed and a reason that names the URL part without quoting it. Before upgrading, percent-encode a # there as %23; a password that ends in whitespace or holds a line break can't be staged even percent-encoded, so change that password first (#355)
  • Over cloudfall mcp serve in a project, a confirmed call of deploy, rollback, restart, migrate, data_copy or domains_apply (yes: true), or of converge_baseline, converge_services or converge_domains (confirm_destructive: true), now runs only when an OperatorPolicy grant covers it in every environment the run touches. Any other confirmed call runs nothing: it answers PRECONDITION with error.context.code: operator_approval_required, context.proposal (the id of a proposed agent record it left in the proposal store, record/operator/proposals/ by default), context.command (cloudfall operator approve <id>, which a person runs at a terminal) and context.reasons (one line per environment, naming why no grant covers it). The same call made again while that proposal is proposed, against the same plan, commit and uncommitted changes, returns the same id and writes nothing. A run that changes no declared server, such as a migrate whose pending steps are builds and controller steps like ttl-lower, touches no environment whose grant could cover it, so it always proposes: its record's operation.servers and operation.environments are empty (the OperatorProposal schema now accepts empty lists there) and its reason says only a person approves it. A converge_* call that refuses itself before any playbook, such as converge_domains for an undeclared Domain, still refuses as before and writes no record. With no OperatorPolicy declared every confirmed call proposes; to keep the one-call flow in an environment, declare an OperatorPolicy for it whose spec.autonomy.operations grants those classes (deploy, rollback, restart, migrate, data-copy, domains-apply, converge-baseline, converge-services, converge-domains) and earn its requiredVerifiedRuns. The checks that came first still do and write no record: argument and config validation, S-008's deployment_approval_missing, and a component lock another run holds. The server's instructions say so, and the plan's approval.selfConfirm tells an agent in advance. A migrate with no pending step left still answers complete and writes no record, since there is nothing to run. On the CLI, a command without --proposals runs as before (cloudfall deploy --yes in a shell runs the deploy), while one given --proposals and --yes (deploy, rollback, restart, data copy, domains apply, migrate) is gated like the MCP server's call: no grant, and it proposes instead of running. A repository's server lists the same tools (#476)

Added

  • A Component can declare spec.buildTools (tool name to minimum version), spec.build (ordered steps, each a workingDirectory inside the packaged root, an argv command and an optional timeoutSeconds from 1 to 3600, default 900) and spec.artifact.include (relative paths the build produces); cloudfall config validate refuses a step whose command[0] is not in buildTools (component_build_tool_undeclared), an absolute or .. path, and buildTools or artifact without build. Validation only: artifact build does not run the steps yet. Components without build validate as before (#113)
  • cloudfall-engine artifact build runs a Component's declared spec.build steps on the controller before packaging: it checks every spec.buildTools entry is on PATH at its minimum version before cloning (artifact_build_tool_missing, artifact_build_tool_unsupported), runs each step in the clean checkout with only PATH, HOME, LANG, TMPDIR and CI=true in its environment and its output on stderr, fails without writing an archive on a non-zero exit or a step outrunning its own timeoutSeconds (artifact_build_step_failed), a missing include path (artifact_include_missing), an include path that leads out of the checkout through a symlink (artifact_include_outside_checkout) or a tracked file a step deleted (artifact_tracked_file_missing), packages the tracked files plus every spec.artifact.include path (a gitignored web/dist, say), and records the steps and tool versions as the new optional spec.build of the Artifact metadata. The command's time limit is now the clone time (1800 seconds) plus every declared step timeout, on the CLI and the SDK/MCP path alike; an explicit --timeout replaces it, and --timeout 0 still disables it. A Component without build is packaged as before (#115)
  • The converge_domains MCP tool takes an optional domain: given, domains.yml runs only on that Domain's proxy server (--limit) and renders only that Domain, and an unknown id exits PRECONDITION (lifecycle_domain_missing) before the inventory render; omitted, it runs on every server as before. Each domain receipt now records previousConfigurationSha256, the sha256 of the vhost file the run replaced, or null when none was there; receipts without it still load (#116)
  • cloudfall domains apply <domain> [--issue-certificate] [--yes] converges one Domain: domains.yml runs on its proxy server alone and renders only it, so no baseline, firewall or service task runs. Without --yes it exits 0 with a plan. --issue-certificate refuses a Domain with tls.mode: disabled, and first checks on the controller that the primary name and every alias resolve to the proxy server (PRECONDITION, domains_dns_unverified, naming each failing name); a proxied Domain skips the check (dnsCheck: "skipped-proxied"). With --yes it answers the receipt path under --deployments (#117)
  • cloudfall domains observe <domain> reads one Domain on its proxy server alone, through the new read-only domains-observe.yml: the vhost file and its sha256, the certificate's expiry, certbot.timer, and whether the vhost still matches the last receipt (receipt: null before the first). It changes nothing on the server or in the project, and a missing vhost exits 0. cloudfall mcp serve lists domains_apply (with its yes gate) and domains_observe as tools (#118)
  • Components: a release.command (with optional workingDirectory and timeoutSeconds, 1 to 1800, default 300) runs once per deploy on the first server of deployment.servers, which now deploys first, after the release's dependencies are materialized and before the unit file changes or current moves; it runs as the application's user with the component's env file loaded under systemd's EnvironmentFile= rules. A non-zero exit or a timeout fails the deploy with ENGINE_STEP_FAILED before any server switches, and a passing run is recorded as releaseCommand (server, command, exit code, last 2000 characters of output) in every release receipt of that deploy (#119)
  • deploy without --yes shows the component's release command as release_command (its command, the working directory resolved under the new release directory, timeout_seconds and the release server), or null for a component without one, and says in would_run that it runs once on that server before any switch; the rollback plan of a component with a release command shows release_command_skipped: true, since a rollback never runs it (#120)
  • A PostgreSQL Service database may name an owner role and its passwordRef (environment, path, key in the secrets store) instead of an application, so a project of a server and its database alone validates; spec.postgresql.remoteAccess declares a listen address and the client networks allowed in. config validate refuses an entry with both or neither of application and owner, an owner without passwordRef or a passwordRef next to application, a reserved owner (service_database_owner_reserved), and a remoteAccess with no owner database, no firewall, the port already open, or a source of every address. Converging these declarations comes later: until then the PostgreSQL role refuses them before any task runs (#121)
  • services.yml converges a PostgreSQL database owned by a declared role: it creates the owner login role, sets its password from passwordRef as SCRAM-SHA-256 on every converge, and gives it the database; backups, the restore drill, and data copy (restoring as the owner, its receipt's spec.owner) cover it like any other database. Every path that runs services.yml (the converge_services MCP tool, the services step of cloudfall migrate, operator approve and operator run, and cloudfall-engine playbook run services.yml) reads the password from a --secrets-dir flag (default secrets) and stages it in a 0600 file in a private temp dir outside the project, removed after the run; the value is in no inventory, command line, Ansible output, answer, or receipt, and the answer's owned_roles names the reference, the key, and the value's SHA-256; operator approve records them as ownedRoles in the proposal receipt's outcome (an optional property) and operator run in each executed proposal's entry. A missing secrets file or key fails with PRECONDITION naming the expected path and key before any playbook runs (#122)
  • A PostgreSQL service with spec.postgresql.remoteAccess opens to its declared sources over TLS: services.yml sets listen_addresses = '127.0.0.1,<listenAddress>' and ssl = on with the Debian ssl-cert snakeoil pair (restarting the cluster), and writes one hostssl <database> <owner> <cidr> scram-sha-256 line per owned database and source between Cloudfall markers in pg_hba.conf, reloading only after pg_hba_file_rules reports no error; baseline.yml adds one ip saddr/ip6 saddr <cidr> tcp dport <port> accept rule per source, rendered as a cloudfall_firewall.allowedInbound entry with a new optional sources list. Removing remoteAccess resets the listener and removes the pg_hba.conf block and the firewall rule; databases named by application stay socket-only (#123)
  • cloudfall data copy takes a repeatable --source-schema NAME (and the data_copy MCP tool a source_schema list): the dump and restore copy only the named schemas, the empty-target check and the per-table row-count check read exactly those schemas, and the plan without --yes names them; a name that is empty, longer than 63 bytes, or holds a control character or a brace is refused as a usage error. Without it the dump is the whole database as in 0.7.0, and the checks now cover every non-system schema of the source instead of public only, so a row-count mismatch outside public now fails the copy. The DataMigrationReceipt gains spec.schemas (sorted) and a schema on each spec.tables[] entry beside table and rows (#124)
  • cloudfall data copy, and the data:<database> step of cloudfall migrate, run a preflight on the target host after the empty-target check and before pg_dump: it reads both catalogs and refuses a copy the target can't hold, with nothing dumped, the target untouched and no receipt, exiting PRECONDITION with code data_copy_preflight_refused and four lists in the error context, each item named: missingSchemas (a named schema the source lacks), extensions (one the selected schemas need, through pg_depend or by living in them, or every source extension but plpgsql without --source-schema, that the target's pg_available_extensions lacks), roles (a role a row-level security policy on a selected table names that the target's pg_roles lacks) and authReferences (a function or policy of the selected schemas that references auth. when auth isn't copied). The role writes the findings to <service>-<database>.preflight.json under --data-migrations, so the lists arrive whole. With --source-schema, pg_dump also gets one --extension per extension the schemas need (pg_dump 14 or later), and pg_restore now runs with --single-transaction, so a failed restore rolls back whole, writes no receipt, exits ENGINE_STEP_FAILED, and the next run passes the empty-target check (#125)
  • cloudfall migrate takes a repeatable --data-schema DATABASE=NAME beside --data DATABASE=PATH (and the migrate MCP tool a data_schema list): the data:<database> step copies only the named schemas, as data copy --source-schema does, and the plan names them. A --source-schema or --data-schema name must now match ^[a-z_][a-z0-9_]*$ and be at most 63 characters, and may be given once per copy; a name that doesn't, a repeated name, or a --data-schema for a database with no --data exits PRECONDITION (code data_copy_schema_invalid or migrate_data_schema_without_data) before any playbook runs, where a malformed name was a usage error and a repeat was silently dropped (#126)
  • A Domain can declare spec.routes, path routes that each either proxy to an upstream (address, port) or serve a component's release files (static: component, directory, optional spaFallback). cloudfall config validate refuses a Domain with both proxy.upstream and routes (domain_route_source_invalid), a path that doesn't start with / or uses characters outside letters, digits and ._~/-, or a static path without a trailing / (domain_route_path_invalid), a path declared twice (domain_route_path_duplicate), a path under /.well-known/acme-challenge (domain_route_path_reserved), a missing static component (resource_reference_missing) or one not deployed on the Domain's proxy.server (domain_route_component_not_on_proxy), and a directory or spaFallback that is absolute, climbs out with .., or, for directory, names the release root. Validation only: the domains playbook doesn't render routes yet and still refuses a Domain without proxy.upstream before any change. Domains with only proxy.upstream validate and render as before (#159)
  • cloudfall release approve <component> --release <release> records a person's approval of one built release as a ReleaseApproval (new release-approval.schema.json) under decisions/releases/: the application, component, release, archive SHA-256, git commit, servers, approver, approvedAt, expiresAt (--valid-for hours, 1 to 168, default 24) and via: terminal. It verifies the artifact first, refuses a component of an automatic application with release_approval_not_required, and writes the record only once a person typed the release id at a terminal; --yes never answers it, and it is no MCP tool. It deploys nothing (#160)
  • Components: a healthCheck of type: command (a required non-empty command argv, timeoutSeconds 1 to 300 and attempts 1 to 20, no other key) validates, and deploy, restart and rollback without --yes name the check and its command in their plan's wouldRun (#161)
  • cloudfall domains apply renders a Domain's spec.routes: a proxy route becomes a location that proxies to its upstream with the full request path, and a static route serves <installRoot>/current/<directory> of its component, answering spaFallback for an unknown path when it declares one and 404 when it doesn't. Without a / route, a path no route matches answers 404, not Nginx's default page. Locations render longest path first, whatever the declared order, where the single proxy location rendered before (the port 80 server without active TLS, the 443 server with it); the port 80 redirect and the ACME challenge location stay. The site role grants the Nginx worker (www-data) traverse-only access (POSIX ACL user:www-data:--x, installing acl) from the application root to the install root and its releases/, so it reads the release files but never shared/env/<component>.env, and a deploy or cloudfall rollback switches what a static route serves without re-running the domains playbook or reloading Nginx. The domains playbook's early check now refuses only a Domain with neither proxy.upstream nor routes. A Domain with only proxy.upstream renders a byte-identical vhost (#162)
  • Deploy: a component's healthCheck of type: command now gates deploy --yes. After the switch and the restart it runs on each server in a transient systemd unit cloudfall-health-<component>, as the application's user, in <installRoot>/current, with the service's env file and timeoutSeconds as its runtime limit, up to attempts times 3 seconds apart. When every attempt fails or times out, the previous release is restored (a first release is stopped), the deploy fails with ENGINE_STEP_FAILED naming the check and its exit code or timeout, and the release receipt is written with healthCheckPassed: false. Receipts of a command check carry healthCheck (command, attempts, exit code, timed out), never the command's output, which stays in the server's journal. The release receipt schema accepts healthCheckPassed: false only with a failed command healthCheck, and every receipt written before (#164)
  • restart, rollback and health run a component's type: command health check instead of refusing it: the same transient unit as deploy's gate, with its user, working directory, env file, timeout, attempts and redacted output. health reports unhealthy when the check exits non-zero or times out, a failed check fails restart or rollback without restoring anything, as an http check does, and none of them writes a release receipt. The check starts its unit as root, so health of such a component needs the inventory SSH account to become root, as restart and rollback already do; http and none checks are unchanged (#166)
  • cloudfall config upgrade: rewrites a project from the deprecated ServerType names to Baseline (moves server-types/ to baselines/, rewrites kind and spec.serverType), keeping comments, formatting and line endings; a second run changes nothing, and it refuses, changing nothing, a move onto an existing file or a file or folder reached through a symlink (upgrade_symlink) (#198)
  • audit checks the certificates a server's host requirements name (present, for the right names, more than 30 days left at the observation's time) from new certificates evidence inspect collects, and expects the logging gateway's firewall rule, so a deployed stack audits compliant (S-011)
  • cloudfall logging deploy provides the stack's mTLS material: each host makes its own key and a signing request (logging-requests.yml), the project CA signs the missing and due certificates, and they are delivered with the CA; certificates provided by hand are kept (S-011)
  • cloudfall logging deploy: deploy the declared LoggingStack after meeting its secret requirements; generated secrets are written to the store, an input secret that is missing stops the run (exit 4, INPUT_REQUIRED), and each stored secret is copied to its host path without overwriting a file already there; without --yes it shows the plan (S-011)
  • Logging collectors pin the gateway name to the backend's address in /etc/hosts (loopback on the backend itself), so the gateway needs no DNS record when the backend declares an IP (S-011)
  • The logging gateway's port opens on the backend's firewall to each remote collector's addresses only, rendered beside the remoteAccess rules and enforced by logging.yml; a remote collector with no IP behind a firewall fails config validate (logging_gateway_collector_unaddressed) (S-011)
  • LoggingStack: migration.mode: none for a host with no legacy agents; parallel still requires preserveLegacyAgents: true (S-011)
  • Logging roles check every package pin against the host's APT sources before installing or configuring anything, and stop naming the versions on offer when one is missing (S-011)
  • Certificate provider, controller side: a project CA per environment, kept sops-encrypted under secrets/<environment>/pki/, that signs host requests for exactly the names and usage a requirement grants (S-011)
  • cloudfall requirements show: the volumes, ports, secrets, certificates, DNS names and package pins each declared LoggingStack needs from its servers, by server (S-011)
  • cloudfall secrets generate and cloudfall secrets input: keep the secrets a LoggingStack requires sops-encrypted under secrets/, generated or typed at a terminal with echo off; requirements show gives secrets a state and lists the missing ones (S-011)
  • A decision record and a proposal receipt name the Cloudfall version that last wrote them in spec.writerVersion, an optional schema field: records written before it stay valid (#218)
  • deploy, rollback, restart and an artifact build hold their component's lock under tmp/locks/ on the management host until they end, passed, failed or interrupted, so a second run of the same component exits CONFLICT (component_locked) before any server is contacted, naming who holds the lock, since when, and whether that process still runs; cloudfall lock clear <component> shows a lock a killed run left, with its age, and removes it once a person types the component id at a terminal, never while its process still runs on this host (#221)
  • deploy.yml, rollback.yml and restart.yml hold the component's lock on each server, at /var/lib/cloudfall/locks/<component>.json, from before they change anything there until that server's turn ends, passed or failed, so runs from two controllers, cloudfall-engine playbook run and approved runs of those playbooks no longer race on current, the unit file or the release command's unit: a second run fails on that server, naming the action, user, controller, start time and age of the lock it found (#221)
  • A Component's optional deployment.retainReleases (an integer, at least 2, 5 when absent) says how many releases a deploy keeps on each server; the SDK inventory and the deploy role carry it, and retainUntilCleanup: true still validates and means the cleanup is on (#224)
  • The release receipt names, in each server entry whose gate passed, the releases the deploy removed there under removedReleases; release-receipt.schema.json accepts it and refuses it on a failed entry (#224)
  • The PostgreSQL backup writes globals-<timestamp>.sql from pg_dumpall --globals-only --no-role-passwords beside the per-database dumps, so a restore onto a fresh server has its roles; retention prunes it with the dumps (#225)
  • Baseline storage.pools: existing LVM volume groups that workload volumes go into; each LoggingStack volume names its pool, LV, LV type and physical volumes in requirements show, and config validate refuses a volume with no pool (S-014)
  • audit checks each required volume (volume.<name>: type, physical volumes, size, health, mount) from new volumes evidence inspect collects (S-014)
  • cloudfall logging deploy provides the stack's data volumes: a read-only report from each host, refusals decided on the controller before any host change (volume_refused), then lvcreate/mkfs for missing volumes, lvextend --resizefs for small ones, an fstab line by UUID, the mount and RequiresMountsFor drop-ins (S-014)
  • cloudfall decisions mark-interrupted <id> --confirm-dead and cloudfall operator mark-interrupted <id> --confirm-dead close a record left running by a run that died midway as the new status interrupted, once a person confirmed the run is dead; the record keeps its approval and gains interruption.confirmedBy (--by, default USER) and interruption.confirmedAt. Nothing runs on the fleet, and neither command is an MCP tool (#244)
  • AlertChannel (alert-channels/): a Telegram chat that a Baseline's hosts and a LoggingStack receiver name; its bot token is an input secret on every server that sends, set with cloudfall secrets input telegram-bot-token --owner <channel> (S-015)
  • A LoggingStack receiver may name an AlertChannel: Alertmanager delivers to it with its native telegram_configs and bot_token_file, the token delivered by logging deploy; an Alertmanager older than 0.26 is refused before its configuration is written (S-015)
  • cloudfall alerts test <server>: send one test message through the server's alert channel; audit checks each host's channel (alerting.channel: sender, token and chat id present, mdadm and smartd pointed at the sender) from new alertChannel evidence inspect collects, never the token (S-015)
  • cloudfall baseline converge: run baseline.yml with each host's alert channel; hosts whose baseline names an AlertChannel get /usr/local/sbin/cloudfall-alert, the channel's token and chat id, and mdadm and smartd pointed at the sender; a token not in the store stops it before any server (S-015)
  • A Service can be serviceKind: elasticsearch (S-012): spec.elasticsearch declares version (an exact X.Y.Z of a line Elastic still supports, 8.19 or 9.x), heapMb (512 to 31744), and optionally nodes, clusterName, existingCluster, users (each with indexPatterns and a passwordRef, never a password), remoteAccess, snapshotRepository and restoreCheckIndices. cloudfall config validate refuses an unsupported version (service_elasticsearch_version), bad nodes (service_elasticsearch_nodes), an existingCluster without clusterName, with a CA file that is missing, outside the project, not a CA or holding a key, or with a seed host that is a declared node (service_elasticsearch_existing_cluster), a server in two clusters (service_elasticsearch_server_shared), several nodes or an existing cluster without a snapshotRepository (service_elasticsearch_snapshot_repository_local), a reserved or repeated user name (service_elasticsearch_user_reserved, service_elasticsearch_user_duplicate), a pattern of every or a hidden index (service_elasticsearch_pattern), and remoteAccess under the PostgreSQL rules; it warns ELASTICSEARCH_VERSION_ENDING with the date when a line is within 90 days of its end of life. The rendered inventory carries the service, with nodes and clusterName resolved, on its server; nothing converges it yet (#309)
  • services.yml converges a one-node serviceKind: elasticsearch service on Debian 13 through the new cloudfall_elasticsearch role (S-012). Its first task refuses a heapMb above half of the host's memory before anything changes. It installs elasticsearch=X.Y.Z from Elastic's APT repository for the declared major and holds it, and writes elasticsearch.yml (cluster.name, node.name, HTTP on the loopback bind only, bootstrap.memory_lock: true, security on, never discovery.type), a jvm.options.d heap file and a unit drop-in with LimitMEMLOCK=infinity, all before the package, so its security auto-configuration skips the node. cluster.initial_master_nodes is written only while the data path holds no cluster state and leaves the file in the same converge once the cluster has formed, with no restart, so a second converge changes nothing; the converge stops when the node runs without its memory locked. Cloudfall's own cloudfall account is a superuser in the file realm, with only its bcrypt hash in users and its password in a 0600 file owned by elasticsearch; the elastic password is set once, as a bcrypt hash computed on the controller (D-9). Each node of the service yields the elasticsearch package and the generated elasticsearch-cloudfall-password as S-011 requirements, and cloudfall secrets generate also generates the elastic password (ELASTIC_PASSWORD beside CLOUDFALL_PASSWORD in secrets/<environment>/services/<id>.env); every path that runs services.yml decrypts both first and stops before any playbook when one is missing. More nodes, declared users, remoteAccess and existingCluster stop the converge until their slices land (#310)
  • An Elasticsearch service's declared users converge (S-012): from the service's home server, each user gets a role of its own name limited to its indexPatterns (read: read and view_index_metadata; write adds write, create_index and manage) and a native user holding that role alone. Every path that runs services.yml decrypts each user's passwordRef first, stops before any playbook when one is missing, and stages only its bcrypt hash, computed on the controller and sent as password_hash (D-9); the salt is derived from the service's cloudfall password, so an unchanged password puts nothing and a second converge changes nothing. Users and roles Cloudfall makes carry metadata.cloudfall_managed_by: <service id>; a converge deletes the marked ones no longer declared and never changes or deletes an unmarked one, and refuses a declared name an unmarked user or role already holds. inspect reads the cluster's users and roles on the home server (elasticsearchUsers in the observation), and audit adds elasticsearch.users[<id>]: drift when a declared user or role is missing or a marked one is left over, with users and roles not made by Cloudfall listed as unmanaged (#311)
  • An Elasticsearch service grows to two or more nodes and rolls a version one node at a time (S-012). Each node yields an elasticsearch data volume of 10 GiB at /var/lib/elasticsearch in its baseline's only storage pool, mirrored for a cluster of one and expendable for two or more; a node of a cluster of two or more also yields port 9300/tcp open to the other nodes only and an elasticsearch-transport certificate from the project CA for its hostname and addresses, of the usage server-and-client: the one certificate remoteAccess serves HTTP with, which the services converge now issues for a node with remoteAccess too. The role refuses remoteAccess on a cluster of two or more, since its one listenAddress exists on one server only. config validate refuses a node whose baseline has no pool (volume_requirement_pool_missing), several (volume_requirement_pool_ambiguous), or, for a cluster of one, no secondary physical volume (volume_requirement_pool_unmirrored), and a node of a cluster of two or more with no IPv4 (service_elasticsearch_node_unaddressed). The rendered inventory carries the service on every node with its transport address and seed hosts, adds the 9300 rule to each node's firewall, and lists each cluster's home first. Every path that runs services.yml through cloudfall first reads each node's volume facts and decides its volume, then has each node make its key and a signing request, signs the missing and due certificates with the project CA and delivers them, all before any host changes. The role creates and mounts the volume, enforces the firewall, binds the transport to the node's address with TLS verification_mode: full, bootstraps a cluster only from the home node and refuses when another node of it already runs, sets the elastic password from the home only, and restarts a running node only from a green cluster (yellow with one node), with replica allocation held and a flush (released again even when the restart, the rejoin or the version check fails), then waits for the node to rejoin on the declared version and the cluster to return to its status before the next server (#312)
  • An Elasticsearch service's S3 snapshotRepository is converged instead of refused (S-012). Its credentialsRef names one key of the secret store holding <access key id>:<secret access key>; every path that runs services.yml decrypts it on the controller, stages the two keys in 0600 files and passes only their paths, and stops before any playbook when the key is missing or malformed. On every node the role puts them in the keystore as s3.client.default.access_key and s3.client.default.secret_key through stdin, keeps only their SHA-256 beside it so a second converge changes nothing, names the endpoint as s3.client.default.endpoint in elasticsearch.yml, reloads a running node's secure settings, and registers the repository cloudfall as type: s3 with the bucket and base path, replacing the filesystem repository of a service that grew; the snapshot and restore-drill scripts use it unchanged (#312)
  • config validate refuses an Elasticsearch cluster of two or more nodes on a baseline without a firewall, since nothing would narrow 9300 to the other nodes (service_elasticsearch_cluster_firewall_missing); the inventory render refuses it too (#312)
  • cloudfall backup run and cloudfall backup verify work for a serviceKind: elasticsearch service (S-012). The converge adds backup.directory to path.repo, which restarts each existing node once on the first converge after the upgrade, registers it as the filesystem snapshot repository cloudfall, and installs cloudfall-elasticsearch-backup and cloudfall-elasticsearch-restore-check with sandboxed units and timers on backup.onCalendar and backup.restoreCheckOnCalendar. The backup counts the documents of every index restoreCheckIndices matches (or of the smallest index) and writes the counts as one document per snapshot into Cloudfall's own index cloudfall-snapshot-counts. It then takes the snapshot cloudfall-<timestamp> of every index, that one included, and the cluster state, waits for it, and fails unless its state is SUCCESS, so a PARTIAL or FAILED snapshot writes no receipt. Finally it deletes Cloudfall's snapshots of the service older than retentionDays with their counts, always keeping the newest and never touching a snapshot it did not take. The restore drill verifies the repository, restores the snapshot's counts and every recorded index as cloudfall-restore-check-<index> with no replicas and without aliases, checks the low disk watermark, waits for green, compares each document count, and deletes the restored copies even when a step failed; it refuses to start over an index of a copy's name, and fails when no snapshot exists. Both run as the elasticsearch user, reach the node as the role does (with remoteAccess, over HTTPS on the listen address, verified against the project CA) and write the usual BackupReceipt. An S3 snapshotRepository stops the converge before it changes the host, with a message that S3 repository registration is not built yet (#313)
  • An Elasticsearch service (S-012) is audited, opened by remoteAccess, and scraped for metrics. cloudfall audit reports drift on each node when the installed elasticsearch package differs from version, when elasticsearch.service is not running and enabled, and when the managed settings of elasticsearch.yml (network.host, http.port, http.host, xpack.security.enabled, xpack.security.http.ssl.enabled) differ from what the role writes, so a hand edit is drift before the node restarts with it; it reports the node as running an unsupported version once its line is past Elastic's End of Maintenance date at the observation's time (D-12). inspect reads elasticsearch.yml as text on a server with an Elasticsearch service (spec.elasticsearch of the observation). With remoteAccess, the firewall opens bind.port to allowedSources only, HTTP listens on loopback and listenAddress and is TLS-only on both, with the node's one certificate from the project CA (the elasticsearch-transport certificate requirement of the new usage server-and-client, for the server's hostname and every address, the listen address included, at /etc/elasticsearch/certs/transport.crt, transport.key and ca.crt), which the role checks before writing a configuration that names it, and stops on while nothing issues it yet; the role's own API calls then go over HTTPS to the listen address, verifying the project CA. With metrics.enabled, the collector renders one Alloy prometheus.exporter.elasticsearch for the local node, as the cloudfall account, labelled with the service id; Alloy reads the password, and with remoteAccess the project CA, as systemd credentials (LoadCredential), and logging deploy stops before changing the collector when the node's files are not there yet (#314)
  • An Elasticsearch service with existingCluster is converged instead of refused: its nodes join a cluster Cloudfall did not create (S-012). Each node's transport leaves loopback even when it is the only Cloudfall node: discovery.seed_hosts lists the existing cluster's seed hosts, then the Cloudfall nodes; its firewall admits 9300 from the other Cloudfall nodes and the seed hosts' addresses only; it gets its transport certificate from the project CA and trusts the existing cluster's caCertificateFile beside it, which the services converge delivers to /etc/elasticsearch/certs/existing-cluster-ca.crt, with verification_mode: certificate. A joining node never writes cluster.initial_master_nodes and never sets the elastic password, which stays the existing cluster's, and the existing nodes are only read. Before the data volume and the package, the role asks the seed hosts over HTTPS on port 9200, verified against that CA, for GET /_nodes, and stops, naming the seed host or node and its value, when the cluster's name differs from clusterName, when a node runs another major, when a node is newer than version, or when no seed host answers. Once existingCluster is removed, a converge drops the seed hosts and the CA from the configuration, restarts one node at a time, removes the CA file, and waits for the cluster to be green with two or more Cloudfall nodes, or yellow with a single one, whose replicas have nowhere to go; clusterName keeps the name. config validate refuses a seed host given by name rather than address, and a node that joins an existing cluster without a firewall or an IPv4, as for a node of a cluster of two or more (#315)
  • existingCluster.credentialsRef names one key of the project's store holding <username>:<password> of an account the team already has on the existing cluster with the monitor privilege; the join check logs in with it, so nobody creates a user on the old cluster by hand. Without it, the check logs in as the cloudfall account, which the team then grants monitor there. A missing key, or a value not of that form, stops the converge before any playbook, naming the file and the key; the login is staged in 0600 files on the controller only, removed after the run, and never reaches a node, the inventory, task output or an error (#315)
  • cloudfall operator mark-interrupted <id> --confirm-dead also closes a receipt left executed, whose verification never recorded an outcome, as interrupted, as it does a running one; it then stops blocking new proposals for the same alert or drift (#358)
  • A project states its format as [tool.cloudfall] format in its pyproject.toml, and cloudfall init writes the current one (format = 1). cloudfall config validate checks it before validating: a project without the key, or without a pyproject.toml, reads as format 0 and still validates, with one PROJECT_FORMAT_OLD warning naming cloudfall config upgrade; a format outside the ones this release reads (0 to 1), or one that is not a non-negative integer, stops with the new exit PROJECT_FORMAT_UNSUPPORTED (95) and error.context.code project_format_too_new, project_format_too_old or project_format_invalid (#375)
  • cloudfall changelog lists entries: each release now records its interface changes (commands, flags, output fields, exit codes) in schema-changelog.json and its manifest snapshot, both in the wheel, as the release bot's after_stamp step runs treaty changelog-add cloudfall.app:app. The first release seeds the snapshot, so its entry lists the whole interface as added; a release whose interface did not change adds no entry. The Releases section of CONTRIBUTING.md names no manual step any more (#378)
  • docs/compatibility.md publishes the compatibility policy: the stable surfaces (resource schemas, the CLI envelope, exit codes, record and receipt schemas, MCP tool names), what a patch, minor and major release may change before and from 1.0, how long a deprecated name stays, the project format version and cloudfall config upgrade; README.md and CONTRIBUTING.md link it (#381)
  • A Service can be serviceKind: mariadb (S-019): spec.mariadb declares version (an exact X.Y.Z of a long-term series MariaDB still maintains: 10.11, 11.4, 11.8 or 12.3), innodbBufferPoolMb (128 or more), and optionally databases (each with an application, or an owner with a passwordRef, never a password, and authentication: ed25519 if wanted), remoteAccess, and replication (role primary or replica, serverId, a primary's listenAddress, a replica's primary service or source with host, port, credentialsRef and caCertificateFile). cloudfall config validate refuses an unsupported version (service_mariadb_version), a server with two MariaDB services (service_mariadb_server_shared), a reserved owner or database name (service_database_owner_reserved), a repeated serverId (service_mariadb_server_id_duplicate), a primary without listenAddress or a replica with one (service_mariadb_replication_listen), a replica without exactly one primary or source, or whose primary is no primary MariaDB service of its environment on another server (service_mariadb_replication_primary), a replica with databases (service_mariadb_replica_databases), of another series than its primary or older (service_mariadb_replica_version), a pair without a firewall (service_mariadb_replication_unfirewalled), a replica server with no address its primary can admit (service_mariadb_replica_unaddressed), a source CA file that is missing, outside the project or not a CA (service_mariadb_replication_source), a server without one mirrored storage pool (the S-014 codes), and remoteAccess under the PostgreSQL rules; it warns MARIADB_VERSION_ENDING with the date when a series is within 90 days of its end of life, and MARIADB_REPLICATION_UNENCRYPTED for a source without a CA file. The kind yields its mariadb volume, package, replication port, certificate and generated replication password as host requirements, and the rendered inventory carries the service with its databases and references, never a password; nothing converges it yet (#419)
  • services converges a standalone MariaDB service (S-019) with the new cloudfall_mariadb role: it refuses, before any task changes the host, an innodbBufferPoolMb above three quarters of the server's memory, a series MariaDB publishes no package of for the server's Debian release, and a series other than the installed one; it mounts the data directory on the mirrored mariadb volume the services converge decides with the Elasticsearch volumes, installs mariadb-server at the exact declared release from MariaDB's archive and holds it, and writes the managed settings to /etc/mysql/mariadb.conf.d/90-cloudfall.cnf (listener on bind.address only, buffer pool, utf8mb4; no binary log). An application database gets a user named after the application's Linux user that logs in through the unix socket only; an owner database gets a user whose password reaches the host only as the hash its plugin stores (mysql_native_password, or ed25519 with authentication: ed25519), computed on the controller and set only when it differs from the stored one, so a second converge changes nothing. Each user has every privilege on its own database. A user Cloudfall made that is no longer declared is locked, its database left with its rows, and users and databases made by hand are left alone; cloudfall audit lists both kinds as undeclared and unmanaged. A passwordRef whose file or key is missing stops the converge with PRECONDITION before any playbook runs. remoteAccess and replication are still refused by the role until their own issues land (#420)
  • A MariaDB service converges its remoteAccess: the services converge has the server make its key and the project CA sign its certificate for its hostname and addresses, the role installs it (the key 0600 for the mysql account), listens on loopback and listenAddress with ssl_cert, ssl_key and ssl_ca, and gives each owner with a password one account per allowed source from the same controller-computed hash, requiring TLS, so a plain connection from a source is refused; the firewall admits the port from the allowed sources only, and audit expects that rule, the listener and those accounts. An account of a source no longer declared is locked, never dropped; a renewed certificate is loaded with FLUSH SSL, without a restart. replication is still refused by the role (#421)
  • A MariaDB service is backed up and its restore drilled (S-019): services installs cloudfall-mariadb-backup and cloudfall-mariadb-restore-check with their systemd timers (the drill's only with restoreCheckOnCalendar), run as root through the socket, and cloudfall backup run <service> and cloudfall backup verify <service> run them and write their BackupReceipt. A backup writes, for every declared database, one gzip dump taken with mariadb-dump --single-transaction (one consistent snapshot of the InnoDB tables, no table locks; tables of another engine are named in the output) and beside it a count file of every table and its rows, counted from the stored dump, both 0600 in a 0700 directory; then it deletes each database's dumps older than retentionDays, never the newest, and only files of the names it writes. The drill loads the newest dump of each database into the scratch database cloudfall-restore-check without writing to the binary log, compares its tables and row counts with the count file (never with the live source, so later writes cannot fail it), and drops it on every path; it fails, and says so, when a database has no dump or no count file, when the load fails, and on any difference, and leaves a database of the scratch name it did not make alone. The role checks onCalendar and restoreCheckOnCalendar before it changes the host. The dumps hold tables, views, routines and triggers but no scheduled events (--events is not passed, so no event can run in the drill's scratch database): a team that uses events recreates them after a restore (#422)
  • A MariaDB service converges its replication between two Cloudfall servers. Both servers write a row-based binary log with GTIDs in strict mode, kept seven days, and get their server certificate from the project CA. A primary listens on listenAddress too and holds one cloudfall_repl account at the IPv4 of each replica's server, requiring TLS, set from the hash of its password, which cloudfall secrets generate makes and keeps sops-encrypted: the primary receives only the hash, a replica the password in a 0600 file for the run only, and its firewall admits the port from those addresses only. An empty replica that never replicated is seeded from one consistent dump of its primary at a recorded GTID position, then follows it by GTID over TLS verified against the project CA, read-only, and the converge waits until it reached the primary's position and holds the same tables; a second converge changes nothing. A replica that holds data attaches from its own position only when its primary saw every GTID it holds; otherwise, and when it has no GTID history, the converge stops and drops nothing. Changing a replica to role: primary promotes it when its source can't be reached, or is read-only and applied, and the answer names the last GTID it applied; over a running, writable source it stops with PRECONDITION naming both positions, and the replica still replicates. Swapping the roles of a pair in one change ends within two converges, with no seed. On a replica, cloudfall-mariadb-backup and cloudfall-mariadb-restore-check take every database it holds that is not MariaDB's own, read when they run, since a replica declares none. A replica of a source Cloudfall does not manage is still refused by the role. The primary's replication password is no longer a host requirement at /etc/mysql/cloudfall/replication-password: no host keeps it (#423)
  • A MariaDB replica declared with replication.source, a server Cloudfall does not manage, converges. Before anything is installed on its server, the role logs in to the source with the login credentialsRef names, over TLS verified against caCertificateFile when declared, through a client module of its own (no mariadb client is needed yet), and stops with PRECONDITION service_mariadb_source_unfit when the source is not MariaDB, writes no binary log, writes one that is not row-based, runs a release newer than version, has the replica's server_id, or its account lacks a right cloudfall_repl holds; it sends the source SELECT and SHOW only. An empty replica is seeded from one consistent dump of the source's databases without its users, follows it by GTID, read-only, and ignores mysql.% (replicate_wild_ignore_table), so a user created or granted on the source afterwards is not applied. The login is staged 0600 on the controller and reaches the replica's server for the run only; the rendered inventory names the source's host, port and whether it uses TLS, never its login. After a promotion, a declared owner database the server holds is adopted with its rows (#424)
  • A MariaDB service (S-019) is audited and scraped for metrics. cloudfall audit adds six checks per service, none renamed or removed: mariadb.version is drift when the installed mariadb-server is another release than version; mariadb.series reports the server as running an unsupported version once its series is past its end of life at the observation's time; mariadb.unit is drift when mariadb.service is not running and enabled; mariadb.configuration is drift when the settings of the managed /etc/mysql/mariadb.conf.d/90-cloudfall.cnf differ from what the role writes (a changed, added or removed setting; comments are not compared); mariadb.role is drift when a replica is writable (read_only off) or its replication threads are not both running, or when a standalone server or a primary replicates from somewhere; and, for a replica only, mariadb.lag is drift when it is more than five minutes behind its primary. inspect reads the managed file as text and, as root through the socket, read_only and SHOW SLAVE STATUS (the threads' states, the lag and the source's server_id, never a password) into a new spec.mariadb of the observation; a record an older inspect wrote has none, so those checks report unknown until the next inspect. With metrics.enabled, the role makes the alloy account on every server of the service, a replica's too, authenticated by the unix socket only and outside the binary log, with PROCESS and SLAVE MONITOR; the collector renders one Alloy prometheus.exporter.mysql for the service, connecting through the server's socket as that account with no password, labelled with the service id, and none without metrics.enabled. The bundled Services dashboard gains MariaDB panels: up, connections, queries and replication lag (#425)
  • A MariaDB service guide (docs/mariadb-service-guide.md), linked from the README: declaring a standalone server, adding a replica and what each services converge answers (seed, attach, promote and the waits), backups and the restore drill on the replica, a planned switch and a promotion after the primary is lost, bringing a former primary back, and each refusal with what to do about it; moving in from a server Cloudfall does not manage gives the steps on the source and says its converge is not built yet (#466)
  • An OperatorPolicy grant may name the operation classes of a project's server-changing MCP tools, deploy, rollback, restart, migrate, data-copy, domains-apply and converge-domains, beside converge-services and converge-baseline; the OperatorProposal schema gains, additively, trigger kind agent with its request, operation.servers and operation.environments, approval mode agent with policy, grant and client, and binding, for the agent records spec 018 writes. Over cloudfall mcp serve in a project, the plan of deploy, rollback, restart, migrate, data_copy and domains_apply called without yes gains approval.selfConfirm and, per environment its servers are in, the policy that applies or none and the reason from the operator's grant rules; nothing a confirmed call does changes yet (#474)
  • Over cloudfall mcp serve in a project, a confirmed call of deploy, rollback, restart, migrate, data_copy, domains_apply (yes: true) or a converge_* tool (confirm_destructive: true) that an OperatorPolicy grant covers in every environment it touches now writes an agent OperatorProposal record to the proposal store (record/operator/proposals/ by default): running before the first engine step, with the tool, its arguments and the agent's reason, approval.mode: agent with the policy and grant, and binding.planSha256 (the sha256 of the plan the unconfirmed call answers) with the git commit and worktree digest, then verified or failed by the tool's own check (the health gate, every migrate step, the copy's receipt, domains observe, or an audit of the changed servers). The result gains approval with mode, policy, grant, record and status; on deploy the release approval's own policy is then named release_policy. Each gated tool takes an optional reason (at most 1000 characters), also a --reason flag on the CLI commands, where it records nothing. A confirmed call no grant covers, and every CLI command, runs as before and writes no record (#475)
  • cloudfall operator approve <id> now runs an agent's proposal (one a confirmed cloudfall mcp serve call no grant covered left in the proposal store) instead of refusing it with operator_agent_proposal_unsupported. As for the operator's own proposals, a person types the id at a terminal (PERSON_REQUIRED off one, and nothing runs). The command then recomputes the plan of the stored tool call, the git commit and the digest of uncommitted changes outside decisions/, record/ and tmp/, and refuses any difference before anything runs with PRECONDITION and context.code: proposal_binding_changed, naming what changed in context.changed; the proposal stays proposed, and the agent calls the tool again for a new one. Otherwise the stored call runs through the tool's own command, S-008's release approval included, recorded with approval.mode: human and via: terminal, and settles verified or failed by the tool's own check (the health gate, every migrate step, the copy's receipt, domains observe, or an audit of the changed servers). A verified person-approved run counts toward the requiredVerifiedRuns of the policy grant for its class in its environment. cloudfall operator run still leaves agent proposals proposed, listed as withheld because only a person approves them. operator approve lists CONFLICT and PARTIAL_FAILURE among its exit codes, which a stored deploy, rollback, restart or migrate can end with (#477)
  • cloudfall why now answers the agent records and agent proposals of the proposal store beside the decisions: the run a policy grant let a confirmed cloudfall mcp serve call make, and the proposal a call no grant covered left for a person. A new --proposals flag names the store (default record/operator/proposals). --host matches any server in the record's operation.servers, --status its status and --since/--until its call, run and verify times; --operation and --stale ask about decisions only. Each such answer carries agent_record (the tool, the class, the approver: agent with its client or human with how it arrived, the policy and grant when there is one, the reason, plan_sha256 and the status), the record itself under proposal, and a story in which the agent's reason is escaped so a terminal prints it as the characters it is; --format html draws it as a card. An agent record that cannot be read is skipped with a RECORD_UNREADABLE warning, as an unreadable decision is. The operator's own alert and drift proposals are not part of the answer, and the answer for decisions alone is unchanged (#478)
  • Components: an optional service.stop.timeoutSeconds (1 to 1800, no default) renders TimeoutStopSec= in the component's unit, so a worker that finishes its running task on SIGTERM gets that long before systemd kills it, in place of systemd's 90 seconds; a component without it renders the same unit as before. The runner's limit on a deploy, restart or rollback playbook grows by that timeout for every restart it may make, so a slow stop is waited for rather than reported as a failed run, and the CI deploy guide says how to deploy a worker (#497)
  • A Service can be serviceKind: rabbitmq (S-023): spec.rabbitmq declares version (an exact X.Y.Z of a series Team RabbitMQ still supports for the open-source edition, 4.3 or 4.4, held in one table with each series' end of community support and its Erlang range, 27.0 or later), memoryHighWatermarkMb (512 or more), and optionally nodes (one server or three, containing spec.server), vhosts (each with an owner and a passwordRef, never a password) and clientAccess with its allowedSources. cloudfall config validate refuses an unsupported version (service_rabbitmq_version), nodes without the service's server, with an undeclared or repeated server, or of another size than one or three (service_rabbitmq_nodes), a server that is a node of two RabbitMQ services (service_rabbitmq_server_shared), a node of another environment (service_rabbitmq_environment), two nodes of one hostname (service_rabbitmq_hostname), remoteAccess (service_rabbitmq_remote_access), a virtual host without an owner or passwordRef, with an application, declared twice, or whose owner owns another (service_rabbitmq_vhost), the owner guest, the virtual host / and names starting cloudfall_ (service_rabbitmq_reserved), with three nodes or clientAccess a server without a firewall (service_rabbitmq_unfirewalled) or an IPv4 (service_rabbitmq_server_unaddressed), a client source of every address (the S-006 rule), and a server without one storage pool, mirrored for one node (the S-014 codes); it warns RABBITMQ_VERSION_ENDING with the date when a series is within 90 days of its end of community support, and RABBITMQ_SINGLE_NODE for one node in the production environment. The rendered inventory carries the service, with nodes resolved and its virtual hosts' password references, never a password, on each of its nodes; nothing converges it yet (#505)
  • services.yml converges a RabbitMQ service of one node with no clientAccess (S-023) through the new cloudfall_rabbitmq role: rabbitmq-server at the declared X.Y.Z-1 from Team RabbitMQ's repository and the Erlang of the Debian release, all held; the data directory on a rabbitmq volume (20 GB, mirrored for one node) that the converge decides with the other volumes; the Erlang cookie Cloudfall generates (cloudfall secrets generate) at /var/lib/rabbitmq/.erlang.cookie, 0400; managed settings in /etc/rabbitmq/conf.d/9*-cloudfall-*.conf (clients, the inter-node port, epmd and metrics on loopback only, vm_memory_high_watermark.absolute from memoryHighWatermarkMb, disk_free_limit of 2 GB, the node list, continuous membership reconciliation of quorum queues toward three members) and enabled_plugins naming the Prometheus plugin alone. The role refuses, before it changes the host, a watermark above six tenths of the server's memory, a hostname that is not the declared one, and a change of the installed version; it refuses an Erlang older than the series needs, and a cluster of three or clientAccess, which come later. On the service's server it creates each virtual host with quorum as its default queue type, gives its owner every permission there and none elsewhere, and sets the owner's password as RabbitMQ's salted SHA-256 hash, computed on the controller with the salt the user already has, so the password never reaches a node and a second converge changes nothing; guest is removed, and / while it holds no queue. A virtual host Cloudfall made that is no longer declared is kept with its queues, and its owner's password cleared; inspect reads the cluster's virtual hosts and users, and audit lists those as undeclared and anything Cloudfall did not make as unmanaged (#506)
  • services.yml converges a RabbitMQ node with clientAccess (S-023): besides plain AMQP on loopback, the node takes clients over TLS alone on its server's IPv4 (5671), with its own certificate from the project CA (usage server and client, names its hostname and addresses, at /etc/rabbitmq/cloudfall/, the key made on the node and readable by rabbitmq alone), which the services converge has the node request and the project CA sign as it does for MariaDB and Elasticsearch. The node's firewall admits 5671 from allowedSources only (ip saddr 10.0.0.0/24 tcp dport 5671 accept), no RabbitMQ port is open to every address, and plain 5672 stays on loopback, so a plain connection from outside the host is refused. audit expects the rule. A node without clientAccess renders and converges as before (#507)
  • services.yml converges a RabbitMQ service of three nodes (S-023): each node runs node discovery (4369) and the inter-node port (25672) on its server's IPv4, which its firewall admits from the other two nodes only, resolves every node's hostname to its IPv4 through a managed block of /etc/hosts, and speaks TLS between nodes, with the command-line tools, using its own certificate from the project CA (usage server and client, at /etc/rabbitmq/cloudfall/, the key made on the node and readable by rabbitmq alone) verified against that CA in both directions. The service's home converges first and the other nodes join it one at a time; once every declared node is a member, the role waits up to an hour until the service is healthy (every node a running member, no alarm, no partition, every quorum queue with a leader and a member online on every node) and otherwise stops with what is missing. Adding two nodes to a service of one grows it the same way, with no queue redeclared: continuous membership reconciliation grows each quorum queue to three members. Before it changes or restarts anything, the role reads the cluster and, when no more than half of its members run, stops the converge with PRECONDITION (service_rabbitmq_no_majority) and restarts no node. inspect reads a cluster's members on its home, and audit reports a declared node that is not a running member and a cluster without majority (#508)
  • A RabbitMQ service of three nodes may declare clientAccess: each node takes clients over TLS on its own IPv4 (5671), admitted from allowedSources only, with the same certificate from the project CA it meets the other nodes with (#508)
  • A change of a RabbitMQ service's version to a later patch of its series, or to the next accepted series, converges as a rolling change instead of being refused (S-023): from the service's home, which services.yml runs first, every other node in declared order and the home last, one at a time. Before each node stops, the cluster must be healthy (every member running, no alarm, every quorum queue with its members online), rabbitmq-upgrade await_online_quorum_plus_one must find that every quorum queue keeps its majority without it, and rabbitmq-upgrade drain moves its clients and queue leaders to the other nodes; every stable feature flag is enabled before the first node stops. Each node's rabbitmq-server is released from its hold, installed at the new version and held again, started, revived, and the cluster must be healthy again before the next one. After the home, every stable feature flag is enabled and every member must report the declared version. A service of one node changes with a short stop and no quorum wait. An older version than one a node runs, and a series before the installed one or past the next accepted one, are refused before any task changes a host; a node other than the home refuses a version change of its own (#509)
  • A RabbitMQ service is backed up and its restore drilled (S-023): on the service's server, services installs cloudfall-rabbitmq-backup and cloudfall-rabbitmq-restore-check with their systemd timers (the drill's only with restoreCheckOnCalendar), run as root, and cloudfall backup run <service> and cloudfall backup verify <service> run them and write their BackupReceipt. A backup exports the definitions of the cluster (virtual hosts, users, permissions, queues, exchanges, bindings, policies) with rabbitmqctl export_definitions into one file of backup.directory, root's and 0600 from its first byte as it holds password hashes, refuses one that does not parse or misses a declared virtual host, and then deletes the exports older than retentionDays, never the newest. A backup holds no message: its receipt says so in the new optional holdsMessages field (false), which only a RabbitMQ receipt carries. The drill starts a second node on the server with a node name, Erlang cookie, data directory, configuration, epmd and loopback ports of its own and no node list, imports the newest export into it at boot, compares the virtual hosts, users (and their password hashes, never printed), queues, exchanges, bindings and policies it then holds with the export, checks that the cluster does not list it, and stops it and removes its directory on every path; it fails, and says so, when there is no export, on any difference, when the server has less free memory than the scratch node's 256 MB watermark or its ports are taken, and when a process or listener of the scratch node is left (#511)
  • A Service can be serviceKind: clickhouse (S-022): spec.clickhouse declares version (an exact X.Y.Z.N of a long-term support series ClickHouse still maintains, 26.3 or 26.8, held in one table with each series' end-of-support date), maxServerMemoryMb (2048 or more), and optionally replicas (one server or two, containing spec.server), keeper.servers (one voter for one replica, three for two, containing every replica), databases (each with an owner and a passwordRef, never a password) and clientAccess with its allowedSources. cloudfall config validate refuses an unsupported version (service_clickhouse_version), replicas without the service's server, with an undeclared or repeated server, or of more than two (service_clickhouse_replicas), keeper.servers that misses a replica, names an undeclared or repeated server, or holds another number of voters (service_clickhouse_keeper_servers), a server that is a replica or voter of two ClickHouse services (service_clickhouse_server_shared), a server of another environment (service_clickhouse_environment), remoteAccess (service_clickhouse_remote_access), a database without an owner or passwordRef, with an application, or whose owner owns another (service_clickhouse_database), the names default, cloudfall, system, information_schema, INFORMATION_SCHEMA and names starting cloudfall_ (service_database_owner_reserved), with two replicas or clientAccess a server without a firewall (service_clickhouse_unfirewalled) or an IPv4 (service_clickhouse_server_unaddressed), a client source of every address (the S-006 rule), and a server without one storage pool, mirrored for one replica (the S-014 codes); it warns CLICKHOUSE_VERSION_ENDING with the date when a series is within 90 days of its end of support, and CLICKHOUSE_SINGLE_REPLICA for one replica in the production environment. The rendered inventory carries the service, with replicas and keeper.servers resolved and its databases' password references, never a password, on each replica and voter; nothing converges it yet (#566)
  • A Service can be serviceKind: kibana (S-024): spec.kibana declares elasticsearch (the id of the Elasticsearch service Kibana reads, of the same environment, with spec.server one of its nodes), heapMb (1024 to 8192), and optionally domain (the Domain that publishes it) and users (users of that Elasticsearch service, each with access of view or edit). The kind declares no backup, which stays required for every other kind. cloudfall config validate refuses backup, metrics or remoteAccess on the kind (service_kibana_field_refused), an elasticsearch that names no Elasticsearch service (resource_reference_missing) or one of another environment (service_kibana_environment), a server that is not one of its nodes (service_kibana_server_not_node), an Elasticsearch service that declares existingCluster (service_kibana_existing_cluster), a second Kibana for one Elasticsearch service or on one server (service_kibana_duplicate), a heapMb outside its range (service_kibana_heap), a user the Elasticsearch service does not declare or named twice (service_kibana_user_unknown), and a domain that names no Domain (resource_reference_missing) or one whose proxy.server, upstream or tls.mode does not publish this Kibana, or that declares routes (service_kibana_domain); it warns KIBANA_PUBLISHED when domain is declared. An Elasticsearch user may no longer be named cloudfall-kibana-view or cloudfall-kibana-edit (service_elasticsearch_user_reserved), the roles the Kibana role will keep. cloudfall backup run and backup verify refuse a Kibana service with PRECONDITION (lifecycle_service_no_backup) before reaching any server, naming the Elasticsearch service whose snapshots hold its saved objects. The rendered inventory carries the service on its server, beside the Elasticsearch service it reads; services.yml still refuses the kind until its role lands (#567)
  • A CI deploy guide (docs/ci-deploy-guide.md): one GitHub Actions workflow that renders secrets, builds the artifact with cloudfall-engine artifact build, deploys with cloudfall deploy --yes and uploads the receipts, with one SSH key and one age key per environment and required reviewers standing in for approval: manual, which deploy does not enforce; not yet run against a real server (#89)
  • An open proposal whose basis hosts have been observed again since is now reported as stale: decisions list, decisions show and cloudfall why add stale and staleHosts to each proposed decision, --stale on decisions list and why keeps only those, and decisions approve names each stale host and its newer observation time before asking for the id. Worked out on read from the observation files; the record and its schema do not change (#26)
  • audit checks a logging backend's dashboards (logging.dashboards): inspect records the name and sha256 of each file in Grafana's Cloudfall dashboards directory (new loggingDashboards evidence, never anything else of Grafana's), and the check is compliant when they are exactly the dashboards the installed release bundles; drift names each missing, extra or changed file (S-016)
  • logging deploy provisions four dashboards into a read-only Cloudfall folder in the stack's Grafana: Fleet (every server's freshness, CPU, memory, fullest filesystem, load and the firing alerts), Server (one server's host metrics and log volume by unit), Logs (log volume, error lines and the lines themselves by server and unit, with a filter) and Services (PostgreSQL and Redis exporters by service). They read servers from the data and name the provisioned data sources by uid, so they work on any project; a dashboard dropped by a later release is removed on the next deploy (S-016)
  • cloudfall logging open opens the LoggingStack's Grafana: it forwards a local port over ssh to Grafana on the backend's loopback, built from the backend server's declared address, SSH user and port, answers with the URL of the Cloudfall dashboards once the forward is up, and holds it until stopped. --port picks the local port (one in use exits 4 before ssh starts), --stack picks the stack, and --print-command answers with the ssh command and runs nothing. The answer names where the admin password is kept, never the password (S-016)

Changed

  • Breaking: cloudfall config validate now refuses a Component with runtime.type node or pm2, or runtime.packageManager npm, pnpm or yarn, with CONFIG_INVALID and code component_runtime_unsupported naming the field: these configs validated before but could never deploy, since the deploy role runs only Python with uv. The schema keeps those values as reserved. The Render importer no longer writes a Component for a Node service; it lists the service as an unsupported gap instead (#114)
  • migrate --yes checks a deploy:<component> step's approval as deploy --yes does: for an application that declares approval: manual and no approval that counts, it pauses at that step with PARTIAL_FAILURE, context.code: deployment_approval_missing, the built release and the cloudfall release approve command, and once a person approved it, migrate --yes resumes there with the release the completed build:<component> step recorded, without building again and without needing --build again. The deploy step's detail gains approval (the record it went on), migrate gains --approvals (default decisions/releases, bound by cloudfall mcp serve) and the RECORD_INVALID exit for an approvals file that isn't a valid ReleaseApproval. rollback needs no approval: its plan gains approval, {"policy": "manual", "required": false, "reason": ...} for a manual application and {"policy": "automatic"} otherwise (#165)
  • The resource every server runs is now kind: Baseline, stored under baselines/, referenced as Server.spec.baseline and checked by baseline.schema.json (was ServerType, server-types/, spec.serverType, server-type.schema.json); cloudfall add baseline declares one and cloudfall add server --baseline names it (#198)
  • Outputs carry the new names: inventory show writes baselines and each server's baseline, config validate counts Baseline in by_kind, audit writes each server's baseline and checks baseline.id (was server_type.id; an operator proposal recorded with server_type.id verifies once baseline.id is compliant, and blocks a new proposal for the same drift, with no removal date, since a record's fields are never deprecated within cloudfall/v1), the inspect role records baseline in observed server snapshots (a snapshot recorded before with spec.serverType is still read as spec.baseline, also with no removal date; one with both fails with observation_baseline_conflict), the rendered Ansible inventory has the host variable cloudfall_baseline, the brownfield catalog variable is cloudfall_baselines, and the error code server_type_entry_duplicate is baseline_entry_duplicate (#198)
  • Site: cloudfall.tech is the primary domain; cloudfall.dev and both www hosts redirect to it with a 301, and the PyPI homepage and the README cloudfall init writes point there (#199)
  • Receipts and operator proposals now land in the project's committed record/ directory instead of the git-ignored tmp/, so the audit trail survives losing the management host (D-16): the defaults of --deployments, --releases, --backups, --data-migrations, --env-receipts and --proposals (on every command and on cloudfall mcp serve) are record/deployments, record/releases, record/backups, record/data-migrations, record/env-receipts and record/operator/proposals. tmp/ keeps only what can be rebuilt: observations, rendered inventory, built artifacts, rendered environment files and the migration plan. A path you pass explicitly keeps working as before. cloudfall config upgrade now also moves an existing project's receipts and proposals from tmp/ into record/, listing each move in upgraded with its moved_from; it drops a file whose identical copy is already in record/, refuses with CONFLICT and code upgrade_target_exists, naming both paths and moving nothing, when a different file is, and a second run changes nothing. Until it runs, operator list, audit, services status and the operator's autonomy pass don't see the receipts left under tmp/, so run cloudfall config upgrade once after upgrading, then commit record/. cloudfall init writes the new layout into the project's README, AGENTS.md and .gitignore, and the operator guide says how to copy record/ off a management host with no git remote (#219)
  • A release receipt no longer holds its release command's output: the command is the application's and may print a secret, and the receipt is now committed. The last 2000 characters stay in tmp/output/releases/<component>.<release>.log (0600), and spec.releaseCommand names that file and its SHA-256 under output (path, sha256) instead of outputTail. The ReleaseReceipt schema stays cloudfall/v1: outputTail becomes optional, so receipts written before still validate, and a receipt may not hold both. cloudfall config upgrade moves the tail of an old receipt, under tmp/ or already in record/, out to that file the same way, refusing to replace a different file there. A reader of releaseCommand.outputTail reads the named file instead, which is gone on another host or once tmp/ is cleared (#219)
  • Contract: cloudfall lock clear <component> now also reads the component's lock on each of its servers through the new lock.yml, shows them with their ages in the same prompt, and once the component id is typed removes exactly the locks it showed; it takes --engine and --inventory-file, answers serverLocks and unreadServers (a server it could not reach), may exit ENGINE_STEP_FAILED, and is listed among the commands that change servers (#221)
  • The release receipt holds one entry per server under spec.servers (the release the server ended on, previousRelease, deployedAt, healthCheckPassed, rolledBack and the command healthCheck), with the deployed release, artifactSha256 and releaseCommand once per receipt; release-receipt.schema.json accepts it and still accepts a receipt written before, with one server, which reads as a one-entry list (#222)
  • Site: contact addresses moved to cloudfall.tech (pilot@, conduct@); mail to cloudfall.dev still arrives (#260)
  • A mirrored volume now meets an expendable requirement (D-18): when a one-node Elasticsearch service grows, its first node keeps its raid1 data volume, and the volume provider keeps and grows it, and audit checks it, as the mirror it is; a linear volume still never meets a mirrored requirement (#312)
  • When installing a new Elasticsearch version fails after the node was stopped for it, the role starts the node again on the package it had and lets its cluster move replicas again before the converge stops (#312)
  • Open proposals made before the upgrade carry no binding, so decisions approve refuses them with decision_binding_changed: propose them again (#352)
  • decisions approve gives the approved run and its verify step their own time limit of 6 hours (21600 seconds) instead of the check run's 1800 seconds, which check mode keeps. A run that reaches it is stopped with its whole process group and the record settles failed with the verdict the run timed out after 21600s and was stopped; the run's output now goes to its log as it comes, so a stopped run keeps what it printed. Before, a run past 1800 seconds was killed, its record stayed running with no outcome, and the CLI's log of it was lost (#359)
  • A release candidate or release is cut and published only from a commit whose converge run passed: release.yml and publish.yml now run the converge job (the roles on a real Debian 13 with systemd) on the commit they release, beside CI, and land or publish nothing unless both passed (D-23). The converge job also restarts the component with cloudfall restart, asserting a new process of the same release that passes its health check, and applies an nginx Domain with a self-signed certificate through cloudfall domains apply, asserting HTTPS serves the probe with that certificate and HTTP redirects to it (#361)
  • Every command that reads a project now checks its [tool.cloudfall] format first, as cloudfall config validate already did: cloudfall audit, deploy, add, health and the rest, cloudfall mcp serve once at start, and cloudfall-engine inventory render and artifact build. A project written by a newer Cloudfall, one below the formats this release reads, or one with a malformed format stops the command with exit PROJECT_FORMAT_UNSUPPORTED (95, now on cloudfall-engine too) before the project is validated, before anything is written under its tmp/ and before any Ansible run, and cloudfall mcp serve refuses to serve it. A format 0 project (no [tool.cloudfall] table, or no pyproject.toml) still runs every command, each now answering with one PROJECT_FORMAT_OLD warning naming cloudfall config upgrade. cloudfall init, changelog, --schema and cloudfall-engine manifest read no project and run whatever its format (#379)
  • cloudfall config upgrade brings a project to the current format: it runs each rewrite from the project's [tool.cloudfall] format to the current one (the step from format 0 is the rename of ServerType, server-types/ and serverType and the move of receipts and proposals into record/), then writes format = 1 into the project's pyproject.toml, creating the table, or the file, and keeping every other byte of it. A format 0 project with nothing to rename now answers effect: updated, since its format is written; a project already at the current format is a noop. The answer gains from_format and format. It runs on a project too old for every other command, refuses one written by a newer Cloudfall (project_format_too_new) or with an unreadable format (project_format_invalid) with exit PROJECT_FORMAT_UNSUPPORTED (95) and nothing changed, and refuses with CONFLICT and code upgrade_format_unwritable a pyproject.toml whose tool.cloudfall is an inline table or dotted keys (#380)
  • The managed vhost's HTTPS server no longer serves HTTP/2 by default: the cloudfall_nginx_site role takes cloudfall_nginx_site_http2 (bool, default false), and the next domains run renders listen 443 ssl; and listen [::]:443 ssl; with no http2 on any Nginx version. Once the installed Nginx carries the HTTP/2 fix, set it to true in the inventory or with --extra-vars cloudfall_nginx_site_http2=true to render HTTP/2 as before (#551)
  • The managed vhost's HTTPS server serves HTTP/2 by default again: cloudfall_nginx_site_http2 defaults to true, so the next domains run renders HTTP/2 in the form the installed Nginx parses. It was off while Nginx's HTTP/2 flow control was exploitable (CVE-2026-123007); Debian has shipped the fix for both targets, bookworm 1.22.1-9+deb12u11 (DLA-4918-1) and trixie 1.26.3-3+deb13u11 (DSA-6642-1), from the security suite only. Install that Nginx on a host before its next domains run, or keep HTTP/1.1 only with cloudfall_nginx_site_http2: false in the inventory or --extra-vars cloudfall_nginx_site_http2=false (#555)
  • The logging backend and collector roles share one copy of the tasks they both ran: the delivery of provided files, the reach preflight, the package pin check and the package holds, with the reach filter, now live in the cloudfall_logging_common role, which both include by task file. What a logging deploy does on a host is unchanged
  • The logging backend and collector roles declare Debian 13 (trixie) next to Debian 12, like every other bundled role, so a trixie host built from the bundled Debian 13 baseline can carry the logging stack. Loki, Grafana and Alloy come from Grafana's distribution-independent APT repository; Prometheus comes from the host's Debian release, so a software.prometheusPackageVersion pin must name a version that release ships

Security

  • An owner role's password no longer reaches the PostgreSQL host or server: services.yml and every path that runs it compute the password's SCRAM-SHA-256 verifier on the controller (a fresh 16-byte salt, 4096 iterations, the password prepared with SASLprep as PostgreSQL does) and stage that verifier in the 0600 file instead of the password, so ALTER ROLE ... PASSWORD sends a verifier PostgreSQL stores as it is and the plaintext can't show in the server log under log_statement = 'ddl' or 'all', in a failing statement, or in pg_stat_activity; answers and receipts still name the reference, the key, and the password's SHA-256 (#140)
  • A declared config value is never run as an Ansible template: cloudfall-engine inventory render and the overlay cloudfall observe writes mark every string unsafe ({"__ansible_unsafe": ...}), so a value holding {{ ... }} reaches the unit file, the release command and the health check as written. Before, a service.command, healthCheck.command or release.command argument such as {{ lookup('pipe', '...') }} ran on the management host during cloudfall health or cloudfall deploy, and an honest one such as docker inspect --format {{.State.Health.Status}} app failed with a template syntax error. A declared value that relied on Ansible evaluating {{ ... }} is now literal, and the rendered inventory's strings read as {"__ansible_unsafe": "..."} in its JSON and in ansible-inventory --list (#174)
  • A catalog operation's inputs are never run as an Ansible template: cloudfall operations propose, cloudfall decisions approve and cloudfall agent investigate hand them to the team's playbook in --extra-vars with every string marked unsafe ({"__ansible_unsafe": ...}), so an input holding {{ ... }} reaches the playbook as written, and integers and booleans keep their types. Before, an input such as {{ lookup('pipe', '...') }} ran on the management host with the operator's rights when the playbook started, and an honest input holding {{ failed with a template syntax error. A team playbook that relied on an input being evaluated as Jinja now gets the literal text. The decision record's inputs are unchanged, and recordings made before this change replay as before (#181)
  • cloudfall data copy and cloudfall migrate --data no longer put the source database URL, password included, on the command line of psql and pg_dump, where every local user could read it in /proc/<pid>/cmdline while the copy ran: the URL becomes a 0600 libpq service file on the target host, named to the clients through PGSERVICEFILE and PGSERVICE and removed when the copy ends, on failure too. A URL libpq can't read (another scheme, a bad percent-encoding, a parameter set twice) is refused before anything is staged, without quoting it (#355)
  • A declared operation's check diff and the logs of its approved run and verify step are no longer written to the committed decisions/ directory: they go to the project's ignored tmp/output/decisions/ (0600), and the decision record keeps each file's path, sha256 and size as before, so a secret a team's playbook prints with --diff (a templated password without no_log or diff: false) never reaches git. A --decisions folder elsewhere in the project keeps its output under the same path below tmp/output/; one already under tmp/ keeps it beside its records. decisions approve refuses with PRECONDITION and decision_diff_missing, naming the file and its sha256, when the raw diff is not on this machine, such as on another clone. Records written before keep citing their files in decisions/ and are read and approved in place; a file of theirs already committed stays in git's history, so one that may hold a secret is for you to purge (#357)

Fixed

  • cloudfall audit no longer reports drift on a server whose PostgreSQL service declares spec.postgresql.remoteAccess: firewall.allowedInbound expects one tcp/<port> from <cidr> entry per declared source and reads the ip saddr/ip6 saddr match of each accepted nftables rule (a single address printed bare by nft compares equal to its /32 or /128), and services.bind[<id>] requires a listener on the declared listenAddress as well as on the bind address, reporting loopbackOnly: false and a remoteAccess.listenAddress key for such a service; a missing listenAddress listener, any other non-loopback listener, an unrestricted rule on the port and an undeclared source are drift. Servers without remoteAccess audit exactly as before. The engine and the audit now derive the source-restricted rules from one SDK function, remote_access_rules_for_server (#143)
  • A pg_hba.conf reload that failed is no longer forgotten: services.yml leaves a marker file at /var/lib/cloudfall/postgresql/<service>.pg_hba-reload-pending when it closes remote access after a failed check or reload, and the next converge checks and reloads pg_hba.conf even though the file is unchanged, then removes the marker, so the old hostssl lines no longer stay live until a restart (#144)
  • cloudfall config validate refuses an owned database or owner named all, sameuser, samerole, samegroup or replication on a PostgreSQL service with remoteAccess, as CONFIG_INVALID with code service_remote_access_name_reserved, instead of leaving the refusal to the converge; without remoteAccess those names still validate (#144)
  • A first deploy over a hand-made systemd unit of the same name no longer loses it when the release fails its health gate: the deploy role keeps a unit without the # Managed by Cloudfall header at /etc/systemd/system/<name>.service.pre-cloudfall before installing the managed unit, and on a failed first release copies it back, restores its enabled state and starts it again when it was active; check mode reports the unit it would keep, and a hand-made unit that differs from an earlier kept copy refuses the deploy (#151)
  • The data copy preflight now names an extension that a selected table needs through an array or a domain of the extension's type, such as a citext[] column or a column of a domain over citext (chained domains included), under extensions and passes it to pg_dump --extension; before, only a direct use of the extension's type was seen, so the copy skipped the extension and the restore failed (#173)
  • cloudfall migrate --yes no longer crashes with HANDLER_CRASHED when it reaches its audit or verify-routes step on a project with no server observations: the step fails with migrate_observations_missing, naming the missing directory and cloudfall observe, and a rerun after observe resumes at that step (#183)
  • Decision records and operator proposal receipts are written through a temporary file and renamed into place, so a crash or a full disk mid-write no longer leaves a truncated record; a new record is created exclusively, so two writers of one id can no longer both succeed, and each read-modify-write of a record holds its directory's lock (#217)
  • An approval is recorded, with the new status running, before the first playbook or the operator's executor starts. A run killed midway stays running and is refused by decisions approve and operator approve with PRECONDITION (decision_run_interrupted, operator_proposal_run_interrupted) instead of being approvable and run a second time; running is added to both record schemas (#217)
  • One decision record or proposal receipt that cannot be read no longer stops every reader of the store: operations propose, decisions list, why and operator list go on with the records they can read and add a RECORD_UNREADABLE warning naming each skipped file and why; an operator pass reports the receipts it skipped under unreadable, and its autonomy pass grants nothing while a receipt cannot be read. decisions show and decisions approve still answer RECORD_INVALID for the one record they read (#218)
  • Each record is read with the schema its apiVersion names, so a later schema version can sit beside cloudfall/v1; a record of a version this Cloudfall does not read is reported, not fatal. A golden set of records in the shapes released versions wrote is loaded by the tests (#218)
  • deploy, rollback and restart contact only the component's own servers (its component_<id> inventory group, still in inventory order), and a backup contacts only the service's server, so an unreachable server the component or service is not on no longer fails them; a component with no servers in the inventory, or a service on no inventory server, now fails before any host is contacted (#220)
  • A component on several servers no longer ends a failed deploy on two releases: when a later server fails its health gate, the servers that switched to the release earlier in the run go back to the release and service unit they had (a first release is stopped), last one first, each gated again, and the failure names them (#222)
  • The release receipt <receipts>/<component>.json names every server of the deploy instead of only the last one, which overwrote the others (#222)
  • A failed deploy whose component has a release command says the command already ran and is not reverted; the engine README says a release command must work with the previous release too (#222)
  • Deploying the release that is already current unpacks into a fresh releases/<release>.redeploy-<UTC time> directory instead of writing into the live one, so a failed health gate rolls back to the untouched copy; rollback to that release id takes its newest copy, and receipts still name the release id (#223)
  • A failed health gate restores the service unit the previous release ran under, not only the current symlink, then gates the restored release on its health check; a restored release that fails it fails the deploy with its own message (#223)
  • deploy.yml --check runs to its end: the dry run reports the uv release it would install and skips the digest check, the unpack and uv sync of an archive it never transferred (#223)
  • Old releases are removed: once a deploy passes its health gate on a server, it keeps deployment.retainReleases releases under releases/ and removes the older ones, a release's redeploy copies with it, always keeping the deployed release, current and the release before it whatever their order; a failed gate removes nothing, check mode only reports what would go, and a directory that is not a release id is left alone (#224)
  • A rollback to a release that is no longer on the server says it may have been removed by a deploy's cleanup and names the releases still kept there (#224)
  • The PostgreSQL restore drill compares the row count of every table the dump holds with the restored table and fails on a difference or on a dump without a table; it counted pg_class rows before, which an empty database has too, so a restore that brought back nothing passed (#225)
  • The restore drill's scratch database keeps to PostgreSQL's 63-byte names (<database>_restore_check, the name cut to 49 characters), and cloudfall config validate and the converge refuse a declared database named as another's scratch database, as CONFIG_INVALID with code service_database_restore_check_name (#225)
  • The test suite passes on a checkout whose config/examples/tmp/ holds rendered inventory or observations (as task inventory and CI leave it): every test that copies the examples now goes through one helper, sdk/tests/examples_project.py, that leaves that folder behind, and a test refuses a direct copy (#226)
  • Docs that no longer matched the code: ARCHITECTURE.md now says a project pins cloudfall==<version>, that decision records are committed while runtime state stays under tmp/, and which Taskfile.yml tasks call Ansible directly; the AGENTS.md that cloudfall init writes names the cloudfall==<version> pin instead of a rev pin; CONTRIBUTING.md names config/ instead of a state/ directory; the CLI evaluation under evaluations/cloudfall/ uses repository-relative paths (#229)
  • An operator proposal left running no longer blocks the operator for good: once closed as interrupted it stops blocking new proposals for the same alert or drift, and the refusals of decisions approve and operator approve name the command that closes it (#244)
  • A decision record or proposal receipt write now flushes its directory after the rename, so a power loss cannot bring the old record back (#244)
  • A redeploy copy (releases/<release>.redeploy-<UTC time>) that fails its health gate is removed by the deploy's rollback once the previous copy runs again, so rollback <release> takes the newest copy that passed instead of switching to the failed one; the copy's time now has microseconds, so two redeploys in the same second get two directories (#245)
  • health contacts only the component's own servers and fails before any host when the component has no servers in the inventory, instead of reporting success with nothing probed; and deploy, rollback and restart stop at the first of the component's servers they could not reach, where before Ansible skipped the unreachable server and went on to the next one (#246)
  • The PostgreSQL restore drill counts each restored table without its inheritance children (count(*) FROM ONLY), as the dump's COPY data holds them, so a database with classic table inheritance no longer fails the drill every night after a correct restore (#247)
  • config validate refuses a storage pool device path with .., a volumeGroup of . or .., and a pool naming one device as both primary and secondary (storage_pool_same_physical_volume); before, the first two passed validation and crashed requirements show (#248)
  • baseline, bootstrap, services, data, domains, logging, time and backup stop at the first server they could not reach, as deploy, rollback and restart do since #246, where before Ansible skipped the unreachable server and changed the next one; the check lives in one shared task file, playbooks/tasks/rollout-start.yml, and inspect still goes on past a server it cannot reach, reporting it as missing (#254)
  • An operator proposal closed as interrupted now suspends autonomy for its operation kind the way a failed one does, until a human-approved run verifies, and an autonomous run closed as interrupted counts toward maxAutonomousPerHour from its interruption.confirmedAt; before, a killed unattended run left the next proposal for the same alert free to run unattended while what the first run changed was unknown (#258)
  • cloudfall lock clear <component> no longer deletes a server's lock that a run took again after it was shown: lock.yml compares the shown token and removes the lock in one command on the server, instead of reading it and deleting it in two tasks, so a lock taken since is kept and the command exits CONFLICT; and with no lock found while a server could not be reached it exits CONFLICT with component_lock_not_cleared and unreadServers, instead of answering noop as if no run held the lock (#271)
  • cloudfall baseline converge keeps a host's own smartd DEVICESCAN options (-s self-test schedules, -W temperature limits) and replaces only its -M exec, adding -m root when the line has no -m; a failed restart of mdadm's monitor or smartd now fails the run instead of reporting success, and only a unit the host does not have is skipped (#281)
  • logging deploy's plan shows each receiver channel's bot token: a new channels field lists {channel, tokenStored}, and a token not in the store is in missing as telegram-bot-token of <channel>, the name --yes refuses with, where the plan used to say nothing was missing (#282)
  • A command that runs a playbook on one server refuses a Server whose id is one of Ansible's implicit group names (all, ungrouped) with lifecycle_limit_reserved_group before anything runs: --limit all reached every host and --limit ungrouped none (#283)
  • A storage pool's primary and secondary may be stable paths such as /dev/disk/by-id/nvme-<model>_<serial>-part5: logging deploy and inspect resolve each one on the host with realpath, the volume provider and audit's volume.<name> compare the device it names with LVM's physical volumes, lvcreate and lvextend name the physical volume as LVM reports it, and audit shows each declared path next to its device (resolvedPhysicalVolumes); before, any path but the kernel name failed with "is not in vg_data", and NVMe kernel names can swap between boots. A pool path that does not exist on the host, and a pool whose primary and secondary resolve to one device, are refused naming the server, pool and path (#293)
  • logging deploy holds Loki, Grafana, Alloy, and Prometheus and Alertmanager when they have a declared version, at the LoggingStack's declared versions: each role writes /etc/apt/preferences.d/cloudfall-logging-<package> (Pin: version <declared>, Pin-Priority: 999) before its pinned install, so a later apt-get dist-upgrade no longer moves them, and removes the file of a package now installed without a declared version; cloudfall audit adds a logging.packages[<package>] check per pinned package on each of the stack's hosts, reporting drift when the installed version is not the declared one (#294)
  • logging deploy checks, before anything is installed, that each service user can reach the files it reads (Grafana's admin password, Alertmanager's SMTP password and Telegram token files, Alloy's client TLS files) through every existing parent directory, and stops naming the directory that blocks it, as in /etc/cloudfall is 0700 root:root; grafana can't reach /etc/cloudfall/logging/grafana-admin-password, where Grafana used to fail only when it started; the host's directory modes are left as they are (#295)
  • cloudfall-engine playbook run <playbook> --engine <dir> runs a playbook of an engine directory that has no services.yml, where it used to crash with HANDLER_CRASHED and playbook does not exist: .../services.yml whatever playbook was asked for; a services.yml run still decrypts and stages the owner passwords as before (#296)
  • logging deploy no longer leaves a host whose pinned logging package is already newer than declared (Grafana 13.3.1 installed, 13.2.6 declared) with a hold that makes every later apt-get dist-upgrade and unattended-upgrades abort on a downgrade: the holds are written at Pin-Priority: 999, which keeps a held package from moving up without ever forcing a downgrade, and before writing them each role compares the installed version with the declared one (dpkg --compare-versions, a dry run too) and stops on a newer one, naming the package and both versions and saying to declare the installed version or downgrade the package by hand, with no hold written on that host (#302)
  • A missing Elasticsearch exporter credential no longer stops Alloy on the whole host: the collector's systemd drop-in gives each LoadCredential= a SetCredential= default, so when the node's cloudfall password or project CA is gone (the service taken out of the project, its files cleaned up) Alloy still starts and only that exporter's scrapes fail; the next logging deploy drops the exporter and its credential lines (#326)
  • Adding or removing remoteAccess on an Elasticsearch service with metrics.enabled no longer leaves the collector scraping the node the old way until the next logging deploy: the services converge re-renders the Alloy configuration and its systemd drop-in on each logging collector where logging deploy already set Alloy up, validates it, and restarts Alloy only when a file changed, so the exporter switches between plain HTTP and HTTPS with the project CA in the same run; a host that is not a collector, or whose Alloy is not set up yet, is left untouched (#329)
  • The PostgreSQL restore drill (cloudfall backup verify and the nightly restore-check timer) passes --file=- to pg_restore when it reads the dump's row counts; without it every PostgreSQL since 12 refuses with one of -d/--dbname and -f/--file must be specified, so the drill failed on every run (#340)
  • A bootstrap or baseline after a deploy no longer reports changes on each component's releases/ and shared/: the deploy now creates them 0750, as bootstrap does, instead of flipping them to 0755, so a converge after a deploy is idempotent; the Nginx worker still reaches a static route's release through its own ACL entry (#341)
  • On a host running more than one service, such as h1 in config/examples with PostgreSQL and Redis, the Redis backup no longer fails with Permission denied: the PostgreSQL, Redis and Elasticsearch roles now create the shared backup root /var/backups/cloudfall root-owned 0755 before their own backup directory, instead of letting the first service's directory task create it with its own owner and 0700, and the next services converge repairs a root already owned by postgres; each service's own directory keeps its owner and mode (#342)
  • An approved operator proposal now runs on the one server it names: every playbook of converge-services and converge-baseline, the Elasticsearch node preparation included, runs with --limit <server>, a drift proposal's verify step inspects that server alone, and the receipt names only the owner passwords set there; before, a staging alert approved by the staging policy converged every declared server, production included (#350)
  • Operator autonomy is now earned per environment: the history an OperatorPolicy grant is judged on, and the rule that suspends it after a failed or interrupted receipt, count only the receipts of servers in the policy's environment, and a receipt of a server no longer declared counts for none; before, verified runs on staging earned autonomy in production, and a failed staging run suspended production (#351)
  • decisions approve runs only what was checked: a proposal's record now binds the sha256 of its playbook and verify playbook and, when the project is in a git work tree, its HEAD commit and a digest of its uncommitted changes (leaving out the decisions directory, record/ and tmp/), and approve recomputes them and the diff file's sha256 before anything runs; any difference is refused with PRECONDITION and decision_binding_changed, naming what changed and asking for a new proposal, and the record stays proposed. Before, an edited playbook, role or diff file ran unseen under the old diff's approval. Outside git only the two playbooks and the diff are bound (#352)
  • A failed deploy or a rollback brings the previous environment back with the previous release: each release now reads its own environment file, shared/env/<component>.<release>.env (a redeploy copy gets one of its own), named by its unit and read by its release command and command health check, so a deploy that fails only because of a bad environment value leaves the previous release running, and restarting, on the environment it had; rollback points the unit at the target release's file. A deploy without --env-file copies in the file the previous release reads, a component with no environment still names shared/env/<component>.env, which no deploy writes or removes any more, and retention removes a removed release's environment files (#353)
  • A baseline run now removes an SSH key declared lifecycle: revoked from root's authorized_keys, after the active keys are in, matching the key's type and body whatever its comment or options; keys the project doesn't declare stay (D-22). The rendered Ansible inventory's cloudfall_ssh_public_keys now carries a server's revoked keys too, each with its lifecycle, so the role can remove them (#354)
  • Two proposals of one operation in the same second no longer share a decision id: the id is reserved before the check run writes anything, and a second proposal in that second takes <operation>-<stamp>-2, then -3, so each record keeps its own diff and host stats, instead of the second overwriting the first one's diff and then failing with decision_exists; a proposal whose check run fails frees its id and leaves no record (#356)
  • An operator verify step that errors (the alert feed unreachable, the audit failing) now settles the receipt failed with the error as its detail, so autonomy for that operation kind is suspended, and the command still fails with the error; before, the receipt stayed executed for good and blocked its trigger (#358)
  • decisions approve no longer settles a decision verified when its verify step observed no host: every host the check run or the approved run left recap counts for must have counts from the run and a task the verify step ran on it, and a verify play that matched no host, skipped a targeted one or skipped every task on it settles the decision failed, with a verdict naming the hosts that were not observed. Before, a verify step that exited 0 and listed no host as changed proved the run, even when it looked at nothing. A verify playbook aimed at other hosts than the run's (such as localhost checking a URL) now settles failed too; aim it at the hosts the operation targets (#360)
  • Docs: D-10 in docs/decisions.md read as if every component already ran in the full systemd sandbox; a status line now says the unit sets only User, Group, NoNewPrivileges=true and PrivateTmp=true, that secrets reach it through the per-release EnvironmentFile (D-21) and deploy tasks run as root, and that the rest of the sandbox is not built yet (#362)
  • Docs gave the deploy order as health-check gate before symlink switch: ARCHITECTURE.md now says the switch and the restart come first, so the gate runs on the live release, and ARCHITECTURE.md, docs/ci-deploy-guide.md and the Render and Neon migration guides say a rollback, automatic or cloudfall rollback, does not undo the release command, so a migration must stay compatible with the release before it (#363)
  • A deploy no longer fails on one slow download of the pinned uv release: the deploy role retries the download five times, ten seconds apart, still checking its SHA-256, before failing, and a test over every role now requires each get_url download to retry a bounded number of times (#372)
  • A redeploy copy (releases/<release>.redeploy-<UTC time>) whose release command fails is removed with its environment file, as one that fails its gate is, so rollback <release> takes the newest copy that passed instead of starting one that never passed its release command or gate; the previous copy and its own environment file stay (#373)
  • A deploy run with Ansible's --diff no longer prints the component's environment file, secrets included: the tasks that install a release's environment file or carry the previous one over set diff: false (#374)
  • An observed server snapshot written before 0.8.0, which names its baseline spec.serverType, validates against observed-server.schema.json again: the schema takes spec.serverType or spec.baseline, exactly one of the two, and the reader keeps reading serverType as the baseline with no removal date, since a record's fields are never deprecated within cloudfall/v1 (#376)
  • An operator proposal recorded with the audit check server_type.id keeps verifying once baseline.id is compliant, and keeps blocking a new proposal for the same drift, with no removal date instead of until 0.9.0, since a record's fields are never deprecated within cloudfall/v1 (#387)
  • A re-converge of a PostgreSQL, Elasticsearch or logging service no longer needs the network for its package repository's signing key: the PostgreSQL, Elastic and Grafana roles download the key only when it is missing (a get_url without force still asked the server whether the key had changed, so a slow www.postgresql.org failed a converge that should change nothing), keep the installed key's owner and mode, and retry a failed download five times, ten seconds apart, before failing (#388)
  • Docs promised a gate the agent does not control for every server-changing action: ARCHITECTURE.md, docs/brownfield-design.md and MANIFESTO.md now say it holds for declared operations (a person approves at a terminal) and the operator (a person at a terminal or an OperatorPolicy grant), while in project mode the cloudfall mcp serve confirmation (yes: true, confirm_destructive: true) is the agent's own, except for a release whose application declares approval: manual; policy-gated self-confirmation is planned (#391)
  • SECURITY.md says who the private-reporting rule is for: outside reporters use private vulnerability reporting; the maintainers track findings that need existing access as public issues, and keep findings a stranger could exploit remotely private until a fix is released (#392)
  • A PostgreSQL owner role whose database names a passwordRef no longer reports a change on every services converge: the role reads the owner's stored verifier, computes the password's SCRAM-SHA-256 verifier on the controller with that verifier's salt and iteration count (a fresh salt when the role has no SCRAM verifier), and runs ALTER ROLE only when the two differ or the role can't log in, so a converge with the same password changes nothing (D-25). The controller now stages the password itself in its private 0600 file, which never leaves the controller; only the verifier reaches the host, as before (D-9). A stored password that is neither a SCRAM-SHA-256 nor an md5 verifier stops the run, naming the role, never the value (#398)
  • config validate refuses an IPv6 network in a MariaDB service's remoteAccess.allowedSources with service_mariadb_remote_access_ipv6, naming the source and saying a single IPv6 address (/128) or an IPv4 network is accepted. Before, such a config validated, the firewall rendered a rule for it, and the converge refused it only after the host had made its certificate key. MariaDB's account host has no netmask form for IPv6, so an IPv6 network never converged; PostgreSQL keeps accepting IPv6 networks (#434)
  • audit now sees REQUIRE SSL removed by hand from a MariaDB owner account at an allowed source (ALTER USER ... REQUIRE NONE): inspect reads each remote account's ssl_type into a new optional sslTypes list of mariadbAccounts, and the mariadb.accounts check reports drift for a remote owner account whose ssl_type isn't ANY (what converge's REQUIRE SSL sets), naming the account and its type. A record an older inspect wrote, which has no sslTypes, now reports that check as unknown instead of compliant when the service has remote owner accounts, until the next inspect; local accounts are checked as before (#435)
  • A MariaDB replica declared primary whose link has not connected (its IO thread still Connecting, so it reports Master_Server_Id: 0) is no longer promoted over a running, writable primary: the role finds its source by the link's Master_Host and Master_Port too, matched against the hostname, addresses and port the inventory now renders for each server of the pair, reads it, and stops with PRECONDITION while it is writable. A link to a source that is no declared service and is not running is promoted only when the controller can't reach the source's port either (#440)
  • cloudfall audit no longer reports services.bind drift on a compliant MariaDB primary that listens on its replication.listenAddress as well as loopback: a MariaDB listener on the service's declared replication.listenAddress or remoteAccess.listenAddress at its port is accepted, compared as a parsed IP, and the check's desired state names the replication address. A primary not listening on its replication address is drift, as a remoteAccess service not listening on its listenAddress already was. Any other non-loopback listener, a wildcard (0.0.0.0, ::) or an undeclared IPv6 address included, is still drift, and the check is unchanged for other service kinds (#445)
  • A MariaDB replica declared primary whose link's source is no declared service is no longer promoted over a source the controller merely can't see: the role reads the replica's own Last_IO_Errno first and refuses with PRECONDITION when its last error came from a running source (access denied, a TLS failure), and the controller's probe of the source's port now tells a refused connection from a timeout, no route or a name it can't resolve. Only a refused connection promotes; the others leave the source unknown and refuse with PRECONDITION, naming what the controller couldn't read. A link whose server_id matches no declared service is matched by its host and port whatever server_id it reports, as the code already did, and the comments now say so (#446)
  • A new MariaDB replica on an ext4 data volume is seeded on its first converge instead of refused with service_mariadb_replica_unattachable ... holds 1 database(s): the role no longer counts or lists a #mysql50# name, which is how MariaDB shows a directory that is no database (the volume's lost+found) until the managed settings ignore it. A replica's backup and restore drill leave it out the same way, so they never try to dump or check it (#449)
  • On MariaDB 11.8, where read_only reads ON rather than 1 in the role's replication state query, the role now parses both servers' state through one filter that takes ON, NO_LOCK and NO_LOCK_NO_ADMIN (or 1 to 3) as read-only and OFF (or 0) as writable, and fails on any other value. A second converge of a healthy replica no longer turns writes off again and reports a change (S-019-13), and a promotion from a declared source that is read-only and applied now goes through instead of waiting for it to turn read-only (S-019-15) (#450)
  • A MariaDB role change now converges in either host order of the serial services.yml play. In a planned swap, a server declared the replica of a primary that still replicates from it turns read-only and waits (demote) when that primary's link names its host and port but has not connected yet (Master_Server_Id: 0, as after a restart), instead of refusing with service_mariadb_replica_unattachable while the other server waits for it. A replica whose primary holds no cloudfall_repl account for it yet, and whose primary's server converges later in the same run, turns read-only and answers the new wait-primary outcome instead of failing with MariaDB's "is not allowed to connect"; it attaches in the next converge. A primary that already converged, or whose declaration does not admit the replica, still stops the converge. The answer of a wait (demote, wait-promotion, wait-primary) now carries its reason, which says to converge again (#451)
  • On MariaDB 11.8, where JSON_VALUE(Priv, '$.account_locked') reads 1 rather than true for a locked account, the role and inspect now find locked accounts by IN ('true', '1'), through one condition in the role's defaults that every read of locked accounts uses. A declared user that was locked is unlocked again, a replica's cloudfall_repl account locked while it was no replica is unlocked so its link logs in, and locking the account of a server that left the pair no longer reports a change on every converge (S-019-13); audit sees the accounts the role locked (S-019-7) (#452)
  • cloudfall services converge enforces each host's declared firewall, the ruleset cloudfall baseline converge applies with the services' port rules in it, before it converges that host's services, so after a MariaDB role change the new primary admits its new replica in the same converge instead of timing out until a baseline converge ran. A host whose baseline declares no firewall is left alone (#453)
  • A MariaDB replica declared primary whose link's source is no declared service is no longer promoted over a running source that only rejects the controller: the probe of the source's port now runs on the replica's own server (a small Python script of the role, files/mariadb_source_probe.py), which the source admitted while it replicated from it, instead of on the controller, where a firewall that REJECTs everyone but the replica gave the same refused connection. The replica's Last_IO_Errno is still read first; a timeout, no route or a name the replica's server can't resolve still leaves the source unknown and refuses with PRECONDITION. Spec 019's Promotion section now says "can't be reached" is read from the replica's side (#455)
  • cloudfall changelog no longer lists the whole interface as added in every release: the release flow now puts back the manifest snapshot and schema-changelog.json from the previous release's tag before treaty changelog-add runs, so an entry lists only what changed since that release and a release whose interface did not change adds no entry. Release commits are not on main, so each release had started with no snapshot (#457)

Cut to fit GitHub's limit of 125,000 characters: the full text is at https://github.com/cloudfall-dev/cloudfall/blob/v0.8.0rc90/CHANGELOG.md