Repository navigation
cloudfall 0.8.0rc59
Pre-release
Pre-release
·
80 commits
to main
since this release
Changes since v0.7.0:
Breaking
deploy --yesandmigrate --yeson an application that declaresapproval: manualnow need a person's approval of the release: without acloudfall release approverecord that counts (the same archive SHA-256, exactly the component's declared servers, not expired),deploy --yesexits 4PRECONDITIONwithdeployment_approval_missingbefore any playbook runs, naming therelease approvecommand, and lists each record that doesn't count withdigest_mismatch,servers_changedorexpired. This hits every project that declaresmanualand deploys today, the examplecrm.yamland every Render import included. Two ways out: keepmanualand approve each release first (build it, runcloudfall release approveat a terminal, commit the record underdecisions/releases/when CI deploys, then deploy), or switch the application toapproval: automaticwhen another gate, such as GitHub's required reviewers, is the real one. Thedeployplan and result gainapproval(policy, and formanualthestatus, the counting record'spath,approverandexpires_atorapproved_at, the records that don't count and thecommandto run),deploygains--approvals(defaultdecisions/releases) and theRECORD_INVALIDexit for an approvals file that isn't a validReleaseApproval, andcloudfall mcp servebinds the approvals folder (its own--approvals) so adeploycall can't name its own; anautomaticapplication's deploy never reads the folder (#163)- A
LoggingStacknow needs a storage pool on its backend's baseline:cloudfall config validaterefuses aLoggingStackwhose backend server'sBaselinedeclares nostorage.poolswithCONFIG_INVALIDand codevolume_requirement_pool_missing, so a project whoseLoggingStackvalidated on 0.8.0rc22 stops validating. Before upgrading, declare the volume group the Loki and Prometheus volumes go into as a pool on that baseline (storage.poolswithid,volumeGroupandprimary); with more than one pool, name it on each volume inspec.backend.storage.poolandspec.metrics.backend.storagePool, or validation refuses withvolume_requirement_pool_ambiguous(S-014-2) cloudfall config validaterefuses one SSH key (same type and body, any comment) declaredactiveandrevokedin the same environment, withCONFIG_INVALIDand codessh_public_key_lifecycle_conflict, so a project that declares a key that way stops validating; before, the baseline installed it. Before upgrading, delete the revoked declaration if the key should keep access, or mark the active onerevokedif it should not (#354)- Some source URLs that
cloudfall data copyandcloudfall migrate --dataread through--dbnamebefore are now refused, because a libpq service file can't hold them as written: a#in the database name or a query parameter value, a password that ends in whitespace, and a value holding a line break. The copy stops before anything is staged, withlifecycle_execution_failedand a reason that names the URL part without quoting it. Before upgrading, percent-encode a#there as%23; a password that ends in whitespace or holds a line break can't be staged even percent-encoded, so change that password first (#355)
Added
- A Component can declare
spec.buildTools(tool name to minimum version),spec.build(ordered steps, each aworkingDirectoryinside the packaged root, an argvcommandand an optionaltimeoutSecondsfrom 1 to 3600, default 900) andspec.artifact.include(relative paths the build produces);cloudfall config validaterefuses a step whosecommand[0]is not inbuildTools(component_build_tool_undeclared), an absolute or..path, andbuildToolsorartifactwithoutbuild. Validation only:artifact builddoes not run the steps yet. Components withoutbuildvalidate as before (#113) cloudfall-engine artifact buildruns a Component's declaredspec.buildsteps on the controller before packaging: it checks everyspec.buildToolsentry is onPATHat its minimum version before cloning (artifact_build_tool_missing,artifact_build_tool_unsupported), runs each step in the clean checkout with onlyPATH,HOME,LANG,TMPDIRandCI=truein its environment and its output on stderr, fails without writing an archive on a non-zero exit or a step outrunning its owntimeoutSeconds(artifact_build_step_failed), a missing include path (artifact_include_missing), an include path that leads out of the checkout through a symlink (artifact_include_outside_checkout) or a tracked file a step deleted (artifact_tracked_file_missing), packages the tracked files plus everyspec.artifact.includepath (a gitignoredweb/dist, say), and records the steps and tool versions as the new optionalspec.buildof the Artifact metadata. The command's time limit is now the clone time (1800 seconds) plus every declared step timeout, on the CLI and the SDK/MCP path alike; an explicit--timeoutreplaces it, and--timeout 0still disables it. A Component withoutbuildis packaged as before (#115)- The
converge_domainsMCP tool takes an optionaldomain: given,domains.ymlruns only on that Domain's proxy server (--limit) and renders only that Domain, and an unknown id exitsPRECONDITION(lifecycle_domain_missing) before the inventory render; omitted, it runs on every server as before. Each domain receipt now recordspreviousConfigurationSha256, the sha256 of the vhost file the run replaced, ornullwhen none was there; receipts without it still load (#116) cloudfall domains apply <domain> [--issue-certificate] [--yes]converges one Domain:domains.ymlruns on its proxy server alone and renders only it, so no baseline, firewall or service task runs. Without--yesit exits 0 with a plan.--issue-certificaterefuses a Domain withtls.mode: disabled, and first checks on the controller that the primary name and every alias resolve to the proxy server (PRECONDITION,domains_dns_unverified, naming each failing name); aproxiedDomain skips the check (dnsCheck: "skipped-proxied"). With--yesit answers the receipt path under--deployments(#117)cloudfall domains observe <domain>reads one Domain on its proxy server alone, through the new read-onlydomains-observe.yml: the vhost file and its sha256, the certificate's expiry,certbot.timer, and whether the vhost still matches the last receipt (receipt: nullbefore the first). It changes nothing on the server or in the project, and a missing vhost exits 0.cloudfall mcp servelistsdomains_apply(with itsyesgate) anddomains_observeas tools (#118)- Components: a
release.command(with optionalworkingDirectoryandtimeoutSeconds, 1 to 1800, default 300) runs once per deploy on the first server ofdeployment.servers, which now deploys first, after the release's dependencies are materialized and before the unit file changes orcurrentmoves; it runs as the application's user with the component's env file loaded under systemd'sEnvironmentFile=rules. A non-zero exit or a timeout fails the deploy withENGINE_STEP_FAILEDbefore any server switches, and a passing run is recorded asreleaseCommand(server, command, exit code, last 2000 characters of output) in every release receipt of that deploy (#119) deploywithout--yesshows the component's release command asrelease_command(its command, the working directory resolved under the new release directory,timeout_secondsand the release server), ornullfor a component without one, and says inwould_runthat it runs once on that server before any switch; therollbackplan of a component with a release command showsrelease_command_skipped: true, since a rollback never runs it (#120)- A PostgreSQL
Servicedatabase may name anownerrole and itspasswordRef(environment,path,keyin the secrets store) instead of anapplication, so a project of a server and its database alone validates;spec.postgresql.remoteAccessdeclares a listen address and the client networks allowed in.config validaterefuses an entry with both or neither ofapplicationandowner, anownerwithoutpasswordRefor apasswordRefnext toapplication, a reserved owner (service_database_owner_reserved), and aremoteAccesswith no owner database, no firewall, the port already open, or a source of every address. Converging these declarations comes later: until then the PostgreSQL role refuses them before any task runs (#121) services.ymlconverges a PostgreSQL database owned by a declared role: it creates theownerlogin role, sets its password frompasswordRefas SCRAM-SHA-256 on every converge, and gives it the database; backups, the restore drill, anddata copy(restoring as the owner, its receipt'sspec.owner) cover it like any other database. Every path that runsservices.yml(theconverge_servicesMCP tool, theservicesstep ofcloudfall migrate,operator approveandoperator run, andcloudfall-engine playbook run services.yml) reads the password from a--secrets-dirflag (defaultsecrets) and stages it in a 0600 file in a private temp dir outside the project, removed after the run; the value is in no inventory, command line, Ansible output, answer, or receipt, and the answer'sowned_rolesnames the reference, the key, and the value's SHA-256;operator approverecords them asownedRolesin the proposal receipt'soutcome(an optional property) andoperator runin each executed proposal's entry. A missing secrets file or key fails withPRECONDITIONnaming the expected path and key before any playbook runs (#122)- A PostgreSQL service with
spec.postgresql.remoteAccessopens to its declared sources over TLS:services.ymlsetslisten_addresses = '127.0.0.1,<listenAddress>'andssl = onwith the Debianssl-certsnakeoil pair (restarting the cluster), and writes onehostssl <database> <owner> <cidr> scram-sha-256line per owned database and source between Cloudfall markers inpg_hba.conf, reloading only afterpg_hba_file_rulesreports no error;baseline.ymladds oneip saddr/ip6 saddr <cidr> tcp dport <port> acceptrule per source, rendered as acloudfall_firewall.allowedInboundentry with a new optionalsourceslist. RemovingremoteAccessresets the listener and removes thepg_hba.confblock and the firewall rule; databases named byapplicationstay socket-only (#123) cloudfall data copytakes a repeatable--source-schema NAME(and thedata_copyMCP tool asource_schemalist): the dump and restore copy only the named schemas, the empty-target check and the per-table row-count check read exactly those schemas, and the plan without--yesnames them; a name that is empty, longer than 63 bytes, or holds a control character or a brace is refused as a usage error. Without it the dump is the whole database as in 0.7.0, and the checks now cover every non-system schema of the source instead ofpubliconly, so a row-count mismatch outsidepublicnow fails the copy. TheDataMigrationReceiptgainsspec.schemas(sorted) and aschemaon eachspec.tables[]entry besidetableandrows(#124)cloudfall data copy, and thedata:<database>step ofcloudfall migrate, run a preflight on the target host after the empty-target check and beforepg_dump: it reads both catalogs and refuses a copy the target can't hold, with nothing dumped, the target untouched and no receipt, exitingPRECONDITIONwith codedata_copy_preflight_refusedand four lists in the error context, each item named:missingSchemas(a named schema the source lacks),extensions(one the selected schemas need, throughpg_dependor by living in them, or every source extension butplpgsqlwithout--source-schema, that the target'spg_available_extensionslacks),roles(a role a row-level security policy on a selected table names that the target'spg_roleslacks) andauthReferences(a function or policy of the selected schemas that referencesauth.whenauthisn't copied). The role writes the findings to<service>-<database>.preflight.jsonunder--data-migrations, so the lists arrive whole. With--source-schema,pg_dumpalso gets one--extensionper extension the schemas need (pg_dump 14 or later), andpg_restorenow runs with--single-transaction, so a failed restore rolls back whole, writes no receipt, exitsENGINE_STEP_FAILED, and the next run passes the empty-target check (#125)cloudfall migratetakes a repeatable--data-schema DATABASE=NAMEbeside--data DATABASE=PATH(and themigrateMCP tool adata_schemalist): thedata:<database>step copies only the named schemas, asdata copy --source-schemadoes, and the plan names them. A--source-schemaor--data-schemaname must now match^[a-z_][a-z0-9_]*$and be at most 63 characters, and may be given once per copy; a name that doesn't, a repeated name, or a--data-schemafor a database with no--dataexitsPRECONDITION(codedata_copy_schema_invalidormigrate_data_schema_without_data) before any playbook runs, where a malformed name was a usage error and a repeat was silently dropped (#126)- A Domain can declare
spec.routes, path routes that each either proxy to anupstream(address,port) or serve a component's release files (static:component,directory, optionalspaFallback).cloudfall config validaterefuses a Domain with bothproxy.upstreamandroutes(domain_route_source_invalid), a path that doesn't start with/or uses characters outside letters, digits and._~/-, or a static path without a trailing/(domain_route_path_invalid), a path declared twice (domain_route_path_duplicate), a path under/.well-known/acme-challenge(domain_route_path_reserved), a missing static component (resource_reference_missing) or one not deployed on the Domain'sproxy.server(domain_route_component_not_on_proxy), and adirectoryorspaFallbackthat is absolute, climbs out with.., or, fordirectory, names the release root. Validation only: the domains playbook doesn't render routes yet and still refuses a Domain withoutproxy.upstreambefore any change. Domains with onlyproxy.upstreamvalidate and render as before (#159) cloudfall release approve <component> --release <release>records a person's approval of one built release as aReleaseApproval(newrelease-approval.schema.json) underdecisions/releases/: the application, component, release, archive SHA-256, git commit, servers, approver,approvedAt,expiresAt(--valid-forhours, 1 to 168, default 24) andvia: terminal. It verifies the artifact first, refuses a component of anautomaticapplication withrelease_approval_not_required, and writes the record only once a person typed the release id at a terminal;--yesnever answers it, and it is no MCP tool. It deploys nothing (#160)- Components: a
healthCheckoftype: command(a required non-emptycommandargv,timeoutSeconds1 to 300 andattempts1 to 20, no other key) validates, anddeploy,restartandrollbackwithout--yesname the check and its command in their plan'swouldRun(#161) cloudfall domains applyrenders a Domain'sspec.routes: a proxy route becomes alocationthat proxies to itsupstreamwith the full request path, and a static route serves<installRoot>/current/<directory>of its component, answeringspaFallbackfor an unknown path when it declares one and 404 when it doesn't. Without a/route, a path no route matches answers 404, not Nginx's default page. Locations render longest path first, whatever the declared order, where the single proxy location rendered before (the port 80 server without active TLS, the 443 server with it); the port 80 redirect and the ACME challenge location stay. The site role grants the Nginx worker (www-data) traverse-only access (POSIX ACLuser:www-data:--x, installingacl) from the application root to the install root and itsreleases/, so it reads the release files but nevershared/env/<component>.env, and a deploy orcloudfall rollbackswitches what a static route serves without re-running the domains playbook or reloading Nginx. The domains playbook's early check now refuses only a Domain with neitherproxy.upstreamnorroutes. A Domain with onlyproxy.upstreamrenders a byte-identical vhost (#162)- Deploy: a component's
healthCheckoftype: commandnow gatesdeploy --yes. After the switch and the restart it runs on each server in a transient systemd unitcloudfall-health-<component>, as the application's user, in<installRoot>/current, with the service's env file andtimeoutSecondsas its runtime limit, up toattemptstimes 3 seconds apart. When every attempt fails or times out, the previous release is restored (a first release is stopped), the deploy fails withENGINE_STEP_FAILEDnaming the check and its exit code or timeout, and the release receipt is written withhealthCheckPassed: false. Receipts of a command check carryhealthCheck(command, attempts, exit code, timed out), never the command's output, which stays in the server's journal. The release receipt schema acceptshealthCheckPassed: falseonly with a failed commandhealthCheck, and every receipt written before (#164) restart,rollbackandhealthrun a component'stype: commandhealth check instead of refusing it: the same transient unit asdeploy's gate, with its user, working directory, env file, timeout, attempts and redacted output.healthreportsunhealthywhen the check exits non-zero or times out, a failed check failsrestartorrollbackwithout restoring anything, as anhttpcheck does, and none of them writes a release receipt. The check starts its unit as root, sohealthof such a component needs the inventory SSH account to become root, asrestartandrollbackalready do;httpandnonechecks are unchanged (#166)cloudfall config upgrade: rewrites a project from the deprecatedServerTypenames toBaseline(movesserver-types/tobaselines/, rewriteskindandspec.serverType), keeping comments, formatting and line endings; a second run changes nothing, and it refuses, changing nothing, a move onto an existing file or a file or folder reached through a symlink (upgrade_symlink) (#198)auditchecks the certificates a server's host requirements name (present, for the right names, more than 30 days left at the observation's time) from newcertificatesevidenceinspectcollects, and expects the logging gateway's firewall rule, so a deployed stack audits compliant (S-011)cloudfall logging deployprovides the stack's mTLS material: each host makes its own key and a signing request (logging-requests.yml), the project CA signs the missing and due certificates, and they are delivered with the CA; certificates provided by hand are kept (S-011)cloudfall logging deploy: deploy the declaredLoggingStackafter meeting its secret requirements; generated secrets are written to the store, an input secret that is missing stops the run (exit 4,INPUT_REQUIRED), and each stored secret is copied to its host path without overwriting a file already there; without--yesit shows the plan (S-011)- Logging collectors pin the gateway name to the backend's address in
/etc/hosts(loopback on the backend itself), so the gateway needs no DNS record when the backend declares an IP (S-011) - The logging gateway's port opens on the backend's firewall to each remote collector's addresses only, rendered beside the
remoteAccessrules and enforced bylogging.yml; a remote collector with no IP behind a firewall failsconfig validate(logging_gateway_collector_unaddressed) (S-011) LoggingStack:migration.mode: nonefor a host with no legacy agents;parallelstill requirespreserveLegacyAgents: true(S-011)- Logging roles check every package pin against the host's APT sources before installing or configuring anything, and stop naming the versions on offer when one is missing (S-011)
- Certificate provider, controller side: a project CA per environment, kept sops-encrypted under
secrets/<environment>/pki/, that signs host requests for exactly the names and usage a requirement grants (S-011) cloudfall requirements show: the volumes, ports, secrets, certificates, DNS names and package pins each declaredLoggingStackneeds from its servers, by server (S-011)cloudfall secrets generateandcloudfall secrets input: keep the secrets aLoggingStackrequires sops-encrypted undersecrets/, generated or typed at a terminal with echo off;requirements showgives secrets a state and lists the missing ones (S-011)- A decision record and a proposal receipt name the Cloudfall version that last wrote them in
spec.writerVersion, an optional schema field: records written before it stay valid (#218) deploy,rollback,restartand an artifact build hold their component's lock undertmp/locks/on the management host until they end, passed, failed or interrupted, so a second run of the same component exitsCONFLICT(component_locked) before any server is contacted, naming who holds the lock, since when, and whether that process still runs;cloudfall lock clear <component>shows a lock a killed run left, with its age, and removes it once a person types the component id at a terminal, never while its process still runs on this host (#221)deploy.yml,rollback.ymlandrestart.ymlhold the component's lock on each server, at/var/lib/cloudfall/locks/<component>.json, from before they change anything there until that server's turn ends, passed or failed, so runs from two controllers,cloudfall-engine playbook runand approved runs of those playbooks no longer race oncurrent, the unit file or the release command's unit: a second run fails on that server, naming the action, user, controller, start time and age of the lock it found (#221)- A Component's optional
deployment.retainReleases(an integer, at least 2, 5 when absent) says how many releases a deploy keeps on each server; the SDK inventory and the deploy role carry it, andretainUntilCleanup: truestill validates and means the cleanup is on (#224) - The release receipt names, in each server entry whose gate passed, the releases the deploy removed there under
removedReleases;release-receipt.schema.jsonaccepts it and refuses it on a failed entry (#224) - The PostgreSQL backup writes
globals-<timestamp>.sqlfrompg_dumpall --globals-only --no-role-passwordsbeside the per-database dumps, so a restore onto a fresh server has its roles; retention prunes it with the dumps (#225) Baselinestorage.pools: existing LVM volume groups that workload volumes go into; eachLoggingStackvolume names its pool, LV, LV type and physical volumes inrequirements show, andconfig validaterefuses a volume with no pool (S-014)auditchecks each required volume (volume.<name>: type, physical volumes, size, health, mount) from newvolumesevidenceinspectcollects (S-014)cloudfall logging deployprovides the stack's data volumes: a read-only report from each host, refusals decided on the controller before any host change (volume_refused), thenlvcreate/mkfsfor missing volumes,lvextend --resizefsfor small ones, an fstab line by UUID, the mount andRequiresMountsFordrop-ins (S-014)cloudfall decisions mark-interrupted <id> --confirm-deadandcloudfall operator mark-interrupted <id> --confirm-deadclose a record leftrunningby a run that died midway as the new statusinterrupted, once a person confirmed the run is dead; the record keeps its approval and gainsinterruption.confirmedBy(--by, defaultUSER) andinterruption.confirmedAt. Nothing runs on the fleet, and neither command is an MCP tool (#244)AlertChannel(alert-channels/): a Telegram chat that aBaseline's hosts and aLoggingStackreceiver name; its bot token is an input secret on every server that sends, set withcloudfall secrets input telegram-bot-token --owner <channel>(S-015)- A
LoggingStackreceiver may name anAlertChannel: Alertmanager delivers to it with its nativetelegram_configsandbot_token_file, the token delivered bylogging deploy; an Alertmanager older than 0.26 is refused before its configuration is written (S-015) cloudfall alerts test <server>: send one test message through the server's alert channel;auditchecks each host's channel (alerting.channel: sender, token and chat id present, mdadm and smartd pointed at the sender) from newalertChannelevidenceinspectcollects, never the token (S-015)cloudfall baseline converge: runbaseline.ymlwith each host's alert channel; hosts whose baseline names anAlertChannelget/usr/local/sbin/cloudfall-alert, the channel's token and chat id, and mdadm and smartd pointed at the sender; a token not in the store stops it before any server (S-015)- A
Servicecan beserviceKind: elasticsearch(S-012):spec.elasticsearchdeclaresversion(an exactX.Y.Zof a line Elastic still supports,8.19or9.x),heapMb(512 to 31744), and optionallynodes,clusterName,existingCluster,users(each withindexPatternsand apasswordRef, never a password),remoteAccess,snapshotRepositoryandrestoreCheckIndices.cloudfall config validaterefuses an unsupported version (service_elasticsearch_version), badnodes(service_elasticsearch_nodes), anexistingClusterwithoutclusterName, with a CA file that is missing, outside the project, not a CA or holding a key, or with a seed host that is a declared node (service_elasticsearch_existing_cluster), a server in two clusters (service_elasticsearch_server_shared), several nodes or an existing cluster without asnapshotRepository(service_elasticsearch_snapshot_repository_local), a reserved or repeated user name (service_elasticsearch_user_reserved,service_elasticsearch_user_duplicate), a pattern of every or a hidden index (service_elasticsearch_pattern), andremoteAccessunder the PostgreSQL rules; it warnsELASTICSEARCH_VERSION_ENDINGwith the date when a line is within 90 days of its end of life. The rendered inventory carries the service, withnodesandclusterNameresolved, on its server; nothing converges it yet (#309) services.ymlconverges a one-nodeserviceKind: elasticsearchservice on Debian 13 through the newcloudfall_elasticsearchrole (S-012). Its first task refuses aheapMbabove half of the host's memory before anything changes. It installselasticsearch=X.Y.Zfrom Elastic's APT repository for the declared major and holds it, and writeselasticsearch.yml(cluster.name,node.name, HTTP on the loopbackbindonly,bootstrap.memory_lock: true, security on, neverdiscovery.type), ajvm.options.dheap file and a unit drop-in withLimitMEMLOCK=infinity, all before the package, so its security auto-configuration skips the node.cluster.initial_master_nodesis written only while the data path holds no cluster state and leaves the file in the same converge once the cluster has formed, with no restart, so a second converge changes nothing; the converge stops when the node runs without its memory locked. Cloudfall's owncloudfallaccount is a superuser in the file realm, with only its bcrypt hash inusersand its password in a0600file owned byelasticsearch; theelasticpassword is set once, as a bcrypt hash computed on the controller (D-9). Each node of the service yields theelasticsearchpackage and the generatedelasticsearch-cloudfall-passwordas S-011 requirements, andcloudfall secrets generatealso generates theelasticpassword (ELASTIC_PASSWORDbesideCLOUDFALL_PASSWORDinsecrets/<environment>/services/<id>.env); every path that runsservices.ymldecrypts both first and stops before any playbook when one is missing. More nodes, declared users,remoteAccessandexistingClusterstop the converge until their slices land (#310)- An Elasticsearch service's declared
usersconverge (S-012): from the service's home server, each user gets a role of its own name limited to itsindexPatterns(read:readandview_index_metadata;writeaddswrite,create_indexandmanage) and a native user holding that role alone. Every path that runsservices.ymldecrypts each user'spasswordReffirst, stops before any playbook when one is missing, and stages only its bcrypt hash, computed on the controller and sent aspassword_hash(D-9); the salt is derived from the service'scloudfallpassword, so an unchanged password puts nothing and a second converge changes nothing. Users and roles Cloudfall makes carrymetadata.cloudfall_managed_by: <service id>; a converge deletes the marked ones no longer declared and never changes or deletes an unmarked one, and refuses a declared name an unmarked user or role already holds.inspectreads the cluster's users and roles on the home server (elasticsearchUsersin the observation), andauditaddselasticsearch.users[<id>]: drift when a declared user or role is missing or a marked one is left over, with users and roles not made by Cloudfall listed as unmanaged (#311) - An Elasticsearch service grows to two or more
nodesand rolls a version one node at a time (S-012). Each node yields anelasticsearchdata volume of 10 GiB at/var/lib/elasticsearchin its baseline's only storage pool,mirroredfor a cluster of one andexpendablefor two or more; a node of a cluster of two or more also yields port 9300/tcp open to the other nodes only and anelasticsearch-transportcertificate from the project CA for its hostname and addresses, of the usageserver-and-client: the one certificateremoteAccessserves HTTP with, which the services converge now issues for a node withremoteAccesstoo. The role refusesremoteAccesson a cluster of two or more, since its onelistenAddressexists on one server only.config validaterefuses a node whose baseline has no pool (volume_requirement_pool_missing), several (volume_requirement_pool_ambiguous), or, for a cluster of one, no secondary physical volume (volume_requirement_pool_unmirrored), and a node of a cluster of two or more with no IPv4 (service_elasticsearch_node_unaddressed). The rendered inventory carries the service on every node with its transport address and seed hosts, adds the 9300 rule to each node's firewall, and lists each cluster's home first. Every path that runsservices.ymlthroughcloudfallfirst reads each node's volume facts and decides its volume, then has each node make its key and a signing request, signs the missing and due certificates with the project CA and delivers them, all before any host changes. The role creates and mounts the volume, enforces the firewall, binds the transport to the node's address with TLSverification_mode: full, bootstraps a cluster only from the home node and refuses when another node of it already runs, sets theelasticpassword from the home only, and restarts a running node only from a green cluster (yellow with one node), with replica allocation held and a flush (released again even when the restart, the rejoin or the version check fails), then waits for the node to rejoin on the declared version and the cluster to return to its status before the next server (#312) - An Elasticsearch service's S3
snapshotRepositoryis converged instead of refused (S-012). ItscredentialsRefnames one key of the secret store holding<access key id>:<secret access key>; every path that runsservices.ymldecrypts it on the controller, stages the two keys in 0600 files and passes only their paths, and stops before any playbook when the key is missing or malformed. On every node the role puts them in the keystore ass3.client.default.access_keyands3.client.default.secret_keythrough stdin, keeps only their SHA-256 beside it so a second converge changes nothing, names the endpoint ass3.client.default.endpointinelasticsearch.yml, reloads a running node's secure settings, and registers the repositorycloudfallastype: s3with the bucket and base path, replacing the filesystem repository of a service that grew; the snapshot and restore-drill scripts use it unchanged (#312) config validaterefuses an Elasticsearch cluster of two or more nodes on a baseline without a firewall, since nothing would narrow 9300 to the other nodes (service_elasticsearch_cluster_firewall_missing); the inventory render refuses it too (#312)cloudfall backup runandcloudfall backup verifywork for aserviceKind: elasticsearchservice (S-012). The converge addsbackup.directorytopath.repo, which restarts each existing node once on the first converge after the upgrade, registers it as the filesystem snapshot repositorycloudfall, and installscloudfall-elasticsearch-backupandcloudfall-elasticsearch-restore-checkwith sandboxed units and timers onbackup.onCalendarandbackup.restoreCheckOnCalendar. The backup counts the documents of every indexrestoreCheckIndicesmatches (or of the smallest index) and writes the counts as one document per snapshot into Cloudfall's own indexcloudfall-snapshot-counts. It then takes the snapshotcloudfall-<timestamp>of every index, that one included, and the cluster state, waits for it, and fails unless its state isSUCCESS, so aPARTIALorFAILEDsnapshot writes no receipt. Finally it deletes Cloudfall's snapshots of the service older thanretentionDayswith their counts, always keeping the newest and never touching a snapshot it did not take. The restore drill verifies the repository, restores the snapshot's counts and every recorded index ascloudfall-restore-check-<index>with no replicas and without aliases, checks the low disk watermark, waits for green, compares each document count, and deletes the restored copies even when a step failed; it refuses to start over an index of a copy's name, and fails when no snapshot exists. Both run as theelasticsearchuser, reach the node as the role does (withremoteAccess, over HTTPS on the listen address, verified against the project CA) and write the usualBackupReceipt. An S3snapshotRepositorystops the converge before it changes the host, with a message that S3 repository registration is not built yet (#313)- An Elasticsearch service (S-012) is audited, opened by
remoteAccess, and scraped for metrics.cloudfall auditreports drift on each node when the installedelasticsearchpackage differs fromversion, whenelasticsearch.serviceis not running and enabled, and when the managed settings ofelasticsearch.yml(network.host,http.port,http.host,xpack.security.enabled,xpack.security.http.ssl.enabled) differ from what the role writes, so a hand edit is drift before the node restarts with it; it reports the node as running an unsupported version once its line is past Elastic's End of Maintenance date at the observation's time (D-12).inspectreadselasticsearch.ymlas text on a server with an Elasticsearch service (spec.elasticsearchof the observation). WithremoteAccess, the firewall opensbind.porttoallowedSourcesonly, HTTP listens on loopback andlistenAddressand is TLS-only on both, with the node's one certificate from the project CA (theelasticsearch-transportcertificate requirement of the new usageserver-and-client, for the server's hostname and every address, the listen address included, at/etc/elasticsearch/certs/transport.crt,transport.keyandca.crt), which the role checks before writing a configuration that names it, and stops on while nothing issues it yet; the role's own API calls then go over HTTPS to the listen address, verifying the project CA. Withmetrics.enabled, the collector renders one Alloyprometheus.exporter.elasticsearchfor the local node, as thecloudfallaccount, labelled with the service id; Alloy reads the password, and withremoteAccessthe project CA, as systemd credentials (LoadCredential), andlogging deploystops before changing the collector when the node's files are not there yet (#314) - An Elasticsearch service with
existingClusteris converged instead of refused: its nodes join a cluster Cloudfall did not create (S-012). Each node's transport leaves loopback even when it is the only Cloudfall node:discovery.seed_hostslists the existing cluster's seed hosts, then the Cloudfall nodes; its firewall admits 9300 from the other Cloudfall nodes and the seed hosts' addresses only; it gets its transport certificate from the project CA and trusts the existing cluster'scaCertificateFilebeside it, which the services converge delivers to/etc/elasticsearch/certs/existing-cluster-ca.crt, withverification_mode: certificate. A joining node never writescluster.initial_master_nodesand never sets theelasticpassword, which stays the existing cluster's, and the existing nodes are only read. Before the data volume and the package, the role asks the seed hosts over HTTPS on port 9200, verified against that CA, forGET /_nodes, and stops, naming the seed host or node and its value, when the cluster's name differs fromclusterName, when a node runs another major, when a node is newer thanversion, or when no seed host answers. OnceexistingClusteris removed, a converge drops the seed hosts and the CA from the configuration, restarts one node at a time, removes the CA file, and waits for the cluster to be green with two or more Cloudfall nodes, or yellow with a single one, whose replicas have nowhere to go;clusterNamekeeps the name.config validaterefuses a seed host given by name rather than address, and a node that joins an existing cluster without a firewall or an IPv4, as for a node of a cluster of two or more (#315) existingCluster.credentialsRefnames one key of the project's store holding<username>:<password>of an account the team already has on the existing cluster with themonitorprivilege; the join check logs in with it, so nobody creates a user on the old cluster by hand. Without it, the check logs in as thecloudfallaccount, which the team then grantsmonitorthere. A missing key, or a value not of that form, stops the converge before any playbook, naming the file and the key; the login is staged in 0600 files on the controller only, removed after the run, and never reaches a node, the inventory, task output or an error (#315)cloudfall operator mark-interrupted <id> --confirm-deadalso closes a receipt leftexecuted, whose verification never recorded an outcome, asinterrupted, as it does arunningone; it then stops blocking new proposals for the same alert or drift (#358)- A project states its format as
[tool.cloudfall] formatin itspyproject.toml, andcloudfall initwrites the current one (format = 1).cloudfall config validatechecks it before validating: a project without the key, or without apyproject.toml, reads as format 0 and still validates, with onePROJECT_FORMAT_OLDwarning namingcloudfall config upgrade; a format outside the ones this release reads (0 to 1), or one that is not a non-negative integer, stops with the new exitPROJECT_FORMAT_UNSUPPORTED(95) anderror.context.codeproject_format_too_new,project_format_too_oldorproject_format_invalid(#375) cloudfall changeloglists entries: each release now records its interface changes (commands, flags, output fields, exit codes) inschema-changelog.jsonand its manifest snapshot, both in the wheel, as the release bot'safter_stampstep runstreaty changelog-add cloudfall.app:app. The first release seeds the snapshot, so its entry lists the whole interface as added; a release whose interface did not change adds no entry. The Releases section ofCONTRIBUTING.mdnames no manual step any more (#378)- A
Servicecan beserviceKind: mariadb(S-019):spec.mariadbdeclaresversion(an exactX.Y.Zof a long-term series MariaDB still maintains:10.11,11.4,11.8or12.3),innodbBufferPoolMb(128 or more), and optionallydatabases(each with anapplication, or anownerwith apasswordRef, never a password, andauthentication: ed25519if wanted),remoteAccess, andreplication(roleprimaryorreplica,serverId, a primary'slistenAddress, a replica'sprimaryservice orsourcewithhost,port,credentialsRefandcaCertificateFile).cloudfall config validaterefuses an unsupported version (service_mariadb_version), a server with two MariaDB services (service_mariadb_server_shared), a reserved owner or database name (service_database_owner_reserved), a repeatedserverId(service_mariadb_server_id_duplicate), a primary withoutlistenAddressor a replica with one (service_mariadb_replication_listen), a replica without exactly oneprimaryorsource, or whoseprimaryis no primary MariaDB service of its environment on another server (service_mariadb_replication_primary), a replica withdatabases(service_mariadb_replica_databases), of another series than its primary or older (service_mariadb_replica_version), a pair without a firewall (service_mariadb_replication_unfirewalled), a replica server with no address its primary can admit (service_mariadb_replica_unaddressed), a source CA file that is missing, outside the project or not a CA (service_mariadb_replication_source), a server without one mirrored storage pool (the S-014 codes), andremoteAccessunder the PostgreSQL rules; it warnsMARIADB_VERSION_ENDINGwith the date when a series is within 90 days of its end of life, andMARIADB_REPLICATION_UNENCRYPTEDfor a source without a CA file. The kind yields itsmariadbvolume, package, replication port, certificate and generated replication password as host requirements, and the rendered inventory carries the service with its databases and references, never a password; nothing converges it yet (#419) servicesconverges a standalone MariaDB service (S-019) with the newcloudfall_mariadbrole: it refuses, before any task changes the host, aninnodbBufferPoolMbabove three quarters of the server's memory, a series MariaDB publishes no package of for the server's Debian release, and a series other than the installed one; it mounts the data directory on the mirroredmariadbvolume the services converge decides with the Elasticsearch volumes, installsmariadb-serverat the exact declared release from MariaDB's archive and holds it, and writes the managed settings to/etc/mysql/mariadb.conf.d/90-cloudfall.cnf(listener onbind.addressonly, buffer pool,utf8mb4; no binary log). Anapplicationdatabase gets a user named after the application's Linux user that logs in through the unix socket only; anownerdatabase gets a user whose password reaches the host only as the hash its plugin stores (mysql_native_password, ored25519withauthentication: ed25519), computed on the controller and set only when it differs from the stored one, so a second converge changes nothing. Each user has every privilege on its own database. A user Cloudfall made that is no longer declared is locked, its database left with its rows, and users and databases made by hand are left alone;cloudfall auditlists both kinds as undeclared and unmanaged. ApasswordRefwhose file or key is missing stops the converge withPRECONDITIONbefore any playbook runs.remoteAccessandreplicationare still refused by the role until their own issues land (#420)- A MariaDB service converges its
remoteAccess: the services converge has the server make its key and the project CA sign its certificate for its hostname and addresses, the role installs it (the key 0600 for the mysql account), listens on loopback andlistenAddresswithssl_cert,ssl_keyandssl_ca, and gives each owner with a password one account per allowed source from the same controller-computed hash, requiring TLS, so a plain connection from a source is refused; the firewall admits the port from the allowed sources only, andauditexpects that rule, the listener and those accounts. An account of a source no longer declared is locked, never dropped; a renewed certificate is loaded withFLUSH SSL, without a restart.replicationis still refused by the role (#421) - A MariaDB service is backed up and its restore drilled (S-019):
servicesinstallscloudfall-mariadb-backupandcloudfall-mariadb-restore-checkwith their systemd timers (the drill's only withrestoreCheckOnCalendar), run as root through the socket, andcloudfall backup run <service>andcloudfall backup verify <service>run them and write theirBackupReceipt. A backup writes, for every declared database, one gzip dump taken withmariadb-dump --single-transaction(one consistent snapshot of the InnoDB tables, no table locks; tables of another engine are named in the output) and beside it a count file of every table and its rows, counted from the stored dump, both0600in a0700directory; then it deletes each database's dumps older thanretentionDays, never the newest, and only files of the names it writes. The drill loads the newest dump of each database into the scratch databasecloudfall-restore-checkwithout writing to the binary log, compares its tables and row counts with the count file (never with the live source, so later writes cannot fail it), and drops it on every path; it fails, and says so, when a database has no dump or no count file, when the load fails, and on any difference, and leaves a database of the scratch name it did not make alone. The role checksonCalendarandrestoreCheckOnCalendarbefore it changes the host. The dumps hold tables, views, routines and triggers but no scheduled events (--eventsis not passed, so no event can run in the drill's scratch database): a team that uses events recreates them after a restore (#422) - A CI deploy guide (
docs/ci-deploy-guide.md): one GitHub Actions workflow that renders secrets, builds the artifact withcloudfall-engine artifact build, deploys withcloudfall deploy --yesand uploads the receipts, with one SSH key and one age key per environment and required reviewers standing in forapproval: manual, whichdeploydoes not enforce; not yet run against a real server (#89) - An open proposal whose basis hosts have been observed again since is now reported as stale:
decisions list,decisions showandcloudfall whyaddstaleandstaleHoststo eachproposeddecision,--staleondecisions listandwhykeeps only those, anddecisions approvenames each stale host and its newer observation time before asking for the id. Worked out on read from the observation files; the record and its schema do not change (#26) auditchecks a logging backend's dashboards (logging.dashboards):inspectrecords the name and sha256 of each file in Grafana's Cloudfall dashboards directory (newloggingDashboardsevidence, never anything else of Grafana's), and the check is compliant when they are exactly the dashboards the installed release bundles; drift names each missing, extra or changed file (S-016)logging deployprovisions four dashboards into a read-onlyCloudfallfolder in the stack's Grafana: Fleet (every server's freshness, CPU, memory, fullest filesystem, load and the firing alerts), Server (one server's host metrics and log volume by unit), Logs (log volume, error lines and the lines themselves by server and unit, with a filter) and Services (PostgreSQL and Redis exporters by service). They read servers from the data and name the provisioned data sources by uid, so they work on any project; a dashboard dropped by a later release is removed on the next deploy (S-016)cloudfall logging openopens the LoggingStack's Grafana: it forwards a local port over ssh to Grafana on the backend's loopback, built from the backend server's declared address, SSH user and port, answers with the URL of the Cloudfall dashboards once the forward is up, and holds it until stopped.--portpicks the local port (one in use exits 4 before ssh starts),--stackpicks the stack, and--print-commandanswers with the ssh command and runs nothing. The answer names where the admin password is kept, never the password (S-016)
Changed
- Breaking:
cloudfall config validatenow refuses a Component withruntime.typenodeorpm2, orruntime.packageManagernpm,pnpmoryarn, withCONFIG_INVALIDand codecomponent_runtime_unsupportednaming the field: these configs validated before but could never deploy, since the deploy role runs only Python with uv. The schema keeps those values as reserved. The Render importer no longer writes a Component for a Node service; it lists the service as an unsupported gap instead (#114) migrate --yeschecks adeploy:<component>step's approval asdeploy --yesdoes: for an application that declaresapproval: manualand no approval that counts, it pauses at that step withPARTIAL_FAILURE,context.code: deployment_approval_missing, the builtreleaseand thecloudfall release approvecommand, and once a person approved it,migrate --yesresumes there with the release the completedbuild:<component>step recorded, without building again and without needing--buildagain. The deploy step's detail gainsapproval(the record it went on),migrategains--approvals(defaultdecisions/releases, bound bycloudfall mcp serve) and theRECORD_INVALIDexit for an approvals file that isn't a validReleaseApproval.rollbackneeds no approval: its plan gainsapproval,{"policy": "manual", "required": false, "reason": ...}for amanualapplication and{"policy": "automatic"}otherwise (#165)- The resource every server runs is now
kind: Baseline, stored underbaselines/, referenced asServer.spec.baselineand checked bybaseline.schema.json(wasServerType,server-types/,spec.serverType,server-type.schema.json);cloudfall add baselinedeclares one andcloudfall add server --baselinenames it (#198) - Outputs carry the new names:
inventory showwritesbaselinesand each server'sbaseline,config validatecountsBaselineinby_kind,auditwrites each server'sbaselineand checksbaseline.id(wasserver_type.id; an operator proposal recorded withserver_type.idverifies oncebaseline.idis compliant, and blocks a new proposal for the same drift, with no removal date, since a record's fields are never deprecated withincloudfall/v1), the inspect role recordsbaselinein observed server snapshots, the rendered Ansible inventory has the host variablecloudfall_baseline, the brownfield catalog variable iscloudfall_baselines, and the error codeserver_type_entry_duplicateisbaseline_entry_duplicate(#198) - Site: cloudfall.tech is the primary domain; cloudfall.dev and both www hosts redirect to it with a 301, and the PyPI homepage and the README
cloudfall initwrites point there (#199) - Receipts and operator proposals now land in the project's committed
record/directory instead of the git-ignoredtmp/, so the audit trail survives losing the management host (D-16): the defaults of--deployments,--releases,--backups,--data-migrations,--env-receiptsand--proposals(on every command and oncloudfall mcp serve) arerecord/deployments,record/releases,record/backups,record/data-migrations,record/env-receiptsandrecord/operator/proposals.tmp/keeps only what can be rebuilt: observations, rendered inventory, built artifacts, rendered environment files and the migration plan. A path you pass explicitly keeps working as before.cloudfall config upgradenow also moves an existing project's receipts and proposals fromtmp/intorecord/, listing each move inupgradedwith itsmoved_from; it drops a file whose identical copy is already inrecord/, refuses withCONFLICTand codeupgrade_target_exists, naming both paths and moving nothing, when a different file is, and a second run changes nothing. Until it runs,operator list,audit,services statusand the operator's autonomy pass don't see the receipts left undertmp/, so runcloudfall config upgradeonce after upgrading, then commitrecord/.cloudfall initwrites the new layout into the project's README,AGENTS.mdand.gitignore, and the operator guide says how to copyrecord/off a management host with no git remote (#219) - A release receipt no longer holds its release command's output: the command is the application's and may print a secret, and the receipt is now committed. The last 2000 characters stay in
tmp/output/releases/<component>.<release>.log(0600), andspec.releaseCommandnames that file and its SHA-256 underoutput(path,sha256) instead ofoutputTail. TheReleaseReceiptschema stayscloudfall/v1:outputTailbecomes optional, so receipts written before still validate, and a receipt may not hold both.cloudfall config upgrademoves the tail of an old receipt, undertmp/or already inrecord/, out to that file the same way, refusing to replace a different file there. A reader ofreleaseCommand.outputTailreads the named file instead, which is gone on another host or oncetmp/is cleared (#219) - Contract:
cloudfall lock clear <component>now also reads the component's lock on each of its servers through the newlock.yml, shows them with their ages in the same prompt, and once the component id is typed removes exactly the locks it showed; it takes--engineand--inventory-file, answersserverLocksandunreadServers(a server it could not reach), may exitENGINE_STEP_FAILED, and is listed among the commands that change servers (#221) - The release receipt holds one entry per server under
spec.servers(the release the server ended on,previousRelease,deployedAt,healthCheckPassed,rolledBackand the commandhealthCheck), with the deployedrelease,artifactSha256andreleaseCommandonce per receipt;release-receipt.schema.jsonaccepts it and still accepts a receipt written before, with oneserver, which reads as a one-entry list (#222) - Site: contact addresses moved to cloudfall.tech (
pilot@,conduct@); mail to cloudfall.dev still arrives (#260) - A
mirroredvolume now meets anexpendablerequirement (D-18): when a one-node Elasticsearch service grows, its first node keeps its raid1 data volume, and the volume provider keeps and grows it, andauditchecks it, as the mirror it is; a linear volume still never meets amirroredrequirement (#312) - When installing a new Elasticsearch version fails after the node was stopped for it, the role starts the node again on the package it had and lets its cluster move replicas again before the converge stops (#312)
- Open proposals made before the upgrade carry no binding, so
decisions approverefuses them withdecision_binding_changed: propose them again (#352) decisions approvegives the approved run and its verify step their own time limit of 6 hours (21600 seconds) instead of the check run's 1800 seconds, which check mode keeps. A run that reaches it is stopped with its whole process group and the record settlesfailedwith the verdictthe run timed out after 21600s and was stopped; the run's output now goes to its log as it comes, so a stopped run keeps what it printed. Before, a run past 1800 seconds was killed, its record stayedrunningwith no outcome, and the CLI's log of it was lost (#359)- Every command that reads a project now checks its
[tool.cloudfall] formatfirst, ascloudfall config validatealready did:cloudfall audit,deploy,add,healthand the rest,cloudfall mcp serveonce at start, andcloudfall-engine inventory renderandartifact build. A project written by a newer Cloudfall, one below the formats this release reads, or one with a malformed format stops the command with exitPROJECT_FORMAT_UNSUPPORTED(95, now oncloudfall-enginetoo) before the project is validated, before anything is written under itstmp/and before any Ansible run, andcloudfall mcp serverefuses to serve it. A format 0 project (no[tool.cloudfall]table, or nopyproject.toml) still runs every command, each now answering with onePROJECT_FORMAT_OLDwarning namingcloudfall config upgrade.cloudfall init,changelog,--schemaandcloudfall-engine manifestread no project and run whatever its format (#379) - The logging backend and collector roles declare Debian 13 (trixie) next to Debian 12, like every other bundled role, so a trixie host built from the bundled Debian 13 baseline can carry the logging stack. Loki, Grafana and Alloy come from Grafana's distribution-independent APT repository; Prometheus comes from the host's Debian release, so a
software.prometheusPackageVersionpin must name a version that release ships
Security
- An owner role's password no longer reaches the PostgreSQL host or server:
services.ymland every path that runs it compute the password's SCRAM-SHA-256 verifier on the controller (a fresh 16-byte salt, 4096 iterations, the password prepared with SASLprep as PostgreSQL does) and stage that verifier in the0600file instead of the password, soALTER ROLE ... PASSWORDsends a verifier PostgreSQL stores as it is and the plaintext can't show in the server log underlog_statement = 'ddl'or'all', in a failing statement, or inpg_stat_activity; answers and receipts still name the reference, the key, and the password's SHA-256 (#140) - A declared config value is never run as an Ansible template:
cloudfall-engine inventory renderand the overlaycloudfall observewrites mark every string unsafe ({"__ansible_unsafe": ...}), so a value holding{{ ... }}reaches the unit file, the release command and the health check as written. Before, aservice.command,healthCheck.commandorrelease.commandargument such as{{ lookup('pipe', '...') }}ran on the management host duringcloudfall healthorcloudfall deploy, and an honest one such asdocker inspect --format {{.State.Health.Status}} appfailed with a template syntax error. A declared value that relied on Ansible evaluating{{ ... }}is now literal, and the rendered inventory's strings read as{"__ansible_unsafe": "..."}in its JSON and inansible-inventory --list(#174) - A catalog operation's inputs are never run as an Ansible template:
cloudfall operations propose,cloudfall decisions approveandcloudfall agent investigatehand them to the team's playbook in--extra-varswith every string marked unsafe ({"__ansible_unsafe": ...}), so an input holding{{ ... }}reaches the playbook as written, and integers and booleans keep their types. Before, an input such as{{ lookup('pipe', '...') }}ran on the management host with the operator's rights when the playbook started, and an honest input holding{{failed with a template syntax error. A team playbook that relied on an input being evaluated as Jinja now gets the literal text. The decision record'sinputsare unchanged, and recordings made before this change replay as before (#181) cloudfall data copyandcloudfall migrate --datano longer put the source database URL, password included, on the command line ofpsqlandpg_dump, where every local user could read it in/proc/<pid>/cmdlinewhile the copy ran: the URL becomes a0600libpq service file on the target host, named to the clients throughPGSERVICEFILEandPGSERVICEand removed when the copy ends, on failure too. A URL libpq can't read (another scheme, a bad percent-encoding, a parameter set twice) is refused before anything is staged, without quoting it (#355)- A declared operation's check diff and the logs of its approved run and verify step are no longer written to the committed
decisions/directory: they go to the project's ignoredtmp/output/decisions/(0600), and the decision record keeps each file's path, sha256 and size as before, so a secret a team's playbook prints with--diff(a templated password withoutno_logordiff: false) never reaches git. A--decisionsfolder elsewhere in the project keeps its output under the same path belowtmp/output/; one already undertmp/keeps it beside its records.decisions approverefuses withPRECONDITIONanddecision_diff_missing, naming the file and its sha256, when the raw diff is not on this machine, such as on another clone. Records written before keep citing their files indecisions/and are read and approved in place; a file of theirs already committed stays in git's history, so one that may hold a secret is for you to purge (#357)
Fixed
cloudfall auditno longer reports drift on a server whose PostgreSQL service declaresspec.postgresql.remoteAccess:firewall.allowedInboundexpects onetcp/<port> from <cidr>entry per declared source and reads theip saddr/ip6 saddrmatch of each accepted nftables rule (a single address printed bare by nft compares equal to its/32or/128), andservices.bind[<id>]requires a listener on the declaredlistenAddressas well as on the bind address, reportingloopbackOnly: falseand aremoteAccess.listenAddresskey for such a service; a missinglistenAddresslistener, any other non-loopback listener, an unrestricted rule on the port and an undeclared source are drift. Servers withoutremoteAccessaudit exactly as before. The engine and the audit now derive the source-restricted rules from one SDK function,remote_access_rules_for_server(#143)- A
pg_hba.confreload that failed is no longer forgotten:services.ymlleaves a marker file at/var/lib/cloudfall/postgresql/<service>.pg_hba-reload-pendingwhen it closes remote access after a failed check or reload, and the next converge checks and reloadspg_hba.confeven though the file is unchanged, then removes the marker, so the oldhostssllines no longer stay live until a restart (#144) cloudfall config validaterefuses an owned database or owner namedall,sameuser,samerole,samegrouporreplicationon a PostgreSQL service withremoteAccess, asCONFIG_INVALIDwith codeservice_remote_access_name_reserved, instead of leaving the refusal to the converge; withoutremoteAccessthose names still validate (#144)- A first deploy over a hand-made systemd unit of the same name no longer loses it when the release fails its health gate: the deploy role keeps a unit without the
# Managed by Cloudfallheader at/etc/systemd/system/<name>.service.pre-cloudfallbefore installing the managed unit, and on a failed first release copies it back, restores its enabled state and starts it again when it was active; check mode reports the unit it would keep, and a hand-made unit that differs from an earlier kept copy refuses the deploy (#151) - The data copy preflight now names an extension that a selected table needs through an array or a domain of the extension's type, such as a
citext[]column or a column of a domain overcitext(chained domains included), underextensionsand passes it topg_dump --extension; before, only a direct use of the extension's type was seen, so the copy skipped the extension and the restore failed (#173) cloudfall migrate --yesno longer crashes withHANDLER_CRASHEDwhen it reaches itsauditorverify-routesstep on a project with no server observations: the step fails withmigrate_observations_missing, naming the missing directory andcloudfall observe, and a rerun afterobserveresumes at that step (#183)- Decision records and operator proposal receipts are written through a temporary file and renamed into place, so a crash or a full disk mid-write no longer leaves a truncated record; a new record is created exclusively, so two writers of one id can no longer both succeed, and each read-modify-write of a record holds its directory's lock (#217)
- An approval is recorded, with the new status
running, before the first playbook or the operator's executor starts. A run killed midway staysrunningand is refused bydecisions approveandoperator approvewithPRECONDITION(decision_run_interrupted,operator_proposal_run_interrupted) instead of being approvable and run a second time;runningis added to both record schemas (#217) - One decision record or proposal receipt that cannot be read no longer stops every reader of the store:
operations propose,decisions list,whyandoperator listgo on with the records they can read and add aRECORD_UNREADABLEwarning naming each skipped file and why; an operator pass reports the receipts it skipped underunreadable, and its autonomy pass grants nothing while a receipt cannot be read.decisions showanddecisions approvestill answerRECORD_INVALIDfor the one record they read (#218) - Each record is read with the schema its
apiVersionnames, so a later schema version can sit besidecloudfall/v1; a record of a version this Cloudfall does not read is reported, not fatal. A golden set of records in the shapes released versions wrote is loaded by the tests (#218) deploy,rollbackandrestartcontact only the component's own servers (itscomponent_<id>inventory group, still in inventory order), and a backup contacts only the service's server, so an unreachable server the component or service is not on no longer fails them; a component with no servers in the inventory, or a service on no inventory server, now fails before any host is contacted (#220)- A component on several servers no longer ends a failed deploy on two releases: when a later server fails its health gate, the servers that switched to the release earlier in the run go back to the release and service unit they had (a first release is stopped), last one first, each gated again, and the failure names them (#222)
- The release receipt
<receipts>/<component>.jsonnames every server of the deploy instead of only the last one, which overwrote the others (#222) - A failed deploy whose component has a release command says the command already ran and is not reverted; the engine README says a release command must work with the previous release too (#222)
- Deploying the release that is already
currentunpacks into a freshreleases/<release>.redeploy-<UTC time>directory instead of writing into the live one, so a failed health gate rolls back to the untouched copy;rollbackto that release id takes its newest copy, and receipts still name the release id (#223) - A failed health gate restores the service unit the previous release ran under, not only the
currentsymlink, then gates the restored release on its health check; a restored release that fails it fails the deploy with its own message (#223) deploy.yml --checkruns to its end: the dry run reports the uv release it would install and skips the digest check, the unpack anduv syncof an archive it never transferred (#223)- Old releases are removed: once a deploy passes its health gate on a server, it keeps
deployment.retainReleasesreleases underreleases/and removes the older ones, a release's redeploy copies with it, always keeping the deployed release,currentand the release before it whatever their order; a failed gate removes nothing, check mode only reports what would go, and a directory that is not a release id is left alone (#224) - A rollback to a release that is no longer on the server says it may have been removed by a deploy's cleanup and names the releases still kept there (#224)
- The PostgreSQL restore drill compares the row count of every table the dump holds with the restored table and fails on a difference or on a dump without a table; it counted
pg_classrows before, which an empty database has too, so a restore that brought back nothing passed (#225) - The restore drill's scratch database keeps to PostgreSQL's 63-byte names (
<database>_restore_check, the name cut to 49 characters), andcloudfall config validateand the converge refuse a declared database named as another's scratch database, asCONFIG_INVALIDwith codeservice_database_restore_check_name(#225) - The test suite passes on a checkout whose
config/examples/tmp/holds rendered inventory or observations (astask inventoryand CI leave it): every test that copies the examples now goes through one helper,sdk/tests/examples_project.py, that leaves that folder behind, and a test refuses a direct copy (#226) - Docs that no longer matched the code:
ARCHITECTURE.mdnow says a project pinscloudfall==<version>, that decision records are committed while runtime state stays undertmp/, and whichTaskfile.ymltasks call Ansible directly; theAGENTS.mdthatcloudfall initwrites names thecloudfall==<version>pin instead of arevpin;CONTRIBUTING.mdnamesconfig/instead of astate/directory; the CLI evaluation underevaluations/cloudfall/uses repository-relative paths (#229) - An operator proposal left
runningno longer blocks the operator for good: once closed asinterruptedit stops blocking new proposals for the same alert or drift, and the refusals ofdecisions approveandoperator approvename the command that closes it (#244) - A decision record or proposal receipt write now flushes its directory after the rename, so a power loss cannot bring the old record back (#244)
- A redeploy copy (
releases/<release>.redeploy-<UTC time>) that fails its health gate is removed by the deploy's rollback once the previous copy runs again, sorollback <release>takes the newest copy that passed instead of switching to the failed one; the copy's time now has microseconds, so two redeploys in the same second get two directories (#245) healthcontacts only the component's own servers and fails before any host when the component has no servers in the inventory, instead of reporting success with nothing probed; anddeploy,rollbackandrestartstop at the first of the component's servers they could not reach, where before Ansible skipped the unreachable server and went on to the next one (#246)- The PostgreSQL restore drill counts each restored table without its inheritance children (
count(*) FROM ONLY), as the dump's COPY data holds them, so a database with classic table inheritance no longer fails the drill every night after a correct restore (#247) config validaterefuses a storage pool device path with.., avolumeGroupof.or.., and a pool naming one device as bothprimaryandsecondary(storage_pool_same_physical_volume); before, the first two passed validation and crashedrequirements show(#248)baseline,bootstrap,services,data,domains,logging,timeandbackupstop at the first server they could not reach, asdeploy,rollbackandrestartdo since #246, where before Ansible skipped the unreachable server and changed the next one; the check lives in one shared task file,playbooks/tasks/rollout-start.yml, andinspectstill goes on past a server it cannot reach, reporting it as missing (#254)- An operator proposal closed as
interruptednow suspends autonomy for its operation kind the way afailedone does, until a human-approved run verifies, and an autonomous run closed asinterruptedcounts towardmaxAutonomousPerHourfrom itsinterruption.confirmedAt; before, a killed unattended run left the next proposal for the same alert free to run unattended while what the first run changed was unknown (#258) cloudfall lock clear <component>no longer deletes a server's lock that a run took again after it was shown:lock.ymlcompares the shown token and removes the lock in one command on the server, instead of reading it and deleting it in two tasks, so a lock taken since is kept and the command exitsCONFLICT; and with no lock found while a server could not be reached it exitsCONFLICTwithcomponent_lock_not_clearedandunreadServers, instead of answeringnoopas if no run held the lock (#271)cloudfall baseline convergekeeps a host's own smartdDEVICESCANoptions (-sself-test schedules,-Wtemperature limits) and replaces only its-M exec, adding-m rootwhen the line has no-m; a failed restart of mdadm's monitor or smartd now fails the run instead of reporting success, and only a unit the host does not have is skipped (#281)logging deploy's plan shows each receiver channel's bot token: a newchannelsfield lists{channel, tokenStored}, and a token not in the store is inmissingastelegram-bot-token of <channel>, the name--yesrefuses with, where the plan used to say nothing was missing (#282)- A command that runs a playbook on one server refuses a Server whose id is one of Ansible's implicit group names (
all,ungrouped) withlifecycle_limit_reserved_groupbefore anything runs:--limit allreached every host and--limit ungroupednone (#283) - A storage pool's
primaryandsecondarymay be stable paths such as/dev/disk/by-id/nvme-<model>_<serial>-part5:logging deployandinspectresolve each one on the host withrealpath, the volume provider and audit'svolume.<name>compare the device it names with LVM's physical volumes,lvcreateandlvextendname the physical volume as LVM reports it, and audit shows each declared path next to its device (resolvedPhysicalVolumes); before, any path but the kernel name failed with "is not in vg_data", and NVMe kernel names can swap between boots. A pool path that does not exist on the host, and a pool whoseprimaryandsecondaryresolve to one device, are refused naming the server, pool and path (#293) logging deployholds Loki, Grafana, Alloy, and Prometheus and Alertmanager when they have a declared version, at the LoggingStack's declared versions: each role writes/etc/apt/preferences.d/cloudfall-logging-<package>(Pin: version <declared>,Pin-Priority: 999) before its pinned install, so a laterapt-get dist-upgradeno longer moves them, and removes the file of a package now installed without a declared version;cloudfall auditadds alogging.packages[<package>]check per pinned package on each of the stack's hosts, reporting drift when the installed version is not the declared one (#294)logging deploychecks, before anything is installed, that each service user can reach the files it reads (Grafana's admin password, Alertmanager's SMTP password and Telegram token files, Alloy's client TLS files) through every existing parent directory, and stops naming the directory that blocks it, as in/etc/cloudfall is 0700 root:root; grafana can't reach /etc/cloudfall/logging/grafana-admin-password, where Grafana used to fail only when it started; the host's directory modes are left as they are (#295)cloudfall-engine playbook run <playbook> --engine <dir>runs a playbook of an engine directory that has noservices.yml, where it used to crash withHANDLER_CRASHEDandplaybook does not exist: .../services.ymlwhatever playbook was asked for; aservices.ymlrun still decrypts and stages the owner passwords as before (#296)logging deployno longer leaves a host whose pinned logging package is already newer than declared (Grafana 13.3.1 installed, 13.2.6 declared) with a hold that makes every laterapt-get dist-upgradeand unattended-upgrades abort on a downgrade: the holds are written atPin-Priority: 999, which keeps a held package from moving up without ever forcing a downgrade, and before writing them each role compares the installed version with the declared one (dpkg --compare-versions, a dry run too) and stops on a newer one, naming the package and both versions and saying to declare the installed version or downgrade the package by hand, with no hold written on that host (#302)- A missing Elasticsearch exporter credential no longer stops Alloy on the whole host: the collector's systemd drop-in gives each
LoadCredential=aSetCredential=default, so when the node'scloudfallpassword or project CA is gone (the service taken out of the project, its files cleaned up) Alloy still starts and only that exporter's scrapes fail; the nextlogging deploydrops the exporter and its credential lines (#326) - Adding or removing
remoteAccesson an Elasticsearch service withmetrics.enabledno longer leaves the collector scraping the node the old way until the nextlogging deploy: the services converge re-renders the Alloy configuration and its systemd drop-in on each logging collector wherelogging deployalready set Alloy up, validates it, and restarts Alloy only when a file changed, so the exporter switches between plain HTTP and HTTPS with the project CA in the same run; a host that is not a collector, or whose Alloy is not set up yet, is left untouched (#329) - The PostgreSQL restore drill (
cloudfall backup verifyand the nightly restore-check timer) passes--file=-topg_restorewhen it reads the dump's row counts; without it every PostgreSQL since 12 refuses withone of -d/--dbname and -f/--file must be specified, so the drill failed on every run (#340) - A bootstrap or baseline after a deploy no longer reports changes on each component's
releases/andshared/: the deploy now creates them0750, as bootstrap does, instead of flipping them to0755, so a converge after a deploy is idempotent; the Nginx worker still reaches a static route's release through its own ACL entry (#341) - On a host running more than one service, such as
h1inconfig/exampleswith PostgreSQL and Redis, the Redis backup no longer fails withPermission denied: the PostgreSQL, Redis and Elasticsearch roles now create the shared backup root/var/backups/cloudfallroot-owned0755before their own backup directory, instead of letting the first service's directory task create it with its own owner and0700, and the next services converge repairs a root already owned bypostgres; each service's own directory keeps its owner and mode (#342) - An approved operator proposal now runs on the one server it names: every playbook of
converge-servicesandconverge-baseline, the Elasticsearch node preparation included, runs with--limit <server>, a drift proposal's verify step inspects that server alone, and the receipt names only the owner passwords set there; before, a staging alert approved by the staging policy converged every declared server, production included (#350) - Operator autonomy is now earned per environment: the history an
OperatorPolicygrant is judged on, and the rule that suspends it after afailedorinterruptedreceipt, count only the receipts of servers in the policy's environment, and a receipt of a server no longer declared counts for none; before, verified runs on staging earned autonomy in production, and a failed staging run suspended production (#351) decisions approveruns only what was checked: a proposal's record now binds the sha256 of its playbook and verify playbook and, when the project is in a git work tree, its HEAD commit and a digest of its uncommitted changes (leaving out the decisions directory,record/andtmp/), and approve recomputes them and the diff file's sha256 before anything runs; any difference is refused withPRECONDITIONanddecision_binding_changed, naming what changed and asking for a new proposal, and the record staysproposed. Before, an edited playbook, role or diff file ran unseen under the old diff's approval. Outside git only the two playbooks and the diff are bound (#352)- A failed deploy or a rollback brings the previous environment back with the previous release: each release now reads its own environment file,
shared/env/<component>.<release>.env(a redeploy copy gets one of its own), named by its unit and read by its release command and command health check, so a deploy that fails only because of a bad environment value leaves the previous release running, and restarting, on the environment it had;rollbackpoints the unit at the target release's file. A deploy without--env-filecopies in the file the previous release reads, a component with no environment still namesshared/env/<component>.env, which no deploy writes or removes any more, and retention removes a removed release's environment files (#353) - A baseline run now removes an SSH key declared
lifecycle: revokedfrom root'sauthorized_keys, after the active keys are in, matching the key's type and body whatever its comment or options; keys the project doesn't declare stay (D-22). The rendered Ansible inventory'scloudfall_ssh_public_keysnow carries a server's revoked keys too, each with itslifecycle, so the role can remove them (#354) - Two proposals of one operation in the same second no longer share a decision id: the id is reserved before the check run writes anything, and a second proposal in that second takes
<operation>-<stamp>-2, then-3, so each record keeps its own diff and host stats, instead of the second overwriting the first one's diff and then failing withdecision_exists; a proposal whose check run fails frees its id and leaves no record (#356) - An operator verify step that errors (the alert feed unreachable, the audit failing) now settles the receipt
failedwith the error as its detail, so autonomy for that operation kind is suspended, and the command still fails with the error; before, the receipt stayedexecutedfor good and blocked its trigger (#358) decisions approveno longer settles a decisionverifiedwhen its verify step observed no host: every host the check run or the approved run left recap counts for must have counts from the run and a task the verify step ran on it, and a verify play that matched no host, skipped a targeted one or skipped every task on it settles the decisionfailed, with a verdict naming the hosts that were not observed. Before, a verify step that exited 0 and listed no host as changed proved the run, even when it looked at nothing. A verify playbook aimed at other hosts than the run's (such aslocalhostchecking a URL) now settlesfailedtoo; aim it at the hosts the operation targets (#360)- Docs: D-10 in
docs/decisions.mdread as if every component already ran in the full systemd sandbox; a status line now says the unit sets onlyUser,Group,NoNewPrivileges=trueandPrivateTmp=true, that secrets reach it through the per-releaseEnvironmentFile(D-21) and deploy tasks run as root, and that the rest of the sandbox is not built yet (#362) - Docs gave the deploy order as health-check gate before symlink switch:
ARCHITECTURE.mdnow says the switch and the restart come first, so the gate runs on the live release, andARCHITECTURE.md,docs/ci-deploy-guide.mdand the Render and Neon migration guides say a rollback, automatic orcloudfall rollback, does not undo the release command, so a migration must stay compatible with the release before it (#363) - A deploy no longer fails on one slow download of the pinned uv release: the deploy role retries the download five times, ten seconds apart, still checking its SHA-256, before failing, and a test over every role now requires each
get_urldownload to retry a bounded number of times (#372) - A redeploy copy (
releases/<release>.redeploy-<UTC time>) whose release command fails is removed with its environment file, as one that fails its gate is, sorollback <release>takes the newest copy that passed instead of starting one that never passed its release command or gate; the previous copy and its own environment file stay (#373) - A deploy run with Ansible's
--diffno longer prints the component's environment file, secrets included: the tasks that install a release's environment file or carry the previous one over setdiff: false(#374) - An operator proposal recorded with the audit check
server_type.idkeeps verifying oncebaseline.idis compliant, and keeps blocking a new proposal for the same drift, with no removal date instead of until 0.9.0, since a record's fields are never deprecated withincloudfall/v1(#387) - A re-converge of a PostgreSQL, Elasticsearch or logging service no longer needs the network for its package repository's signing key: the PostgreSQL, Elastic and Grafana roles download the key only when it is missing (a
get_urlwithoutforcestill asked the server whether the key had changed, so a slowwww.postgresql.orgfailed a converge that should change nothing), keep the installed key's owner and mode, and retry a failed download five times, ten seconds apart, before failing (#388) - Docs promised a gate the agent does not control for every server-changing action:
ARCHITECTURE.md,docs/brownfield-design.mdandMANIFESTO.mdnow say it holds for declared operations (a person approves at a terminal) and the operator (a person at a terminal or anOperatorPolicygrant), while in project mode thecloudfall mcp serveconfirmation (yes: true,confirm_destructive: true) is the agent's own, except for a release whose application declaresapproval: manual; policy-gated self-confirmation is planned (#391) SECURITY.mdsays who the private-reporting rule is for: outside reporters use private vulnerability reporting; the maintainers track findings that need existing access as public issues, and keep findings a stranger could exploit remotely private until a fix is released (#392)- A PostgreSQL owner role whose database names a
passwordRefno longer reports a change on every services converge: the role reads the owner's stored verifier, computes the password's SCRAM-SHA-256 verifier on the controller with that verifier's salt and iteration count (a fresh salt when the role has no SCRAM verifier), and runsALTER ROLEonly when the two differ or the role can't log in, so a converge with the same password changes nothing (D-25). The controller now stages the password itself in its private0600file, which never leaves the controller; only the verifier reaches the host, as before (D-9). A stored password that is neither a SCRAM-SHA-256 nor an md5 verifier stops the run, naming the role, never the value (#398) auditnow seesREQUIRE SSLremoved by hand from a MariaDB owner account at an allowed source (ALTER USER ... REQUIRE NONE): inspect reads each remote account'sssl_typeinto a new optionalsslTypeslist ofmariadbAccounts, and themariadb.accountscheck reports drift for a remote owner account whosessl_typeisn'tANY(what converge'sREQUIRE SSLsets), naming the account and its type. A record an older inspect wrote, which has nosslTypes, now reports that check as unknown instead of compliant when the service has remote owner accounts, until the next inspect; local accounts are checked as before (#435)domains.ymlnow fails before it changes any server when a Domain it would render has noproxy.upstream, and names every such Domain; before, the site role's assert stopped the run only on that Domain's server, afterserial: 1had already converged the servers before it.upstreamstays optional in the schema, for a Domain the fleet routes with its own proxy (#90)- A deploy of a component with
healthCheck: {type: none}no longer fails its health gate on the missing HTTP fields and stops the release: the gate now requires the systemd unit to be active right after the restart and still active, with no automatic restart in between, 10 seconds later, so a worker that crashes or crash-loops on start still fails and rolls back (#96)
Deprecated
kind: ServerType, theserver-types/folder,Server.spec.serverType, thecloudfall_server_typescatalog variable and a host'sserver_type,cloudfall add server-typeandcloudfall add server --typekeep working until 0.9.0;cloudfall config validatewarnsDEPRECATED_NAMEfor each file using an old name, and a server setting bothserverTypeandbaselinefails withserver_baseline_conflict(#198)- Until 0.9.0 the old output names stay beside the new ones with the same values: the rendered Ansible inventory's host variable
cloudfall_server_type,inventory show'sserver_typesand each server'sserver_type,config validate'sby_kind.ServerType, andaudit's per-serverserver_type; an observed server snapshot recorded withspec.serverTypeis still read asspec.baseline, and one with both fails withobservation_baseline_conflict(#198)