Repository navigation
cloudfall 0.8.0rc83
Pre-release
Pre-release
·
20 commits
to main
since this release
Changes since v0.7.0:
Breaking
deploy --yesandmigrate --yeson an application that declaresapproval: manualnow need a person's approval of the release: without acloudfall release approverecord that counts (the same archive SHA-256, exactly the component's declared servers, not expired),deploy --yesexits 4PRECONDITIONwithdeployment_approval_missingbefore any playbook runs, naming therelease approvecommand, and lists each record that doesn't count withdigest_mismatch,servers_changedorexpired. This hits every project that declaresmanualand deploys today, the examplecrm.yamland every Render import included. Two ways out: keepmanualand approve each release first (build it, runcloudfall release approveat a terminal, commit the record underdecisions/releases/when CI deploys, then deploy), or switch the application toapproval: automaticwhen another gate, such as GitHub's required reviewers, is the real one. Thedeployplan and result gainapproval(policy, and formanualthestatus, the counting record'spath,approverandexpires_atorapproved_at, the records that don't count and thecommandto run),deploygains--approvals(defaultdecisions/releases) and theRECORD_INVALIDexit for an approvals file that isn't a validReleaseApproval, andcloudfall mcp servebinds the approvals folder (its own--approvals) so adeploycall can't name its own; anautomaticapplication's deploy never reads the folder (#163)- A
LoggingStacknow needs a storage pool on its backend's baseline:cloudfall config validaterefuses aLoggingStackwhose backend server'sBaselinedeclares nostorage.poolswithCONFIG_INVALIDand codevolume_requirement_pool_missing, so a project whoseLoggingStackvalidated on 0.8.0rc22 stops validating. Before upgrading, declare the volume group the Loki and Prometheus volumes go into as a pool on that baseline (storage.poolswithid,volumeGroupandprimary); with more than one pool, name it on each volume inspec.backend.storage.poolandspec.metrics.backend.storagePool, or validation refuses withvolume_requirement_pool_ambiguous(S-014-2) cloudfall config validaterefuses one SSH key (same type and body, any comment) declaredactiveandrevokedin the same environment, withCONFIG_INVALIDand codessh_public_key_lifecycle_conflict, so a project that declares a key that way stops validating; before, the baseline installed it. Before upgrading, delete the revoked declaration if the key should keep access, or mark the active onerevokedif it should not (#354)- Some source URLs that
cloudfall data copyandcloudfall migrate --dataread through--dbnamebefore are now refused, because a libpq service file can't hold them as written: a#in the database name or a query parameter value, a password that ends in whitespace, and a value holding a line break. The copy stops before anything is staged, withlifecycle_execution_failedand a reason that names the URL part without quoting it. Before upgrading, percent-encode a#there as%23; a password that ends in whitespace or holds a line break can't be staged even percent-encoded, so change that password first (#355) - Over
cloudfall mcp servein a project, a confirmed call ofdeploy,rollback,restart,migrate,data_copyordomains_apply(yes: true), or ofconverge_baseline,converge_servicesorconverge_domains(confirm_destructive: true), now runs only when anOperatorPolicygrant covers it in every environment the run touches. Any other confirmed call runs nothing: it answersPRECONDITIONwitherror.context.code: operator_approval_required,context.proposal(the id of aproposedagent record it left in the proposal store,record/operator/proposals/by default),context.command(cloudfall operator approve <id>, which a person runs at a terminal) andcontext.reasons(one line per environment, naming why no grant covers it). The same call made again while that proposal isproposed, against the same plan, commit and uncommitted changes, returns the same id and writes nothing. A run that changes no declared server, such as amigratewhose pending steps are builds and controller steps likettl-lower, touches no environment whose grant could cover it, so it always proposes: its record'soperation.serversandoperation.environmentsare empty (theOperatorProposalschema now accepts empty lists there) and its reason says only a person approves it. Aconverge_*call that refuses itself before any playbook, such asconverge_domainsfor an undeclared Domain, still refuses as before and writes no record. With noOperatorPolicydeclared every confirmed call proposes; to keep the one-call flow in an environment, declare anOperatorPolicyfor it whosespec.autonomy.operationsgrants those classes (deploy,rollback,restart,migrate,data-copy,domains-apply,converge-baseline,converge-services,converge-domains) and earn itsrequiredVerifiedRuns. The checks that came first still do and write no record: argument and config validation, S-008'sdeployment_approval_missing, and a component lock another run holds. The server's instructions say so, and the plan'sapproval.selfConfirmtells an agent in advance. Amigratewith no pending step left still answerscompleteand writes no record, since there is nothing to run. On the CLI, a command without--proposalsruns as before (cloudfall deploy --yesin a shell runs the deploy), while one given--proposalsand--yes(deploy,rollback,restart,data copy,domains apply,migrate) is gated like the MCP server's call: no grant, and it proposes instead of running. A repository's server lists the same tools (#476)
Added
- A Component can declare
spec.buildTools(tool name to minimum version),spec.build(ordered steps, each aworkingDirectoryinside the packaged root, an argvcommandand an optionaltimeoutSecondsfrom 1 to 3600, default 900) andspec.artifact.include(relative paths the build produces);cloudfall config validaterefuses a step whosecommand[0]is not inbuildTools(component_build_tool_undeclared), an absolute or..path, andbuildToolsorartifactwithoutbuild. Validation only:artifact builddoes not run the steps yet. Components withoutbuildvalidate as before (#113) cloudfall-engine artifact buildruns a Component's declaredspec.buildsteps on the controller before packaging: it checks everyspec.buildToolsentry is onPATHat its minimum version before cloning (artifact_build_tool_missing,artifact_build_tool_unsupported), runs each step in the clean checkout with onlyPATH,HOME,LANG,TMPDIRandCI=truein its environment and its output on stderr, fails without writing an archive on a non-zero exit or a step outrunning its owntimeoutSeconds(artifact_build_step_failed), a missing include path (artifact_include_missing), an include path that leads out of the checkout through a symlink (artifact_include_outside_checkout) or a tracked file a step deleted (artifact_tracked_file_missing), packages the tracked files plus everyspec.artifact.includepath (a gitignoredweb/dist, say), and records the steps and tool versions as the new optionalspec.buildof the Artifact metadata. The command's time limit is now the clone time (1800 seconds) plus every declared step timeout, on the CLI and the SDK/MCP path alike; an explicit--timeoutreplaces it, and--timeout 0still disables it. A Component withoutbuildis packaged as before (#115)- The
converge_domainsMCP tool takes an optionaldomain: given,domains.ymlruns only on that Domain's proxy server (--limit) and renders only that Domain, and an unknown id exitsPRECONDITION(lifecycle_domain_missing) before the inventory render; omitted, it runs on every server as before. Each domain receipt now recordspreviousConfigurationSha256, the sha256 of the vhost file the run replaced, ornullwhen none was there; receipts without it still load (#116) cloudfall domains apply <domain> [--issue-certificate] [--yes]converges one Domain:domains.ymlruns on its proxy server alone and renders only it, so no baseline, firewall or service task runs. Without--yesit exits 0 with a plan.--issue-certificaterefuses a Domain withtls.mode: disabled, and first checks on the controller that the primary name and every alias resolve to the proxy server (PRECONDITION,domains_dns_unverified, naming each failing name); aproxiedDomain skips the check (dnsCheck: "skipped-proxied"). With--yesit answers the receipt path under--deployments(#117)cloudfall domains observe <domain>reads one Domain on its proxy server alone, through the new read-onlydomains-observe.yml: the vhost file and its sha256, the certificate's expiry,certbot.timer, and whether the vhost still matches the last receipt (receipt: nullbefore the first). It changes nothing on the server or in the project, and a missing vhost exits 0.cloudfall mcp servelistsdomains_apply(with itsyesgate) anddomains_observeas tools (#118)- Components: a
release.command(with optionalworkingDirectoryandtimeoutSeconds, 1 to 1800, default 300) runs once per deploy on the first server ofdeployment.servers, which now deploys first, after the release's dependencies are materialized and before the unit file changes orcurrentmoves; it runs as the application's user with the component's env file loaded under systemd'sEnvironmentFile=rules. A non-zero exit or a timeout fails the deploy withENGINE_STEP_FAILEDbefore any server switches, and a passing run is recorded asreleaseCommand(server, command, exit code, last 2000 characters of output) in every release receipt of that deploy (#119) deploywithout--yesshows the component's release command asrelease_command(its command, the working directory resolved under the new release directory,timeout_secondsand the release server), ornullfor a component without one, and says inwould_runthat it runs once on that server before any switch; therollbackplan of a component with a release command showsrelease_command_skipped: true, since a rollback never runs it (#120)- A PostgreSQL
Servicedatabase may name anownerrole and itspasswordRef(environment,path,keyin the secrets store) instead of anapplication, so a project of a server and its database alone validates;spec.postgresql.remoteAccessdeclares a listen address and the client networks allowed in.config validaterefuses an entry with both or neither ofapplicationandowner, anownerwithoutpasswordRefor apasswordRefnext toapplication, a reserved owner (service_database_owner_reserved), and aremoteAccesswith no owner database, no firewall, the port already open, or a source of every address. Converging these declarations comes later: until then the PostgreSQL role refuses them before any task runs (#121) services.ymlconverges a PostgreSQL database owned by a declared role: it creates theownerlogin role, sets its password frompasswordRefas SCRAM-SHA-256 on every converge, and gives it the database; backups, the restore drill, anddata copy(restoring as the owner, its receipt'sspec.owner) cover it like any other database. Every path that runsservices.yml(theconverge_servicesMCP tool, theservicesstep ofcloudfall migrate,operator approveandoperator run, andcloudfall-engine playbook run services.yml) reads the password from a--secrets-dirflag (defaultsecrets) and stages it in a 0600 file in a private temp dir outside the project, removed after the run; the value is in no inventory, command line, Ansible output, answer, or receipt, and the answer'sowned_rolesnames the reference, the key, and the value's SHA-256;operator approverecords them asownedRolesin the proposal receipt'soutcome(an optional property) andoperator runin each executed proposal's entry. A missing secrets file or key fails withPRECONDITIONnaming the expected path and key before any playbook runs (#122)- A PostgreSQL service with
spec.postgresql.remoteAccessopens to its declared sources over TLS:services.ymlsetslisten_addresses = '127.0.0.1,<listenAddress>'andssl = onwith the Debianssl-certsnakeoil pair (restarting the cluster), and writes onehostssl <database> <owner> <cidr> scram-sha-256line per owned database and source between Cloudfall markers inpg_hba.conf, reloading only afterpg_hba_file_rulesreports no error;baseline.ymladds oneip saddr/ip6 saddr <cidr> tcp dport <port> acceptrule per source, rendered as acloudfall_firewall.allowedInboundentry with a new optionalsourceslist. RemovingremoteAccessresets the listener and removes thepg_hba.confblock and the firewall rule; databases named byapplicationstay socket-only (#123) cloudfall data copytakes a repeatable--source-schema NAME(and thedata_copyMCP tool asource_schemalist): the dump and restore copy only the named schemas, the empty-target check and the per-table row-count check read exactly those schemas, and the plan without--yesnames them; a name that is empty, longer than 63 bytes, or holds a control character or a brace is refused as a usage error. Without it the dump is the whole database as in 0.7.0, and the checks now cover every non-system schema of the source instead ofpubliconly, so a row-count mismatch outsidepublicnow fails the copy. TheDataMigrationReceiptgainsspec.schemas(sorted) and aschemaon eachspec.tables[]entry besidetableandrows(#124)cloudfall data copy, and thedata:<database>step ofcloudfall migrate, run a preflight on the target host after the empty-target check and beforepg_dump: it reads both catalogs and refuses a copy the target can't hold, with nothing dumped, the target untouched and no receipt, exitingPRECONDITIONwith codedata_copy_preflight_refusedand four lists in the error context, each item named:missingSchemas(a named schema the source lacks),extensions(one the selected schemas need, throughpg_dependor by living in them, or every source extension butplpgsqlwithout--source-schema, that the target'spg_available_extensionslacks),roles(a role a row-level security policy on a selected table names that the target'spg_roleslacks) andauthReferences(a function or policy of the selected schemas that referencesauth.whenauthisn't copied). The role writes the findings to<service>-<database>.preflight.jsonunder--data-migrations, so the lists arrive whole. With--source-schema,pg_dumpalso gets one--extensionper extension the schemas need (pg_dump 14 or later), andpg_restorenow runs with--single-transaction, so a failed restore rolls back whole, writes no receipt, exitsENGINE_STEP_FAILED, and the next run passes the empty-target check (#125)cloudfall migratetakes a repeatable--data-schema DATABASE=NAMEbeside--data DATABASE=PATH(and themigrateMCP tool adata_schemalist): thedata:<database>step copies only the named schemas, asdata copy --source-schemadoes, and the plan names them. A--source-schemaor--data-schemaname must now match^[a-z_][a-z0-9_]*$and be at most 63 characters, and may be given once per copy; a name that doesn't, a repeated name, or a--data-schemafor a database with no--dataexitsPRECONDITION(codedata_copy_schema_invalidormigrate_data_schema_without_data) before any playbook runs, where a malformed name was a usage error and a repeat was silently dropped (#126)- A Domain can declare
spec.routes, path routes that each either proxy to anupstream(address,port) or serve a component's release files (static:component,directory, optionalspaFallback).cloudfall config validaterefuses a Domain with bothproxy.upstreamandroutes(domain_route_source_invalid), a path that doesn't start with/or uses characters outside letters, digits and._~/-, or a static path without a trailing/(domain_route_path_invalid), a path declared twice (domain_route_path_duplicate), a path under/.well-known/acme-challenge(domain_route_path_reserved), a missing static component (resource_reference_missing) or one not deployed on the Domain'sproxy.server(domain_route_component_not_on_proxy), and adirectoryorspaFallbackthat is absolute, climbs out with.., or, fordirectory, names the release root. Validation only: the domains playbook doesn't render routes yet and still refuses a Domain withoutproxy.upstreambefore any change. Domains with onlyproxy.upstreamvalidate and render as before (#159) cloudfall release approve <component> --release <release>records a person's approval of one built release as aReleaseApproval(newrelease-approval.schema.json) underdecisions/releases/: the application, component, release, archive SHA-256, git commit, servers, approver,approvedAt,expiresAt(--valid-forhours, 1 to 168, default 24) andvia: terminal. It verifies the artifact first, refuses a component of anautomaticapplication withrelease_approval_not_required, and writes the record only once a person typed the release id at a terminal;--yesnever answers it, and it is no MCP tool. It deploys nothing (#160)- Components: a
healthCheckoftype: command(a required non-emptycommandargv,timeoutSeconds1 to 300 andattempts1 to 20, no other key) validates, anddeploy,restartandrollbackwithout--yesname the check and its command in their plan'swouldRun(#161) cloudfall domains applyrenders a Domain'sspec.routes: a proxy route becomes alocationthat proxies to itsupstreamwith the full request path, and a static route serves<installRoot>/current/<directory>of its component, answeringspaFallbackfor an unknown path when it declares one and 404 when it doesn't. Without a/route, a path no route matches answers 404, not Nginx's default page. Locations render longest path first, whatever the declared order, where the single proxy location rendered before (the port 80 server without active TLS, the 443 server with it); the port 80 redirect and the ACME challenge location stay. The site role grants the Nginx worker (www-data) traverse-only access (POSIX ACLuser:www-data:--x, installingacl) from the application root to the install root and itsreleases/, so it reads the release files but nevershared/env/<component>.env, and a deploy orcloudfall rollbackswitches what a static route serves without re-running the domains playbook or reloading Nginx. The domains playbook's early check now refuses only a Domain with neitherproxy.upstreamnorroutes. A Domain with onlyproxy.upstreamrenders a byte-identical vhost (#162)- Deploy: a component's
healthCheckoftype: commandnow gatesdeploy --yes. After the switch and the restart it runs on each server in a transient systemd unitcloudfall-health-<component>, as the application's user, in<installRoot>/current, with the service's env file andtimeoutSecondsas its runtime limit, up toattemptstimes 3 seconds apart. When every attempt fails or times out, the previous release is restored (a first release is stopped), the deploy fails withENGINE_STEP_FAILEDnaming the check and its exit code or timeout, and the release receipt is written withhealthCheckPassed: false. Receipts of a command check carryhealthCheck(command, attempts, exit code, timed out), never the command's output, which stays in the server's journal. The release receipt schema acceptshealthCheckPassed: falseonly with a failed commandhealthCheck, and every receipt written before (#164) restart,rollbackandhealthrun a component'stype: commandhealth check instead of refusing it: the same transient unit asdeploy's gate, with its user, working directory, env file, timeout, attempts and redacted output.healthreportsunhealthywhen the check exits non-zero or times out, a failed check failsrestartorrollbackwithout restoring anything, as anhttpcheck does, and none of them writes a release receipt. The check starts its unit as root, sohealthof such a component needs the inventory SSH account to become root, asrestartandrollbackalready do;httpandnonechecks are unchanged (#166)cloudfall config upgrade: rewrites a project from the deprecatedServerTypenames toBaseline(movesserver-types/tobaselines/, rewriteskindandspec.serverType), keeping comments, formatting and line endings; a second run changes nothing, and it refuses, changing nothing, a move onto an existing file or a file or folder reached through a symlink (upgrade_symlink) (#198)auditchecks the certificates a server's host requirements name (present, for the right names, more than 30 days left at the observation's time) from newcertificatesevidenceinspectcollects, and expects the logging gateway's firewall rule, so a deployed stack audits compliant (S-011)cloudfall logging deployprovides the stack's mTLS material: each host makes its own key and a signing request (logging-requests.yml), the project CA signs the missing and due certificates, and they are delivered with the CA; certificates provided by hand are kept (S-011)cloudfall logging deploy: deploy the declaredLoggingStackafter meeting its secret requirements; generated secrets are written to the store, an input secret that is missing stops the run (exit 4,INPUT_REQUIRED), and each stored secret is copied to its host path without overwriting a file already there; without--yesit shows the plan (S-011)- Logging collectors pin the gateway name to the backend's address in
/etc/hosts(loopback on the backend itself), so the gateway needs no DNS record when the backend declares an IP (S-011) - The logging gateway's port opens on the backend's firewall to each remote collector's addresses only, rendered beside the
remoteAccessrules and enforced bylogging.yml; a remote collector with no IP behind a firewall failsconfig validate(logging_gateway_collector_unaddressed) (S-011) LoggingStack:migration.mode: nonefor a host with no legacy agents;parallelstill requirespreserveLegacyAgents: true(S-011)- Logging roles check every package pin against the host's APT sources before installing or configuring anything, and stop naming the versions on offer when one is missing (S-011)
- Certificate provider, controller side: a project CA per environment, kept sops-encrypted under
secrets/<environment>/pki/, that signs host requests for exactly the names and usage a requirement grants (S-011) cloudfall requirements show: the volumes, ports, secrets, certificates, DNS names and package pins each declaredLoggingStackneeds from its servers, by server (S-011)cloudfall secrets generateandcloudfall secrets input: keep the secrets aLoggingStackrequires sops-encrypted undersecrets/, generated or typed at a terminal with echo off;requirements showgives secrets a state and lists the missing ones (S-011)- A decision record and a proposal receipt name the Cloudfall version that last wrote them in
spec.writerVersion, an optional schema field: records written before it stay valid (#218) deploy,rollback,restartand an artifact build hold their component's lock undertmp/locks/on the management host until they end, passed, failed or interrupted, so a second run of the same component exitsCONFLICT(component_locked) before any server is contacted, naming who holds the lock, since when, and whether that process still runs;cloudfall lock clear <component>shows a lock a killed run left, with its age, and removes it once a person types the component id at a terminal, never while its process still runs on this host (#221)deploy.yml,rollback.ymlandrestart.ymlhold the component's lock on each server, at/var/lib/cloudfall/locks/<component>.json, from before they change anything there until that server's turn ends, passed or failed, so runs from two controllers,cloudfall-engine playbook runand approved runs of those playbooks no longer race oncurrent, the unit file or the release command's unit: a second run fails on that server, naming the action, user, controller, start time and age of the lock it found (#221)- A Component's optional
deployment.retainReleases(an integer, at least 2, 5 when absent) says how many releases a deploy keeps on each server; the SDK inventory and the deploy role carry it, andretainUntilCleanup: truestill validates and means the cleanup is on (#224) - The release receipt names, in each server entry whose gate passed, the releases the deploy removed there under
removedReleases;release-receipt.schema.jsonaccepts it and refuses it on a failed entry (#224) - The PostgreSQL backup writes
globals-<timestamp>.sqlfrompg_dumpall --globals-only --no-role-passwordsbeside the per-database dumps, so a restore onto a fresh server has its roles; retention prunes it with the dumps (#225) Baselinestorage.pools: existing LVM volume groups that workload volumes go into; eachLoggingStackvolume names its pool, LV, LV type and physical volumes inrequirements show, andconfig validaterefuses a volume with no pool (S-014)auditchecks each required volume (volume.<name>: type, physical volumes, size, health, mount) from newvolumesevidenceinspectcollects (S-014)cloudfall logging deployprovides the stack's data volumes: a read-only report from each host, refusals decided on the controller before any host change (volume_refused), thenlvcreate/mkfsfor missing volumes,lvextend --resizefsfor small ones, an fstab line by UUID, the mount andRequiresMountsFordrop-ins (S-014)cloudfall decisions mark-interrupted <id> --confirm-deadandcloudfall operator mark-interrupted <id> --confirm-deadclose a record leftrunningby a run that died midway as the new statusinterrupted, once a person confirmed the run is dead; the record keeps its approval and gainsinterruption.confirmedBy(--by, defaultUSER) andinterruption.confirmedAt. Nothing runs on the fleet, and neither command is an MCP tool (#244)AlertChannel(alert-channels/): a Telegram chat that aBaseline's hosts and aLoggingStackreceiver name; its bot token is an input secret on every server that sends, set withcloudfall secrets input telegram-bot-token --owner <channel>(S-015)- A
LoggingStackreceiver may name anAlertChannel: Alertmanager delivers to it with its nativetelegram_configsandbot_token_file, the token delivered bylogging deploy; an Alertmanager older than 0.26 is refused before its configuration is written (S-015) cloudfall alerts test <server>: send one test message through the server's alert channel;auditchecks each host's channel (alerting.channel: sender, token and chat id present, mdadm and smartd pointed at the sender) from newalertChannelevidenceinspectcollects, never the token (S-015)cloudfall baseline converge: runbaseline.ymlwith each host's alert channel; hosts whose baseline names anAlertChannelget/usr/local/sbin/cloudfall-alert, the channel's token and chat id, and mdadm and smartd pointed at the sender; a token not in the store stops it before any server (S-015)- A
Servicecan beserviceKind: elasticsearch(S-012):spec.elasticsearchdeclaresversion(an exactX.Y.Zof a line Elastic still supports,8.19or9.x),heapMb(512 to 31744), and optionallynodes,clusterName,existingCluster,users(each withindexPatternsand apasswordRef, never a password),remoteAccess,snapshotRepositoryandrestoreCheckIndices.cloudfall config validaterefuses an unsupported version (service_elasticsearch_version), badnodes(service_elasticsearch_nodes), anexistingClusterwithoutclusterName, with a CA file that is missing, outside the project, not a CA or holding a key, or with a seed host that is a declared node (service_elasticsearch_existing_cluster), a server in two clusters (service_elasticsearch_server_shared), several nodes or an existing cluster without asnapshotRepository(service_elasticsearch_snapshot_repository_local), a reserved or repeated user name (service_elasticsearch_user_reserved,service_elasticsearch_user_duplicate), a pattern of every or a hidden index (service_elasticsearch_pattern), andremoteAccessunder the PostgreSQL rules; it warnsELASTICSEARCH_VERSION_ENDINGwith the date when a line is within 90 days of its end of life. The rendered inventory carries the service, withnodesandclusterNameresolved, on its server; nothing converges it yet (#309) services.ymlconverges a one-nodeserviceKind: elasticsearchservice on Debian 13 through the newcloudfall_elasticsearchrole (S-012). Its first task refuses aheapMbabove half of the host's memory before anything changes. It installselasticsearch=X.Y.Zfrom Elastic's APT repository for the declared major and holds it, and writeselasticsearch.yml(cluster.name,node.name, HTTP on the loopbackbindonly,bootstrap.memory_lock: true, security on, neverdiscovery.type), ajvm.options.dheap file and a unit drop-in withLimitMEMLOCK=infinity, all before the package, so its security auto-configuration skips the node.cluster.initial_master_nodesis written only while the data path holds no cluster state and leaves the file in the same converge once the cluster has formed, with no restart, so a second converge changes nothing; the converge stops when the node runs without its memory locked. Cloudfall's owncloudfallaccount is a superuser in the file realm, with only its bcrypt hash inusersand its password in a0600file owned byelasticsearch; theelasticpassword is set once, as a bcrypt hash computed on the controller (D-9). Each node of the service yields theelasticsearchpackage and the generatedelasticsearch-cloudfall-passwordas S-011 requirements, andcloudfall secrets generatealso generates theelasticpassword (ELASTIC_PASSWORDbesideCLOUDFALL_PASSWORDinsecrets/<environment>/services/<id>.env); every path that runsservices.ymldecrypts both first and stops before any playbook when one is missing. More nodes, declared users,remoteAccessandexistingClusterstop the converge until their slices land (#310)- An Elasticsearch service's declared
usersconverge (S-012): from the service's home server, each user gets a role of its own name limited to itsindexPatterns(read:readandview_index_metadata;writeaddswrite,create_indexandmanage) and a native user holding that role alone. Every path that runsservices.ymldecrypts each user'spasswordReffirst, stops before any playbook when one is missing, and stages only its bcrypt hash, computed on the controller and sent aspassword_hash(D-9); the salt is derived from the service'scloudfallpassword, so an unchanged password puts nothing and a second converge changes nothing. Users and roles Cloudfall makes carrymetadata.cloudfall_managed_by: <service id>; a converge deletes the marked ones no longer declared and never changes or deletes an unmarked one, and refuses a declared name an unmarked user or role already holds.inspectreads the cluster's users and roles on the home server (elasticsearchUsersin the observation), andauditaddselasticsearch.users[<id>]: drift when a declared user or role is missing or a marked one is left over, with users and roles not made by Cloudfall listed as unmanaged (#311) - An Elasticsearch service grows to two or more
nodesand rolls a version one node at a time (S-012). Each node yields anelasticsearchdata volume of 10 GiB at/var/lib/elasticsearchin its baseline's only storage pool,mirroredfor a cluster of one andexpendablefor two or more; a node of a cluster of two or more also yields port 9300/tcp open to the other nodes only and anelasticsearch-transportcertificate from the project CA for its hostname and addresses, of the usageserver-and-client: the one certificateremoteAccessserves HTTP with, which the services converge now issues for a node withremoteAccesstoo. The role refusesremoteAccesson a cluster of two or more, since its onelistenAddressexists on one server only.config validaterefuses a node whose baseline has no pool (volume_requirement_pool_missing), several (volume_requirement_pool_ambiguous), or, for a cluster of one, no secondary physical volume (volume_requirement_pool_unmirrored), and a node of a cluster of two or more with no IPv4 (service_elasticsearch_node_unaddressed). The rendered inventory carries the service on every node with its transport address and seed hosts, adds the 9300 rule to each node's firewall, and lists each cluster's home first. Every path that runsservices.ymlthroughcloudfallfirst reads each node's volume facts and decides its volume, then has each node make its key and a signing request, signs the missing and due certificates with the project CA and delivers them, all before any host changes. The role creates and mounts the volume, enforces the firewall, binds the transport to the node's address with TLSverification_mode: full, bootstraps a cluster only from the home node and refuses when another node of it already runs, sets theelasticpassword from the home only, and restarts a running node only from a green cluster (yellow with one node), with replica allocation held and a flush (released again even when the restart, the rejoin or the version check fails), then waits for the node to rejoin on the declared version and the cluster to return to its status before the next server (#312) - An Elasticsearch service's S3
snapshotRepositoryis converged instead of refused (S-012). ItscredentialsRefnames one key of the secret store holding<access key id>:<secret access key>; every path that runsservices.ymldecrypts it on the controller, stages the two keys in 0600 files and passes only their paths, and stops before any playbook when the key is missing or malformed. On every node the role puts them in the keystore ass3.client.default.access_keyands3.client.default.secret_keythrough stdin, keeps only their SHA-256 beside it so a second converge changes nothing, names the endpoint ass3.client.default.endpointinelasticsearch.yml, reloads a running node's secure settings, and registers the repositorycloudfallastype: s3with the bucket and base path, replacing the filesystem repository of a service that grew; the snapshot and restore-drill scripts use it unchanged (#312) config validaterefuses an Elasticsearch cluster of two or more nodes on a baseline without a firewall, since nothing would narrow 9300 to the other nodes (service_elasticsearch_cluster_firewall_missing); the inventory render refuses it too (#312)cloudfall backup runandcloudfall backup verifywork for aserviceKind: elasticsearchservice (S-012). The converge addsbackup.directorytopath.repo, which restarts each existing node once on the first converge after the upgrade, registers it as the filesystem snapshot repositorycloudfall, and installscloudfall-elasticsearch-backupandcloudfall-elasticsearch-restore-checkwith sandboxed units and timers onbackup.onCalendarandbackup.restoreCheckOnCalendar. The backup counts the documents of every indexrestoreCheckIndicesmatches (or of the smallest index) and writes the counts as one document per snapshot into Cloudfall's own indexcloudfall-snapshot-counts. It then takes the snapshotcloudfall-<timestamp>of every index, that one included, and the cluster state, waits for it, and fails unless its state isSUCCESS, so aPARTIALorFAILEDsnapshot writes no receipt. Finally it deletes Cloudfall's snapshots of the service older thanretentionDayswith their counts, always keeping the newest and never touching a snapshot it did not take. The restore drill verifies the repository, restores the snapshot's counts and every recorded index ascloudfall-restore-check-<index>with no replicas and without aliases, checks the low disk watermark, waits for green, compares each document count, and deletes the restored copies even when a step failed; it refuses to start over an index of a copy's name, and fails when no snapshot exists. Both run as theelasticsearchuser, reach the node as the role does (withremoteAccess, over HTTPS on the listen address, verified against the project CA) and write the usualBackupReceipt. An S3snapshotRepositorystops the converge before it changes the host, with a message that S3 repository registration is not built yet (#313)- An Elasticsearch service (S-012) is audited, opened by
remoteAccess, and scraped for metrics.cloudfall auditreports drift on each node when the installedelasticsearchpackage differs fromversion, whenelasticsearch.serviceis not running and enabled, and when the managed settings ofelasticsearch.yml(network.host,http.port,http.host,xpack.security.enabled,xpack.security.http.ssl.enabled) differ from what the role writes, so a hand edit is drift before the node restarts with it; it reports the node as running an unsupported version once its line is past Elastic's End of Maintenance date at the observation's time (D-12).inspectreadselasticsearch.ymlas text on a server with an Elasticsearch service (spec.elasticsearchof the observation). WithremoteAccess, the firewall opensbind.porttoallowedSourcesonly, HTTP listens on loopback andlistenAddressand is TLS-only on both, with the node's one certificate from the project CA (theelasticsearch-transportcertificate requirement of the new usageserver-and-client, for the server's hostname and every address, the listen address included, at/etc/elasticsearch/certs/transport.crt,transport.keyandca.crt), which the role checks before writing a configuration that names it, and stops on while nothing issues it yet; the role's own API calls then go over HTTPS to the listen address, verifying the project CA. Withmetrics.enabled, the collector renders one Alloyprometheus.exporter.elasticsearchfor the local node, as thecloudfallaccount, labelled with the service id; Alloy reads the password, and withremoteAccessthe project CA, as systemd credentials (LoadCredential), andlogging deploystops before changing the collector when the node's files are not there yet (#314) - An Elasticsearch service with
existingClusteris converged instead of refused: its nodes join a cluster Cloudfall did not create (S-012). Each node's transport leaves loopback even when it is the only Cloudfall node:discovery.seed_hostslists the existing cluster's seed hosts, then the Cloudfall nodes; its firewall admits 9300 from the other Cloudfall nodes and the seed hosts' addresses only; it gets its transport certificate from the project CA and trusts the existing cluster'scaCertificateFilebeside it, which the services converge delivers to/etc/elasticsearch/certs/existing-cluster-ca.crt, withverification_mode: certificate. A joining node never writescluster.initial_master_nodesand never sets theelasticpassword, which stays the existing cluster's, and the existing nodes are only read. Before the data volume and the package, the role asks the seed hosts over HTTPS on port 9200, verified against that CA, forGET /_nodes, and stops, naming the seed host or node and its value, when the cluster's name differs fromclusterName, when a node runs another major, when a node is newer thanversion, or when no seed host answers. OnceexistingClusteris removed, a converge drops the seed hosts and the CA from the configuration, restarts one node at a time, removes the CA file, and waits for the cluster to be green with two or more Cloudfall nodes, or yellow with a single one, whose replicas have nowhere to go;clusterNamekeeps the name.config validaterefuses a seed host given by name rather than address, and a node that joins an existing cluster without a firewall or an IPv4, as for a node of a cluster of two or more (#315) existingCluster.credentialsRefnames one key of the project's store holding<username>:<password>of an account the team already has on the existing cluster with themonitorprivilege; the join check logs in with it, so nobody creates a user on the old cluster by hand. Without it, the check logs in as thecloudfallaccount, which the team then grantsmonitorthere. A missing key, or a value not of that form, stops the converge before any playbook, naming the file and the key; the login is staged in 0600 files on the controller only, removed after the run, and never reaches a node, the inventory, task output or an error (#315)cloudfall operator mark-interrupted <id> --confirm-deadalso closes a receipt leftexecuted, whose verification never recorded an outcome, asinterrupted, as it does arunningone; it then stops blocking new proposals for the same alert or drift (#358)- A project states its format as
[tool.cloudfall] formatin itspyproject.toml, andcloudfall initwrites the current one (format = 1).cloudfall config validatechecks it before validating: a project without the key, or without apyproject.toml, reads as format 0 and still validates, with onePROJECT_FORMAT_OLDwarning namingcloudfall config upgrade; a format outside the ones this release reads (0 to 1), or one that is not a non-negative integer, stops with the new exitPROJECT_FORMAT_UNSUPPORTED(95) anderror.context.codeproject_format_too_new,project_format_too_oldorproject_format_invalid(#375) cloudfall changeloglists entries: each release now records its interface changes (commands, flags, output fields, exit codes) inschema-changelog.jsonand its manifest snapshot, both in the wheel, as the release bot'safter_stampstep runstreaty changelog-add cloudfall.app:app. The first release seeds the snapshot, so its entry lists the whole interface as added; a release whose interface did not change adds no entry. The Releases section ofCONTRIBUTING.mdnames no manual step any more (#378)docs/compatibility.mdpublishes the compatibility policy: the stable surfaces (resource schemas, the CLI envelope, exit codes, record and receipt schemas, MCP tool names), what a patch, minor and major release may change before and from 1.0, how long a deprecated name stays, the project format version andcloudfall config upgrade;README.mdandCONTRIBUTING.mdlink it (#381)- A
Servicecan beserviceKind: mariadb(S-019):spec.mariadbdeclaresversion(an exactX.Y.Zof a long-term series MariaDB still maintains:10.11,11.4,11.8or12.3),innodbBufferPoolMb(128 or more), and optionallydatabases(each with anapplication, or anownerwith apasswordRef, never a password, andauthentication: ed25519if wanted),remoteAccess, andreplication(roleprimaryorreplica,serverId, a primary'slistenAddress, a replica'sprimaryservice orsourcewithhost,port,credentialsRefandcaCertificateFile).cloudfall config validaterefuses an unsupported version (service_mariadb_version), a server with two MariaDB services (service_mariadb_server_shared), a reserved owner or database name (service_database_owner_reserved), a repeatedserverId(service_mariadb_server_id_duplicate), a primary withoutlistenAddressor a replica with one (service_mariadb_replication_listen), a replica without exactly oneprimaryorsource, or whoseprimaryis no primary MariaDB service of its environment on another server (service_mariadb_replication_primary), a replica withdatabases(service_mariadb_replica_databases), of another series than its primary or older (service_mariadb_replica_version), a pair without a firewall (service_mariadb_replication_unfirewalled), a replica server with no address its primary can admit (service_mariadb_replica_unaddressed), a source CA file that is missing, outside the project or not a CA (service_mariadb_replication_source), a server without one mirrored storage pool (the S-014 codes), andremoteAccessunder the PostgreSQL rules; it warnsMARIADB_VERSION_ENDINGwith the date when a series is within 90 days of its end of life, andMARIADB_REPLICATION_UNENCRYPTEDfor a source without a CA file. The kind yields itsmariadbvolume, package, replication port, certificate and generated replication password as host requirements, and the rendered inventory carries the service with its databases and references, never a password; nothing converges it yet (#419) servicesconverges a standalone MariaDB service (S-019) with the newcloudfall_mariadbrole: it refuses, before any task changes the host, aninnodbBufferPoolMbabove three quarters of the server's memory, a series MariaDB publishes no package of for the server's Debian release, and a series other than the installed one; it mounts the data directory on the mirroredmariadbvolume the services converge decides with the Elasticsearch volumes, installsmariadb-serverat the exact declared release from MariaDB's archive and holds it, and writes the managed settings to/etc/mysql/mariadb.conf.d/90-cloudfall.cnf(listener onbind.addressonly, buffer pool,utf8mb4; no binary log). Anapplicationdatabase gets a user named after the application's Linux user that logs in through the unix socket only; anownerdatabase gets a user whose password reaches the host only as the hash its plugin stores (mysql_native_password, ored25519withauthentication: ed25519), computed on the controller and set only when it differs from the stored one, so a second converge changes nothing. Each user has every privilege on its own database. A user Cloudfall made that is no longer declared is locked, its database left with its rows, and users and databases made by hand are left alone;cloudfall auditlists both kinds as undeclared and unmanaged. ApasswordRefwhose file or key is missing stops the converge withPRECONDITIONbefore any playbook runs.remoteAccessandreplicationare still refused by the role until their own issues land (#420)- A MariaDB service converges its
remoteAccess: the services converge has the server make its key and the project CA sign its certificate for its hostname and addresses, the role installs it (the key 0600 for the mysql account), listens on loopback andlistenAddresswithssl_cert,ssl_keyandssl_ca, and gives each owner with a password one account per allowed source from the same controller-computed hash, requiring TLS, so a plain connection from a source is refused; the firewall admits the port from the allowed sources only, andauditexpects that rule, the listener and those accounts. An account of a source no longer declared is locked, never dropped; a renewed certificate is loaded withFLUSH SSL, without a restart.replicationis still refused by the role (#421) - A MariaDB service is backed up and its restore drilled (S-019):
servicesinstallscloudfall-mariadb-backupandcloudfall-mariadb-restore-checkwith their systemd timers (the drill's only withrestoreCheckOnCalendar), run as root through the socket, andcloudfall backup run <service>andcloudfall backup verify <service>run them and write theirBackupReceipt. A backup writes, for every declared database, one gzip dump taken withmariadb-dump --single-transaction(one consistent snapshot of the InnoDB tables, no table locks; tables of another engine are named in the output) and beside it a count file of every table and its rows, counted from the stored dump, both0600in a0700directory; then it deletes each database's dumps older thanretentionDays, never the newest, and only files of the names it writes. The drill loads the newest dump of each database into the scratch databasecloudfall-restore-checkwithout writing to the binary log, compares its tables and row counts with the count file (never with the live source, so later writes cannot fail it), and drops it on every path; it fails, and says so, when a database has no dump or no count file, when the load fails, and on any difference, and leaves a database of the scratch name it did not make alone. The role checksonCalendarandrestoreCheckOnCalendarbefore it changes the host. The dumps hold tables, views, routines and triggers but no scheduled events (--eventsis not passed, so no event can run in the drill's scratch database): a team that uses events recreates them after a restore (#422) - A MariaDB service converges its
replicationbetween two Cloudfall servers. Both servers write a row-based binary log with GTIDs in strict mode, kept seven days, and get their server certificate from the project CA. A primary listens onlistenAddresstoo and holds onecloudfall_replaccount at the IPv4 of each replica's server, requiring TLS, set from the hash of its password, whichcloudfall secrets generatemakes and keeps sops-encrypted: the primary receives only the hash, a replica the password in a 0600 file for the run only, and its firewall admits the port from those addresses only. An empty replica that never replicated is seeded from one consistent dump of its primary at a recorded GTID position, then follows it by GTID over TLS verified against the project CA, read-only, and the converge waits until it reached the primary's position and holds the same tables; a second converge changes nothing. A replica that holds data attaches from its own position only when its primary saw every GTID it holds; otherwise, and when it has no GTID history, the converge stops and drops nothing. Changing a replica torole: primarypromotes it when its source can't be reached, or is read-only and applied, and the answer names the last GTID it applied; over a running, writable source it stops withPRECONDITIONnaming both positions, and the replica still replicates. Swapping the roles of a pair in one change ends within two converges, with no seed. On a replica,cloudfall-mariadb-backupandcloudfall-mariadb-restore-checktake every database it holds that is not MariaDB's own, read when they run, since a replica declares none. A replica of a source Cloudfall does not manage is still refused by the role. The primary's replication password is no longer a host requirement at/etc/mysql/cloudfall/replication-password: no host keeps it (#423) - A MariaDB replica declared with
replication.source, a server Cloudfall does not manage, converges. Before anything is installed on its server, the role logs in to the source with the logincredentialsRefnames, over TLS verified againstcaCertificateFilewhen declared, through a client module of its own (nomariadbclient is needed yet), and stops withPRECONDITION service_mariadb_source_unfitwhen the source is not MariaDB, writes no binary log, writes one that is not row-based, runs a release newer thanversion, has the replica'sserver_id, or its account lacks a rightcloudfall_replholds; it sends the sourceSELECTandSHOWonly. An empty replica is seeded from one consistent dump of the source's databases without its users, follows it by GTID, read-only, and ignoresmysql.%(replicate_wild_ignore_table), so a user created or granted on the source afterwards is not applied. The login is staged 0600 on the controller and reaches the replica's server for the run only; the rendered inventory names the source's host, port and whether it uses TLS, never its login. After a promotion, a declaredownerdatabase the server holds is adopted with its rows (#424) - A MariaDB service (S-019) is audited and scraped for metrics.
cloudfall auditadds six checks per service, none renamed or removed:mariadb.versionis drift when the installedmariadb-serveris another release thanversion;mariadb.seriesreports the server as running an unsupported version once its series is past its end of life at the observation's time;mariadb.unitis drift whenmariadb.serviceis not running and enabled;mariadb.configurationis drift when the settings of the managed/etc/mysql/mariadb.conf.d/90-cloudfall.cnfdiffer from what the role writes (a changed, added or removed setting; comments are not compared);mariadb.roleis drift when a replica is writable (read_onlyoff) or its replication threads are not both running, or when a standalone server or a primary replicates from somewhere; and, for a replica only,mariadb.lagis drift when it is more than five minutes behind its primary.inspectreads the managed file as text and, as root through the socket,read_onlyandSHOW SLAVE STATUS(the threads' states, the lag and the source'sserver_id, never a password) into a newspec.mariadbof the observation; a record an older inspect wrote has none, so those checks report unknown until the next inspect. Withmetrics.enabled, the role makes thealloyaccount on every server of the service, a replica's too, authenticated by the unix socket only and outside the binary log, withPROCESSandSLAVE MONITOR; the collector renders one Alloyprometheus.exporter.mysqlfor the service, connecting through the server's socket as that account with no password, labelled with the service id, and none withoutmetrics.enabled. The bundled Services dashboard gains MariaDB panels: up, connections, queries and replication lag (#425) - A MariaDB service guide (
docs/mariadb-service-guide.md), linked from the README: declaring a standalone server, adding a replica and what each services converge answers (seed,attach,promoteand the waits), backups and the restore drill on the replica, a planned switch and a promotion after the primary is lost, bringing a former primary back, and each refusal with what to do about it; moving in from a server Cloudfall does not manage gives the steps on the source and says its converge is not built yet (#466) - An
OperatorPolicygrant may name the operation classes of a project's server-changing MCP tools,deploy,rollback,restart,migrate,data-copy,domains-applyandconverge-domains, besideconverge-servicesandconverge-baseline; theOperatorProposalschema gains, additively, trigger kindagentwith itsrequest,operation.serversandoperation.environments, approval modeagentwithpolicy,grantandclient, andbinding, for the agent records spec 018 writes. Overcloudfall mcp servein a project, the plan ofdeploy,rollback,restart,migrate,data_copyanddomains_applycalled withoutyesgainsapproval.selfConfirmand, per environment its servers are in, thepolicythat applies or none and thereasonfrom the operator's grant rules; nothing a confirmed call does changes yet (#474) - Over
cloudfall mcp servein a project, a confirmed call ofdeploy,rollback,restart,migrate,data_copy,domains_apply(yes: true) or aconverge_*tool (confirm_destructive: true) that anOperatorPolicygrant covers in every environment it touches now writes an agentOperatorProposalrecord to the proposal store (record/operator/proposals/by default):runningbefore the first engine step, with the tool, its arguments and the agent'sreason,approval.mode: agentwith the policy and grant, andbinding.planSha256(the sha256 of the plan the unconfirmed call answers) with the git commit and worktree digest, thenverifiedorfailedby the tool's own check (the health gate, every migrate step, the copy's receipt,domains observe, or anauditof the changed servers). The result gainsapprovalwithmode,policy,grant,recordandstatus; ondeploythe release approval's ownpolicyis then namedrelease_policy. Each gated tool takes an optionalreason(at most 1000 characters), also a--reasonflag on the CLI commands, where it records nothing. A confirmed call no grant covers, and every CLI command, runs as before and writes no record (#475) cloudfall operator approve <id>now runs an agent's proposal (one a confirmedcloudfall mcp servecall no grant covered left in the proposal store) instead of refusing it withoperator_agent_proposal_unsupported. As for the operator's own proposals, a person types the id at a terminal (PERSON_REQUIREDoff one, and nothing runs). The command then recomputes the plan of the stored tool call, the git commit and the digest of uncommitted changes outsidedecisions/,record/andtmp/, and refuses any difference before anything runs withPRECONDITIONandcontext.code: proposal_binding_changed, naming what changed incontext.changed; the proposal staysproposed, and the agent calls the tool again for a new one. Otherwise the stored call runs through the tool's own command, S-008's release approval included, recorded withapproval.mode: humanandvia: terminal, and settlesverifiedorfailedby the tool's own check (the health gate, every migrate step, the copy's receipt,domains observe, or anauditof the changed servers). A verified person-approved run counts toward therequiredVerifiedRunsof the policy grant for its class in its environment.cloudfall operator runstill leaves agent proposalsproposed, listed as withheld because only a person approves them.operator approvelistsCONFLICTandPARTIAL_FAILUREamong its exit codes, which a storeddeploy,rollback,restartormigratecan end with (#477)- A
Servicecan beserviceKind: rabbitmq(S-023):spec.rabbitmqdeclaresversion(an exactX.Y.Zof a series Team RabbitMQ still supports for the open-source edition,4.3or4.4, held in one table with each series' end of community support and its Erlang range, 27.0 or later),memoryHighWatermarkMb(512 or more), and optionallynodes(one server or three, containingspec.server),vhosts(each with anownerand apasswordRef, never a password) andclientAccesswith itsallowedSources.cloudfall config validaterefuses an unsupported version (service_rabbitmq_version),nodeswithout the service's server, with an undeclared or repeated server, or of another size than one or three (service_rabbitmq_nodes), a server that is a node of two RabbitMQ services (service_rabbitmq_server_shared), a node of another environment (service_rabbitmq_environment), two nodes of one hostname (service_rabbitmq_hostname),remoteAccess(service_rabbitmq_remote_access), a virtual host without an owner orpasswordRef, with anapplication, declared twice, or whose owner owns another (service_rabbitmq_vhost), the ownerguest, the virtual host/and names startingcloudfall_(service_rabbitmq_reserved), with three nodes orclientAccessa server without a firewall (service_rabbitmq_unfirewalled) or an IPv4 (service_rabbitmq_server_unaddressed), a client source of every address (the S-006 rule), and a server without one storage pool, mirrored for one node (the S-014 codes); it warnsRABBITMQ_VERSION_ENDINGwith the date when a series is within 90 days of its end of community support, andRABBITMQ_SINGLE_NODEfor one node in theproductionenvironment. The rendered inventory carries the service, withnodesresolved and its virtual hosts' password references, never a password, on each of its nodes; nothing converges it yet (#505) services.ymlconverges a RabbitMQ service of one node with noclientAccess(S-023) through the newcloudfall_rabbitmqrole:rabbitmq-serverat the declaredX.Y.Z-1from Team RabbitMQ's repository and the Erlang of the Debian release, all held; the data directory on arabbitmqvolume (20 GB, mirrored for one node) that the converge decides with the other volumes; the Erlang cookie Cloudfall generates (cloudfall secrets generate) at/var/lib/rabbitmq/.erlang.cookie,0400; managed settings in/etc/rabbitmq/conf.d/9*-cloudfall-*.conf(clients, the inter-node port, epmd and metrics on loopback only,vm_memory_high_watermark.absolutefrommemoryHighWatermarkMb,disk_free_limitof 2 GB, the node list, continuous membership reconciliation of quorum queues toward three members) andenabled_pluginsnaming the Prometheus plugin alone. The role refuses, before it changes the host, a watermark above six tenths of the server's memory, a hostname that is not the declared one, and a change of the installed version; it refuses an Erlang older than the series needs, and a cluster of three orclientAccess, which come later. On the service's server it creates each virtual host withquorumas its default queue type, gives its owner every permission there and none elsewhere, and sets the owner's password as RabbitMQ's salted SHA-256 hash, computed on the controller with the salt the user already has, so the password never reaches a node and a second converge changes nothing;guestis removed, and/while it holds no queue. A virtual host Cloudfall made that is no longer declared is kept with its queues, and its owner's password cleared;inspectreads the cluster's virtual hosts and users, andauditlists those as undeclared and anything Cloudfall did not make as unmanaged (#506)services.ymlconverges a RabbitMQ node withclientAccess(S-023): besides plain AMQP on loopback, the node takes clients over TLS alone on its server's IPv4 (5671), with its own certificate from the project CA (usage server and client, names its hostname and addresses, at/etc/rabbitmq/cloudfall/, the key made on the node and readable byrabbitmqalone), which the services converge has the node request and the project CA sign as it does for MariaDB and Elasticsearch. The node's firewall admits 5671 fromallowedSourcesonly (ip saddr 10.0.0.0/24 tcp dport 5671 accept), no RabbitMQ port is open to every address, and plain 5672 stays on loopback, so a plain connection from outside the host is refused.auditexpects the rule. A node withoutclientAccessrenders and converges as before (#507)- A RabbitMQ service is backed up and its restore drilled (S-023): on the service's server,
servicesinstallscloudfall-rabbitmq-backupandcloudfall-rabbitmq-restore-checkwith their systemd timers (the drill's only withrestoreCheckOnCalendar), run as root, andcloudfall backup run <service>andcloudfall backup verify <service>run them and write theirBackupReceipt. A backup exports the definitions of the cluster (virtual hosts, users, permissions, queues, exchanges, bindings, policies) withrabbitmqctl export_definitionsinto one file ofbackup.directory, root's and0600from its first byte as it holds password hashes, refuses one that does not parse or misses a declared virtual host, and then deletes the exports older thanretentionDays, never the newest. A backup holds no message: its receipt says so in the new optionalholdsMessagesfield (false), which only a RabbitMQ receipt carries. The drill starts a second node on the server with a node name, Erlang cookie, data directory, configuration, epmd and loopback ports of its own and no node list, imports the newest export into it at boot, compares the virtual hosts, users (and their password hashes, never printed), queues, exchanges, bindings and policies it then holds with the export, checks that the cluster does not list it, and stops it and removes its directory on every path; it fails, and says so, when there is no export, on any difference, when the server has less free memory than the scratch node's 256 MB watermark or its ports are taken, and when a process or listener of the scratch node is left (#511) - A CI deploy guide (
docs/ci-deploy-guide.md): one GitHub Actions workflow that renders secrets, builds the artifact withcloudfall-engine artifact build, deploys withcloudfall deploy --yesand uploads the receipts, with one SSH key and one age key per environment and required reviewers standing in forapproval: manual, whichdeploydoes not enforce; not yet run against a real server (#89) - An open proposal whose basis hosts have been observed again since is now reported as stale:
decisions list,decisions showandcloudfall whyaddstaleandstaleHoststo eachproposeddecision,--staleondecisions listandwhykeeps only those, anddecisions approvenames each stale host and its newer observation time before asking for the id. Worked out on read from the observation files; the record and its schema do not change (#26) auditchecks a logging backend's dashboards (logging.dashboards):inspectrecords the name and sha256 of each file in Grafana's Cloudfall dashboards directory (newloggingDashboardsevidence, never anything else of Grafana's), and the check is compliant when they are exactly the dashboards the installed release bundles; drift names each missing, extra or changed file (S-016)logging deployprovisions four dashboards into a read-onlyCloudfallfolder in the stack's Grafana: Fleet (every server's freshness, CPU, memory, fullest filesystem, load and the firing alerts), Server (one server's host metrics and log volume by unit), Logs (log volume, error lines and the lines themselves by server and unit, with a filter) and Services (PostgreSQL and Redis exporters by service). They read servers from the data and name the provisioned data sources by uid, so they work on any project; a dashboard dropped by a later release is removed on the next deploy (S-016)cloudfall logging openopens the LoggingStack's Grafana: it forwards a local port over ssh to Grafana on the backend's loopback, built from the backend server's declared address, SSH user and port, answers with the URL of the Cloudfall dashboards once the forward is up, and holds it until stopped.--portpicks the local port (one in use exits 4 before ssh starts),--stackpicks the stack, and--print-commandanswers with the ssh command and runs nothing. The answer names where the admin password is kept, never the password (S-016)
Changed
- Breaking:
cloudfall config validatenow refuses a Component withruntime.typenodeorpm2, orruntime.packageManagernpm,pnpmoryarn, withCONFIG_INVALIDand codecomponent_runtime_unsupportednaming the field: these configs validated before but could never deploy, since the deploy role runs only Python with uv. The schema keeps those values as reserved. The Render importer no longer writes a Component for a Node service; it lists the service as an unsupported gap instead (#114) migrate --yeschecks adeploy:<component>step's approval asdeploy --yesdoes: for an application that declaresapproval: manualand no approval that counts, it pauses at that step withPARTIAL_FAILURE,context.code: deployment_approval_missing, the builtreleaseand thecloudfall release approvecommand, and once a person approved it,migrate --yesresumes there with the release the completedbuild:<component>step recorded, without building again and without needing--buildagain. The deploy step's detail gainsapproval(the record it went on),migrategains--approvals(defaultdecisions/releases, bound bycloudfall mcp serve) and theRECORD_INVALIDexit for an approvals file that isn't a validReleaseApproval.rollbackneeds no approval: its plan gainsapproval,{"policy": "manual", "required": false, "reason": ...}for amanualapplication and{"policy": "automatic"}otherwise (#165)- The resource every server runs is now
kind: Baseline, stored underbaselines/, referenced asServer.spec.baselineand checked bybaseline.schema.json(wasServerType,server-types/,spec.serverType,server-type.schema.json);cloudfall add baselinedeclares one andcloudfall add server --baselinenames it (#198) - Outputs carry the new names:
inventory showwritesbaselinesand each server'sbaseline,config validatecountsBaselineinby_kind,auditwrites each server'sbaselineand checksbaseline.id(wasserver_type.id; an operator proposal recorded withserver_type.idverifies oncebaseline.idis compliant, and blocks a new proposal for the same drift, with no removal date, since a record's fields are never deprecated withincloudfall/v1), the inspect role recordsbaselinein observed server snapshots (a snapshot recorded before withspec.serverTypeis still read asspec.baseline, also with no removal date; one with both fails withobservation_baseline_conflict), the rendered Ansible inventory has the host variablecloudfall_baseline, the brownfield catalog variable iscloudfall_baselines, and the error codeserver_type_entry_duplicateisbaseline_entry_duplicate(#198) - Site: cloudfall.tech is the primary domain; cloudfall.dev and both www hosts redirect to it with a 301, and the PyPI homepage and the README
cloudfall initwrites point there (#199) - Receipts and operator proposals now land in the project's committed
record/directory instead of the git-ignoredtmp/, so the audit trail survives losing the management host (D-16): the defaults of--deployments,--releases,--backups,--data-migrations,--env-receiptsand--proposals(on every command and oncloudfall mcp serve) arerecord/deployments,record/releases,record/backups,record/data-migrations,record/env-receiptsandrecord/operator/proposals.tmp/keeps only what can be rebuilt: observations, rendered inventory, built artifacts, rendered environment files and the migration plan. A path you pass explicitly keeps working as before.cloudfall config upgradenow also moves an existing project's receipts and proposals fromtmp/intorecord/, listing each move inupgradedwith itsmoved_from; it drops a file whose identical copy is already inrecord/, refuses withCONFLICTand codeupgrade_target_exists, naming both paths and moving nothing, when a different file is, and a second run changes nothing. Until it runs,operator list,audit,services statusand the operator's autonomy pass don't see the receipts left undertmp/, so runcloudfall config upgradeonce after upgrading, then commitrecord/.cloudfall initwrites the new layout into the project's README,AGENTS.mdand.gitignore, and the operator guide says how to copyrecord/off a management host with no git remote (#219) - A release receipt no longer holds its release command's output: the command is the application's and may print a secret, and the receipt is now committed. The last 2000 characters stay in
tmp/output/releases/<component>.<release>.log(0600), andspec.releaseCommandnames that file and its SHA-256 underoutput(path,sha256) instead ofoutputTail. TheReleaseReceiptschema stayscloudfall/v1:outputTailbecomes optional, so receipts written before still validate, and a receipt may not hold both.cloudfall config upgrademoves the tail of an old receipt, undertmp/or already inrecord/, out to that file the same way, refusing to replace a different file there. A reader ofreleaseCommand.outputTailreads the named file instead, which is gone on another host or oncetmp/is cleared (#219) - Contract:
cloudfall lock clear <component>now also reads the component's lock on each of its servers through the newlock.yml, shows them with their ages in the same prompt, and once the component id is typed removes exactly the locks it showed; it takes--engineand--inventory-file, answersserverLocksandunreadServers(a server it could not reach), may exitENGINE_STEP_FAILED, and is listed among the commands that change servers (#221) - The release receipt holds one entry per server under
spec.servers(the release the server ended on,previousRelease,deployedAt,healthCheckPassed,rolledBackand the commandhealthCheck), with the deployedrelease,artifactSha256andreleaseCommandonce per receipt;release-receipt.schema.jsonaccepts it and still accepts a receipt written before, with oneserver, which reads as a one-entry list (#222) - Site: contact addresses moved to cloudfall.tech (
pilot@,conduct@); mail to cloudfall.dev still arrives (#260) - A
mirroredvolume now meets anexpendablerequirement (D-18): when a one-node Elasticsearch service grows, its first node keeps its raid1 data volume, and the volume provider keeps and grows it, andauditchecks it, as the mirror it is; a linear volume still never meets amirroredrequirement (#312) - When installing a new Elasticsearch version fails after the node was stopped for it, the role starts the node again on the package it had and lets its cluster move replicas again before the converge stops (#312)
- Open proposals made before the upgrade carry no binding, so
decisions approverefuses them withdecision_binding_changed: propose them again (#352) decisions approvegives the approved run and its verify step their own time limit of 6 hours (21600 seconds) instead of the check run's 1800 seconds, which check mode keeps. A run that reaches it is stopped with its whole process group and the record settlesfailedwith the verdictthe run timed out after 21600s and was stopped; the run's output now goes to its log as it comes, so a stopped run keeps what it printed. Before, a run past 1800 seconds was killed, its record stayedrunningwith no outcome, and the CLI's log of it was lost (#359)- A release candidate or release is cut and published only from a commit whose converge run passed:
release.ymlandpublish.ymlnow run the converge job (the roles on a real Debian 13 with systemd) on the commit they release, beside CI, and land or publish nothing unless both passed (D-23). The converge job also restarts the component withcloudfall restart, asserting a new process of the same release that passes its health check, and applies an nginx Domain with a self-signed certificate throughcloudfall domains apply, asserting HTTPS serves the probe with that certificate and HTTP redirects to it (#361) - Every command that reads a project now checks its
[tool.cloudfall] formatfirst, ascloudfall config validatealready did:cloudfall audit,deploy,add,healthand the rest,cloudfall mcp serveonce at start, andcloudfall-engine inventory renderandartifact build. A project written by a newer Cloudfall, one below the formats this release reads, or one with a malformed format stops the command with exitPROJECT_FORMAT_UNSUPPORTED(95, now oncloudfall-enginetoo) before the project is validated, before anything is written under itstmp/and before any Ansible run, andcloudfall mcp serverefuses to serve it. A format 0 project (no[tool.cloudfall]table, or nopyproject.toml) still runs every command, each now answering with onePROJECT_FORMAT_OLDwarning namingcloudfall config upgrade.cloudfall init,changelog,--schemaandcloudfall-engine manifestread no project and run whatever its format (#379) cloudfall config upgradebrings a project to the current format: it runs each rewrite from the project's[tool.cloudfall] formatto the current one (the step from format 0 is the rename ofServerType,server-types/andserverTypeand the move of receipts and proposals intorecord/), then writesformat = 1into the project'spyproject.toml, creating the table, or the file, and keeping every other byte of it. A format 0 project with nothing to rename now answerseffect: updated, since its format is written; a project already at the current format is a noop. The answer gainsfrom_formatandformat. It runs on a project too old for every other command, refuses one written by a newer Cloudfall (project_format_too_new) or with an unreadable format (project_format_invalid) with exitPROJECT_FORMAT_UNSUPPORTED(95) and nothing changed, and refuses withCONFLICTand codeupgrade_format_unwritableapyproject.tomlwhosetool.cloudfallis an inline table or dotted keys (#380)- The logging backend and collector roles share one copy of the tasks they both ran: the delivery of provided files, the reach preflight, the package pin check and the package holds, with the
reachfilter, now live in thecloudfall_logging_commonrole, which both include by task file. What a logging deploy does on a host is unchanged - The logging backend and collector roles declare Debian 13 (trixie) next to Debian 12, like every other bundled role, so a trixie host built from the bundled Debian 13 baseline can carry the logging stack. Loki, Grafana and Alloy come from Grafana's distribution-independent APT repository; Prometheus comes from the host's Debian release, so a
software.prometheusPackageVersionpin must name a version that release ships
Security
- An owner role's password no longer reaches the PostgreSQL host or server:
services.ymland every path that runs it compute the password's SCRAM-SHA-256 verifier on the controller (a fresh 16-byte salt, 4096 iterations, the password prepared with SASLprep as PostgreSQL does) and stage that verifier in the0600file instead of the password, soALTER ROLE ... PASSWORDsends a verifier PostgreSQL stores as it is and the plaintext can't show in the server log underlog_statement = 'ddl'or'all', in a failing statement, or inpg_stat_activity; answers and receipts still name the reference, the key, and the password's SHA-256 (#140) - A declared config value is never run as an Ansible template:
cloudfall-engine inventory renderand the overlaycloudfall observewrites mark every string unsafe ({"__ansible_unsafe": ...}), so a value holding{{ ... }}reaches the unit file, the release command and the health check as written. Before, aservice.command,healthCheck.commandorrelease.commandargument such as{{ lookup('pipe', '...') }}ran on the management host duringcloudfall healthorcloudfall deploy, and an honest one such asdocker inspect --format {{.State.Health.Status}} appfailed with a template syntax error. A declared value that relied on Ansible evaluating{{ ... }}is now literal, and the rendered inventory's strings read as{"__ansible_unsafe": "..."}in its JSON and inansible-inventory --list(#174) - A catalog operation's inputs are never run as an Ansible template:
cloudfall operations propose,cloudfall decisions approveandcloudfall agent investigatehand them to the team's playbook in--extra-varswith every string marked unsafe ({"__ansible_unsafe": ...}), so an input holding{{ ... }}reaches the playbook as written, and integers and booleans keep their types. Before, an input such as{{ lookup('pipe', '...') }}ran on the management host with the operator's rights when the playbook started, and an honest input holding{{failed with a template syntax error. A team playbook that relied on an input being evaluated as Jinja now gets the literal text. The decision record'sinputsare unchanged, and recordings made before this change replay as before (#181) cloudfall data copyandcloudfall migrate --datano longer put the source database URL, password included, on the command line ofpsqlandpg_dump, where every local user could read it in/proc/<pid>/cmdlinewhile the copy ran: the URL becomes a0600libpq service file on the target host, named to the clients throughPGSERVICEFILEandPGSERVICEand removed when the copy ends, on failure too. A URL libpq can't read (another scheme, a bad percent-encoding, a parameter set twice) is refused before anything is staged, without quoting it (#355)- A declared operation's check diff and the logs of its approved run and verify step are no longer written to the committed
decisions/directory: they go to the project's ignoredtmp/output/decisions/(0600), and the decision record keeps each file's path, sha256 and size as before, so a secret a team's playbook prints with--diff(a templated password withoutno_logordiff: false) never reaches git. A--decisionsfolder elsewhere in the project keeps its output under the same path belowtmp/output/; one already undertmp/keeps it beside its records.decisions approverefuses withPRECONDITIONanddecision_diff_missing, naming the file and its sha256, when the raw diff is not on this machine, such as on another clone. Records written before keep citing their files indecisions/and are read and approved in place; a file of theirs already committed stays in git's history, so one that may hold a secret is for you to purge (#357)
Fixed
cloudfall auditno longer reports drift on a server whose PostgreSQL service declaresspec.postgresql.remoteAccess:firewall.allowedInboundexpects onetcp/<port> from <cidr>entry per declared source and reads theip saddr/ip6 saddrmatch of each accepted nftables rule (a single address printed bare by nft compares equal to its/32or/128), andservices.bind[<id>]requires a listener on the declaredlistenAddressas well as on the bind address, reportingloopbackOnly: falseand aremoteAccess.listenAddresskey for such a service; a missinglistenAddresslistener, any other non-loopback listener, an unrestricted rule on the port and an undeclared source are drift. Servers withoutremoteAccessaudit exactly as before. The engine and the audit now derive the source-restricted rules from one SDK function,remote_access_rules_for_server(#143)- A
pg_hba.confreload that failed is no longer forgotten:services.ymlleaves a marker file at/var/lib/cloudfall/postgresql/<service>.pg_hba-reload-pendingwhen it closes remote access after a failed check or reload, and the next converge checks and reloadspg_hba.confeven though the file is unchanged, then removes the marker, so the oldhostssllines no longer stay live until a restart (#144) cloudfall config validaterefuses an owned database or owner namedall,sameuser,samerole,samegrouporreplicationon a PostgreSQL service withremoteAccess, asCONFIG_INVALIDwith codeservice_remote_access_name_reserved, instead of leaving the refusal to the converge; withoutremoteAccessthose names still validate (#144)- A first deploy over a hand-made systemd unit of the same name no longer loses it when the release fails its health gate: the deploy role keeps a unit without the
# Managed by Cloudfallheader at/etc/systemd/system/<name>.service.pre-cloudfallbefore installing the managed unit, and on a failed first release copies it back, restores its enabled state and starts it again when it was active; check mode reports the unit it would keep, and a hand-made unit that differs from an earlier kept copy refuses the deploy (#151) - The data copy preflight now names an extension that a selected table needs through an array or a domain of the extension's type, such as a
citext[]column or a column of a domain overcitext(chained domains included), underextensionsand passes it topg_dump --extension; before, only a direct use of the extension's type was seen, so the copy skipped the extension and the restore failed (#173) cloudfall migrate --yesno longer crashes withHANDLER_CRASHEDwhen it reaches itsauditorverify-routesstep on a project with no server observations: the step fails withmigrate_observations_missing, naming the missing directory andcloudfall observe, and a rerun afterobserveresumes at that step (#183)- Decision records and operator proposal receipts are written through a temporary file and renamed into place, so a crash or a full disk mid-write no longer leaves a truncated record; a new record is created exclusively, so two writers of one id can no longer both succeed, and each read-modify-write of a record holds its directory's lock (#217)
- An approval is recorded, with the new status
running, before the first playbook or the operator's executor starts. A run killed midway staysrunningand is refused bydecisions approveandoperator approvewithPRECONDITION(decision_run_interrupted,operator_proposal_run_interrupted) instead of being approvable and run a second time;runningis added to both record schemas (#217) - One decision record or proposal receipt that cannot be read no longer stops every reader of the store:
operations propose,decisions list,whyandoperator listgo on with the records they can read and add aRECORD_UNREADABLEwarning naming each skipped file and why; an operator pass reports the receipts it skipped underunreadable, and its autonomy pass grants nothing while a receipt cannot be read.decisions showanddecisions approvestill answerRECORD_INVALIDfor the one record they read (#218) - Each record is read with the schema its
apiVersionnames, so a later schema version can sit besidecloudfall/v1; a record of a version this Cloudfall does not read is reported, not fatal. A golden set of records in the shapes released versions wrote is loaded by the tests (#218) deploy,rollbackandrestartcontact only the component's own servers (itscomponent_<id>inventory group, still in inventory order), and a backup contacts only the service's server, so an unreachable server the component or service is not on no longer fails them; a component with no servers in the inventory, or a service on no inventory server, now fails before any host is contacted (#220)- A component on several servers no longer ends a failed deploy on two releases: when a later server fails its health gate, the servers that switched to the release earlier in the run go back to the release and service unit they had (a first release is stopped), last one first, each gated again, and the failure names them (#222)
- The release receipt
<receipts>/<component>.jsonnames every server of the deploy instead of only the last one, which overwrote the others (#222) - A failed deploy whose component has a release command says the command already ran and is not reverted; the engine README says a release command must work with the previous release too (#222)
- Deploying the release that is already
currentunpacks into a freshreleases/<release>.redeploy-<UTC time>directory instead of writing into the live one, so a failed health gate rolls back to the untouched copy;rollbackto that release id takes its newest copy, and receipts still name the release id (#223) - A failed health gate restores the service unit the previous release ran under, not only the
currentsymlink, then gates the restored release on its health check; a restored release that fails it fails the deploy with its own message (#223) deploy.yml --checkruns to its end: the dry run reports the uv release it would install and skips the digest check, the unpack anduv syncof an archive it never transferred (#223)- Old releases are removed: once a deploy passes its health gate on a server, it keeps
deployment.retainReleasesreleases underreleases/and removes the older ones, a release's redeploy copies with it, always keeping the deployed release,currentand the release before it whatever their order; a failed gate removes nothing, check mode only reports what would go, and a directory that is not a release id is left alone (#224) - A rollback to a release that is no longer on the server says it may have been removed by a deploy's cleanup and names the releases still kept there (#224)
- The PostgreSQL restore drill compares the row count of every table the dump holds with the restored table and fails on a difference or on a dump without a table; it counted
pg_classrows before, which an empty database has too, so a restore that brought back nothing passed (#225) - The restore drill's scratch database keeps to PostgreSQL's 63-byte names (
<database>_restore_check, the name cut to 49 characters), andcloudfall config validateand the converge refuse a declared database named as another's scratch database, asCONFIG_INVALIDwith codeservice_database_restore_check_name(#225) - The test suite passes on a checkout whose
config/examples/tmp/holds rendered inventory or observations (astask inventoryand CI leave it): every test that copies the examples now goes through one helper,sdk/tests/examples_project.py, that leaves that folder behind, and a test refuses a direct copy (#226) - Docs that no longer matched the code:
ARCHITECTURE.mdnow says a project pinscloudfall==<version>, that decision records are committed while runtime state stays undertmp/, and whichTaskfile.ymltasks call Ansible directly; theAGENTS.mdthatcloudfall initwrites names thecloudfall==<version>pin instead of arevpin;CONTRIBUTING.mdnamesconfig/instead of astate/directory; the CLI evaluation underevaluations/cloudfall/uses repository-relative paths (#229) - An operator proposal left
runningno longer blocks the operator for good: once closed asinterruptedit stops blocking new proposals for the same alert or drift, and the refusals ofdecisions approveandoperator approvename the command that closes it (#244) - A decision record or proposal receipt write now flushes its directory after the rename, so a power loss cannot bring the old record back (#244)
- A redeploy copy (
releases/<release>.redeploy-<UTC time>) that fails its health gate is removed by the deploy's rollback once the previous copy runs again, sorollback <release>takes the newest copy that passed instead of switching to the failed one; the copy's time now has microseconds, so two redeploys in the same second get two directories (#245) healthcontacts only the component's own servers and fails before any host when the component has no servers in the inventory, instead of reporting success with nothing probed; anddeploy,rollbackandrestartstop at the first of the component's servers they could not reach, where before Ansible skipped the unreachable server and went on to the next one (#246)- The PostgreSQL restore drill counts each restored table without its inheritance children (
count(*) FROM ONLY), as the dump's COPY data holds them, so a database with classic table inheritance no longer fails the drill every night after a correct restore (#247) config validaterefuses a storage pool device path with.., avolumeGroupof.or.., and a pool naming one device as bothprimaryandsecondary(storage_pool_same_physical_volume); before, the first two passed validation and crashedrequirements show(#248)baseline,bootstrap,services,data,domains,logging,timeandbackupstop at the first server they could not reach, asdeploy,rollbackandrestartdo since #246, where before Ansible skipped the unreachable server and changed the next one; the check lives in one shared task file,playbooks/tasks/rollout-start.yml, andinspectstill goes on past a server it cannot reach, reporting it as missing (#254)- An operator proposal closed as
interruptednow suspends autonomy for its operation kind the way afailedone does, until a human-approved run verifies, and an autonomous run closed asinterruptedcounts towardmaxAutonomousPerHourfrom itsinterruption.confirmedAt; before, a killed unattended run left the next proposal for the same alert free to run unattended while what the first run changed was unknown (#258) cloudfall lock clear <component>no longer deletes a server's lock that a run took again after it was shown:lock.ymlcompares the shown token and removes the lock in one command on the server, instead of reading it and deleting it in two tasks, so a lock taken since is kept and the command exitsCONFLICT; and with no lock found while a server could not be reached it exitsCONFLICTwithcomponent_lock_not_clearedandunreadServers, instead of answeringnoopas if no run held the lock (#271)cloudfall baseline convergekeeps a host's own smartdDEVICESCANoptions (-sself-test schedules,-Wtemperature limits) and replaces only its-M exec, adding-m rootwhen the line has no-m; a failed restart of mdadm's monitor or smartd now fails the run instead of reporting success, and only a unit the host does not have is skipped (#281)logging deploy's plan shows each receiver channel's bot token: a newchannelsfield lists{channel, tokenStored}, and a token not in the store is inmissingastelegram-bot-token of <channel>, the name--yesrefuses with, where the plan used to say nothing was missing (#282)- A command that runs a playbook on one server refuses a Server whose id is one of Ansible's implicit group names (
all,ungrouped) withlifecycle_limit_reserved_groupbefore anything runs:--limit allreached every host and--limit ungroupednone (#283) - A storage pool's
primaryandsecondarymay be stable paths such as/dev/disk/by-id/nvme-<model>_<serial>-part5:logging deployandinspectresolve each one on the host withrealpath, the volume provider and audit'svolume.<name>compare the device it names with LVM's physical volumes,lvcreateandlvextendname the physical volume as LVM reports it, and audit shows each declared path next to its device (resolvedPhysicalVolumes); before, any path but the kernel name failed with "is not in vg_data", and NVMe kernel names can swap between boots. A pool path that does not exist on the host, and a pool whoseprimaryandsecondaryresolve to one device, are refused naming the server, pool and path (#293) logging deployholds Loki, Grafana, Alloy, and Prometheus and Alertmanager when they have a declared version, at the LoggingStack's declared versions: each role writes/etc/apt/preferences.d/cloudfall-logging-<package>(Pin: version <declared>,Pin-Priority: 999) before its pinned install, so a laterapt-get dist-upgradeno longer moves them, and removes the file of a package now installed without a declared version;cloudfall auditadds alogging.packages[<package>]check per pinned package on each of the stack's hosts, reporting drift when the installed version is not the declared one (#294)logging deploychecks, before anything is installed, that each service user can reach the files it reads (Grafana's admin password, Alertmanager's SMTP password and Telegram token files, Alloy's client TLS files) through every existing parent directory, and stops naming the directory that blocks it, as in/etc/cloudfall is 0700 root:root; grafana can't reach /etc/cloudfall/logging/grafana-admin-password, where Grafana used to fail only when it started; the host's directory modes are left as they are (#295)cloudfall-engine playbook run <playbook> --engine <dir>runs a playbook of an engine directory that has noservices.yml, where it used to crash withHANDLER_CRASHEDandplaybook does not exist: .../services.ymlwhatever playbook was asked for; aservices.ymlrun still decrypts and stages the owner passwords as before (#296)logging deployno longer leaves a host whose pinned logging package is already newer than declared (Grafana 13.3.1 installed, 13.2.6 declared) with a hold that makes every laterapt-get dist-upgradeand unattended-upgrades abort on a downgrade: the holds are written atPin-Priority: 999, which keeps a held package from moving up without ever forcing a downgrade, and before writing them each role compares the installed version with the declared one (dpkg --compare-versions, a dry run too) and stops on a newer one, naming the package and both versions and saying to declare the installed version or downgrade the package by hand, with no hold written on that host (#302)- A missing Elasticsearch exporter credential no longer stops Alloy on the whole host: the collector's systemd drop-in gives each
LoadCredential=aSetCredential=default, so when the node'scloudfallpassword or project CA is gone (the service taken out of the project, its files cleaned up) Alloy still starts and only that exporter's scrapes fail; the nextlogging deploydrops the exporter and its credential lines (#326) - Adding or removing
remoteAccesson an Elasticsearch service withmetrics.enabledno longer leaves the collector scraping the node the old way until the nextlogging deploy: the services converge re-renders the Alloy configuration and its systemd drop-in on each logging collector wherelogging deployalready set Alloy up, validates it, and restarts Alloy only when a file changed, so the exporter switches between plain HTTP and HTTPS with the project CA in the same run; a host that is not a collector, or whose Alloy is not set up yet, is left untouched (#329) - The PostgreSQL restore drill (
cloudfall backup verifyand the nightly restore-check timer) passes--file=-topg_restorewhen it reads the dump's row counts; without it every PostgreSQL since 12 refuses withone of -d/--dbname and -f/--file must be specified, so the drill failed on every run (#340) - A bootstrap or baseline after a deploy no longer reports changes on each component's
releases/andshared/: the deploy now creates them0750, as bootstrap does, instead of flipping them to0755, so a converge after a deploy is idempotent; the Nginx worker still reaches a static route's release through its own ACL entry (#341) - On a host running more than one service, such as
h1inconfig/exampleswith PostgreSQL and Redis, the Redis backup no longer fails withPermission denied: the PostgreSQL, Redis and Elasticsearch roles now create the shared backup root/var/backups/cloudfallroot-owned0755before their own backup directory, instead of letting the first service's directory task create it with its own owner and0700, and the next services converge repairs a root already owned bypostgres; each service's own directory keeps its owner and mode (#342) - An approved operator proposal now runs on the one server it names: every playbook of
converge-servicesandconverge-baseline, the Elasticsearch node preparation included, runs with--limit <server>, a drift proposal's verify step inspects that server alone, and the receipt names only the owner passwords set there; before, a staging alert approved by the staging policy converged every declared server, production included (#350) - Operator autonomy is now earned per environment: the history an
OperatorPolicygrant is judged on, and the rule that suspends it after afailedorinterruptedreceipt, count only the receipts of servers in the policy's environment, and a receipt of a server no longer declared counts for none; before, verified runs on staging earned autonomy in production, and a failed staging run suspended production (#351) decisions approveruns only what was checked: a proposal's record now binds the sha256 of its playbook and verify playbook and, when the project is in a git work tree, its HEAD commit and a digest of its uncommitted changes (leaving out the decisions directory,record/andtmp/), and approve recomputes them and the diff file's sha256 before anything runs; any difference is refused withPRECONDITIONanddecision_binding_changed, naming what changed and asking for a new proposal, and the record staysproposed. Before, an edited playbook, role or diff file ran unseen under the old diff's approval. Outside git only the two playbooks and the diff are bound (#352)- A failed deploy or a rollback brings the previous environment back with the previous release: each release now reads its own environment file,
shared/env/<component>.<release>.env(a redeploy copy gets one of its own), named by its unit and read by its release command and command health check, so a deploy that fails only because of a bad environment value leaves the previous release running, and restarting, on the environment it had;rollbackpoints the unit at the target release's file. A deploy without--env-filecopies in the file the previous release reads, a component with no environment still namesshared/env/<component>.env, which no deploy writes or removes any more, and retention removes a removed release's environment files (#353) - A baseline run now removes an SSH key declared
lifecycle: revokedfrom root'sauthorized_keys, after the active keys are in, matching the key's type and body whatever its comment or options; keys the project doesn't declare stay (D-22). The rendered Ansible inventory'scloudfall_ssh_public_keysnow carries a server's revoked keys too, each with itslifecycle, so the role can remove them (#354) - Two proposals of one operation in the same second no longer share a decision id: the id is reserved before the check run writes anything, and a second proposal in that second takes
<operation>-<stamp>-2, then-3, so each record keeps its own diff and host stats, instead of the second overwriting the first one's diff and then failing withdecision_exists; a proposal whose check run fails frees its id and leaves no record (#356) - An operator verify step that errors (the alert feed unreachable, the audit failing) now settles the receipt
failedwith the error as its detail, so autonomy for that operation kind is suspended, and the command still fails with the error; before, the receipt stayedexecutedfor good and blocked its trigger (#358) decisions approveno longer settles a decisionverifiedwhen its verify step observed no host: every host the check run or the approved run left recap counts for must have counts from the run and a task the verify step ran on it, and a verify play that matched no host, skipped a targeted one or skipped every task on it settles the decisionfailed, with a verdict naming the hosts that were not observed. Before, a verify step that exited 0 and listed no host as changed proved the run, even when it looked at nothing. A verify playbook aimed at other hosts than the run's (such aslocalhostchecking a URL) now settlesfailedtoo; aim it at the hosts the operation targets (#360)- Docs: D-10 in
docs/decisions.mdread as if every component already ran in the full systemd sandbox; a status line now says the unit sets onlyUser,Group,NoNewPrivileges=trueandPrivateTmp=true, that secrets reach it through the per-releaseEnvironmentFile(D-21) and deploy tasks run as root, and that the rest of the sandbox is not built yet (#362) - Docs gave the deploy order as health-check gate before symlink switch:
ARCHITECTURE.mdnow says the switch and the restart come first, so the gate runs on the live release, andARCHITECTURE.md,docs/ci-deploy-guide.mdand the Render and Neon migration guides say a rollback, automatic orcloudfall rollback, does not undo the release command, so a migration must stay compatible with the release before it (#363) - A deploy no longer fails on one slow download of the pinned uv release: the deploy role retries the download five times, ten seconds apart, still checking its SHA-256, before failing, and a test over every role now requires each
get_urldownload to retry a bounded number of times (#372) - A redeploy copy (
releases/<release>.redeploy-<UTC time>) whose release command fails is removed with its environment file, as one that fails its gate is, sorollback <release>takes the newest copy that passed instead of starting one that never passed its release command or gate; the previous copy and its own environment file stay (#373) - A deploy run with Ansible's
--diffno longer prints the component's environment file, secrets included: the tasks that install a release's environment file or carry the previous one over setdiff: false(#374) - An observed server snapshot written before 0.8.0, which names its baseline
spec.serverType, validates againstobserved-server.schema.jsonagain: the schema takesspec.serverTypeorspec.baseline, exactly one of the two, and the reader keeps readingserverTypeas the baseline with no removal date, since a record's fields are never deprecated withincloudfall/v1(#376) - An operator proposal recorded with the audit check
server_type.idkeeps verifying oncebaseline.idis compliant, and keeps blocking a new proposal for the same drift, with no removal date instead of until 0.9.0, since a record's fields are never deprecated withincloudfall/v1(#387) - A re-converge of a PostgreSQL, Elasticsearch or logging service no longer needs the network for its package repository's signing key: the PostgreSQL, Elastic and Grafana roles download the key only when it is missing (a
get_urlwithoutforcestill asked the server whether the key had changed, so a slowwww.postgresql.orgfailed a converge that should change nothing), keep the installed key's owner and mode, and retry a failed download five times, ten seconds apart, before failing (#388) - Docs promised a gate the agent does not control for every server-changing action:
ARCHITECTURE.md,docs/brownfield-design.mdandMANIFESTO.mdnow say it holds for declared operations (a person approves at a terminal) and the operator (a person at a terminal or anOperatorPolicygrant), while in project mode thecloudfall mcp serveconfirmation (yes: true,confirm_destructive: true) is the agent's own, except for a release whose application declaresapproval: manual; policy-gated self-confirmation is planned (#391) SECURITY.mdsays who the private-reporting rule is for: outside reporters use private vulnerability reporting; the maintainers track findings that need existing access as public issues, and keep findings a stranger could exploit remotely private until a fix is released (#392)- A PostgreSQL owner role whose database names a
passwordRefno longer reports a change on every services converge: the role reads the owner's stored verifier, computes the password's SCRAM-SHA-256 verifier on the controller with that verifier's salt and iteration count (a fresh salt when the role has no SCRAM verifier), and runsALTER ROLEonly when the two differ or the role can't log in, so a converge with the same password changes nothing (D-25). The controller now stages the password itself in its private0600file, which never leaves the controller; only the verifier reaches the host, as before (D-9). A stored password that is neither a SCRAM-SHA-256 nor an md5 verifier stops the run, naming the role, never the value (#398) config validaterefuses an IPv6 network in a MariaDB service'sremoteAccess.allowedSourceswithservice_mariadb_remote_access_ipv6, naming the source and saying a single IPv6 address (/128) or an IPv4 network is accepted. Before, such a config validated, the firewall rendered a rule for it, and the converge refused it only after the host had made its certificate key. MariaDB's account host has no netmask form for IPv6, so an IPv6 network never converged; PostgreSQL keeps accepting IPv6 networks (#434)auditnow seesREQUIRE SSLremoved by hand from a MariaDB owner account at an allowed source (ALTER USER ... REQUIRE NONE): inspect reads each remote account'sssl_typeinto a new optionalsslTypeslist ofmariadbAccounts, and themariadb.accountscheck reports drift for a remote owner account whosessl_typeisn'tANY(what converge'sREQUIRE SSLsets), naming the account and its type. A record an older inspect wrote, which has nosslTypes, now reports that check as unknown instead of compliant when the service has remote owner accounts, until the next inspect; local accounts are checked as before (#435)- A MariaDB replica declared
primarywhose link has not connected (its IO thread stillConnecting, so it reportsMaster_Server_Id: 0) is no longer promoted over a running, writable primary: the role finds its source by the link'sMaster_HostandMaster_Porttoo, matched against the hostname, addresses and port the inventory now renders for each server of the pair, reads it, and stops withPRECONDITIONwhile it is writable. A link to a source that is no declared service and is not running is promoted only when the controller can't reach the source's port either (#440) cloudfall auditno longer reportsservices.binddrift on a compliant MariaDB primary that listens on itsreplication.listenAddressas well as loopback: a MariaDB listener on the service's declaredreplication.listenAddressorremoteAccess.listenAddressat its port is accepted, compared as a parsed IP, and the check's desired state names the replication address. A primary not listening on its replication address is drift, as a remoteAccess service not listening on its listenAddress already was. Any other non-loopback listener, a wildcard (0.0.0.0,::) or an undeclared IPv6 address included, is still drift, and the check is unchanged for other service kinds (#445)- A MariaDB replica declared
primarywhose link's source is no declared service is no longer promoted over a source the controller merely can't see: the role reads the replica's ownLast_IO_Errnofirst and refuses withPRECONDITIONwhen its last error came from a running source (access denied, a TLS failure), and the controller's probe of the source's port now tells a refused connection from a timeout, no route or a name it can't resolve. Only a refused connection promotes; the others leave the source unknown and refuse withPRECONDITION, naming what the controller couldn't read. A link whoseserver_idmatches no declared service is matched by its host and port whateverserver_idit reports, as the code already did, and the comments now say so (#446) - A new MariaDB replica on an ext4 data volume is seeded on its first converge instead of refused with
service_mariadb_replica_unattachable ... holds 1 database(s): the role no longer counts or lists a#mysql50#name, which is how MariaDB shows a directory that is no database (the volume'slost+found) until the managed settings ignore it. A replica's backup and restore drill leave it out the same way, so they never try to dump or check it (#449) - On MariaDB 11.8, where
read_onlyreadsONrather than1in the role's replication state query, the role now parses both servers' state through one filter that takesON,NO_LOCKandNO_LOCK_NO_ADMIN(or1to3) as read-only andOFF(or0) as writable, and fails on any other value. A second converge of a healthy replica no longer turns writes off again and reports a change (S-019-13), and a promotion from a declared source that is read-only and applied now goes through instead of waiting for it to turn read-only (S-019-15) (#450) - A MariaDB role change now converges in either host order of the serial
services.ymlplay. In a planned swap, a server declared the replica of a primary that still replicates from it turns read-only and waits (demote) when that primary's link names its host and port but has not connected yet (Master_Server_Id: 0, as after a restart), instead of refusing withservice_mariadb_replica_unattachablewhile the other server waits for it. A replica whose primary holds nocloudfall_replaccount for it yet, and whose primary's server converges later in the same run, turns read-only and answers the newwait-primaryoutcome instead of failing with MariaDB's "is not allowed to connect"; it attaches in the next converge. A primary that already converged, or whose declaration does not admit the replica, still stops the converge. The answer of a wait (demote,wait-promotion,wait-primary) now carries itsreason, which says to converge again (#451) - On MariaDB 11.8, where
JSON_VALUE(Priv, '$.account_locked')reads1rather thantruefor a locked account, the role and inspect now find locked accounts byIN ('true', '1'), through one condition in the role's defaults that every read of locked accounts uses. A declared user that was locked is unlocked again, a replica'scloudfall_replaccount locked while it was no replica is unlocked so its link logs in, and locking the account of a server that left the pair no longer reports a change on every converge (S-019-13); audit sees the accounts the role locked (S-019-7) (#452) cloudfall services convergeenforces each host's declared firewall, the rulesetcloudfall baseline convergeapplies with the services' port rules in it, before it converges that host's services, so after a MariaDB role change the new primary admits its new replica in the same converge instead of timing out until a baseline converge ran. A host whose baseline declares no firewall is left alone (#453)- A MariaDB replica declared
primarywhose link's source is no declared service is no longer promoted over a running source that only rejects the controller: the probe of the source's port now runs on the replica's own server (a small Python script of the role,files/mariadb_source_probe.py), which the source admitted while it replicated from it, instead of on the controller, where a firewall that REJECTs everyone but the replica gave the same refused connection. The replica'sLast_IO_Errnois still read first; a timeout, no route or a name the replica's server can't resolve still leaves the source unknown and refuses withPRECONDITION. Spec 019's Promotion section now says "can't be reached" is read from the replica's side (#455) cloudfall-engine inventory render --outputkeeps thecloudfall_servershosts in rollout order instead of sorting them by name, so aserial: 1play run from the file converges a multi-node Elasticsearch cluster's home and each MariaDB primary's server first, as S-012 and S-019 require, whatever the servers are named. Every other key in the file stays sorted, so a project without such a home renders the same file as before. The stdout answer holds the same inventory, with its keys sorted as every treaty answer's are (#465)- On MariaDB 11.8,
cloudfall auditno longer reports a MariaDB server as unavailable becauseread_onlyreadsONrather than1, and no longer lists the data volume's#mysql50#lost+foundas an undeclared database. Inspect now readsread_onlyin the forms the role's replication filter takes (ON,NO_LOCKandNO_LOCK_NO_ADMIN, or1to3, as read-only;OFFor0as writable), reads any other value as a role it could not read, so audit reports that server's role unknown and still inspects the rest of the fleet, and leaves every#mysql50#name out of the databases it reads, by the same pattern as the role. Tests keep inspect's values and pattern equal to the role's (S-019-7, S-019-19) (#468) cloudfall observeno longer reads a snapshot an earlier run left behind as this run's: it deletes the requested servers' snapshots before the inspect playbook runs, so a host that failed inspect, or one the run stopped before reaching, is listedmissingand audit reads it asunknowninstead of judging its old state; a--limitrun keeps the other servers' snapshots (#470)- A MariaDB primary's and replica's scheduled dumps now record the GTID position of their snapshot (
mariadb-dump --gtid --master-data=2, as the replica seed does), in commented lines a load never runs, so the restore drill still loads them as they are. A replica records its own binary log position, which holds its primary's GTIDs, and its replication is not stopped for the dump. A standalone server writes no binary log, so its dumps carry no position. Spec 019 and the service guide now say every MariaDB service keeps its own backup instead of suggesting backups on the replica alone, and ROADMAP.md and ARCHITECTURE.md list MariaDB as built (#473) - A granted MCP run whose agent record the proposal store can't save now exits
PRECONDITIONbefore any engine step, naming the store folder (agent_record_unwritable) or, for an identical call started in the same second, the existing record (agent_record_exists), instead of an internal error; a record that can't be settled after the run started no longer masks the run's own error (#489) services.ymlrefuses a server that declares a service of a kind it has no role for, instead of converging it as a silent no-op (#506)backup.ymlruns a service's backup on its declared server alone, so a service rendered on several nodes no longer fails the one-server check, and knows the RabbitMQ kind (#506)config validatecompares the nodes of a RabbitMQ service by short hostname, the name RabbitMQ gives a node, soh2.aandh2.bare refused withservice_rabbitmq_hostname(#506)- 0.7.x now declares
requires-python = ">=3.14,<3.15": itsansible-core2.21 fails at start on Python 3.15 (AttributeError: module 'dataclasses' has no attribute '_is_type'), so an install on 3.15 now picks Python 3.14 or is refused, instead of installing a release whose every playbook fails; 0.8.x runs on 3.15 (#516) cloudfall doctorreads theansible-playbookversion only from ansible's ownansible-playbook [core X.Y.Z]line: an ansible that fails at start no longer passes the check with the Python version its traceback names, such as3.15; the check fails withno version in the output of ansible-playbook --version, anddata.dependenciesnow listsansible-playbook(#524)cloudfall-engine doctorreads theansible-playbookversion only from ansible's ownansible-playbook [core X.Y.Z]line, ascloudfall doctordoes: an ansible that fails at start no longer passes the check with the Python version its traceback names, such as3.15; the check fails withno version in the output of ansible-playbook --version, anddata.dependenciesnow listsansible-playbook(#529)- The RabbitMQ restore drill no longer writes into its scratch directory by path as root: the
rabbitmquser owns that directory's parent and could swap the fresh directory for a link in the window before the copy, so root's copy of the export (which holds password hashes) and the scratch node's configuration could land wherever the link pointed. The scratch tree is now made, written (the export0600, the cookie0400), read and removed as therabbitmquser, root only reads the export from its backup directory and streams it in, and a link left at the scratch path is removed, not followed (#544) domains.ymlnow fails before it changes any server when a Domain it would render has noproxy.upstream, and names every such Domain; before, the site role's assert stopped the run only on that Domain's server, afterserial: 1had already converged the servers before it.upstreamstays optional in the schema, for a Domain the fleet routes with its own proxy (#90)- A deploy of a component with
healthCheck: {type: none}no longer fails its health gate on the missing HTTP fields and stops the release: the gate now requires the systemd unit to be active right after the restart and still active, with no automatic restart in between, 10 seconds later, so a worker that crashes or crash-loops on start still fails and rolls back (#96)
Deprecated
kind: ServerType, theserver-types/folder,Server.spec.serverType, thecloudfall_server_typescatalog variable and a host'sserver_type,cloudfall add server-typeandcloudfall add server --typekeep working until 0.9.0;cloudfall config validatewarnsDEPRECATED_NAMEfor each file using an old name, and a server setting bothserverTypeandbaselinefails withserver_baseline_conflict(#198)- Until 0.9.0 the old output names stay beside the new ones with the same values: the rendered Ansible inventory's host variable
cloudfall_server_type,inventory show'sserver_typesand each server'sserver_type,config validate'sby_kind.ServerType, andaudit's per-serverserver_type(#198)