v0.4.0

clusterctl v0.4.0

This release shows how far a long command has got, runs on many hosts at once what still went one host at a time, and decrypts Secret documents with the sops command, which takes the binary to a third of its size. It also removes the configuration keys nothing read and the -o jsonpath format, and gives the exit code of a command on many hosts one rule. Some of these changes refuse configuration, command lines or Secret documents that v0.3.0 accepted, which a minor release before 1.0.0 may do; Upgrading from v0.3.0 lists them.

Progress

• --progress, or CLUSTERCTL_PROGRESS, chooses what standard error shows while a command runs. auto, the default, draws a live tree of the steps and the hosts under way when standard error is a terminal and standard output does not go into a pipe, and nothing otherwise; tty asks for the tree, counter for a single line, plain for lines a CI log can keep, and none for nothing. Standard output never carries progress, and without a terminal standard error holds what it held in v0.3.0, byte for byte, unless plain is asked for.
• A display that has run for a second or more ends with one summary line, such as clusterctl: provision reinstall: failed in 18m03s: 478 ok, 2 failed. The summary and the plain lines are written for people and may change; programs read the event log.
• --progress-log FILE, or CLUSTERCTL_PROGRESS_LOG, appends every event to a file as a line of JSON, in format version 1, and creates the file with mode 0600. A TRACEPARENT in the environment gives the log its trace; neither it nor TRACESTATE is handed on to the programs clusterctl runs.
• --progress tty or --progress counter where neither can be drawn, and a --progress-log that cannot be used, are usage errors (exit 2). The variables fail no command: they show nothing, or write no log, and say why in one line.
• The MCP server sends progress notifications for its tool calls, and its audit log records each call's trace.

The Progress page of the manual describes the displays and the event log.

Many hosts at once

• dns lookup, provision status, secrets push, doctor --remote, bmc power through the ipmitool backend, fabric state, and the MCP tools describe_nodes and plan_change now work on many hosts at once rather than one after another.
• Two new settings bound them: services.dns.maxConcurrent, 16 by default, and bmc.ipmi.maxConcurrent, 8. --fanout on the command line lowers these and the Redfish limit but never raises them, and is handed to ipmipower; CLUSTERCTL_FANOUT and fanout.max leave them alone.
• doctor --remote asks each role in one session. A role whose account logs in but runs nothing, such as one with a nologin shell, now fails.
• The MCP server runs at most two tool calls at once; the others wait, reported as waiting for another tool call.

Secret documents are decrypted by the sops command

A workstation that reads a sops-encrypted Secret needs sops 3.10.0 or later, in PATH or named by workstation.sopsBinary or CLUSTERCTL_SOPS_BINARY; clusterctl doctor checks for it. Commands that use no secret do not need sops. The linux/amd64 binary shrinks from 50.5 MB to 17.3 MB, and fixes to sops and its cloud SDKs now come with your sops update rather than with a clusterctl release. clusterctl still makes every check it made before sops runs, and hands sops only the bytes it has checked. It now also refuses sops metadata with a field it does not know, and master keys both inside and outside key_groups, where sops would ignore the groups.

Exit codes

A command on many hosts now exits with the worst of its hosts' codes, in one order: 130, then 3, then 2, then 1.

• secrets push with one node refused and one unreachable exits 3 (was 1).
• exec, copy, cinc, fabric hca and node hardware exit 2 (was 1) when a node's configuration is at fault.
• cinc with one node interrupted and one unreachable exits 130 (was 3).
• hostkey verify, hostkey refresh, bmc ping, dns lookup and dns aliases keep rules of their own, which the exit code reference now lists.
• secrets push refuses two secrets for one target (exit 2), and exits 3 when a node that failed could not be reached for any of its secrets.
• A remote command is also stopped locally, with exit 3, once its timeout, the grace period and the time it may take to reach the host have passed, so a host that hangs without dropping the connection no longer holds a command up.
• A command on an infrastructure host whose output was cut off at the limit exits 3 (was 1), as Slurm's commands already did.
• An ageFile credential whose file or identities cannot be read exits 2 (was 1), as an ageFile secret does.

Configuration

• Keys nothing read are gone, and refused on load as unknown: bmc.ipmi.passwordTransport, bmc.pdu.credential, fanout.connectTimeout, services.cinc.archivePath, services.cinc.baseCookbook, services.cinc.rolesPath, services.fabric.guidFormat, services.http.root, services.http.baseUrl, services.mail.* and workstation.pager. CLUSTERCTL_PAGER goes with the last.
• An explicit "" or 0 for a key the defaults set, such as services.dhcp.configPath or bmc.redfish.maxConcurrent, is refused; it used to be read as the default. services.tftp.root and services.tftp.logPath may no longer be empty.
• The context, tunnel and PDU users must be portable POSIX user names. alice@EXAMPLE.ORG and svc$ now fail config validate, with their line, where they used to validate and then fail every connection, and config init --user refuses them too. .alice is now accepted, as connecting already did.
• A ProxyJump host that is not a role is checked as a host name, so gw_1.example.org is refused.

Boot

• New: boot grub log shows what the TFTP server, in.tftpd, tftpd, atftpd or dnsmasq-tftp, wrote into services.tftp.logPath.
• boot grub set refuses a target that is not a file under services.tftp.root (exit 2, with --dry-run too), and writes the link relative, so that a TFTP server confined to its root, such as in.tftpd -s, finds it.

Output

• -o jsonpath=… is gone: it exits 2, as an unknown format. -o jq=… does what its templates did, and the output guide gives the jq for each.
• boot log, boot sync, dhcp log and fabric counters honour -o json and -o yaml, as a list of lines.
• Table columns that hold CJK characters or emoji line up.
• A failed remote command reads <program> on <host> exited N: <what it said> everywhere.
• slurm job summary -o json rows have the key order of the MCP tool's, and credential names and ties in the job summary come in a fixed order.
• fabric state prints the ports that answered when the fabric stops early, and asks the fabric in a dry run too.

Other fixes

• A Ctrl-C at a password prompt ends the reading of credentials: provision status used to ask again for every node, and could leave the terminal without echo. A credential lookup that failed is remembered rather than tried again for every node.
• A worker's panic is written after a password prompt another node asks, never into it.
• Host key scans stop once interrupted.
• cinc run gets its 30 minutes, or fanout.commandTimeout when that is longer.
• Idle Redfish connections time out, and are closed after a fan-out.
• scp's progress meter is shown only for a single transfer.
• ipmitool lines longer than 400 bytes are cut and end in ….
• tunnel start, tunnel stop and config use-context complete their names when --config is given.
• mcp serve --set is parsed as every other command's is.
• A Redfish client or an IPMI backend printed in a debug line no longer shows its password.
• describe_nodes no longer stops reading groups at the first node that fails.
• A path such as ~bob/x is no longer read as $HOME followed by bob/x.
• A cached remote file with a modification time in the future is no longer trusted for ever. The cache format changed, so old entries read as misses once.

For Go programs that import nodeset

nodeset.Resolver now holds only what parsing calls, Resolve and All. List and DefaultSource moved to the new, optional Lister interface, which MapResolver implements. A resolver written for v0.3.0 still satisfies Resolver; code that called List or DefaultSource through a Resolver asserts Lister now. Folded output and error messages are unchanged. The package is to move to a module of its own, github.com/GSI-HPC/go-nodeset; a later release will say when.

Upgrading from v0.3.0

The command line gains --progress, --progress-log and boot grub log, and loses -o jsonpath; the schema is still clusterctl/v1alpha1. Check these before you upgrade:

• sops on every workstation that reads a Secret document, 3.10.0 or later, in PATH or named by workstation.sopsBinary.
• Every workstation before the site's configuration. v0.3.0 refuses a key it does not know, so upgrade everyone before a site sets services.dns.maxConcurrent or bmc.ipmi.maxConcurrent.
• The removed keys, in every layer, and explicit empty values for keys the defaults set.
• User names and jump hosts that the stricter rules refuse.
• Scripts that read exit codes or use -o jsonpath. See Exit codes and Output.
• xargs on the IPMI gateway and the fabric host. The ipmitool backend and fabric state now run xargs -0 -P there, which a BusyBox built without those options lacks.
• GRUB targets outside services.tftp.root, which boot grub set refuses.
• Secret documents whose sops metadata has a field clusterctl does not know, or master keys both inside and outside key_groups.

clusterctl config validate reports the configuration changes, with file and line; clusterctl doctor finds a missing sops, and clusterctl doctor --remote an xargs the hosts lack.

mise offers a release only once it is 24 hours old. To take this one sooner, name it:

$ mise use -g github:GSI-HPC/clusterctl@0.4.0

Download the binary for your platform below, or:

$ go install github.com/GSI-HPC/clusterctl/cmd/clusterctl@v0.4.0

Changelog

Features

• eafa6a2: feat(app): let --fanout lower the Redfish fan-out, never raise it (@claude)
• 363cad9: feat(app): tell whether standard error and output are terminals, and their size (@claude)
• 13d9f47: feat(boot): check a GRUB target against services.tftp.root (@claude)
• 77835de: feat(boot): show the TFTP service log with boot grub log (@claude)
• 51d8eeb: feat(cli): add --progress and a counter on standard error (@claude)
• 7ff3721: feat(cli): name the steps of reinstall, and end IPMI and fabric targets as answers arrive (@claude)
• d594448: feat(cli): open a command span around every leaf command, and trace every ssh call (@claude)
• b391e85: feat(cli): write progress events to --progress-log, and adopt TRACEPARENT (@claude)
• 03d3af5: feat(display): hold the lines written beside the command while a question is asked (@claude)
• 6223bc7: feat(display): name the program and the noun of the targets as parameters (@claude)
• b000043: feat(fanout): add Batches, which runs batches in turn with a pause between them (@claude)
• 1bb915c: feat(fanout): add Map, one bounded pool for any kind of work, and run the executor on it (@claude)
• dab4621: feat(mcp): send progress notifications, and record each call's trace in the audit log (@claude)
• 9ed8578: feat(progress): add span-shaped progress events that nothing emits yet (@claude)
• 1b023f5: feat(progress): add the plain renderer and an end summary line (@claude)
• de7c967: feat(progress): draw a live tree for --progress=tty (@claude)
• 5ccefcb: feat(progress): name the program in the panic line and the event log (@claude)
• a4bbc90: feat(progresstest): export Checked and Watch, with one rule for when Check runs (@claude)
• 2aa586b: feat(secrets): decrypt Secret documents with the sops command (@claude)
• 295cdb7: feat(termtext): add Truncate, write the escape policy down, and test to 100% (@claude)
• b9b9a2b: feat(transport): tee remote lines for a display, and hand complete lines to a parser (@claude)
• 6b062da: feat: classify remote timeouts, refused accounts and pin mismatches (@claude)
• 3f71a83: feat: report Redfish requests, copies, lookups and the confirmation as spans (@claude)

Fixes

• a7fa9a4: fix(cli)!: refuse two secrets for one target in secrets push (@claude)
• 015b9ff: fix(cli): complete tunnel names and use-context from the deduplicated flags (@claude)
• 9c0b692: fix(cli): give cinc run the 30 minutes it was meant to have (@claude)
• b78d76e: fix(cli): honour -o in the commands that pass on what a host printed (@claude)
• dd44bb7: fix(cli): keep the fabric ports that answered when fabric state stops early (@claude)
• fa713d5: fix(cli): read the DHCP configuration once per command (@claude)
• 5413ac7: fix(cli): stop scanning host keys once interrupted, through one bounded loop (@claude)
• 8483231: fix(cli): write the panics of pool workers after a question, never into it (@claude)
• cf590bc: fix(credentials): remember a failed lookup, and read nothing once interrupted (@claude)
• 0d11ce1: fix(fanout): end an interrupted step canceled, and class a fan-out's failure by its exit code (@claude)
• 2a24c3c: fix(fanout): name the failed items as a list when a name is no host name (@claude)
• b1a460d: fix(groups): look a group up once however many callers ask, and remember what a source lacks (@claude)
• 01dc92f: fix(mcp): parse --set of mcp serve as every other command does (@claude)
• 59418fe: fix(mcp): run at most two tool calls at once (@claude)
• 261dc21: fix(output): line up table columns that hold wide characters (@claude)
• 21a9e20: fix(redfish): time out idle connections and close them after a fan-out (@claude)
• b374654: fix(slurm): count queued jobs one way for the command line and the agent (@claude)
• f8e2523: fix(transport): bound a remote command locally as well as on the host (@claude)
• 7f7e845: fix(transport): keep scp's progress meter off the terminal when copies run side by side (@claude)
• 5be67b9: fix: print a Redfish client or an IPMI backend without its password (@claude)
• 369ed96: fix: run the ssh -V probes under the command's context (@claude)
• 951259a: fix: send a password helper's stderr and a panic's stack to the front end's diagnostics (@claude)

Documentation

• 00ccbbd: docs(nodeset): write the package documentation for any program (@claude)
• 156b929: docs(progresstest): say which parts of a tree stay as they are (@claude)
• 0e47946: docs: describe progress in the manual (@claude)
• c445da2: docs: describe the package graph as it is (@claude)
• 07979d8: docs: give the node set engine's tests a document of their own (@claude)
• 190f3d8: docs: leave the user's view of the configuration to the manual (@claude)
• dd403d9: docs: name termtext as the one escaper (@claude)
• e6000a4: docs: record why Secret documents are decrypted with the sops command (@claude)
• 15bea5f: docs: record why a release tag may be signed with OpenPGP (@claude)
• c65132a: docs: record why progress is our own events, and how pools bound work (@claude)
• 19caafb: docs: relate ADR 0019 to #90 measure 19 and requirement R69 (@claude)
• fcc3f57: docs: split selecting nodes out of the node set language (@claude)

Other

• e7b4c3d: build(deps): Bump golang.org/x/net in the go-modules group (@dependabot[bot])
• cad0b8e: perf(cli): ask the fabric about four ports at a time, the list on stdin (@claude)
• a3a345c: perf(cli): check the roles of doctor --remote in parallel, one session each (@claude)
• 36dc000: perf(cli): push each node's secrets without waiting for the others (@claude)
• e25cbab: perf(cli): read the power state and ssh of provision status at once (@claude)
• e39552c: perf(cli): resolve the names of dns lookup in parallel (@claude)
• 0c091dd: perf(ipmi): run ipmitool for several processors at once (@claude)
• 53e99e1: perf(mcp): read the groups, nodes and jobs of describe_nodes and plan_change at once (@claude)
• d87409e: refactor(app): name the Redfish pin store in one place (@claude)
• f7b2ad4: refactor(app): read workstation.identities once for age files and sops (@claude)
• c9e2571: refactor(bmc): run the power batches through fanout.Batches (@claude)
• dff7491: refactor(cli): declare tables by column name, marking the wide and right ones (@claude)
• 768b287: refactor(cli): delete bmcUnreachable, which never matched (@claude)
• a19da92: refactor(cli): drop Go fallbacks that repeat defaults.yaml (@claude)
• e49ccf1: refactor(cli): let the dry run signal reach report (@claude)
• 68d11f5: refactor(cli): open remote sessions through one session helper (@claude)
• fe02889: refactor(cli): pass the context to the helpers that reach hosts (@claude)
• d17337f: refactor(cli): resolve the command context in r.run, not in every command (@claude)
• 6ea9cbc: refactor(cli): run the Redfish fan-out and host key scans on fanout.Map (@claude)
• f70c968: refactor(config): remove bmc.pdu.credential, which nothing read (@claude)
• 16f61f7: refactor(config): remove fanout.connectTimeout, which nothing read (@claude)
• efb4e30: refactor(config): remove the configuration fields nothing reads (@claude)
• 902a991: refactor(config): report unknown keys from the validator's own errors (@claude)
• 505bb06: refactor(credentials): take paths and age decryption from app (@claude)
• 91d6ebf: refactor(exitcode): one rule for the exit code of many hosts (@claude)
• 211e886: refactor(fanout): separate the pools, which know no program, into internal/clikit/fanout (@claude)
• a269779: refactor(hostname): one rule for user names, CheckHost for jump hosts (@claude)
• 0e88b96: refactor(nodeset)!: narrow Resolver to what parsing calls (@claude)
• 5652181: refactor(nodeset): delete the branches no input can reach (@claude)
• 51ec070: refactor(output): drop -o jsonpath for the embedded jq (@claude)
• 199e23c: refactor(progress)!: take the exit-code rule as a Classify fallback (@claude)
• 17bc387: refactor(progress): escape and measure text with termtext (@claude)
• 924b8f0: refactor(shellquote): delete Split, which only the tests called (@claude)
• 9bc0952: refactor(slurm): check for control characters with unicode.IsControl (@claude)
• 6b16aeb: refactor(slurm): own the power-off job check in internal/slurm (@claude)
• 780e702: refactor(termtext): move the escaper and display widths out of output (@claude)
• f454025: refactor: adopt the Go 1.26 idioms go fix and cmp offer (@claude)
• 6c8df75: refactor: collect sorted map keys with slices.Sorted(maps.Keys(m)) (@claude)
• 767f264: refactor: delete exported functions and fields production never reaches (@claude)
• 5f94d60: refactor: drop results every caller discards, and an unused parameter (@claude)
• 2b98fac: refactor: fill the defaults of a collected request in App.Collect (@claude)
• 8d4a9a7: refactor: keep one disk cache, fileutil.ReadCache and WriteCache (@claude)
• 16207c3: refactor: say why a remote command failed in one place, Result.Check (@claude)
• 2bc04f7: refactor: take the short name of a host from naming.Short (@claude)

---

The manual covers every command.
Verify a download against checksums.txt.