isothrum 2.1.0

Added

• Error::ServerAuthFailed, code server_auth_failed: the server a run is on (Proxmox VE,
  ESXi; the proxmox and esxi targets, Vagrant on vmware_esxi) refused the run's credentials,
  an API token revoked or a password changed. A step whose tool failed in the server's words for
  a refused sign-in (an HTTP 401 or 403 from its API, Terraform's Proxmox provider's
  authentication failure, ovftool's incorrect user name or password, vagrant-vmware-esxi's
  ESXi host access and Authentication failed for user) fails with it instead of
  tool_failed, in an operation and in a status read, naming the server as the run's params do
  (server: the API's host, the ESXi host), so an embedder can say which credentials to update.

Fixed

• A Proxmox VM removed or powered off in Proxmox's web UI reads so. A Terraform run's status
  read its local state alone, which changes only at the next apply or destroy: a VM removed on the
  server read as running. The status of a run whose VMs have Proxmox's console now asks the
  server's API where they stand (cluster/resources, with the run's connection: its token, else a
  ticket for its password): a VM the server no longer has reads absent (a resume refuses it as
  gone, a start again rebuilds the run), one powered off stopped. A server that doesn't answer,
  or a run whose connection gives no credentials, leaves the state's view; a server refusing the
  credentials is server_auth_failed.

• The tools a killed operation left running are ended by the next operation. Each tool runs
  in a process group of its own (a job object on Windows), so a process killed outright (SIGKILL,
  Task Manager, a crash) stopped none of them: a docker compose up ran to its end seconds later,
  a vagrant up could run an hour, and the next start waited on Vagrant's machine lock. An
  operation's tools are now recorded in the run's state folder while they run (tools/: each
  one's process id and start, under a lock the recording process holds while it lives), and
  every operation, settling a killed one with the run locked, ends the tools a dead process left
  first: its process group asked to stop (SIGINT), killed after 30 seconds; on Windows its tree
  (taskkill /T /F). Only the very process recorded is ended: an id that now names another
  process is left alone. ProcessSpec has a tools field (the folder a tool is recorded in).

• A download the network cut runs again. One connection reset by peer in the middle of a
  1.93 GB image layer, pulled in a VM, failed the whole start. A download (a step the manifest
  marks download, a Compose start's pull, a Vagrant start's vagrant box add) that fails with a
  network error that passes (a connection reset, refused or timed out, a TLS handshake timeout, an
  unexpected EOF, a name not resolved for now, a registry's gateway answering 502, 503 or 504,
  curl's 18, 28, 35 or 56) now runs again, up to three more times, 5, 15 and 45 seconds apart, each
  new try said in a retrying warning with the step and the line that failed (as the runner's
  other retries are); Docker keeps the layers it finished, so the next pull takes up where the last
  one ended. A refusal (a sign-in, toomanyrequests, an image or a box the registry doesn't have)
  still fails at once, as does a download that stalled, and an operation cut short stops waiting
  with it.

• A status read's errors mask the connection's secrets. A status read emits nothing, so its
  errors quoted what a tool printed of the connection's values as it was (a status step's failure,
  and now a Proxmox API refusing the run's token or password): they are masked as an
  operation's are.