Skip to content

spawn-ts 0.7.0 — Go-parity audit + browser-native data movement

Latest

Choose a tag to compare

@scttfrdmn scttfrdmn released this 01 Aug 22:14
· 12 commits to main since this release
888816b

The Go-parity audit (#57),
plus browser-native data movement. Six per-command gaps were filed and closed;
two of the six were cost-safety bugs rather than missing features (#54, #55),
which is what a per-command judgement surfaces and a line-count comparison never
would. MINOR, not PATCH: launch now refuses a spec it previously accepted
(#51), and the unbounded-launch guard changed which options satisfy it (#55).

Added

  • --min-viable for job arrays (#52) — a threshold on the set, which is a
    different thing from onFailure. onFailure decides whether to keep
    launching; stop leaves the members that already came up running, so a
    100-member array that reaches 2-of-100 is two instances billing indefinitely for
    a job that cannot be done. --min-viable N states the count below which the
    whole array is pointless, and spawn-ts then does what Go's cohort.Reconciler
    does for the stated reason "Drain surviving instances so nothing idles and
    bills"
    :
    • Fast-fail — the moment the threshold becomes unreachable, the unstarted
      members are skipped rather than launched. Without this, an array with
      --min-viable 50 that lost 51 members would still launch the other 49.
    • Drain — JobArray.enforceViability() terminates the survivors, and
      SpawnClient.pumpFanOuts calls it on every tick, so the wind-down needs no
      caller action. It emits an action/terminate event with rule: "min-viable";
      a survivor that could not be terminated emits a warning instead of being
      swallowed, because that is exactly the case where money keeps being spent.
      Each survivor is reported exactly once, including when two pumps overlap
      (startJobArray kicks one without awaiting it) — one instance, one event, not
      two that read as two instances wound down. A failed terminate stays
      retryable.
    • The gate is a monotone latch (skipping raises the lost count, which keeps
      it non-viable), and nonViable: false means "not yet ruled out" — never
      "confirmed viable". completed counts toward viability, or a fully successful
      array would turn non-viable as it drained.
    • The value is clamped to [1, size] as Go clamps it, since --min-viable 200 on a 100-member array is an obviously-intended "all of them". A
      malformed value is rejected by the CLI rather than clamped: Number("hlaf")
      would land on the no-op 1 and silently disable the guard.
    • FanOutSummary gains minViable/viableCandidates/nonViable/
      missingIndexes. The last is the sparse-index view Go's missingIndexes
      provides: "97 of 100 running" hides which three slices have no worker, and a
      terminated member's index counts as missing too.
    • The dashboard card states the threshold, and when the array goes
      non-viable it says so in red with what follows ("survivors are being
      terminated") and its state chip reads non-viable rather than the green
      done. Without that a user watches running members disappear with no reason
      given, and an array that was torn down looks like one that succeeded.
      minviable.harness.html reaches that state — it needs capacity failures, which
      no control in the UI can produce, so it is the state most likely to be left
      broken.
    • Lives in FanOut (report) + JobArray (enforce) because FanOut owns no
      lifecycle authority — every other state change it makes is a launch, and a
      shared engine that silently terminated instances would surprise the sweep and
      queue callers.
  • MPI declaration tags for arrays (#52) — --mpi and
    --mpi-processes-per-node N stamp spawn:mpi-enabled /
    spawn:mpi-processes-per-node on every member, decoded onto
    ManagedInstance.mpi and shown in spawn status, so a spawn-ts-launched array
    is recognisable as an MPI job by the Go CLI and the portal.
    This is the tag half only, and that boundary is deliberate rather than
    unfinished: Go's pkg/mpicohort is a self-declared spike whose header states the
    unresolved problem — cohort's Placement is per-entity while a placement group
    and an EFA fabric are collective constraints. Porting a spike would commit
    spawn-ts to a shape Go is still deciding. EFA validation must run in the launch
    region (blocked on truffle-ts#33) and --auto-placement-group creates a real AWS
    resource. Absence stays absence: no mpi-enabled=false is ever written and
    status never prints "mpi: no".
  • docs/execution-shapes.md — the three shapes (single node / job array /
    MPI), what --min-viable guarantees, sparse indexes, and an explicit out-of-reach
    table for logs/collect/retry --failed quoting Go's own reason (a local
    launch record under ~/.config/spore/arrays/ that "must run from the machine
    that launched the array").
  • Browser-native Globus Transfer (#53, new ./transfer subpath) — data
    movement in and out of launched instances with no local machine and nothing
    installed. Globus Transfer is not Globus Connect Personal: it moves data
    between managed collections (an HPC DTN, an S3 collection) over REST, and every
    endpoint is CORS-enabled. transferClient() covers endpoint_search,
    submit_transfer, task/task_list, cancel, and an awaitTask polling loop —
    all over an injected fetch, so it is unit-tested with no credentials.
    • submitTransfer fetches a submission_id first: that is Globus's idempotency
      mechanism, and skipping it lets a network retry move the data twice.
    • An empty item list is refused — Globus would accept it and return a task that
      succeeds having moved nothing, which reads as a silent failure.
    • GlobusTransferError carries Globus's own code, and needsConsent
      separates a fixable missing consent from a flat permission denial (both 403).
    • INACTIVE is not terminal, and niceStatus is surfaced so a stuck task says
      why.
  • The Globus Transfer scope, opt-in — GlobusConfig.requestTransfer adds
    TRANSFER_SCOPE and completeLogin returns GlobusTokens.transferToken from
    the same sign-in (Globus issues one token per resource server). Deliberately
    not in DEFAULT_SCOPE: that would show every signing-in user a consent
    screen about managing their transfers and reading their files, including users
    who only want to see their instances. An absent transferToken is a normal
    outcome, not an error.
  • Plugin detection and launch-time declaration (#53). Two columns, not one:
    the browser can declare 7 of 12 plugins at launch and detect all 12.
    • parsePluginTag / detectPlugins / instancePlugins decode the
      spore:plugin:<name> provenance tag. An absent or unparseable tag reads as
      "unknown", never "not installed"; verify=none stays distinct from a missing
      verify=; unknown key=value pairs are preserved rather than dropped.
      Go's "(none)" digest placeholder is normalised away.
    • LaunchSpec.plugins writes /etc/spawn/plugins.json in user-data, which
      spored reads at startup — byte-compatible with Go's plugin.Declaration.
      Written before spored starts, unlike the Go bootstrap, which appends it
      after systemctl start spored; spored reads the file once at startup, so that
      ordering is a race.
    • canDeclareAtLaunch / validateDeclarations refuse the other five with
      distinct reasons
      . Four need a locally-minted secret pushed to the instance
      and would park at StatusWaitingForPush — a limitation of Go's own async path
      (pkg/pluginruntime/runtime.go:62), not a browser gap. spore-sync's local
      half is mutagen on the developer's machine and stays the CLI's job.
      Rejections are returned, not thrown, so a caller can launch with what works
      and still say what was dropped.
    • See docs/data-movement.md.
  • The four tag-derived status notices (#56), as pure src/core/notices.ts
    returning structured { kind, level, text, detail } so the CLI, dashboard and
    portal render one source three ways. spawn status previously answered "what did
    you configure" and nothing else. The headline gap: spored writes
    spawn:dns-status + spawn:dns-error when registration fails, specifically so
    the failure isn't buried in the instance's journal (spawn#435) — and nothing read
    them, so a portal user got an FQDN that never resolved and no explanation, while
    the diagnostic sat in a tag DescribeInstances had already returned.
    • lifecycleProtection() — who enforces the deadline, when it falls, and the
      worst-case compute cost to it. The counterpart to accumulatedCost(), which
      reports only what's been spent; the ceiling is the number that says whether to
      worry.
    • dnsNotice() — the failure, with spored's own recorded reason.
    • sporedUpgrade() + compareSemver() — a port of libs/update's comparison,
      so the two tools never disagree about whether an upgrade exists.
    • elasticIpNotice() — an EIP on a stopped instance keeps billing (~$3.60/mo)
      precisely because nothing is using it, which is exactly when the user believes
      they've stopped paying.
    • Absence is never smoothed into reassurance: no dns-status tag does not mean
      "registered", and no supplied latest version does not mean "up to date".

Changed

  • launch now refuses a spec whose identity can't be resolved (#51) —
    buildLaunchTags takes an optional LaunchIdentity and stamps the base-identity
    block Go writes (root, created-by, version, account-id, account-base36,
    iam-user, account-name, plus os and local-username). This was the audit's
    load-bearing find: a spawn-ts launch was visible in spawn list yet invisible
    and unterminatable in the portal, which filters its list, lookup and terminate
    on spawn:iam-user (the last 403s on a mismatch) while spawn list filters on
    spawn:managed alone. The divergence therefore surfaced only when someone tried
    to clean up — by which time the instance had been billing the whole time.
    • Identity arrives as data, so the tag builder stays pure; EC2Provider
      resolves it once via GetCallerIdentity and caches it, or takes it from the
      caller (the federated BYOA path already has the ARN and account id back from
      AssumeRoleWithWebIdentity).
    • Refusing to launch is deliberate, including for the 200-with-empty-fields case
      a bare try/catch would sail past: omitting the tag and launching anyway produces
      an orphaned billable instance nobody can terminate from the portal — strictly
      worse than a failed launch.
    • local-username is resolved once and used for both the tag and user-data.
      They must agree or spawn connect SSHes to a user that doesn't exist, so a
      custom username with no tag silently sent it to ec2-user.
    • Also stamps spawn:active-ports.

Fixed

  • The unbounded-launch guard was inverted (#55) — it accepted the weakest of
    the three bounds and refused a stronger one:

    bound Go accepts spawn-ts accepted
    --ttl yes yes
    --idle-timeout yes no
    --cost-limit no yes

    costLimit is a soft limit — spored polls accumulated compute-seconds
    against spawn:cost-limit, so if spored never starts (failed bootstrap, wrong
    instance profile, crash-loop) nothing enforces it. Only the TTL is enforced from
    outside the box, by the ttl-reaper Lambda reading spawn:ttl-deadline without
    the instance's cooperation. Worse, findOrphans skips any instance whose deadline
    is 0, so a cost-limit-only instance was invisible to orphan detection too: the
    guard was waving through exactly the launches nothing downstream can catch, and
    saying nothing while it did. New pure src/core/bounds.ts holds one predicate
    carrying the enforcement distinction (external vs on-instance) that was the
    missing type. idleTimeout now counts; costLimit alone still permits the launch
    — refusing it would be a new, harsher divergence from Go — but warns, naming the
    consequence and the orphan-detection blind spot. Both launch paths share it.

  • extend could hand back a deadline in the past (#54). It added the duration
    to the existing deadline with no floor, so extending an instance already overdue
    produced a deadline still behind now — reported as success, then reaped on the
    ttl-reaper's next pass. It failed precisely when it mattered most: the instance
    you are trying to rescue is by definition the overdue one. As with #55 the bug
    existed twice — SpawnClient.extend and the CLI's extend each had their own
    copy of the arithmetic and the same three defects — so the fix is one shared pure
    computeExtension, placed next to ttlDeadline, the rule it has to agree with.

    • The floor: max(old + by, now + by). The two rules compose — a live
      instance still gets old + by, so stop/start still buys nothing; only an
      overdue one is pulled forward. Both paths say when the floor engaged, because
      the user asked for one deadline and got another.
    • Both TTL tags: spawn:ttl is recomputed from the launch anchor alongside
      spawn:ttl-deadline. Two TTL tags that disagree are a trap for whichever
      enforcer reads the stale one. With no usable anchor the tag is omitted rather
      than guessed.
    • Nudges spored to reload rather than waiting for its next poll.
  • Expired federated credentials in the direct demo (#50) — Globus→STS sessions
    are capped at 1 hour, and a tab left open past that failed at the next action with
    a raw Request has expired, which says nothing about what to do. The expiration
    was already on AwsCreds; nothing read it. The demo now mirrors the portal's
    SessionController shape (two surfaces shouldn't invent two models): a session
    clock, an amber warning 5 minutes out so an in-flight launch isn't started against
    lapsing creds, and at expiry a red banner offering re-sign-in with launch/terminal
    disabled, the monitor loop stopped and any SSM terminal torn down. Checked at the
    moment of each click, not only on the timer — a laptop asleep past the deadline
    doesn't fire timeouts on schedule. The loop stops with a note that the instance
    still self-terminates on its TTL, which is the reassuring part: spored owns the
    deadline, not the tab.

  • Two stale claims in docs/integration.md (#58) that changed which features
    read as portable. The "live AWS data needs credentials a browser can't safely hold
    and hits CORS" paragraph was false on both counts — ec2 / api.pricing /
    servicequotas each return access-control-allow-origin: * when preflighted from
    Origin: https://spore.host, and credsFromIdToken already puts short-lived STS
    credentials in the tab. BundledFinder stays the default for cost and latency,
    which is the real reason. And "wire-compatible" read as complete when Go stamps 55
    launch tags to spawn-ts's 33; the gap is now named in the same tier language as
    the rest of the section.

  • The SSM session tests no longer race a fixed sleep (#61). Four assertions
    waited setTimeout(r, 5) for an async chain — WebCrypto digests, then
    markReady's un-awaited flushPending — whose duration is a property of the
    machine, not of the code. 5 ms was enough idle and not enough alongside 29 other
    test files in parallel CI workers. Each now waits on the condition it is actually
    about (ready, a flushed Output frame, onClose having fired) via a shared
    waitFor(pred, describe) poll with a 2 s ceiling, so the test's duration tracks
    the machine while its verdict tracks only correctness. This mattered more than
    an ordinary flake because the flakiest assertion is the one covering input
    being queued rather than dropped
    — the terminal's core correctness property, and
    a test everyone learns to re-run is exactly how a real regression in the flush
    path gets waved through. Verified by slowing the flush path to 25 ms: the old
    sleeps fail, the waits pass; and by four mutations of session.ts, each still
    caught.