Repository navigation
The Go-parity audit (#57),
plus browser-native data movement. Six per-command gaps were filed and closed;
two of the six were cost-safety bugs rather than missing features (#54, #55),
which is what a per-command judgement surfaces and a line-count comparison never
would. MINOR, not PATCH: launch now refuses a spec it previously accepted
(#51), and the unbounded-launch guard changed which options satisfy it (#55).
Added
--min-viablefor job arrays (#52) — a threshold on the set, which is a
different thing fromonFailure.onFailuredecides whether to keep
launching;stopleaves the members that already came up running, so a
100-member array that reaches 2-of-100 is two instances billing indefinitely for
a job that cannot be done.--min-viable Nstates the count below which the
whole array is pointless, and spawn-ts then does what Go'scohort.Reconciler
does for the stated reason "Drain surviving instances so nothing idles and
bills":- Fast-fail — the moment the threshold becomes unreachable, the unstarted
members are skipped rather than launched. Without this, an array with
--min-viable 50that lost 51 members would still launch the other 49. - Drain —
JobArray.enforceViability()terminates the survivors, and
SpawnClient.pumpFanOutscalls it on every tick, so the wind-down needs no
caller action. It emits anaction/terminateevent withrule: "min-viable";
a survivor that could not be terminated emits a warning instead of being
swallowed, because that is exactly the case where money keeps being spent.
Each survivor is reported exactly once, including when two pumps overlap
(startJobArraykicks one without awaiting it) — one instance, one event, not
two that read as two instances wound down. A failed terminate stays
retryable. - The gate is a monotone latch (skipping raises the lost count, which keeps
it non-viable), andnonViable: falsemeans "not yet ruled out" — never
"confirmed viable".completedcounts toward viability, or a fully successful
array would turn non-viable as it drained. - The value is clamped to
[1, size]as Go clamps it, since--min-viable 200on a 100-member array is an obviously-intended "all of them". A
malformed value is rejected by the CLI rather than clamped:Number("hlaf")
would land on the no-op1and silently disable the guard. FanOutSummarygainsminViable/viableCandidates/nonViable/
missingIndexes. The last is the sparse-index view Go'smissingIndexes
provides: "97 of 100 running" hides which three slices have no worker, and a
terminated member's index counts as missing too.- The dashboard card states the threshold, and when the array goes
non-viable it says so in red with what follows ("survivors are being
terminated") and its state chip reads non-viable rather than the green
done. Without that a user watches running members disappear with no reason
given, and an array that was torn down looks like one that succeeded.
minviable.harness.htmlreaches that state — it needs capacity failures, which
no control in the UI can produce, so it is the state most likely to be left
broken. - Lives in
FanOut(report) +JobArray(enforce) becauseFanOutowns no
lifecycle authority — every other state change it makes is a launch, and a
shared engine that silently terminated instances would surprise the sweep and
queue callers.
- Fast-fail — the moment the threshold becomes unreachable, the unstarted
- MPI declaration tags for arrays (#52) —
--mpiand
--mpi-processes-per-node Nstampspawn:mpi-enabled/
spawn:mpi-processes-per-nodeon every member, decoded onto
ManagedInstance.mpiand shown inspawn status, so a spawn-ts-launched array
is recognisable as an MPI job by the Go CLI and the portal.
This is the tag half only, and that boundary is deliberate rather than
unfinished: Go'spkg/mpicohortis a self-declared spike whose header states the
unresolved problem — cohort'sPlacementis per-entity while a placement group
and an EFA fabric are collective constraints. Porting a spike would commit
spawn-ts to a shape Go is still deciding. EFA validation must run in the launch
region (blocked on truffle-ts#33) and--auto-placement-groupcreates a real AWS
resource. Absence stays absence: nompi-enabled=falseis ever written and
statusnever prints "mpi: no". docs/execution-shapes.md— the three shapes (single node / job array /
MPI), what--min-viableguarantees, sparse indexes, and an explicit out-of-reach
table forlogs/collect/retry --failedquoting Go's own reason (a local
launch record under~/.config/spore/arrays/that "must run from the machine
that launched the array").- Browser-native Globus Transfer (#53, new
./transfersubpath) — data
movement in and out of launched instances with no local machine and nothing
installed. Globus Transfer is not Globus Connect Personal: it moves data
between managed collections (an HPC DTN, an S3 collection) over REST, and every
endpoint is CORS-enabled.transferClient()coversendpoint_search,
submit_transfer,task/task_list, cancel, and anawaitTaskpolling loop —
all over an injectedfetch, so it is unit-tested with no credentials.submitTransferfetches asubmission_idfirst: that is Globus's idempotency
mechanism, and skipping it lets a network retry move the data twice.- An empty item list is refused — Globus would accept it and return a task that
succeeds having moved nothing, which reads as a silent failure. GlobusTransferErrorcarries Globus's owncode, andneedsConsent
separates a fixable missing consent from a flat permission denial (both 403).INACTIVEis not terminal, andniceStatusis surfaced so a stuck task says
why.
- The Globus Transfer scope, opt-in —
GlobusConfig.requestTransferadds
TRANSFER_SCOPEandcompleteLoginreturnsGlobusTokens.transferTokenfrom
the same sign-in (Globus issues one token per resource server). Deliberately
not inDEFAULT_SCOPE: that would show every signing-in user a consent
screen about managing their transfers and reading their files, including users
who only want to see their instances. An absenttransferTokenis a normal
outcome, not an error. - Plugin detection and launch-time declaration (#53). Two columns, not one:
the browser can declare 7 of 12 plugins at launch and detect all 12.parsePluginTag/detectPlugins/instancePluginsdecode the
spore:plugin:<name>provenance tag. An absent or unparseable tag reads as
"unknown", never "not installed";verify=nonestays distinct from a missing
verify=; unknownkey=valuepairs are preserved rather than dropped.
Go's"(none)"digest placeholder is normalised away.LaunchSpec.pluginswrites/etc/spawn/plugins.jsonin user-data, which
spored reads at startup — byte-compatible with Go'splugin.Declaration.
Written before spored starts, unlike the Go bootstrap, which appends it
aftersystemctl start spored; spored reads the file once at startup, so that
ordering is a race.canDeclareAtLaunch/validateDeclarationsrefuse the other five with
distinct reasons. Four need a locally-minted secret pushed to the instance
and would park atStatusWaitingForPush— a limitation of Go's own async path
(pkg/pluginruntime/runtime.go:62), not a browser gap.spore-sync's local
half is mutagen on the developer's machine and stays the CLI's job.
Rejections are returned, not thrown, so a caller can launch with what works
and still say what was dropped.- See
docs/data-movement.md.
- The four tag-derived
statusnotices (#56), as puresrc/core/notices.ts
returning structured{ kind, level, text, detail }so the CLI, dashboard and
portal render one source three ways.spawn statuspreviously answered "what did
you configure" and nothing else. The headline gap: spored writes
spawn:dns-status+spawn:dns-errorwhen registration fails, specifically so
the failure isn't buried in the instance's journal (spawn#435) — and nothing read
them, so a portal user got an FQDN that never resolved and no explanation, while
the diagnostic sat in a tagDescribeInstanceshad already returned.lifecycleProtection()— who enforces the deadline, when it falls, and the
worst-case compute cost to it. The counterpart toaccumulatedCost(), which
reports only what's been spent; the ceiling is the number that says whether to
worry.dnsNotice()— the failure, with spored's own recorded reason.sporedUpgrade()+compareSemver()— a port oflibs/update's comparison,
so the two tools never disagree about whether an upgrade exists.elasticIpNotice()— an EIP on a stopped instance keeps billing (~$3.60/mo)
precisely because nothing is using it, which is exactly when the user believes
they've stopped paying.- Absence is never smoothed into reassurance: no
dns-statustag does not mean
"registered", and no supplied latest version does not mean "up to date".
Changed
launchnow refuses a spec whose identity can't be resolved (#51) —
buildLaunchTagstakes an optionalLaunchIdentityand stamps the base-identity
block Go writes (root, created-by, version, account-id, account-base36,
iam-user, account-name, plusosandlocal-username). This was the audit's
load-bearing find: a spawn-ts launch was visible inspawn listyet invisible
and unterminatable in the portal, which filters its list, lookup and terminate
onspawn:iam-user(the last 403s on a mismatch) whilespawn listfilters on
spawn:managedalone. The divergence therefore surfaced only when someone tried
to clean up — by which time the instance had been billing the whole time.- Identity arrives as data, so the tag builder stays pure;
EC2Provider
resolves it once viaGetCallerIdentityand caches it, or takes it from the
caller (the federated BYOA path already has the ARN and account id back from
AssumeRoleWithWebIdentity). - Refusing to launch is deliberate, including for the 200-with-empty-fields case
a bare try/catch would sail past: omitting the tag and launching anyway produces
an orphaned billable instance nobody can terminate from the portal — strictly
worse than a failed launch. local-usernameis resolved once and used for both the tag and user-data.
They must agree orspawn connectSSHes to a user that doesn't exist, so a
custom username with no tag silently sent it toec2-user.- Also stamps
spawn:active-ports.
- Identity arrives as data, so the tag builder stays pure;
Fixed
-
The unbounded-launch guard was inverted (#55) — it accepted the weakest of
the three bounds and refused a stronger one:bound Go accepts spawn-ts accepted --ttlyes yes --idle-timeoutyes no --cost-limitno yes costLimitis a soft limit — spored polls accumulated compute-seconds
againstspawn:cost-limit, so if spored never starts (failed bootstrap, wrong
instance profile, crash-loop) nothing enforces it. Only the TTL is enforced from
outside the box, by the ttl-reaper Lambda readingspawn:ttl-deadlinewithout
the instance's cooperation. Worse,findOrphansskips any instance whose deadline
is 0, so a cost-limit-only instance was invisible to orphan detection too: the
guard was waving through exactly the launches nothing downstream can catch, and
saying nothing while it did. New puresrc/core/bounds.tsholds one predicate
carrying the enforcement distinction (external vs on-instance) that was the
missing type.idleTimeoutnow counts;costLimitalone still permits the launch
— refusing it would be a new, harsher divergence from Go — but warns, naming the
consequence and the orphan-detection blind spot. Both launch paths share it. -
extendcould hand back a deadline in the past (#54). It added the duration
to the existing deadline with no floor, so extending an instance already overdue
produced a deadline still behindnow— reported as success, then reaped on the
ttl-reaper's next pass. It failed precisely when it mattered most: the instance
you are trying to rescue is by definition the overdue one. As with #55 the bug
existed twice —SpawnClient.extendand the CLI'sextendeach had their own
copy of the arithmetic and the same three defects — so the fix is one shared pure
computeExtension, placed next tottlDeadline, the rule it has to agree with.- The floor:
max(old + by, now + by). The two rules compose — a live
instance still getsold + by, so stop/start still buys nothing; only an
overdue one is pulled forward. Both paths say when the floor engaged, because
the user asked for one deadline and got another. - Both TTL tags:
spawn:ttlis recomputed from the launch anchor alongside
spawn:ttl-deadline. Two TTL tags that disagree are a trap for whichever
enforcer reads the stale one. With no usable anchor the tag is omitted rather
than guessed. - Nudges spored to reload rather than waiting for its next poll.
- The floor:
-
Expired federated credentials in the direct demo (#50) — Globus→STS sessions
are capped at 1 hour, and a tab left open past that failed at the next action with
a rawRequest has expired, which says nothing about what to do. The expiration
was already onAwsCreds; nothing read it. The demo now mirrors the portal's
SessionControllershape (two surfaces shouldn't invent two models): a session
clock, an amber warning 5 minutes out so an in-flight launch isn't started against
lapsing creds, and at expiry a red banner offering re-sign-in with launch/terminal
disabled, the monitor loop stopped and any SSM terminal torn down. Checked at the
moment of each click, not only on the timer — a laptop asleep past the deadline
doesn't fire timeouts on schedule. The loop stops with a note that the instance
still self-terminates on its TTL, which is the reassuring part: spored owns the
deadline, not the tab. -
Two stale claims in
docs/integration.md(#58) that changed which features
read as portable. The "live AWS data needs credentials a browser can't safely hold
and hits CORS" paragraph was false on both counts —ec2/api.pricing/
servicequotaseach returnaccess-control-allow-origin: *when preflighted from
Origin: https://spore.host, andcredsFromIdTokenalready puts short-lived STS
credentials in the tab.BundledFinderstays the default for cost and latency,
which is the real reason. And "wire-compatible" read as complete when Go stamps 55
launch tags to spawn-ts's 33; the gap is now named in the same tier language as
the rest of the section. -
The SSM session tests no longer race a fixed sleep (#61). Four assertions
waitedsetTimeout(r, 5)for an async chain — WebCrypto digests, then
markReady's un-awaitedflushPending— whose duration is a property of the
machine, not of the code. 5 ms was enough idle and not enough alongside 29 other
test files in parallel CI workers. Each now waits on the condition it is actually
about (ready, a flushedOutputframe,onClosehaving fired) via a shared
waitFor(pred, describe)poll with a 2 s ceiling, so the test's duration tracks
the machine while its verdict tracks only correctness. This mattered more than
an ordinary flake because the flakiest assertion is the one covering input
being queued rather than dropped — the terminal's core correctness property, and
a test everyone learns to re-run is exactly how a real regression in the flush
path gets waved through. Verified by slowing the flush path to 25 ms: the old
sleeps fail, the waits pass; and by four mutations ofsession.ts, each still
caught.