feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
Closed
bussyjd wants to merge 7 commits into
Closed
feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487bussyjd wants to merge 7 commits into
bussyjd wants to merge 7 commits into
Conversation
`obol sell inference` was producing ServiceOffers with three latent defects that together stopped /.well-known/agent-registration.json from ever being routed for inference-typed sells: 1. The inference subcommand never exposed `--register-*` flags (compare with `obol sell http` which does), even though the help text on `--register` says "registration is enabled by default". 2. `buildInferenceServiceOfferSpec` never wrote `spec.registration` onto the offer at all, so the controller's `reconcileRegistrationStatus` saw `Enabled=false` (zero value) and emitted `Registered=True Disabled` with no RegistrationRequest CR and no /.well-known HTTPRoute. 3. `buildInferenceServiceOfferSpec` hardcoded `spec.model.name = "ollama"` regardless of the actual `--model` value, so anything downstream that keyed off the model id (the controller's description default included) was looking at the wrong string. Surfaced today on spark2 while trying to fetch `https://inference.v1337.org/.well-known/agent-registration.json` — got the Traefik fall-through 404. Manual `kubectl patch` enabled registration, exposed a fourth defect: 4. `buildActiveRegistrationDocument` in the serviceoffer-controller unconditionally overwrote `Spec.Registration.Description` for inference offers with `"<model.name> inference via x402 micropayments"`, even when the operator had supplied an explicit description at sell time. This PR fixes all four: - Add `--no-register`, `--register-name`, `--register-description`, `--register-image`, `--register-skills`, `--register-domains`, `--register-metadata` to `obol sell inference`. - Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig` to the unqualified `sellRegistrationInput` / `buildSellRegistrationConfig` since they now serve both inference and http call sites. - Extend `buildInferenceServiceOfferSpec` to accept the resolved model name and the registration block, write `spec.model.name = <real model id>`, and merge the registration block into `spec.registration` when non-empty. - In the controller's `buildActiveRegistrationDocument`, only fall back to the model-aware description string when the operator left `Spec.Registration.Description` empty. Tests that would have caught the regression earlier: - `TestSellInference_Flags` now requires the six registration flags; their absence on `main` was the bug. - `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault` pins that defaults produce `spec.registration.enabled = true` and the offer name as `spec.registration.name`. - `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration` pins the `--no-register` opt-out. - `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins that operator-supplied name/description/image/skills/domains all survive into the spec verbatim. - `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins that `spec.model.name` reflects `--model`, not the historical "ollama" literal. - `TestBuildActiveRegistrationDocument_KeepsOperatorDescription` pins the controller-side fix: an operator description survives the buildActiveRegistrationDocument pass. - `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference` pins the other branch — inference offers with no operator description still get the model-aware default, not the generic one. Drive-by: `TestSellInference_Flags` was failing on `main` after #470 removed the `--price` default but didn't update the corresponding assertion. Updated to assert `--price` default = "" with a comment explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a populated spec.registration block". That made /.well-known/agent-registration.json get routed, but left the offer in Registered=False AwaitingExternalRegistration until someone manually ran `obol sell register`. The serviceoffer-controller's storefront filter (buildServiceCatalogJSON in render.go) requires Ready=True, which transitively requires Registered=True, so the offer was silently excluded from /api/services.json — the very feed the operator's own storefront UI consumes. obol sell http already auto-registers on the same code path. Mirror that here so the inference path reaches the same end state without a follow-up obol sell register step. Changes: - Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared decision predicate. The same gate now drives both the http and inference auto-register call sites; defensively returns false when the registration block is missing, disabled, malformed, or the tunnel URL is empty. - sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell succeed, if shouldAutoRegisterSell says yes, call autoRegisterServiceOffer with the resolved name/description/wallet pulled from the spec. Surface failures as warnings + a re-run hint rather than aborting the gateway start, because the underlying call needs gas on the target chain and we do not want a one-off RPC hiccup to block local dev. Test coverage that would have caught the regression: - TestShouldAutoRegisterSell - table-driven over six scenarios including the defensive cases (registration not a map, registration.enabled not a bool). Both call sites use the helper. - TestSellInferenceAction_InvokesAutoRegister - source-level guard that scans the sellInferenceCommand body for shouldAutoRegisterSell and autoRegisterServiceOffer. The bug we just fixed was "Action calls neither"; an innocent refactor could remove the calls without any unit-level signal otherwise. Operational note: the agent's remote-signer wallet needs a small ETH balance on the target chain (~0.20-0.50 USD typical) for the on-chain register tx. If the wallet is unfunded the warning fires and the offer stays in AwaitingExternalRegistration; the operator can fund the wallet and re-run obol sell register to finish.
…straint) The autoRegisterServiceOffer pre-flight check rejected any registration whose signer didn't match the offer's payTo wallet: registration signer 0xA... does not match the payment wallet 0xB... Use a matching signer, omit --wallet so the remote-signer wallet is used, or pass --no-register The error wording read like an ERC-8004 limitation but isn't. ERC-8004 treats the agent OWNER (msg.sender at register time) and the agent WALLET (settable post-mint via setAgentWallet) as independent addresses. x402 settlement honors the offer's spec.payment.payTo directly — buyers pay that address regardless of what the registry's getAgentWallet returns. The "hot signer, cold/multisig payee" split is the canonical pattern. The historic guard existed because the obol CLI never exposed setAgentWallet, so a mismatched registration left operators with no in-CLI recovery path. This change instead surfaces the split as an informational note + adds `obol sell update <name> --pay-to <new>` as the recovery surface (already in tree; just needed test coverage and the connection wired into the diagnostic). Changes: - signerPayeeDelegationNote(signer, payTo) returns a human-readable note when the two diverge (case-insensitive, whitespace-tolerant, empty on either side) and "" otherwise. Used by autoRegisterServiceOffer instead of the previous early-return. - buildSellUpdatePatch(payTo, chain, price) extracted from the inline sellUpdateCommand Action so the patch shape — the thing that actually hits the cluster — is testable without a live offer. Action calls the helper instead of inlining the same logic. Tests: - TestSignerPayeeDelegationNote — 6-case table: match, case-insensitive, whitespace, empty payTo, empty signer (defensive), true mismatch (assertions name the addresses + advise sell update). - TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard asserting the banned error wording is gone from autoRegisterServiceOffer and the soft-notice path is wired. Anyone re-introducing the check has to delete this test too, which forces them to read the rationale. - TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name> --pay-to 0xBooB` builds a patch that touches only spec.payment.payTo, not network or price. - TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over perRequest/perMTok/perHour: the unused keys are explicitly null so a switch (e.g. perRequest → perMTok) doesn't leave the previous key fighting through merge semantics. - TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no fields are set, and the error names the flags the operator should pass. - TestSellUpdate_PayToFlagSurface — `obol sell update` exposes --pay-to (with --wallet/--recipient/-w aliases via payToFlag), and --namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap. After `obol stack down` destroys the k3d cluster, all ServiceOffer custom resources are wiped from etcd. The descriptors on disk at ~/.workspace/config/inference/<name>/ survive, but nothing replays them when the cluster comes back, so operators had to manually re-run `obol sell inference <name>` for every offer after every stack up. This wires a resume step into `obol stack up`: after stack.Up returns successfully, the action iterates inference.Store, rebuilds the Service+Endpoints+ServiceOffer for each persisted deployment from the on-disk descriptor, and kubectl-applies them. The foreground host gateway is NOT auto-started — it is an interactive operator action and stack-up shouldn't launch long-running processes — but the cluster side is fully reattached so the operator's eventual `obol sell inference <name>` rerun hits a "service already healthy" reconcile instead of "build from scratch." Storage model: - inference.Deployment gains ModelName, ServiceNamespace, and Registration fields, persisted with `omitempty` so legacy descriptors written by older binaries still load (the new fields come back as zero values; the resume path either defaults sensibly or refuses with an actionable message). - The inference Action now resolves the registration block once and passes the same map to both store.Create() and the in-process ServiceOffer apply — guaranteeing on-disk state matches what the cluster sees. Resume path: - resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no kubeconfig (no cluster yet), warns + continues per-offer on errors. - resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec + kubectlApply, no foreground process. Returns an actionable error on missing ModelName so legacy descriptors surface a clear "recreate the offer" prompt rather than producing a broken ServiceOffer. - Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A resume failure is logged as a warning, not propagated — stack-up must succeed even if one descriptor is malformed. Tests: - TestStoreCreate_PersistsResumeFields: pins ModelName, ServiceNamespace, and Registration round-trip through JSON. Without this round-trip, the resume path silently loses operator customizations. - TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins backwards-compatibility — a JSON written by an older binary still loads, with the new fields as zero values. - TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil. - TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on disk, kubeconfig absent (post-purge, pre-up state) — resume must skip cleanly. - TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor with empty ModelName surfaces an actionable error naming the missing field and the recovery command. - TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty descriptor guard so one bad entry can't crash the loop. - TestStackUpAction_CallsResumeSellOffers: source-level guard that cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER stack.Up. Pinning the order — running resume before stack.Up would see no kubeconfig and skip every offer silently. Operational note: `obol sell http` offers don't yet have an on-disk descriptor (only `obol sell inference` persists), so they aren't covered by this resume pass. Filed as a follow-up — adding a parallel manifest store for `sell http` plus a third resume branch is its own PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit: 1. Gateway PID printed as -1. cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1 to the operator and pins -1 in the PID file. Snapshot the PID before the Release call. 2. Stale "Host gateways are not auto-started" footer. Carried over from the pre-auto-spawn version of the resume function. With the gateway now auto-spawned, the message is misleading. Replaced with a pointer to the gateway log path so operators can `tail -f` immediately. 3. /skill.md catalog used the strict Ready=True filter while /api/services.json used the new operationally-ready filter. The two surfaces should stay consistent — an offer that's usable for x402 payments shows up in BOTH the storefront feed and the operator- facing skill catalog. Switched buildSkillCatalogMarkdown to the same offerOperationallyReady predicate. No new tests; the existing TestOfferOperationallyReady_* / catalog tests cover the filter relax for both call sites since they share the predicate. The PID-reading order is a one-line semantic fix; the footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
This was referenced May 12, 2026
Collaborator
Author
|
Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the |
6 tasks
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #485 and #486. Single PR for the full "operator does
obol stack uponce and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.The work spans three originally-separate threads that turned out to be one coherent feature:
obol sell inferenceregistration parity —--register-*flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, andobol sell update --pay-topatch builder extracted so operators can re-point payees post-creation.obol stack upwalks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess soUpstreamHealthyflips True without operator intervention.Ready=Truerequirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and addsregistrationPending=trueto/api/services.jsonso UIs can badge the offer. Same store-and-resume pattern wired up forobol sell httptoo.Commits (chronological story for review)
4708407fix(sell inference): emit registration spec + honor operator overrides— adds--register-*, fixesbuildInferenceServiceOfferSpecto honor operator description / model nameb92b78efix(sell inference): auto-register on-chain to match sell http parity—shouldAutoRegisterSellhelper, source-level guard for the call sitedbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint)— drops the over-strict obol-CLI guard, addssignerPayeeDelegationNote, extractsbuildSellUpdatePatchforobol sell update --pay-to2d36745feat(stack): resume sell-inference offers on stack up— descriptor → cluster artifacts replay6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter— detached subprocess +offerOperationallyReady+RegistrationPendingfield9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume— PID-before-Release, footer wording,skill.mdfilter aligned with storefront702d7c6feat(stack): resume obol sell http offers on stack up— sell-http manifest store + matching resume branchTests added across the PR
TestSellInference_Flagsextended to require the six--register-*flags +--no-registerTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault,_NoRegisterOmitsRegistration,_OperatorOverridesWin,_ModelNameNotHardcodedTestShouldAutoRegisterSell(6 cases),TestSellInferenceAction_InvokesAutoRegister(source guard)TestSignerPayeeDelegationNote(6 cases),TestAutoRegister_AllowsSignerPayeeMismatch(source guard)obol sell updateTestBuildSellUpdatePatch_PayToOnly,_PriceSwitchNullsOldKeys,_NoFieldsErrors,TestSellUpdate_PayToFlagSurfaceTestStoreCreate_PersistsResumeFields,TestStoreCreate_LegacyDescriptorWithoutResumeFieldsTestResumeSellOffers_EmptyStoreNoOp,_DescriptorPresentButNoCluster,TestResumeOneInferenceOffer_RequiresModelName,_NilDescriptor,TestStackUpAction_CallsResumeSellOffersTestBuildResumeGatewayArgs(3 cases),TestReadGatewayPID(8 cases),TestProcessAlive_SelfAndBogusTestOfferOperationallyReady_IncludesAwaitingExternalRegistration,_RejectsRealNotReady,TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers,_RegistrationPendingFalseForFullyReadyTestPersistSellHTTPOffer_RoundTrip,_NamespaceIsolation,TestRemoveSellHTTPOffer_DropsPersistedManifest,TestResumeSellHTTPOffers_EmptyStoreNoOp,TestSellDeleteAction_CallsRemoveSellHTTPOffer,TestResumeSellOffers_HTTPOnlyStoreLive spark2 validation
Walked through end-to-end multiple times during development:
obol sell inference aeon … --register-* …writes a descriptor withmodel_name,service_namespace,registrationfields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).obol stack downthenobol stack up(with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back,/api/services.jsonreturns 1 service withregistrationPending=true, and the storefront landing page no longer says "No services are currently available."obol sell delete aeon -n llm --forceremoves the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack updoes NOT resurrect the offer.obol sell inference aeon --pay-to 0xCold…against an agent signer of0xA5d4aF96…no longer errors at the auto-register pre-flight; an informational note explains the delegation.Follow-ups noted in the code
startDetachedInferenceGatewaycalls this out.obol sell deletedoesn't drop the inference descriptor — by design (the descriptor is whatobol sell inference list/statusreads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.Closes
Generated with Claude Code