diff --git a/.gitignore b/.gitignore index 57af644..74e2f24 100644 --- a/.gitignore +++ b/.gitignore @@ -9,5 +9,8 @@ apps/*/log_*.json # Local runtime config files config/ -# Local live supervisor artifacts -logs/live-supervisor/ +# Local logs +logs/ + +# Local scratch files +.scratch/ diff --git a/README.md b/README.md index 7113c05..91030ab 100644 --- a/README.md +++ b/README.md @@ -111,15 +111,15 @@ Rebuild disposable live configs from `ICKB_TESTNET_BOT_PRIVATE_KEY` and `ICKB_TE `pnpm live:preflight -- --config config/bot-testnet.json --role bot` prints public balance evidence for funding checks. Use `key.recommendedAddress` as the funding address, then rerun preflight and check `balances.CKB.available`, `balances.CKB.reserve`, `balances.CKB.spendable`, `balances.CKB.projectedAvailable`, `balances.CKB.total`, and `capital.minimumCkbCapital`; `available` and `spendable` are actual plain-cell values, while `projectedAvailable` and `total` are projected accounting values. For machine-readable JSON without package-manager output, run `node scripts/ickb-live-preflight.mjs --config config/bot-testnet.json --role bot` directly. -For repeated bounded invocations, keep loop-owned options before `--` and supervisor options after it: +For repeated bounded invocations, keep loop-owned options before `--` and supervisor options after it. The loop owns child run directories through `--out-root`, so do not pass supervisor `--out-dir` after `--`: ```bash pnpm live:supervisor:loop --max-runs 1 -- --scenario standard-cycle --max-cycles 1 ``` -Use loop-owned `--child-timeout-seconds` to bound the outer supervisor child process when running long watches; keep it long enough for the whole supervisor invocation, including actor preflights and actor commands, so the supervisor remains alive to enforce its own `--command-timeout-seconds` process-group cleanup. +By default the loop prebuilds the local CCC fork plus bot, tester, and supervisor runtime before the first run. Use loop-owned `--skip-build` only when another wrapper has already built those artifacts. Use loop-owned `--child-timeout-seconds` to bound the outer supervisor child process when running long watches; keep it long enough for the whole supervisor invocation, including actor preflights and actor commands, so the supervisor remains alive to enforce its own `--command-timeout-seconds` process-group cleanup. -For continuous live matching, use the dynamic external loop. It reads only tester preflight balance summaries, chooses a fundable tester stimulus (`all-ckb-limit-order` when plain CKB can preserve reserve plus overhead, otherwise `ickb-to-ckb-limit-order` with the smaller live fee when iCKB is available), then runs bounded `scripts/ickb-supervisor-loop.mjs` chunks: +For continuous live matching, use the dynamic external loop. It reads only tester preflight balance summaries, chooses `all-ckb-limit-order` when `CKB.available >= 3001`, otherwise chooses `ickb-to-ckb-limit-order` with `--tester-fee 1 --tester-fee-base 1000` when `CKB.available >= 2100` and `ICKB.available >= 100`, otherwise leaves the tester scenario as `auto`, then runs bounded `scripts/ickb-supervisor-loop.mjs` chunks: ```bash pnpm live:supervisor:dynamic-loop diff --git a/apps/supervisor/README.md b/apps/supervisor/README.md index 020e29c..f5fe2a4 100644 --- a/apps/supervisor/README.md +++ b/apps/supervisor/README.md @@ -81,11 +81,11 @@ The KISS watcher script runs one deterministic supervisor invocation per child o node scripts/ickb-supervisor-loop.mjs --max-runs 1 --stable-limit 2 --backoff-seconds 0 -- --scenario standard-cycle --max-cycles 1 ``` -Loop-owned options go before `--`; supervisor options go after `--`. If using `pnpm live:supervisor:loop`, keep loop-owned options before the first `--` so they are not passed through to the supervisor. The loop stops on supervisor nonzero exit, incident artifacts listed in `summary.json`, tx-creating outcomes or tx hashes for tx-creating outcomes, a new outcome after the first run, repeated no-progress signatures, or `--max-runs`. `-- --help` and `-- -h` are child help passthroughs: the delegated help is printed and the wrapper exits with the child status. +Loop-owned options go before `--`; supervisor options go after `--`. If using `pnpm live:supervisor:loop`, keep loop-owned options before the first `--` so they are not passed through to the supervisor. The loop owns child run directories through `--out-root`, so do not pass supervisor `--out-dir` after `--`. The loop stops on supervisor nonzero exit, incident artifacts listed in `summary.json`, tx-creating outcomes or tx hashes for tx-creating outcomes, a new outcome after the first run, repeated no-progress signatures, or `--max-runs`. `-- --help` and `-- -h` are child help passthroughs: the delegated help is printed and the wrapper exits with the child status. -The external loop also has a loop-owned `--child-timeout-seconds` guard for the supervisor child process. Keep it long enough for the whole delegated supervisor run, including actor preflights and actor commands, not just one `--command-timeout-seconds` window. The dynamic loop defaults this guard to the supervisor-loop default so the supervisor keeps ownership of killing funded actor process groups on command timeout. +By default the loop prebuilds the local CCC fork plus bot, tester, and supervisor runtime before the first run. Use loop-owned `--skip-build` only when another wrapper has already built those artifacts. The external loop also has a loop-owned `--child-timeout-seconds` guard for the supervisor child process. Keep it long enough for the whole delegated supervisor run, including actor preflights and actor commands, not just one `--command-timeout-seconds` window. The dynamic loop defaults this guard to the supervisor-loop default so the supervisor keeps ownership of killing funded actor process groups on command timeout. -For continuous tester-bot matching, use `node scripts/ickb-supervisor-dynamic-loop.mjs` or `pnpm live:supervisor:dynamic-loop`. This remains outside `apps/supervisor`: it reads tester preflight balance summaries, chooses a currently fundable tester scenario, and delegates each bounded chunk to `scripts/ickb-supervisor-loop.mjs`. When `--target-outcome tester_fresh_order_skip` is passed through, supervisor auto-planning can choose `tester-fresh-skip-two-pass`; the dynamic loop itself only chooses fundable tester stimuli. The dynamic loop also treats `-- --help` and `-- -h` as child help passthroughs and exits with the delegated status. +For continuous tester-bot matching, use `node scripts/ickb-supervisor-dynamic-loop.mjs` or `pnpm live:supervisor:dynamic-loop`. This remains outside `apps/supervisor`: it reads tester preflight balance summaries, chooses `all-ckb-limit-order` when `CKB.available >= 3001`, otherwise chooses `ickb-to-ckb-limit-order` with `--tester-fee 1 --tester-fee-base 1000` when `CKB.available >= 2100` and `ICKB.available >= 100`, otherwise leaves tester selection as `auto`, and delegates each bounded chunk to `scripts/ickb-supervisor-loop.mjs`. When `--target-outcome tester_fresh_order_skip` is passed through, supervisor auto-planning can choose `tester-fresh-skip-two-pass`; the dynamic loop itself only chooses fundable tester stimuli. The dynamic loop also treats `-- --help` and `-- -h` as child help passthroughs and exits with the delegated status. Loop and dynamic-loop exit codes are operator-visible control flow: tx/new-outcome stops exit `0`, incidents exit `2`, `max_runs` and `stable_no_progress` inspection stops exit `3`, and child nonzero statuses are preserved. diff --git a/apps/supervisor/src/index.test.ts b/apps/supervisor/src/index.test.ts index ffe57a8..28c1b4a 100644 --- a/apps/supervisor/src/index.test.ts +++ b/apps/supervisor/src/index.test.ts @@ -186,11 +186,60 @@ describe("supervisor CLI", () => { const plan = resolvePlan(args, "/repo", { spawnSyncCommand: ignoredChecker(true) }); await expect(supervise(args, plan, { - stat: () => Promise.resolve({} as never), - mkdir: () => Promise.resolve(undefined), + mkdir: ((path: string) => { + if (pathToString(path) === "/repo/logs/live-supervisor/existing") { + const error = new Error("exists") as NodeJS.ErrnoException; + error.code = "EEXIST"; + throw error; + } + return Promise.resolve(undefined); + }) as never, })).rejects.toThrow("Output directory already exists: logs/live-supervisor/existing"); }); + it("creates only parent directories recursively before reserving a fresh output directory", async () => { + const args = parseArgs(["--dry-run", "--out-dir", "logs/live-supervisor/fresh"]); + const plan = resolvePlan(args, "/repo", { spawnSyncCommand: ignoredChecker(true) }); + const mkdirs: Array<{ path: string; recursive?: boolean }> = []; + + await supervise(args, plan, { + lstat: missingStat, + realpath: (path) => Promise.resolve(pathToString(path)), + mkdir: ((path: string, options?: { recursive?: boolean }) => { + mkdirs.push({ path: pathToString(path), recursive: options?.recursive }); + return Promise.resolve(undefined); + }) as never, + writeFile: () => Promise.resolve(), + appendFile: () => Promise.resolve(), + }); + + expect(mkdirs).toContainEqual({ path: "/repo/logs/live-supervisor", recursive: true }); + expect(mkdirs).toContainEqual({ path: "/repo/logs/live-supervisor/fresh", recursive: undefined }); + }); + + it("refuses output directories created after ancestor checks", async () => { + const args = parseArgs(["--dry-run", "--out-dir", "logs/live-supervisor/raced"]); + const plan = resolvePlan(args, "/repo", { spawnSyncCommand: ignoredChecker(true) }); + + await expect(supervise(args, plan, { + lstat: missingStat, + mkdir: ((path: string) => { + if (pathToString(path) === "/repo/logs/live-supervisor/raced") { + const error = new Error("exists") as NodeJS.ErrnoException; + error.code = "EEXIST"; + throw error; + } + return Promise.resolve(undefined); + }) as never, + writeFile: () => { + throw new Error("should not write artifacts after raced output directory"); + }, + appendFile: () => { + throw new Error("should not write events after raced output directory"); + }, + })).rejects.toThrow("Output directory already exists: logs/live-supervisor/raced"); + }); + it("refuses symlinked supervisor artifact parents", async () => { const args = parseArgs(["--dry-run", "--out-dir", "logs/live-supervisor/symlink-parent"]); const plan = resolvePlan(args, "/repo", { spawnSyncCommand: ignoredChecker(true) }); @@ -1272,6 +1321,17 @@ describe("classification", () => { expect(classifyActorResult("tester", commandResult("tester", JSON.stringify({ skip: { reason: "sampled-amount-too-small" }, }))).outcome).toBe("tester_sampled_too_small_skip"); + expect(classifyActorResult("tester", commandResult("tester", JSON.stringify({ + skip: { + reason: "estimated-conversion-too-small", + requestedTesterScenario: "auto", + attemptedTesterScenarios: ["random-order", "sdk-conversion", "bounded-ickb-to-ckb-limit-order"], + }, + })))).toMatchObject({ + outcome: "tester_estimated_too_small_skip", + terminal: false, + skipReason: "estimated-conversion-too-small", + }); expect(classifyActorResult("tester", commandResult("tester", JSON.stringify({ skip: { reason: "post-tx-ckb-reserve" }, })))).toMatchObject({ diff --git a/apps/supervisor/src/index.ts b/apps/supervisor/src/index.ts index 50e932e..cf0fa92 100644 --- a/apps/supervisor/src/index.ts +++ b/apps/supervisor/src/index.ts @@ -1,7 +1,7 @@ import { spawn, spawnSync, type SpawnSyncReturns } from "node:child_process"; import { existsSync } from "node:fs"; import { appendFile, lstat, mkdir, realpath, stat, writeFile } from "node:fs/promises"; -import { isAbsolute, join, parse, relative, resolve, sep } from "node:path"; +import { dirname, isAbsolute, join, parse, relative, resolve, sep } from "node:path"; import process from "node:process"; import { fileURLToPath, pathToFileURL } from "node:url"; @@ -947,18 +947,18 @@ async function runDryRun(plan: SupervisorPlan, ledger: CoverageLedger, dependenc } async function prepareOutputDirectory(plan: SupervisorPlan, dependencies: Dependencies): Promise { - const statFn = dependencies.stat ?? stat; const mkdirFn = dependencies.mkdir ?? mkdir; await assertNoSymlinkedOutputAncestors(plan, dependencies); + await mkdirFn(dirname(plan.outDir), { recursive: true }); + await assertNoSymlinkedOutputAncestors(plan, dependencies); try { - await statFn(plan.outDir); - throw new Error(`Output directory already exists: ${plan.relativeOutDir}`); + await mkdirFn(plan.outDir); } catch (error) { - if (!isNotFoundError(error)) { - throw error; + if (isAlreadyExistsError(error)) { + throw new Error(`Output directory already exists: ${plan.relativeOutDir}`); } + throw error; } - await mkdirFn(plan.outDir, { recursive: true }); await assertRealOutputDirectory(plan, dependencies); } @@ -2460,6 +2460,10 @@ function isNotFoundError(error: unknown): boolean { return isRecord(error) && error["code"] === "ENOENT"; } +function isAlreadyExistsError(error: unknown): boolean { + return isRecord(error) && error["code"] === "EEXIST"; +} + function jsonReplacer(_key: string, value: unknown): unknown { return typeof value === "bigint" ? value.toString() : value; } diff --git a/apps/tester/src/index.test.ts b/apps/tester/src/index.test.ts index 3210454..b15b6a3 100644 --- a/apps/tester/src/index.test.ts +++ b/apps/tester/src/index.test.ts @@ -21,6 +21,7 @@ import { testerAttemptedTransactionEvidence, testerEstimatedTooSmallSkip, testerExecutionActions, + testerNoActionableAutoScenarioSkip, resolveTesterScenario, testerReserveSkip, TesterTerminalError, @@ -351,26 +352,64 @@ describe("planTesterTransaction", () => { }); const random = vi.spyOn(Math, "random").mockReturnValue(0.999999); try { - expect(planTesterTransaction(state, 1000n, "random-order")).toEqual({ + const plan = planTesterTransaction(state, 1000n, "random-order"); + expect(plan).toMatchObject({ direction: "ickb-to-ckb", - amount: ccc.fixedPointFrom(123), ckbAmount: 0n, - udtAmount: ccc.fixedPointFrom(123), orderCount: 1, }); + expect(plan.amount).toBeGreaterThan(1n << 33n); + expect(plan.amount).toBeLessThanOrEqual(ccc.fixedPointFrom(123)); + expect(plan.udtAmount).toBe(plan.amount); } finally { random.mockRestore(); } }); - it("does not auto-select unbuildable tiny SDK conversions before other funded auto choices", () => { + it("does not auto-select random orders when only dust is available", () => { expect(resolveTesterScenario( testerState({ availableCkbBalance: ccc.fixedPointFrom(2000) + 1n, availableIckbBalance: 0n }), "auto", undefined, 1000n, () => 0.99, - )).toBe("random-order"); + )).toBeUndefined(); + }); + + it("treats auto with capital but no actionable scenario as a nonterminal estimate skip", () => { + const liveNearReserveState = testerState({ + availableCkbBalance: 229423868188n, + availableIckbBalance: 147394003472899n, + exchangeRatio: { ckbScale: 10000000000000000n, udtScale: 11845567055823930n }, + feeRate: 33222n, + }); + + expect(resolveTesterScenario( + liveNearReserveState, + "auto", + undefined, + 11845567055823n, + () => 0, + )).toBeUndefined(); + expect(testerNoActionableAutoScenarioSkip()).toEqual({ + reason: "estimated-conversion-too-small", + requestedTesterScenario: "auto", + attemptedTesterScenarios: ["random-order", "sdk-conversion", "bounded-ickb-to-ckb-limit-order"], + }); + }); + + it("samples random order amounts above the matcher minimum", () => { + const state = testerState({ availableCkbBalance: ccc.fixedPointFrom(3000), availableIckbBalance: 0n }); + const random = vi.spyOn(Math, "random").mockReturnValue(0); + try { + const plan = planTesterTransaction(state, ccc.fixedPointFrom(1000), "random-order"); + expect(plan.direction).toBe("ckb-to-ickb"); + expect(plan.amount).toBeGreaterThan(1n << 33n); + expect(plan.amount).toBeLessThanOrEqual(ccc.fixedPointFrom(1000)); + expect(plan.ckbAmount).toBe(plan.amount); + } finally { + random.mockRestore(); + } }); it("plans SDK conversions only in a buildable funded direction", () => { @@ -420,13 +459,13 @@ describe("planTesterTransaction", () => { it("resolves auto only to scenarios funded by current balances", () => { const ckbOnlyState = testerState({ availableCkbBalance: ccc.fixedPointFrom(3000), availableIckbBalance: 0n }); - expect(resolveTesterScenario(ckbOnlyState, "auto", undefined, 1000n, () => 0)).toBe("random-order"); - expect(resolveTesterScenario(ckbOnlyState, "auto", undefined, 1000n, () => 0.99)).toBe("sdk-conversion"); + expect(resolveTesterScenario(ckbOnlyState, "auto", undefined, ccc.fixedPointFrom(1000), () => 0)).toBe("random-order"); + expect(resolveTesterScenario(ckbOnlyState, "auto", undefined, ccc.fixedPointFrom(1000), () => 0.99)).toBe("sdk-conversion"); - const ickbOnlyState = testerState({ availableCkbBalance: 0n, availableIckbBalance: ccc.fixedPointFrom(10) }); - expect(resolveTesterScenario(ickbOnlyState, "auto", undefined, 1000n, () => 0)).toBe("random-order"); - expect(resolveTesterScenario(ickbOnlyState, "auto", undefined, 1000n, () => 0.5)).toBe("sdk-conversion"); - expect(resolveTesterScenario(ickbOnlyState, "auto", undefined, 1000n, () => 0.99)).toBe("bounded-ickb-to-ckb-limit-order"); + const ickbOnlyState = testerState({ availableCkbBalance: 0n, availableIckbBalance: ccc.fixedPointFrom(123) }); + expect(resolveTesterScenario(ickbOnlyState, "auto", undefined, ccc.fixedPointFrom(1000), () => 0)).toBe("random-order"); + expect(resolveTesterScenario(ickbOnlyState, "auto", undefined, ccc.fixedPointFrom(1000), () => 0.5)).toBe("sdk-conversion"); + expect(resolveTesterScenario(ickbOnlyState, "auto", undefined, ccc.fixedPointFrom(1000), () => 0.99)).toBe("bounded-ickb-to-ckb-limit-order"); const mixedMultiOrderState = testerState({ availableCkbBalance: ccc.fixedPointFrom(650000), @@ -436,7 +475,7 @@ describe("planTesterTransaction", () => { mixedMultiOrderState, "auto", undefined, - 1000n, + ccc.fixedPointFrom(1000), () => sample, )); expect(autoSamples).not.toContain("all-ckb-limit-order"); @@ -447,13 +486,13 @@ describe("planTesterTransaction", () => { expect(autoSamples).not.toContain("dust-ckb-conversion"); expect(autoSamples).not.toContain("dust-ickb-conversion"); - expect(() => resolveTesterScenario( + expect(resolveTesterScenario( testerState({ availableCkbBalance: ccc.fixedPointFrom(2000), availableIckbBalance: 0n }), "auto", undefined, 1000n, () => 0, - )).toThrow("Not enough funds for auto tester scenario"); + )).toBeUndefined(); }); it("computes post-transaction plain CKB reserve from unspent inputs and account outputs", () => { @@ -875,14 +914,18 @@ function estimatedOrder( function testerState(values: { availableCkbBalance: bigint; availableIckbBalance?: bigint; + exchangeRatio?: TesterState["system"]["exchangeRatio"]; + feeRate?: bigint; capacityCells?: ccc.Cell[]; userOrders?: never[]; }): TesterState { const availableIckbBalance = values.availableIckbBalance ?? 0n; + const exchangeRatio = values.exchangeRatio ?? { ckbScale: 1n, udtScale: 1n }; + const feeRate = values.feeRate ?? 1000n; return { system: { - exchangeRatio: { ckbScale: 1n, udtScale: 1n }, - feeRate: 1000n, + exchangeRatio, + feeRate, tip: headerLike({ timestamp: 0n }), orderPool: [], ckbAvailable: values.availableCkbBalance, @@ -899,8 +942,8 @@ function testerState(values: { userOrders: values.userOrders ?? [], conversionContext: { system: { - exchangeRatio: { ckbScale: 1n, udtScale: 1n }, - feeRate: 1000n, + exchangeRatio, + feeRate, tip: headerLike({ timestamp: 0n }), orderPool: [], ckbAvailable: values.availableCkbBalance, diff --git a/apps/tester/src/index.ts b/apps/tester/src/index.ts index d398d90..2805f3a 100644 --- a/apps/tester/src/index.ts +++ b/apps/tester/src/index.ts @@ -1,6 +1,6 @@ import { ccc } from "@ckb-ccc/core"; import { ICKB_DEPOSIT_CAP, convert } from "@ickb/core"; -import { IckbSdk, getConfig, sendAndWaitForCommit } from "@ickb/sdk"; +import { IckbSdk, estimateMaturityFeeThreshold, getConfig, sendAndWaitForCommit } from "@ickb/sdk"; import { accountPlainCkbBalance, createPublicClient, @@ -177,7 +177,20 @@ async function main(): Promise { executionLog.ratio = state.system.exchangeRatio; const effectiveTesterScenario = resolveTesterScenario(state, testerScenario, feePolicy, depositCapacity); - const plan = planTesterTransaction(state, depositCapacity, effectiveTesterScenario); + if (effectiveTesterScenario === undefined) { + if (totalEquivalentCkb < depositCapacity / MIN_TOTAL_CAPITAL_DIVISOR) { + executionLog.error = + "Not enough funds to continue testing, shutting down..."; + logExecution(executionLog, startTime); + return; + } + executionLog.skip = testerNoActionableAutoScenarioSkip(); + if (logTerminalIteration(executionLog, startTime, ++completedIterations, maxIterations)) { + return; + } + continue; + } + const plan = planTesterTransaction(state, depositCapacity, effectiveTesterScenario, feePolicy); const rawOrders = plannedRawOrders(plan, effectiveTesterScenario); if (rawOrders.length === 0) { @@ -442,6 +455,14 @@ export function testerEstimatedTooSmallSkip( }; } +export function testerNoActionableAutoScenarioSkip(): Record { + return { + reason: "estimated-conversion-too-small", + requestedTesterScenario: "auto", + attemptedTesterScenarios: [...AUTO_TESTER_SCENARIOS], + }; +} + export function testerAttemptedTransactionEvidence( requestedScenario: TesterScenarioSelection, effectiveScenario: TesterScenario, @@ -460,9 +481,10 @@ export function planTesterTransaction( state: Pick, depositCapacity: bigint, scenario: TesterScenario, + feePolicy: TesterFeePolicy = DEFAULT_TESTER_FEE_POLICY, ): TesterPlan { if (scenario === "multi-order-limit-orders") { - return planTesterTransaction(state, depositCapacity, resolveTesterScenario(state, scenario)); + return planTesterTransaction(state, depositCapacity, resolveMultiOrderScenario(state, feePolicy), feePolicy); } if (scenario === "sdk-conversion") { return planSdkConversionTransaction(state, depositCapacity); @@ -539,33 +561,7 @@ export function planTesterTransaction( } const spendableCkbBalance = max(0n, state.availableCkbBalance - CKB_RESERVE); - const ickbEquivalentBalance = convert( - true, - spendableCkbBalance, - state.system.exchangeRatio, - ); - const totalIckbBalance = ickbEquivalentBalance + state.availableIckbBalance; - const isCkb2Udt = sampleRatio(totalIckbBalance) <= ickbEquivalentBalance; - const ckbAmount = isCkb2Udt - ? min( - isSdkConversionScenario(scenario) ? depositCapacity : sampleRatio(depositCapacity), - spendableCkbBalance, - ) - : 0n; - const udtAmount = isCkb2Udt - ? 0n - : min( - isSdkConversionScenario(scenario) ? ICKB_DEPOSIT_CAP : sampleRatio(ICKB_DEPOSIT_CAP), - state.availableIckbBalance, - ); - - return { - direction: isCkb2Udt ? "ckb-to-ickb" : "ickb-to-ckb", - amount: isCkb2Udt ? ckbAmount : udtAmount, - ckbAmount, - udtAmount, - orderCount: 1, - }; + return planRandomOrderTransaction(state, depositCapacity, feePolicy, spendableCkbBalance); } export function resolveTesterScenario( @@ -574,17 +570,24 @@ export function resolveTesterScenario( feePolicy: TesterFeePolicy = DEFAULT_TESTER_FEE_POLICY, depositCapacity = 0n, random: () => number = Math.random, -): TesterScenario { +): TesterScenario | undefined { if (scenario === "auto") { const fundedScenarios = fundedTesterScenarios(state, depositCapacity, feePolicy, AUTO_TESTER_SCENARIOS); if (fundedScenarios.length === 0) { - throw new TesterTerminalError("Not enough funds for auto tester scenario"); + return undefined; } return randomTesterScenario(random, fundedScenarios); } if (scenario !== "multi-order-limit-orders") { return scenario; } + return resolveMultiOrderScenario(state, feePolicy); +} + +function resolveMultiOrderScenario( + state: Pick, + feePolicy: TesterFeePolicy, +): TesterScenario { const selected = MULTI_ORDER_SCENARIOS.find((candidate) => hasPositiveMultiOrderEstimates(state, candidate, feePolicy)); if (selected !== undefined) { return selected; @@ -600,7 +603,7 @@ function fundedTesterScenarios( ): TesterScenario[] { return candidates.filter((scenario) => { if (scenario === "random-order") { - return hasPositiveRandomOrderEstimate(state, depositCapacity, feePolicy); + return hasActionableRandomOrderEstimate(state, depositCapacity, feePolicy); } if (scenario === "sdk-conversion") { return hasBuildableSdkConversionEstimate(state, depositCapacity); @@ -678,25 +681,91 @@ function buildableSdkConversionPlan( return isBuildableSdkConversionOrder(plan, order, estimate, state.system, depositCapacity) ? plan : undefined; } -function hasPositiveRandomOrderEstimate( +function hasActionableRandomOrderEstimate( state: Pick, depositCapacity: bigint, feePolicy: TesterFeePolicy, ): boolean { const spendableCkbBalance = max(0n, state.availableCkbBalance - CKB_RESERVE); - const orders: PlannedRawOrder[] = []; - if (spendableCkbBalance > 0n) { - const amount = min(depositCapacity, spendableCkbBalance); - orders.push({ direction: "ckb-to-ickb", amounts: { ckbValue: amount, udtValue: 0n }, amount }); - } - if (state.availableIckbBalance > 0n) { - const amount = min(ICKB_DEPOSIT_CAP, state.availableIckbBalance); - orders.push({ direction: "ickb-to-ckb", amounts: { ckbValue: 0n, udtValue: amount }, amount }); - } - return orders.some((order) => { - const estimate = estimateRawOrder(order, state.system, feePolicy); - return estimate !== undefined && estimate.convertedAmount > 0n; - }); + return minimumActionableRandomOrderAmount( + "ckb-to-ickb", + min(depositCapacity, spendableCkbBalance), + state.system, + feePolicy, + ) !== undefined || minimumActionableRandomOrderAmount( + "ickb-to-ckb", + min(ICKB_DEPOSIT_CAP, state.availableIckbBalance), + state.system, + feePolicy, + ) !== undefined; +} + +function planRandomOrderTransaction( + state: Pick, + depositCapacity: bigint, + feePolicy: TesterFeePolicy, + spendableCkbBalance: bigint, +): TesterPlan { + const ckbMax = min(depositCapacity, spendableCkbBalance); + const udtMax = min(ICKB_DEPOSIT_CAP, state.availableIckbBalance); + const ckbMin = minimumActionableRandomOrderAmount("ckb-to-ickb", ckbMax, state.system, feePolicy); + const udtMin = minimumActionableRandomOrderAmount("ickb-to-ckb", udtMax, state.system, feePolicy); + const ckbWeight = ckbMin === undefined ? 0n : convert(true, ckbMax, state.system.exchangeRatio); + const udtWeight = udtMin === undefined ? 0n : udtMax; + const isCkb2Udt = ckbWeight > 0n && (udtWeight === 0n || sampleRatio(ckbWeight + udtWeight) < ckbWeight); + if (isCkb2Udt && ckbMin !== undefined) { + const ckbAmount = sampleAmount(ckbMin, ckbMax); + return { direction: "ckb-to-ickb", amount: ckbAmount, ckbAmount, udtAmount: 0n, orderCount: 1 }; + } + if (udtMin !== undefined) { + const udtAmount = sampleAmount(udtMin, udtMax); + return { direction: "ickb-to-ckb", amount: udtAmount, ckbAmount: 0n, udtAmount, orderCount: 1 }; + } + return { direction: "ckb-to-ickb", amount: 0n, ckbAmount: 0n, udtAmount: 0n, orderCount: 1 }; +} + +function minimumActionableRandomOrderAmount( + direction: TesterDirection, + maxAmount: bigint, + system: TesterState["system"], + feePolicy: TesterFeePolicy, +): bigint | undefined { + if (maxAmount <= 0n || !isActionableRandomOrderAmount(direction, maxAmount, system, feePolicy)) { + return undefined; + } + let low = 1n; + let high = maxAmount; + while (low < high) { + const mid = (low + high) / 2n; + if (isActionableRandomOrderAmount(direction, mid, system, feePolicy)) { + high = mid; + } else { + low = mid + 1n; + } + } + return low; +} + +function isActionableRandomOrderAmount( + direction: TesterDirection, + amount: bigint, + system: TesterState["system"], + feePolicy: TesterFeePolicy, +): boolean { + const order: PlannedRawOrder = { direction, amounts: planAmounts(direction, amount), amount }; + const estimate = estimateRawOrder(order, system, feePolicy); + return estimate !== undefined && + estimate.convertedAmount >= minimumMatcherOutput(direction, estimate.info) && + estimate.ckbFee >= estimateMaturityFeeThreshold(system); +} + +function minimumMatcherOutput(direction: TesterDirection, info: ReturnType["info"]): bigint { + const minimumCkb = info.getCkbMinMatch(); + if (direction === "ckb-to-ickb") { + const { ckbScale, udtScale } = info.ckbToUdt; + return (minimumCkb * udtScale + ckbScale - 1n) / ckbScale; + } + return minimumCkb; } function hasPositiveMultiOrderEstimates( @@ -864,6 +933,10 @@ function sampleRatio(amount: bigint): bigint { return (amount * randomScaled()) / RANDOM_SCALE; } +function sampleAmount(minimum: bigint, maximum: bigint): bigint { + return minimum + sampleRatio(maximum - minimum + 1n); +} + function randomScaled(): bigint { return BigInt(Math.floor(Math.random() * Number(RANDOM_SCALE))); } diff --git a/scripts/ickb-supervisor-dynamic-loop.mjs b/scripts/ickb-supervisor-dynamic-loop.mjs index 94b9f48..19f82b4 100644 --- a/scripts/ickb-supervisor-dynamic-loop.mjs +++ b/scripts/ickb-supervisor-dynamic-loop.mjs @@ -1,13 +1,15 @@ #!/usr/bin/env node import { spawnSync } from "node:child_process"; import { appendFile, lstat, mkdir, stat, writeFile } from "node:fs/promises"; -import { isAbsolute, join, parse, relative, resolve, sep } from "node:path"; +import { dirname, isAbsolute, join, parse, relative, resolve, sep } from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import { fileURLToPath, pathToFileURL } from "node:url"; import { parseNonNegativeInteger, parsePositiveInteger, valueAfter } from "./ickb-script-helpers.mjs"; import { DEFAULT_CHILD_TIMEOUT_SECONDS_VALUE as DEFAULT_SUPERVISOR_LOOP_CHILD_TIMEOUT_SECONDS, INSPECTION_REQUIRED_EXIT_CODE, + formatPrebuildFailure, + prebuildRuntime, } from "./ickb-supervisor-loop.mjs"; const rootDir = fileURLToPath(new URL("..", import.meta.url)); @@ -43,6 +45,7 @@ const DYNAMIC_LOOP_OWNED_FLAGS = [ ]; const ALL_CKB_MIN_CKB = 3001n; const ICKB_STIMULUS_MIN_CKB = 2100n; +const ICKB_STIMULUS_MIN_ICKB = 100n; const MAX_SAFE_INTEGER = BigInt(Number.MAX_SAFE_INTEGER); export function parseArgs(argv) { @@ -178,7 +181,7 @@ export function usage() { " -h, --help", " -- [supervisor-options]", "Dynamic-loop options must appear before --; supervisor --out-dir is owned by the dynamic loop.", - "Reads tester preflight balance summaries, chooses a fundable tester scenario, then runs bounded supervisor-loop chunks.", + "Builds local runtime once, reads tester preflight balance summaries, then runs bounded supervisor-loop chunks with --skip-build.", ].join("\n"); } @@ -200,7 +203,7 @@ export function chooseTesterScenario({ ckb, ickb }) { if (ckb >= ALL_CKB_MIN_CKB * CKB_UNIT) { return { scenario: "all-ckb-limit-order", feeArgs: [] }; } - if (ckb >= ICKB_STIMULUS_MIN_CKB * CKB_UNIT && ickb > 0n) { + if (ckb >= ICKB_STIMULUS_MIN_CKB * CKB_UNIT && ickb >= ICKB_STIMULUS_MIN_ICKB * CKB_UNIT) { return { scenario: "ickb-to-ckb-limit-order", feeArgs: ["--tester-fee", "1", "--tester-fee-base", "1000"], @@ -241,7 +244,20 @@ export async function runDynamicSupervisorLoop({ argv, root = rootDir, dependenc let session; try { - session = await prepareValidationSession(args, root, dependencies); + session = await resolveValidationSession(args, root, dependencies); + } catch (error) { + stderr.write(`${errorMessage(error)}\n${usage()}\n`); + return 1; + } + + const prebuild = prebuildRuntime(root, dependencies); + if (prebuild.status !== 0) { + stdout.write(formatPrebuildFailure(prebuild) + "\n"); + return prebuild.status; + } + + try { + await createValidationSession(session, dependencies); await writeLaunchArtifact(session, args, root, dependencies); await writeOperatorEvent(session, { type: "session_started", @@ -390,6 +406,7 @@ function runSupervisorChunk(args, choice, root, dependencies, session, chunkInde "--stable-limit", String(args.stableLimit), "--backoff-seconds", String(args.chunkBackoffSeconds), "--child-timeout-seconds", String(args.childTimeoutSeconds), + "--skip-build", "--", ...testerScenarioArgs, "--max-cycles", "1", @@ -399,7 +416,7 @@ function runSupervisorChunk(args, choice, root, dependencies, session, chunkInde ], root, dependencies, { timeout: args.chunkTimeoutSeconds * 1000 }); } -async function prepareValidationSession(args, root, dependencies) { +async function resolveValidationSession(args, root, dependencies) { const now = dependencies.now ?? Date.now; const pid = dependencies.pid ?? process.pid; const logRoot = resolveConfiguredPath(root, args.logRoot ?? DEFAULT_LOG_ROOT, "--log-root"); @@ -427,12 +444,8 @@ async function prepareValidationSession(args, root, dependencies) { throw error; } } - - const mkdirFn = dependencies.mkdir ?? mkdir; const operatorDir = join(sessionRoot, "operator"); const chunksDir = join(sessionRoot, "chunks"); - await mkdirFn(operatorDir, { recursive: true }); - await mkdirFn(chunksDir, { recursive: true }); return { logRoot, sessionRoot, @@ -443,6 +456,22 @@ async function prepareValidationSession(args, root, dependencies) { }; } +async function createValidationSession(session, dependencies) { + const mkdirFn = dependencies.mkdir ?? mkdir; + await mkdirFn(dirname(session.sessionRoot), { recursive: true }); + await assertNoSymlinkedPath(session.sessionRoot, "session root", dependencies); + try { + await mkdirFn(session.sessionRoot); + } catch (error) { + if (isAlreadyExistsError(error)) { + throw new Error(`Validation session root already exists: ${session.displaySessionRoot}`); + } + throw error; + } + await mkdirFn(session.operatorDir); + await mkdirFn(session.chunksDir); +} + async function writeLaunchArtifact(session, args, root, dependencies) { const writeFileFn = dependencies.writeFile ?? writeFile; const startedAt = new Date((dependencies.now ?? Date.now)()).toISOString(); @@ -595,6 +624,10 @@ function isNotFoundError(error) { return typeof error === "object" && error !== null && "code" in error && error.code === "ENOENT"; } +function isAlreadyExistsError(error) { + return typeof error === "object" && error !== null && "code" in error && error.code === "EEXIST"; +} + function padChunk(index) { return String(index).padStart(4, "0"); } diff --git a/scripts/ickb-supervisor-dynamic-loop.test.mjs b/scripts/ickb-supervisor-dynamic-loop.test.mjs index bf991ca..d0d00db 100644 --- a/scripts/ickb-supervisor-dynamic-loop.test.mjs +++ b/scripts/ickb-supervisor-dynamic-loop.test.mjs @@ -73,6 +73,7 @@ test("dynamic supervisor loop parses options", () => { assert.deepEqual(parseArgs(["--", "--command-timeout-seconds", "9"]).supervisorArgs, ["--command-timeout-seconds", "9"]); assert.match(usage(), /tester-config/u); assert.match(usage(), /--log-root/u); + assert.match(usage(), /--skip-build/u); }); test("dynamic supervisor loop passes child help through visibly", async () => { @@ -123,6 +124,9 @@ test("dynamic supervisor loop creates a default validation session root", async appendFile: async (path, text) => appended.set(path, (appended.get(path) ?? "") + text), spawnSync: (_command, args, options) => { commands.push({ args, options }); + if (isPrebuildCommand(args)) { + return { status: 0, signal: null, stdout: "", stderr: "" }; + } if (args[0] === "scripts/ickb-live-preflight.mjs") { return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "3200" }, ICKB: { available: "0" } } }), stderr: "" }; } @@ -134,15 +138,23 @@ test("dynamic supervisor loop creates a default validation session root", async assert.equal(exitCode, 3); assert.equal(writes.has("/repo/log/validation/dynamic-1700000000-4321/operator/launch.json"), true); assert.equal(appended.has("/repo/log/validation/dynamic-1700000000-4321/operator/events.ndjson"), true); - assert.deepEqual(commands[1].args.slice(0, 3), [ + assert.deepEqual(commands.slice(0, 4).map((item) => item.args), [ + ["forks:ccc"], + ["bot:build"], + ["--filter", "@ickb/tester", "build"], + ["--filter", "@ickb/supervisor", "build"], + ]); + assert.deepEqual(commands[5].args.slice(0, 3), [ "scripts/ickb-supervisor-loop.mjs", "--out-root", "log/validation/dynamic-1700000000-4321/chunks/chunk-0001", ]); - assert.equal(commands[0].options.env.NODE_OPTIONS, "--disable-warning=DEP0040"); - assert.equal(commands[1].options.env.NODE_OPTIONS, "--disable-warning=DEP0040"); - assert.equal(commands[0].options.env.PRIVATE_KEY, undefined); - assert.equal(commands[1].options.env.PRIVATE_KEY, undefined); + assert.equal(commands[5].args.includes("--skip-build"), true); + assert.equal(commands[4].options.env.NODE_OPTIONS, "--disable-warning=DEP0040"); + assert.equal(commands[5].options.env.NODE_OPTIONS, "--disable-warning=DEP0040"); + for (const command of commands) { + assert.equal(command.options.env.PRIVATE_KEY, undefined); + } assert.match(output.text, /"type":"chunk_finished"/u); assert.match(output.text, /log\/validation\/dynamic-1700000000-4321\/chunks\/chunk-0001/u); } finally { @@ -231,10 +243,14 @@ test("dynamic supervisor loop chooses fundable tester scenarios", () => { scenario: "all-ckb-limit-order", feeArgs: [], }); - assert.deepEqual(chooseTesterScenario({ ckb: 2100n * ckb, ickb: 1n }), { + assert.deepEqual(chooseTesterScenario({ ckb: 2100n * ckb, ickb: 100n * ckb }), { scenario: "ickb-to-ckb-limit-order", feeArgs: ["--tester-fee", "1", "--tester-fee-base", "1000"], }); + assert.deepEqual(chooseTesterScenario({ ckb: 2100n * ckb, ickb: 1n }), { + scenario: "auto", + feeArgs: [], + }); assert.deepEqual(chooseTesterScenario({ ckb: 2099n * ckb, ickb: 1n }), { scenario: "auto", feeArgs: [], @@ -265,10 +281,14 @@ test("dynamic supervisor loop runs selected bounded chunks", async () => { appendFile: async () => undefined, spawnSync: (_command, args, options) => { commands.push({ args, options }); + if (isPrebuildCommand(args)) { + return { status: 0, signal: null, stdout: "", stderr: "" }; + } if (args[0] === "scripts/ickb-live-preflight.mjs") { - const stdout = commands.length === 1 + const preflightCount = commands.filter((command) => command.args[0] === "scripts/ickb-live-preflight.mjs").length; + const stdout = preflightCount === 1 ? JSON.stringify({ balances: { CKB: { available: "3200" }, ICKB: { available: "0" } } }) - : JSON.stringify({ balances: { CKB: { available: "2100" }, ICKB: { available: "1" } } }); + : JSON.stringify({ balances: { CKB: { available: "2100" }, ICKB: { available: "100" } } }); return { status: 0, signal: null, stdout, stderr: "" }; } return { status: 0, signal: null, stdout: "loop run=1 status=0 stopped=max_cycles outcomes=- tx=0 new=- stable=1 state=- decision=max_runs out=logs/live-supervisor/test\n", stderr: "" }; @@ -281,16 +301,129 @@ test("dynamic supervisor loop runs selected bounded chunks", async () => { assert.equal(exitCode, 3); assert.equal(sleeps.length, 0); - assert.equal(commands.length, 2); - assert.equal(commands[1].args.includes("all-ckb-limit-order"), true); - assert.deepEqual(commands[1].args.slice(0, 3), ["scripts/ickb-supervisor-loop.mjs", "--out-root", "log/validation/test-session/chunks/chunk-0001"]); - const separator = commands[1].args.indexOf("--"); - assert.equal(commands[1].args.slice(0, separator).includes("--out-root"), true); - assert.equal(commands[1].args.slice(separator + 1).includes("--target-outcome"), true); + assert.equal(commands.length, 6); + assert.deepEqual(commands.slice(0, 4).map((item) => item.args), [ + ["forks:ccc"], + ["bot:build"], + ["--filter", "@ickb/tester", "build"], + ["--filter", "@ickb/supervisor", "build"], + ]); + assert.equal(commands[5].args.includes("all-ckb-limit-order"), true); + assert.deepEqual(commands[5].args.slice(0, 3), ["scripts/ickb-supervisor-loop.mjs", "--out-root", "log/validation/test-session/chunks/chunk-0001"]); + const separator = commands[5].args.indexOf("--"); + assert.equal(commands[5].args.slice(0, separator).includes("--out-root"), true); + assert.equal(commands[5].args.slice(0, separator).includes("--skip-build"), true); + assert.equal(commands[5].args.slice(separator + 1).includes("--target-outcome"), true); assert.match(output.text, /"type":"selected"/u); assert.match(output.text, /testerScenario":"all-ckb-limit-order"/u); }); +test("dynamic supervisor loop reports prebuild failures before opening sessions", async () => { + const output = { text: "", write(chunk) { this.text += chunk; } }; + let mkdirCalled = false; + const exitCode = await runDynamicSupervisorLoop({ + root: "/repo", + argv: ["--log-root", "log", "--session-root", "log/validation/prebuild-failure", "--max-chunks", "1"], + io: { stdout: output, stderr: output }, + dependencies: { + checkIgnored: () => true, + stat: missingStat, + lstat: missingStat, + mkdir: async () => { + mkdirCalled = true; + }, + spawnSync: () => ({ + status: 1, + signal: null, + stdout: "privateKey 0x1111\n", + stderr: "operator secret 0x2222\n", + }), + }, + }); + + assert.equal(exitCode, 1); + assert.equal(mkdirCalled, false); + assert.match(output.text, /loop prebuild_failed/u); + assert.doesNotMatch(output.text, /privateKey|0x1111|operator secret|0x2222/u); +}); + +test("dynamic supervisor loop refuses sessions created during prebuild", async () => { + const commands = []; + const mkdirs = []; + const output = { text: "", write(chunk) { this.text += chunk; } }; + const exitCode = await runDynamicSupervisorLoop({ + root: "/repo", + argv: ["--log-root", "log", "--session-root", "log/validation/raced-session", "--max-chunks", "1"], + io: { stdout: output, stderr: output }, + dependencies: { + checkIgnored: () => true, + stat: missingStat, + lstat: missingStat, + mkdir: async (path) => { + mkdirs.push(path); + if (path === "/repo/log/validation/raced-session") { + const error = new Error("exists"); + error.code = "EEXIST"; + throw error; + } + }, + writeFile: async () => { + throw new Error("should not write launch artifact after raced session"); + }, + appendFile: async () => { + throw new Error("should not write events after raced session"); + }, + spawnSync: (_command, args) => { + commands.push(args); + if (isPrebuildCommand(args)) { + return okResult(); + } + throw new Error("should not spawn preflight or supervisor after raced session"); + }, + }, + }); + + assert.equal(exitCode, 1); + assert.deepEqual(commands, [ + ["forks:ccc"], + ["bot:build"], + ["--filter", "@ickb/tester", "build"], + ["--filter", "@ickb/supervisor", "build"], + ]); + assert.deepEqual(mkdirs, ["/repo/log/validation", "/repo/log/validation/raced-session"]); + assert.match(output.text, /Validation session root already exists: log\/validation\/raced-session/u); +}); + +test("dynamic supervisor loop refuses symlinked session parents created during prebuild", async () => { + const output = { text: "", write(chunk) { this.text += chunk; } }; + const exitCode = await runDynamicSupervisorLoop({ + root: "/repo", + argv: ["--log-root", "log", "--session-root", "log/validation/raced-symlink", "--max-chunks", "1"], + io: { stdout: output, stderr: output }, + dependencies: { + checkIgnored: () => true, + stat: missingStat, + lstat: (path) => ({ isSymbolicLink: () => path === "/repo/log/validation" }), + mkdir: async () => undefined, + writeFile: async () => { + throw new Error("should not write launch artifact through raced symlink"); + }, + appendFile: async () => { + throw new Error("should not write events through raced symlink"); + }, + spawnSync: (_command, args) => { + if (isPrebuildCommand(args)) { + return okResult(); + } + throw new Error("should not spawn preflight or supervisor through raced symlink"); + }, + }, + }); + + assert.equal(exitCode, 1); + assert.match(output.text, /Refusing to use session root through symlinked path: \/repo\/log\/validation/u); +}); + test("dynamic supervisor loop stops after inspection-worthy supervisor-loop reasons", async () => { const commands = []; const output = { text: "", write(chunk) { this.text += chunk; } }; @@ -312,6 +445,9 @@ test("dynamic supervisor loop stops after inspection-worthy supervisor-loop reas appendFile: async () => undefined, spawnSync: (_command, args, options) => { commands.push({ args, options }); + if (isPrebuildCommand(args)) { + return okResult(); + } if (args[0] === "scripts/ickb-live-preflight.mjs") { return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "3200" }, ICKB: { available: "0" } } }), stderr: "" }; } @@ -321,7 +457,9 @@ test("dynamic supervisor loop stops after inspection-worthy supervisor-loop reas }); assert.equal(exitCode, 0); - assert.equal(commands.length, 2); + assert.equal(commands.filter((command) => isPrebuildCommand(command.args)).length, 4); + assert.equal(commands.filter((command) => command.args[0] === "scripts/ickb-live-preflight.mjs").length, 1); + assert.equal(commands.filter((command) => command.args[0] === "scripts/ickb-supervisor-loop.mjs").length, 1); assert.match(output.text, /"supervisorLoopStopReason":"tx_observed"/u); }); @@ -346,6 +484,9 @@ test("dynamic supervisor loop preserves supervisor-loop inspection-required stat appendFile: async () => undefined, spawnSync: (_command, args, options) => { commands.push({ args, options }); + if (isPrebuildCommand(args)) { + return okResult(); + } if (args[0] === "scripts/ickb-live-preflight.mjs") { return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "3200" }, ICKB: { available: "0" } } }), stderr: "" }; } @@ -355,7 +496,9 @@ test("dynamic supervisor loop preserves supervisor-loop inspection-required stat }); assert.equal(exitCode, 3); - assert.equal(commands.length, 2); + assert.equal(commands.filter((command) => isPrebuildCommand(command.args)).length, 4); + assert.equal(commands.filter((command) => command.args[0] === "scripts/ickb-live-preflight.mjs").length, 1); + assert.equal(commands.filter((command) => command.args[0] === "scripts/ickb-supervisor-loop.mjs").length, 1); assert.match(output.text, /"supervisorLoopStopReason":"max_runs"/u); }); @@ -381,6 +524,9 @@ test("dynamic supervisor loop leaves supervisor target steering intact for auto appendFile: async () => undefined, spawnSync: (_command, args, options) => { commands.push({ args, options }); + if (isPrebuildCommand(args)) { + return okResult(); + } if (args[0] === "scripts/ickb-live-preflight.mjs") { return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "1000" }, ICKB: { available: "0" } } }), stderr: "" }; } @@ -389,7 +535,7 @@ test("dynamic supervisor loop leaves supervisor target steering intact for auto }, }); - const supervisorArgs = commands[1].args; + const supervisorArgs = commands.find((command) => command.args[0] === "scripts/ickb-supervisor-loop.mjs").args; const separator = supervisorArgs.indexOf("--"); const passthrough = supervisorArgs.slice(separator + 1); assert.equal(exitCode, 3); @@ -421,15 +567,18 @@ test("dynamic supervisor loop leaves fresh-order skip target planning to supervi appendFile: async () => undefined, spawnSync: (_command, args, options) => { commands.push({ args, options }); + if (isPrebuildCommand(args)) { + return okResult(); + } if (args[0] === "scripts/ickb-live-preflight.mjs") { - return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "2100" }, ICKB: { available: "1" } } }), stderr: "" }; + return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "2100" }, ICKB: { available: "100" } } }), stderr: "" }; } return { status: 0, signal: null, stdout: "loop run=1 status=0 stopped=max_cycles outcomes=tester_fresh_order_skip tx=0 new=- stable=1 state=- decision=max_runs out=logs/live-supervisor/test\n", stderr: "" }; }, }, }); - const supervisorArgs = commands[1].args; + const supervisorArgs = commands.find((command) => command.args[0] === "scripts/ickb-supervisor-loop.mjs").args; const separator = supervisorArgs.indexOf("--"); const passthrough = supervisorArgs.slice(separator + 1); assert.equal(exitCode, 3); @@ -453,7 +602,9 @@ test("dynamic supervisor loop stops on preflight failures", async () => { mkdir: async () => undefined, writeFile: async () => undefined, appendFile: async () => undefined, - spawnSync: () => ({ status: 2, signal: null, stdout: "", stderr: "" }), + spawnSync: (_command, args) => isPrebuildCommand(args) + ? okResult() + : { status: 2, signal: null, stdout: "", stderr: "" }, }, }); @@ -474,7 +625,9 @@ test("dynamic supervisor loop reports preflight spawn errors", async () => { mkdir: async () => undefined, writeFile: async () => undefined, appendFile: async () => undefined, - spawnSync: () => ({ status: null, signal: null, stdout: "", stderr: "", error: new Error("spawn ETIMEDOUT") }), + spawnSync: (_command, args) => isPrebuildCommand(args) + ? okResult() + : { status: null, signal: null, stdout: "", stderr: "", error: new Error("spawn ETIMEDOUT") }, }, }); @@ -497,6 +650,9 @@ test("dynamic supervisor loop preserves supervisor chunk spawn errors", async () writeFile: async () => undefined, appendFile: async (path, text) => appended.set(path, `${appended.get(path) ?? ""}${text}`), spawnSync: (_command, args) => { + if (isPrebuildCommand(args)) { + return okResult(); + } if (args[0] === "scripts/ickb-live-preflight.mjs") { return { status: 0, signal: null, stdout: JSON.stringify({ balances: { CKB: { available: "3200" }, ICKB: { available: "0" } } }), stderr: "" }; } @@ -524,7 +680,9 @@ test("dynamic supervisor loop preserves malformed preflight stderr", async () => mkdir: async () => undefined, writeFile: async () => undefined, appendFile: async (path, text) => appended.set(path, `${appended.get(path) ?? ""}${text}`), - spawnSync: () => ({ status: 0, signal: null, stdout: "not json", stderr: "preflight diagnostic\n" }), + spawnSync: (_command, args) => isPrebuildCommand(args) + ? okResult() + : { status: 0, signal: null, stdout: "not json", stderr: "preflight diagnostic\n" }, }, }); @@ -538,3 +696,14 @@ function missingStat() { error.code = "ENOENT"; throw error; } + +function isPrebuildCommand(args) { + return args[0] === "forks:ccc" || + args[0] === "bot:build" || + args.includes("@ickb/tester") || + args.includes("@ickb/supervisor"); +} + +function okResult() { + return { status: 0, signal: null, stdout: "", stderr: "" }; +} diff --git a/scripts/ickb-supervisor-loop.mjs b/scripts/ickb-supervisor-loop.mjs index a035e61..2f7afed 100644 --- a/scripts/ickb-supervisor-loop.mjs +++ b/scripts/ickb-supervisor-loop.mjs @@ -11,13 +11,22 @@ const DEFAULT_STABLE_LIMIT = 3; const DEFAULT_BACKOFF_SECONDS = 30; const DEFAULT_CHILD_TIMEOUT_SECONDS = 65 * 60; export const DEFAULT_CHILD_TIMEOUT_SECONDS_VALUE = DEFAULT_CHILD_TIMEOUT_SECONDS; +const DEFAULT_PREBUILD_TIMEOUT_SECONDS = DEFAULT_CHILD_TIMEOUT_SECONDS; +export const DEFAULT_PREBUILD_TIMEOUT_SECONDS_VALUE = DEFAULT_PREBUILD_TIMEOUT_SECONDS; const DEFAULT_SUPERVISOR_SCRIPT = "apps/supervisor/dist/index.js"; const SUPERVISOR_OUTPUT_ROOT = "logs/live-supervisor"; export const INSPECTION_REQUIRED_EXIT_CODE = 3; -const LOOP_OWNED_FLAGS = ["--out-root", "--max-runs", "--stable-limit", "--backoff-seconds", "--child-timeout-seconds", "--supervisor-script"]; +const LOOP_OWNED_FLAGS = ["--out-root", "--max-runs", "--stable-limit", "--backoff-seconds", "--child-timeout-seconds", "--supervisor-script", "--skip-build"]; +const PREBUILD_COMMANDS = [ + { target: "ccc", command: "pnpm", args: ["forks:ccc"] }, + { target: "bot", command: "pnpm", args: ["bot:build"] }, + { target: "tester", command: "pnpm", args: ["--filter", "@ickb/tester", "build"] }, + { target: "supervisor", command: "pnpm", args: ["--filter", "@ickb/supervisor", "build"] }, +]; export function parseArgs(argv) { const args = { help: false, + skipBuild: false, maxRuns: DEFAULT_MAX_RUNS, stableLimit: DEFAULT_STABLE_LIMIT, backoffSeconds: DEFAULT_BACKOFF_SECONDS, @@ -35,6 +44,10 @@ export function parseArgs(argv) { args.help = true; continue; } + if (arg === "--skip-build") { + args.skipBuild = true; + continue; + } if (arg === "--out-root") { args.outRoot = valueAfter(argv, ++index, arg); continue; @@ -81,8 +94,9 @@ export function usage() { ` --backoff-seconds Default: ${String(DEFAULT_BACKOFF_SECONDS)}`, ` --child-timeout-seconds Default: ${String(DEFAULT_CHILD_TIMEOUT_SECONDS)}`, ` --supervisor-script Default: ${DEFAULT_SUPERVISOR_SCRIPT}`, + " --skip-build Do not rebuild local runtime packages before launching the supervisor", " -h, --help", - "Reads only each child run summary.json.", + "Builds local CCC/bot/tester/supervisor runtime, then reads only each child run summary.json.", "Loop options must appear before --; supervisor --out-dir is owned by the loop.", ].join("\n"); } @@ -197,6 +211,14 @@ export async function runSupervisorLoop({ argv, root = rootDir, dependencies = { let previousSignature; let stableCount = 0; + if (!args.skipBuild) { + const prebuild = prebuildRuntime(root, dependencies); + if (prebuild.status !== 0) { + stdout.write(formatPrebuildFailure(prebuild) + "\n"); + return prebuild.status; + } + } + for (let runIndex = 1; runIndex <= args.maxRuns; runIndex += 1) { const runOutDir = join(outRoot.absolutePath, `run-${padRun(runIndex)}`); const relativeOutDir = displayPath(root, runOutDir); @@ -214,7 +236,7 @@ export async function runSupervisorLoop({ argv, root = rootDir, dependencies = { const summary = await readSummary(join(runOutDir, "summary.json"), dependencies); run = summarizeRun(summary, { runIndex, relativeOutDir, status }); } catch (error) { - stdout.write(formatMissingSummaryLine({ runIndex, relativeOutDir, status, error }) + "\n"); + stdout.write(formatMissingSummaryLine({ runIndex, relativeOutDir, status, error, spawnResult }) + "\n"); return status === 0 ? 1 : status; } @@ -276,6 +298,24 @@ function spawnSupervisorHelp({ root, supervisorScript, supervisorArgs, dependenc }); } +export function prebuildRuntime(root, dependencies) { + const spawnSyncFn = dependencies.spawnSync ?? spawnSync; + for (const step of PREBUILD_COMMANDS) { + const result = spawnSyncFn(step.command, step.args, { + cwd: root, + env: minimalProcessEnv(process.env), + stdio: "ignore", + timeout: DEFAULT_PREBUILD_TIMEOUT_SECONDS * 1000, + killSignal: "SIGTERM", + }); + const status = typeof result.status === "number" ? result.status : 1; + if (status !== 0) { + return { ...step, status, signal: result.signal, error: result.error }; + } + } + return { status: 0 }; +} + function hasHelpFlag(args) { return args.some((arg) => arg === "-h" || arg === "--help"); } @@ -329,14 +369,41 @@ function formatRunLine(run, decision) { ].join(" "); } -function formatMissingSummaryLine({ runIndex, relativeOutDir, status, error }) { - return [ +function formatMissingSummaryLine({ runIndex, relativeOutDir, status, error, spawnResult }) { + const fields = [ `loop run=${String(runIndex)}`, `status=${String(status)}`, "summary=missing_or_invalid", `error=${shellWord(errorMessage(error))}`, - `out=${relativeOutDir}`, - ].join(" "); + ]; + const childError = spawnResult?.error === undefined ? undefined : childSpawnError(spawnResult.error); + if (childError !== undefined) { + fields.push(`child_error=${shellWord(childError)}`); + } + if (typeof spawnResult?.signal === "string" && spawnResult.signal.length > 0) { + fields.push(`signal=${shellWord(spawnResult.signal)}`); + } + fields.push(`out=${relativeOutDir}`); + return fields.join(" "); +} + +export function formatPrebuildFailure(prebuild) { + const fields = [ + "loop prebuild_failed", + `target=${shellWord(prebuild.target ?? "unknown")}`, + `status=${String(prebuild.status)}`, + ]; + if (prebuild.command !== undefined) { + fields.push(`command=${shellWord([prebuild.command, ...(prebuild.args ?? [])].join(" "))}`); + } + if (typeof prebuild.signal === "string" && prebuild.signal.length > 0) { + fields.push(`signal=${shellWord(prebuild.signal)}`); + } + const childError = prebuild.error === undefined ? undefined : childSpawnError(prebuild.error); + if (childError !== undefined) { + fields.push(`child_error=${shellWord(childError)}`); + } + return fields.join(" "); } function formatCounts(counts) { @@ -491,6 +558,13 @@ function errorMessage(error) { return error instanceof Error ? error.message : String(error ?? "Unknown error"); } +function childSpawnError(error) { + if (error instanceof Error && "code" in error && typeof error.code === "string") { + return error.code; + } + return errorMessage(error); +} + if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) { process.exitCode = await runSupervisorLoop({ argv: process.argv.slice(2) }); } diff --git a/scripts/ickb-supervisor-loop.test.mjs b/scripts/ickb-supervisor-loop.test.mjs index a950308..9dd3bfd 100644 --- a/scripts/ickb-supervisor-loop.test.mjs +++ b/scripts/ickb-supervisor-loop.test.mjs @@ -4,6 +4,7 @@ import test from "node:test"; import { decideNext, DEFAULT_CHILD_TIMEOUT_SECONDS_VALUE, + DEFAULT_PREBUILD_TIMEOUT_SECONDS_VALUE, INSPECTION_REQUIRED_EXIT_CODE, parseArgs, runSupervisorLoop, @@ -21,11 +22,13 @@ test("supervisor loop parses loop options and supervisor passthrough", () => { "--stable-limit", "2", "--backoff-seconds", "0", "--child-timeout-seconds", "120", + "--skip-build", "--supervisor-script", "apps/supervisor/dist/custom.js", "--", "--scenario", "bot-only", ]), { help: false, + skipBuild: true, outRoot: "logs/live-supervisor/loop-test", maxRuns: 5, stableLimit: 2, @@ -48,6 +51,7 @@ test("supervisor loop parses loop options and supervisor passthrough", () => { assert.throws(() => parseArgs(["--", "--out-dir", "logs/live-supervisor/x"]), /Do not pass supervisor --out-dir/u); assert.throws(() => parseArgs(["--", "--out-dir=logs/live-supervisor/x"]), /Do not pass supervisor --out-dir/u); assert.throws(() => parseArgs(["--", "--max-runs", "1", "--scenario", "standard-cycle"]), /Do not pass loop option --max-runs after --/u); + assert.throws(() => parseArgs(["--", "--skip-build"]), /Do not pass loop option --skip-build after --/u); assert.throws(() => parseArgs(["--", "--stable-limit=2"]), /Do not pass loop option --stable-limit after --/u); assert.throws(() => parseArgs(["--scenario", "standard-cycle", "--out-dir=logs/live-supervisor/x"]), /Unknown argument before --: --scenario/u); assert.equal(parseArgs([]).childTimeoutSeconds, DEFAULT_CHILD_TIMEOUT_SECONDS_VALUE); @@ -266,6 +270,7 @@ test("supervisor loop runs bounded supervisor commands until stable", async () = "--max-runs", "5", "--stable-limit", "2", "--backoff-seconds", "0", + "--skip-build", "--", "--scenario", "bot-only", ], @@ -297,6 +302,112 @@ test("supervisor loop runs bounded supervisor commands until stable", async () = } }); +test("supervisor loop prebuilds runtime before launching supervisor", async () => { + const root = "/repo"; + const originalPrivateKey = process.env.PRIVATE_KEY; + process.env.PRIVATE_KEY = "operator-secret"; + const commands = []; + const output = { text: "", write(chunk) { this.text += chunk; } }; + try { + const exitCode = await runSupervisorLoop({ + argv: [ + "--out-root", "logs/live-supervisor/loop-prebuild", + "--max-runs", "1", + "--backoff-seconds", "0", + "--", + "--scenario", "bot-only", + ], + root, + io: { stdout: output, stderr: output }, + dependencies: { + spawnSync: (command, args, options) => { + commands.push({ command, args, options }); + return { status: 0 }; + }, + readFile: async () => JSON.stringify({ + stopped: "max_cycles", + aggregateCounts: { bot_no_action_skip: 1 }, + txCreatingTxHashCount: 0, + txCreatingOutcomeCount: 0, + artifacts: [], + }), + }, + }); + + assert.equal(exitCode, INSPECTION_REQUIRED_EXIT_CODE); + assert.deepEqual(commands.map((item) => [item.command, ...item.args]), [ + ["pnpm", "forks:ccc"], + ["pnpm", "bot:build"], + ["pnpm", "--filter", "@ickb/tester", "build"], + ["pnpm", "--filter", "@ickb/supervisor", "build"], + [process.execPath, "/repo/apps/supervisor/dist/index.js", "--scenario", "bot-only", "--out-dir", "logs/live-supervisor/loop-prebuild/run-0001"], + ]); + for (const command of commands) { + assert.equal(command.options.env.PRIVATE_KEY, undefined); + } + for (const command of commands.slice(0, 4)) { + assert.equal(command.options.stdio, "ignore"); + assert.equal(command.options.timeout, DEFAULT_PREBUILD_TIMEOUT_SECONDS_VALUE * 1000); + assert.equal(command.options.killSignal, "SIGTERM"); + } + } finally { + if (originalPrivateKey === undefined) { + delete process.env.PRIVATE_KEY; + } else { + process.env.PRIVATE_KEY = originalPrivateKey; + } + } +}); + +test("supervisor loop reports prebuild failures without child output", async () => { + const output = { text: "", write(chunk) { this.text += chunk; } }; + const exitCode = await runSupervisorLoop({ + argv: ["--out-root", "logs/live-supervisor/loop-prebuild-fail"], + root: "/repo", + io: { stdout: output, stderr: output }, + dependencies: { + spawnSync: () => ({ + status: 1, + stdout: "privateKey 0x1111\n", + stderr: "operator secret 0x2222\n", + }), + readFile: async () => { + throw new Error("should not read summary after prebuild failure"); + }, + }, + }); + + assert.equal(exitCode, 1); + assert.match(output.text, /loop prebuild_failed/u); + assert.match(output.text, /target=ccc/u); + assert.match(output.text, /command=pnpm_forks:ccc/u); + assert.doesNotMatch(output.text, /privateKey|0x1111|operator secret|0x2222/u); +}); + +test("supervisor loop reports timed out prebuild failures", async () => { + const output = { text: "", write(chunk) { this.text += chunk; } }; + const exitCode = await runSupervisorLoop({ + argv: ["--out-root", "logs/live-supervisor/loop-prebuild-timeout"], + root: "/repo", + io: { stdout: output, stderr: output }, + dependencies: { + spawnSync: () => ({ + status: null, + signal: "SIGTERM", + error: Object.assign(new Error("spawn ETIMEDOUT"), { code: "ETIMEDOUT" }), + }), + readFile: async () => { + throw new Error("should not read summary after prebuild timeout"); + }, + }, + }); + + assert.equal(exitCode, 1); + assert.match(output.text, /loop prebuild_failed/u); + assert.match(output.text, /signal=SIGTERM/u); + assert.match(output.text, /child_error=ETIMEDOUT/u); +}); + test("supervisor loop accepts validation session out roots", async () => { const root = "/repo"; const reads = new Map([ @@ -315,6 +426,7 @@ test("supervisor loop accepts validation session out roots", async () => { argv: [ "--out-root", "log/validation/dynamic-test/chunks/chunk-0001", "--max-runs", "1", + "--skip-build", "--", "--scenario", "bot-only", ], @@ -376,6 +488,7 @@ test("supervisor loop accepts explicit validation roots outside the repo", async argv: [ "--out-root", "/var/tmp/ickb-log/validation/dynamic-test/chunks/chunk-0001", "--max-runs", "1", + "--skip-build", "--", "--scenario", "bot-only", ], @@ -404,6 +517,7 @@ test("supervisor loop applies child timeout at the outer process boundary", asyn "--out-root", "logs/live-supervisor/loop-timeout", "--child-timeout-seconds", "1", "--backoff-seconds", "0", + "--skip-build", ], root, io: { stdout: output, stderr: output }, @@ -424,13 +538,45 @@ test("supervisor loop applies child timeout at the outer process boundary", asyn assert.equal(commands[0].options.timeout, 1000); assert.equal(commands[0].options.killSignal, "SIGTERM"); assert.match(output.text, /summary=missing_or_invalid/u); + assert.match(output.text, /child_error=ETIMEDOUT/u); + assert.match(output.text, /signal=SIGTERM/u); +}); + +test("supervisor loop does not print arbitrary child output on missing summary", async () => { + const commands = []; + const output = { text: "", write(chunk) { this.text += chunk; } }; + const exitCode = await runSupervisorLoop({ + argv: ["--out-root", "logs/live-supervisor/loop-private-child-output", "--backoff-seconds", "0", "--skip-build"], + root: "/repo", + io: { stdout: output, stderr: output }, + dependencies: { + spawnSync: (command, args, options) => { + commands.push({ command, args, options }); + return { + status: 1, + stdout: "privateKey 0x1111\n", + stderr: "Live supervisor failed: Missing built bot app: apps/bot/dist/index.js\noperator secret 0x2222\n", + }; + }, + readFile: async () => { + const error = new Error("missing"); + error.code = "ENOENT"; + throw error; + }, + }, + }); + + assert.equal(exitCode, 1); + assert.equal(commands[0].options.stdio, "ignore"); + assert.match(output.text, /summary=missing_or_invalid/u); + assert.doesNotMatch(output.text, /privateKey|0x1111|Missing built bot app|operator secret|0x2222/u); }); test("supervisor loop stops for inspection on new tx-bearing summary", async () => { const root = "/repo"; const output = { text: "", write(chunk) { this.text += chunk; } }; const exitCode = await runSupervisorLoop({ - argv: ["--out-root", "logs/live-supervisor/loop-tx", "--backoff-seconds", "0"], + argv: ["--out-root", "logs/live-supervisor/loop-tx", "--backoff-seconds", "0", "--skip-build"], root, io: { stdout: output, stderr: output }, dependencies: { @@ -495,7 +641,7 @@ test("supervisor loop reports invalid out-root as a concise CLI error", async () test("supervisor loop hides invalid summary JSON contents", async () => { const output = { text: "", write(chunk) { this.text += chunk; } }; const exitCode = await runSupervisorLoop({ - argv: ["--out-root", "logs/live-supervisor/loop-invalid", "--backoff-seconds", "0"], + argv: ["--out-root", "logs/live-supervisor/loop-invalid", "--backoff-seconds", "0", "--skip-build"], root: "/repo", io: { stdout: output, stderr: output }, dependencies: {