feat(runtime): coordinate tool batches through resource authorities

[Jarad-z]

Summary

This revision keeps the useful part of the batch scheduler—a thin, deterministic sequencer—but moves resource correctness to resource authorities:

frozen validated args
  -> authority.prepare(...)
  -> one-shot PreparedOperation { authoritative claims, execute(context) }
  -> thin batch sequencer
  -> authority-owned admission / lease / identity checks / effect

The filesystem authority owns canonical execution paths plus process-wide, writer-fair exact/tree read-write leases. A separate process admission plane now makes all() a real process-wide exclusive barrier against participating filesystem work, including direct execution and root/child Runtime compositions.

Preparation retains only stable canonical claims. Mutable filesystem identity is sampled after process and filesystem admission, once earlier conflicting owners have completed, and is then pinned through descriptor/CAS primitives for the actual effect. This allows legitimate ordered transitions such as Write(create) -> Read and atomic replacement -> Read without reopening the race between admission and execution.

The batch layer owns provider-order tickets, stable result slots, cancellation, and fail-stop dispatch. Provider order is only a deterministic tie-breaker for conflicting operations; it is not a general data dependency between tool calls in one assistant step.

Refs #4487.

What changed after review

• Replaced mutable resource planning with immutable, authority-prepared, one-shot operations.
• Moved canonical path claims, exact/tree overlap, writer fairness, and cross-batch exclusion into the filesystem authority.
• Added a writer-fair process shared/exclusive admission coordinator. all() takes exclusive admission; participating filesystem effects take shared admission; explicit none() bypasses it.
• Shared the same process and filesystem coordinators across direct execution and root/child Runtime Host compositions.
• Split stable claim preparation from mutable target identity capture. Identity is now sampled only after admission, immediately before a pinned read or mutation.
• Added descriptor-pinned exact reads and preserved mutation CAS/outcome-unknown behavior.
• Stopped dispatching queued work after a turn-fatal rejection while allowing already-active work to settle.
• Preserved fan-out for explicitly independent web/agent work and the existing subagent limiter instead of treating capacity as a fake resource conflict.
• Added real builtin and cross-batch coverage for Read, Write, Edit, Grep, Glob, and apply_patch, including aliases, tree overlap, multiple write keys, create/read and replacement/read chains, all()/filesystem ordering, and root/child composition.
• Rebased onto current apache/main at 7f7843e.

Current compromises

• Unknown or unclassified executable tools, including dynamic MCP tools without a contract, fail closed to process-exclusive all(). This avoids unsafe concurrency but can reduce throughput.
• Bash and Computer remain all() until they have enforceable resource-owner contracts.
• Explicit none() work intentionally does not participate in the process barrier.
• Preparation still captures stable canonical claims eagerly. If an earlier coarse operation replaces a directory tree so that a later call resolves to a different canonical claim, the later call fails closed and must be retried in a new provider step. Ordinary owner-caused inode/content transitions under an unchanged claim are now supported because identity is captured after admission.
• Todo/Goal/Plan, terminal/browser state, provider capacity, and multi-resource transactions remain follow-up authorities rather than being forced into the filesystem lock model.
• exclusive_step remains a separate admission decision computed before authority preparation.

Verification

Passed locally on Windows after rebasing onto current apache/main:

• @maka/core build
• @maka/storage build
• @maka/mcp build
• @maka/runtime build
• @maka/runtime-host build
• Authority, filesystem admission, stable-read, worker CAS, ToolCallBatch, preparation, and Runtime Host composition matrix: 120 passed, 0 failed
• Biome check across the changed Runtime/Runtime Host/architecture surfaces: passed
• git diff --check: passed

The complete Runtime dist suite was also attempted. It does not complete cleanly on this Windows host because of existing platform/test-harness cases involving /bin/echo, symlink privileges, SQLite EBUSY cleanup, and PTY/pipe timing. The affected and newly added suites above complete cleanly; the platform limitation is reported rather than hidden.

The detailed earlier filesystem lease matrix remains in docs/filesystem-read-tree-lease-test-report.md.

AI use

• No generative tool made a substantive contribution
• Generative tooling made a substantive contribution

Tool(s) and scope: Codex assisted with architecture review, implementation, tests, rebase, and PR preparation. All contributed commits carry a Generated-by: Codex trailer.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck/build, and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described above
• No

[alva-bot01]

Approved via Slack

[likun666661]

A few design thoughts after looking at this through the lens of classic concurrency and Tokio-style async I/O.

Overall, I agree that overlapping local tool effects need deterministic coordination, and the scheduler implementation itself is coherent. The writer-fair rule (a later reader cannot bypass an earlier queued writer, while independent work can bypass) is a sensible async-RwLock policy. My main concern is that the current design may be combining two different responsibilities:

1. Correctness: preventing corruption or inconsistent observation of a resource.
2. Orchestration: deciding which otherwise-safe tasks should start now for determinism, fairness, or throughput.

The concurrency problem is at the resource authority

This is closer to an asynchronous I/O side-effect race than a shared-memory/executor scheduling problem. A Tokio executor schedules runnable futures; it does not inspect application resources to decide whether two futures touch the same Redis key or file. Readiness and exclusivity are owned below the executor by the I/O resource, an async Mutex/RwLock, a semaphore, or a dedicated resource-owning task/actor.

The analogous Maka split would be:

AiSdkBackend / batch runner
  - creates lazy ToolTasks
  - preserves admission and result slots
  - propagates cancellation

Resource authority
  - canonicalizes the real resource identity
  - owns mutual exclusion / version checks
  - defines the linearization point
  - wakes the next waiter when the resource becomes available

Underlying effect
  - filesystem / session state / terminal / browser / remote service

With this split, the batch runner could remain a simple fan-out/fan-in over lazy run() closures. A task that is not ready awaits its resource authority instead of being kept out of a central batch scheduler. Correctness would still hold if the batch planner were disabled.

This is especially important because a batch-local scheduler cannot enforce the invariant against another batch, turn, agent, process, or any code path that reaches the same resource without going through this scheduler. The existing filesystem write lock already points toward the lower-level authority model; it also canonicalizes closer to the actual execution boundary.

Tokio concepts map to different Maka primitives

I do not think one generic access graph should carry every responsibility:

• Files: keyed, writer-fair async RwLease at the filesystem authority, plus the existing inode/version/atomic-write safeguards.
• Todo/Goal/Plan: immutable state + revision CAS, or a session-state actor.
• Terminal/browser: one resource-owning actor per session/tab/window, with commands and reply promises.
• Web/MCP/subagent capacity: semaphore/capacity policy, not fabricated resource conflicts.
• Multi-resource atomic operations: an explicit transaction/acquireMany contract if atomicity is truly required. Holding multiple batch-local accesses prevents some interleavings, but does not by itself provide crash atomicity or cross-batch isolation.

In that model, the equivalent of a Tokio waker is simply the authority releasing a lease or consuming a command and waking the next waiter. We do not need work stealing or lock-free queue machinery here; the hard part is ownership and cancellation-safe I/O effects, not CPU scheduling.

Concrete concerns in the current phase

1. resolveAccesses omission => all is a behavioral change, not only a safe default. This PR only adds precise declarations to file tools. Existing tools such as WebSearch and agent spawning appear to remain undeclared, so multiple calls that previously fanned out can now serialize behind all. In particular, the existing subagent concurrency limiter may effectively become batch-local concurrency 1. If this is intentional, it should be an explicit compatibility decision with regression tests; if not, it is a fairly large hidden throughput regression.

2. The first demonstrated problem is narrower than the architecture. Mutation/mutation on one file is already serialized by withFileWriteLock. The immediate uncovered cases are read/write, search/write, multi-file interleaving, and cross-tool cases such as Bash versus file tools. I would prefer real end-to-end tests using builtin Read/Edit/Grep/apply_patch, not only the synthetic ScheduledFile integration test, so the user-visible failure and the new guarantee are falsifiable.

3. Provider array order needs an explicit semantic status. Calls in one assistant step are concurrent in the sense that the model cannot observe an earlier result before emitting the later call. Provider order can be a useful deterministic tie-breaker, but it should not silently become a data-dependency mechanism. If a later call must consume an earlier result, it belongs in the next model step.

4. all should ideally name a real coarse authority. “Conflicts with everything in the universe” creates unrelated head-of-line blocking. workspace:{id}, session:{id}, and provider:{id} make the ownership and blast radius explicit. Unknown provider-only work should not automatically conflict with local filesystem work unless a verified contract requires it.

Suggested simplification

I would frame the first milestone as:

Make builtin filesystem operations linearizable against overlapping reads and writes at the filesystem authority. Use provider order only as a deterministic tie-breaker for same-batch conflicts. Preserve fan-out for tools whose resource/capacity contract is independent.

Then introduce only three reusable coordination concepts when concrete users exist:

ResourceLease   keyed Mutex/RwLock semantics
ResourceActor   single owner + bounded mailbox
CapacityPermit  semaphore/backpressure

A thin batch planner can still be valuable later to avoid wasted contention and improve observability, but it should be advisory rather than the sole correctness boundary.

So my conclusion is: the PR has a reasonable scheduling algorithm, but I think the design would become both simpler and stronger if correctness moved to resource owners and the batch layer were narrowed to orchestration. At minimum, I would clarify the intended regression for undeclared tools and add real builtin behavior tests before merging this phase.

[M4n5ter]

English

The problem is real, and the scheduling algorithm itself is coherent. However, I do not think resolveAccesses plus a batch-local ToolScheduler should become Maka’s long-term source of truth for resource correctness.

My preferred end state is:

thin deterministic batch sequencer
    over
authority-prepared operations

The batch layer should own step admission, provider-order tickets, result slots, cancellation, and fail-stop behavior. Resource authorities should own canonical identity, mutual exclusion, revision/CAS checks, and the actual effect. In other words, the scheduler is still useful, but it should decide when a prepared operation may proceed—not independently decide what the real resource is.

Ideally, validated immutable arguments would be prepared once:

validated args
    → authority.prepare(...)
    → PreparedOperation { authoritative claims, execute() }

The same prepared operation would then drive both scheduling and execution. Files, session state, terminal/browser state, and provider capacity do not need to pretend to have identical semantics: they may use keyed leases, revisioned state/actors, or semaphores behind this lifecycle contract.

There are also two concrete correctness concerns in the current implementation:

1. The scheduler key is not the identity used at the execution boundary.

   normalizeToolFilePath is explicitly lexical, and the builtin access resolvers use it directly. The filesystem executor later derives a canonical enforcement path independently. Consequently, Read("link/a") and Edit("real/a") can be considered independent by this scheduler even when a symlink makes them the same physical file. The architecture document says aliases should be canonicalized during preparation, but that does not currently happen.

   If the intended guarantee is ordering accesses to the same resource—not merely identical path spellings—the claim should come from the same filesystem authority used by execution, with a real builtin alias regression test.

2. A turn-fatal rejection releases the resource and starts queued conflicting work.

   settleToolCall normally resolves tool/business failures, while T1/T2 commit failures can reject. However, finishTask drains the queue after every rejection, and the backend only promotes the rejection after the entire batch has settled.

   This permits the following sequence:

   A performs a write
   → A's T2 outcome commit fails
   → scheduler releases A's access
   → queued conflicting operation B starts
   → only after B settles does the turn surface A's fatal error
   

   Waiting for already-active tasks to unwind is necessary; starting work that has not crossed T1 is not. A business failure may continue the queue, but a turn-fatal/infrastructure rejection should freeze dispatch, cancel queued entries, and then wait for active settlements.

A smaller contract concern is that resolveAccesses currently receives the original mutable executionInput before ToolRuntime snapshots and validates it. Its purity is only documentary. If this hook influences correctness, it should receive an immutable validated snapshot, or be replaced eventually by the prepared-operation seam above.

I am not suggesting that this PR must implement cross-turn coordination, every resource actor, or the complete target architecture. I would, however, address the identity mismatch and fatal-rejection behavior before treating this as a correctness foundation, and document resolveAccesses as a batch orchestration bridge rather than the final resource authority. The previously raised undefined => all fan-out regression also still needs an explicit compatibility decision and tests.

中文

这个问题确实值得解决，当前这套调度算法本身也说得通。不过我不希望 resolveAccesses + batch 内 ToolScheduler 就此变成 Maka 长期的资源正确性边界。

我更期待的最终形态是：

一层很薄、负责确定性编排的 batch sequencer
    +
由各个资源权威准备好的 operation

batch 层负责 step admission、provider 顺序、结果槽位、取消和致命失败后的停止调度；文件系统、Session 状态、终端或浏览器等真正拥有资源的一层，负责确认资源身份、互斥、版本检查以及执行副作用。

换句话说，调度器仍然有价值，但它只该决定“什么时候可以执行”，不该自己猜“这个调用真正访问的是哪个资源”。

比较干净的契约应该是先校验并冻结参数，再由资源权威生成一份 operation：

校验后的参数
    → authority.prepare(...)
    → PreparedOperation { 权威资源声明, execute() }

之后调度和执行使用同一份结果，避免一边按字符串路径排队，另一边执行时又重新解析路径。文件、Session 状态、终端、浏览器和并发容量也不必硬塞进同一种锁模型：它们可以分别使用 keyed lease、revision/CAS、actor 或 semaphore，只共享准备、执行和完成这套生命周期。

当前实现里还有两个比较实在的正确性问题：

1. 排队时认的资源，和真正执行时认的资源不是一回事。

   normalizeToolFilePath 只做字符串层面的路径整理，几个 builtin 的 resolveAccesses 直接拿它生成 key；真正写文件时，filesystem executor 又会单独解析 canonical path。

   这样一来，Read("link/a") 和 Edit("real/a") 即使通过符号链接指向同一个文件，也可能被调度器当成互不相关的任务并发执行。设计文档里写了别名应当在 preparation 阶段归一，但目前代码没有做到。

   如果这里承诺的是“同一资源按顺序执行”，而不只是“路径字符串恰好相同才按顺序执行”，那么资源 claim 应该来自实际执行所使用的 filesystem authority，并补一个真实 builtin 的路径别名测试。

2. 遇到 Turn 级致命失败后，调度器仍会继续放行排队中的冲突任务。

   settleToolCall 会把普通工具错误变成正常的错误结果；真正 reject 的可能是 T1/T2 持久化失败之类的基础设施问题。但 finishTask 不区分失败类型，只要当前任务结束就释放资源并继续扫队列，而 backend 要等整个 batch 全部结束后才抛出致命错误。

   实际上可能发生：

   A 已经修改文件
   → A 的 T2 结果持久化失败
   → 调度器释放 A 占用的资源
   → 排队中的冲突操作 B 开始执行
   → B 结束后，Turn 才报告 A 的致命错误
   

   已经在运行的任务需要等待它们正常收尾，但尚未跨过 T1 的任务没有理由继续启动。普通业务失败可以继续队列；一旦确认是 Turn 级基础设施错误，就应该停止派发、取消 queued tasks，然后等待 active tasks 结束。

还有一个相对小一些的契约问题：resolveAccesses 现在拿到的是原始、可变的 executionInput，之后 ToolRuntime 才会复制并校验参数。“必须是纯函数”目前只靠注释约束。如果它要参与正确性判断，至少应该传入已经校验并冻结的参数；长期则可以被上面的 prepared-operation 契约取代。

我的意思不是要求这个 PR 一步做完跨 Turn 协调、各种 actor 和完整终局架构。但如果要把这套机制当成后续资源调度的基础，我认为路径身份不一致和致命失败后继续启动任务这两点应该先处理；同时应把 resolveAccesses 定位成 batch 编排阶段的过渡接口，而不是最终的资源权威。前面已经有人提到的 undefined => all 导致 fan-out 退化，也仍然需要明确的兼容性决定和回归测试。

[Jarad-z]

@likun666661 @M4n5ter I pushed a substantial revision that follows the authority-owned correctness model from your reviews. The PR now uses immutable authority-prepared operations, canonical filesystem identity with process-wide exact/tree leases, fail-stop queued dispatch, real builtin/cross-batch tests, and explicit fan-out/E2E trade-offs. I also rebased onto current main and rewrote the PR description around the remaining compromises and follow-up authorities. When you have time, could you please take another look?

[likun666661]

重新 review 了当前 revision 67b0881999d802d78207e6ae8bd6568a9436d95d。上轮提到的 canonical filesystem identity 和 turn-fatal rejection 后继续派发的问题已经得到实质修复，整体架构也已经收敛到 authority-prepared operation。不过当前仍有两个 correctness 问题需要处理：

1. all() 目前只在单个 batch 内互斥，并不是 process-wide global barrier

allResourceAuthority 只生成 { kind: 'all' } claim；而 settleToolCallBatch 每次调用都会创建一个新的、batch-local ToolScheduler。

这意味着不同 batch/session 中的 all 彼此不互斥，也不会与 process-wide FilesystemLeaseCoordinator 协调。当前显式映射到 all 的 Bash、maka_computer、request_sandbox_boundary，以及 registry miss 的未知/dynamic tool，都可能与另一个 batch 的真实 Read/Write 并发执行。

我用两个独立的 settleToolCallBatch() 做了复现：batch A 持有一个尚未释放的 Bash/all operation，batch B 执行真实 builtin Write。结果 Write 在 all 仍 active 时就完成了。

因此现在的 all 只能称为 batch-global，不能满足 PR 文案和代码注释中的 “global serialization / fail closed”，也没有覆盖 PR 强调的 cross-batch correctness。建议让 coarse all 和各 process-scoped authority 进入同一个 process-level admission plane，并补一个跨 batch/session 的 all -> Read/Write 阻塞测试。

2. eager identity capture 会把合法的前序 filesystem effect 识别成外部 target replacement

settleToolCallBatch 会在任何 operation 开始执行前并行 prepare 整个 batch。filesystem authority 在 prepare 阶段捕获当前 target identity（filesystem-executor.ts#L515-L527），取得 lease 后重新 resolve，并要求 identity 未发生变化（filesystem-executor.ts#L558-L567）。

我使用真实 builtin filesystem owner 运行同一 batch：

Write("created.txt", "created")
Read("created.txt")

Write 成功且文件已经创建，但后续 Read 因 prepare 时 identity 为 missing、执行时 identity 为 file，以 FilesystemPreparedTargetChangedError 失败。现有 3 calls: queued writer prevents a later reader from bypassing it 测试没有发现这一点，因为 ControlledFilesystemWorker 的 Write 只返回成功结果，并不实际修改磁盘。

PR 已明确不把 provider order 当作通用数据依赖，但对于已经因资源冲突被强制排序的 filesystem operations，目前的实现仍把前序 owner 自己造成的预期变化与 lease 外的篡改混为一谈。除了文案中记录的 Bash -> Write，Write/create -> Read、atomic replacement 后的 Read，以及类似 patch/create 链路也会遇到同样问题。

建议把稳定的 canonical claim/lease key preparation 与可变 identity snapshot 分开：先准备不会变化的 claims，在 operation 真正获得 authority admission、前序冲突 owner 已完成后捕获或刷新 identity，同时继续保证 admission 后到 effect 之间的 race closure。并建议用真实 filesystem backend 增加 create→read、replace→read 回归测试，而不是只依赖不修改磁盘的 controlled worker。

Verification

本地验证通过：

• @maka/core、@maka/storage、@maka/runtime、@maka/mcp、@maka/runtime-host 相关构建
• PR 文档中的定向矩阵：62 tests passed, 0 failed
• git diff --check
• 与当前 main 的本地 merge-tree 无冲突

截至 2026-09-04，当前 head 的 CI、Dependency audit 和 Windows recovery 都仍为 action_required，尚未产生实际 CI 结果。

第一个问题直接破坏 cross-batch correctness 和 all 的 fail-closed 语义，因此我认为当前 revision 仍不宜合并。

[Jarad-z]

@likun666661 @M4n5ter I pushed revision c7d2ac8 and rebased it onto current main.

This addresses the two blockers from the latest review:

1. all() now takes writer-fair process-exclusive admission, while participating filesystem paths take process-shared admission. Direct execution, independent ToolCallBatches, and root/child Runtime Host compositions share the same coordinator. Explicit none() remains outside the barrier.
2. Filesystem preparation now retains only the stable canonical claim. Mutable target identity is sampled after process/filesystem admission, immediately before the pinned effect. Real Write(create) -> Read, replacement -> Read, ApplyPatch chains, and cross-batch cases now pass while post-admission replacement still fails closed.

I also added descriptor-pinned exact reads and updated the worker identity contract so Read and mutation effects use the admitted object rather than re-resolving an unchecked pathname.

Verification after rebase:

• @maka/core, @maka/storage, @maka/mcp, @maka/runtime, and @maka/runtime-host builds pass
• affected authority/filesystem/worker/batch/host matrix: 120 passed, 0 failed
• Biome check and git diff --check pass

The full Runtime dist suite still has the previously reported Windows-only harness/platform failures (/bin/echo, symlink privileges, SQLite EBUSY cleanup, and PTY timing), so I have kept that limitation explicit in the PR description.

When you have time, could you please re-review the current revision?