Enforce execution limits with a balance gate and per-org rate-limit backstop#1274
Merged
Conversation
Check the org's Autumn execution balance before execute/executeWithPause (never resume, so paused executions can always complete). Blocked orgs get a descriptive ExecuteResult.error instead of running. Fails open on any billing error or a 2s timeout, and caches per-org outcomes for 60s.
1000 execute calls per org per hour, fixed window, counted in a minimal
per-org Durable Object (EXECUTION_RATE_LIMITER, one instance per org via
idFromName). Independent of billing so runaway automation is caught even
when Autumn is down; fails open if the counter DO errors or is slow. The
DO stores a single {windowId, count} record and purges itself by alarm
after two idle windows.
Order: rate-limit backstop first (cheap counter), then the balance gate, then usage tracking. Guards wrap outside the tracker so a blocked execution is neither run nor tracked. Covers both planes (HTTP executor plane and MCP session DO) since both build engines through this layer.
Covers allow/block, the typed errors, fail-open on billing errors and timeouts (asserting the check was attempted AND the execution ran), the 60s per-org outcome cache (allowed and blocked cached, errors never), window rollover, per-org isolation, and that resume is never gated.
Contributor
Cloudflare previewTorn down — the PR is closed. |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
Greptile SummaryThis PR adds two pre-execution guards to
Confidence Score: 5/5Safe to merge. The guards fail open, the decoration order is correct (rate-limit then balance-gate then tracker), and all four critical behaviors are covered by e2e tests against real workerd. The core guard logic is well-structured: both gates wrap outside the usage tracker so blocked executions are neither run nor billed, resume is correctly ungated, and the fail-open paths are exercised by e2e fault injection. The two findings are quality nits in the test file only and do not affect production behavior. e2e/cloud/mcp-execution-limits.test.ts imports an app-internal module and carries a manually-synced rate-limit constant. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Client as MCP Client
participant RL as RateLimiter (outermost)
participant BG as BalanceGate
participant UT as UsageTracker
participant Eng as ExecutionEngine
participant DO as CounterDO
participant Autumn as Autumn (billing)
Client->>RL: execute(code)
RL->>DO: increment(windowId)
DO-->>RL: count
alt "count > hourly limit"
RL-->>Client: error: rate limit exceeded (not metered)
else within limit
RL->>BG: execute(code)
BG->>Autumn: checkExecutionBalance(orgId)
note over BG,Autumn: 60s cache, 2s timeout, fail open on error
Autumn-->>BG: "{ allowed }"
alt "allowed == false"
BG-->>Client: error: execution limit reached (not metered)
else "allowed == true"
BG->>UT: execute(code)
UT->>Eng: execute(code)
Eng-->>UT: result
UT->>Autumn: trackExecution (fire-and-forget)
UT-->>Client: result
end
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Client as MCP Client
participant RL as RateLimiter (outermost)
participant BG as BalanceGate
participant UT as UsageTracker
participant Eng as ExecutionEngine
participant DO as CounterDO
participant Autumn as Autumn (billing)
Client->>RL: execute(code)
RL->>DO: increment(windowId)
DO-->>RL: count
alt "count > hourly limit"
RL-->>Client: error: rate limit exceeded (not metered)
else within limit
RL->>BG: execute(code)
BG->>Autumn: checkExecutionBalance(orgId)
note over BG,Autumn: 60s cache, 2s timeout, fail open on error
Autumn-->>BG: "{ allowed }"
alt "allowed == false"
BG-->>Client: error: execution limit reached (not metered)
else "allowed == true"
BG->>UT: execute(code)
UT->>Eng: execute(code)
Eng-->>UT: result
UT->>Autumn: trackExecution (fire-and-forget)
UT-->>Client: result
end
end
Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+87
to
+94
| const checkExecutionBalance = (organizationId: string) => | ||
| Effect.gen(function* () { | ||
| yield* Effect.annotateCurrentSpan({ "autumn.customer.id": organizationId }); | ||
| const check = yield* use((c) => | ||
| c.check({ customerId: organizationId, featureId: "executions" }), | ||
| ); | ||
| return { allowed: check.allowed }; | ||
| }).pipe(Effect.withSpan("autumn.checkExecutionBalance")); |
There was a problem hiding this comment.
check() for unlimited-plan orgs may silently block executions
checkExecutionBalance returns { allowed: check.allowed } verbatim. The balance gate's fail-open path only triggers on an AutumnError (i.e., when use(...) throws). If Autumn's check() returns { allowed: false } for customers on plans where the executions feature is unlimited or not feature-gated (rather than throwing), those orgs will be blocked without any fail-open. The PR description says it "fails open on missing customer", which implies Autumn throws for unknown customers — but orgs on enterprise or legacy plans that have executions either unconfigured or returned as false would be silently blocked with no warning. Worth verifying the exact Autumn response shape for unlimited/non-metered plan customers before this ships.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
executor-cloud | 78f7b83 | Jul 03 2026, 07:42 PM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 78f7b83 | Commit Preview URL Branch Preview URL |
Jul 03 2026, 07:40 PM |
Delete the balance-gate and rate-limit unit tests: the guards are now exercised end to end against the real workerd + Autumn emulator, which is a truer test of behaviour that mattered here (blocked executions not reaching the meter, failing open on billing errors). Drop the test-only affordances the unit tests were the only callers of: the gate's `timeoutMs` option knob and both limiters' `check` diagnostic methods. Move the two blocked-execution messages into a small Cloudflare-free module so the e2e suite can import them as assertion ground truth without pulling the guards' worker-runtime dependencies into its typecheck.
Add an optional EXECUTION_RATE_LIMIT_PER_HOUR override, parsed as a positive integer and falling back to the 1000/hour default when unset or unparseable. Production leaves it unset; it exists so e2e can drive the abuse backstop with a handful of real executions instead of a thousand. The e2e cloud dev server sets it to 3, reaching the worker the same way ALLOW_LOCAL_NETWORK does.
Bump the emulator to 0.13.2 for its balances.check support and one-shot fault injection, and extend the Autumn surface to use them: armFault/clearFaults drive the emulator's fault control plane, exhaustExecutions burns an org's whole included allotment in one track (the amount read from the plan seed, never hardcoded), and ledgerFor reads the request ledger so a scenario can prove a faulted balances.check was actually attempted.
Four cloud scenarios drive a real MCP client against the workerd + session-DO topology and read Autumn's ledger as ground truth: an org at its cap is blocked before the code runs and not metered; a balance-check 500 and a balance-check timeout each fail open and still run the execution (with the faulted check visible in the ledger as proof the gate consulted Autumn); and the rate-limit backstop blocks the execution past the hourly cap without metering it. Faults are one-shot and cleared in finalizers, and each scenario uses a fresh org.
The e2e worker's per-org hourly cap of 3 was low enough that ordinary scenarios (toolkits-mcp, no-auth-connection, connect-handoff) tripped the backstop mid-test once the suite ran them against the shared worker. Lift it to 20 (still exhaustible by the backstop scenario) and move the value into one shared constant consumed by both the boot env and the test.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
executor-cloud | 6ec66c4 | Jul 03 2026, 08:14 PM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 6ec66c4 | Commit Preview URL Branch Preview URL |
Jul 03 2026, 08:14 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Execution usage is tracked to the billing provider after every execution, but nothing ever checks the balance before running. An org on the free plan (10k executions/month, no overage) can keep executing indefinitely past its quota: the ledger clamps at the cap while executions continue at full speed. Observed in production as one tenant sustaining automated polling around the clock, far past its included usage, for over a week.
What this adds
Two pre-execution guards in
CloudMeteringEngineDecorator, the one decorator both cloud planes (MCP session DO and HTTP executor plane) build engines through. They wrap outside usage tracking, so a blocked execution is neither run nor billed.resumeis never gated: a paused execution already consumed its slot, and blocking resume would strand approved work.1. Balance gate (
execution-gate.ts)executionsfeature via the billing service beforeexecute/executeWithPause.trackExecution.ExecuteResult.error(the same channel compilation errors use, since engine error-channel failures are deliberately rendered opaque by the host): a cleanisErrortool result telling the user their plan's included executions are used up.2. Rate-limit backstop (
execution-rate-limit.ts)idFromName(orgId), single{windowId, count}record, alarm-purged after two idle windows). NewEXECUTION_RATE_LIMITERbinding + migrationv3.EXECUTION_RATE_LIMIT_PER_HOUR) so the e2e worker can exercise the backstop with a small cap; production uses the default.Testing: end to end, with fault injection
Coverage is e2e rather than unit tests:
e2e/cloud/mcp-execution-limits.test.tsdrives a real MCP client over StreamableHTTP against the production workerd + session-DO topology, with the Autumn emulator (@executor-js/emulate0.13.2, which addsbalances.checkand one-shot fault injection) as the billing backend and its request ledger as ground truth:balances.check: the execution succeeds, and the ledger's faulted entry proves the gate consulted the provider and failed open rather than skipping the check.The fail-open scenarios pin the operational guarantee directly: a billing-provider outage or slowdown cannot take executions down.
Verification
vitest run --project cloud cloud/mcp-execution-limits.test.ts— 4/4 passed against real workerd + emulatorsbun run typecheck— 42/42 greenbun run lint,bun run format:check— cleanDeploy notes
v3creates the counter DO class; standard single deploy. In workers without the binding (tests, older local setups) the limiter logs a warning and disables itself.