feat(bench): EOP by default + --query for query-context measurement#601
Merged
Conversation
mops bench compiled benchmark canisters with --legacy-persistence and ran every cell as an update call — neither reflects how a real query canister executes. Two changes: - Compile under enhanced orthogonal persistence (moc's default since 0.15) instead of forcing --legacy-persistence. The bench canister is already a `persistent actor`, so this just drops the flag. - Add --query: measure each cell via the canister's existing runCellQuery (a query method) instead of runCellUpdateAwait. Queries run no GC on the IC, so instruction counts exclude GC work an update would incur — realistic for query/read-only workloads. Synchronous runners only (async runners need the update path). Verbose pipeline now prints context (query/update) and persistence. Docs + changelog updated. mops-bench library needs no change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Cursor AI review👍 APPROVE — looks safe to merge
VerdictDecision: APPROVE Generated for commit 6c3d9aa |
EOP stays the default, but add a --legacy-persistence flag so a canister that still uses legacy persistence can be benchmarked. Only emitted for moc >= 0.15 (legacy is already the default below that). Verbose pipeline prints the active persistence mode. Docs + changelog updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kamirus
added a commit
that referenced
this pull request
Jun 30, 2026
…ence for legacy collectors (#606) `mops bench` (no `--gc` flag) crashed on moc 0.15+ — a regression from 2.16.0 ([#601](#601)), which made enhanced orthogonal persistence the default bench mode but left the default `--gc` as `copying`. moc fixes the GC to `incremental` under EOP and rejects the legacy collectors there, so the default run was an invalid flag combination. **Before** — default `mops bench`: ``` moc-wrapper -c --idl canister.mo --force-gc --copying-gc --release ... moc: Invalid compiler flag combination: --copying-gc is not supported with --enhanced-orthogonal-persistence Error: benchmarks failed ``` **After** — default `mops bench` compiles under EOP with no collector flag (incremental is moc's fixed GC there): ``` moc-wrapper -c --idl canister.mo --force-gc --release ... ``` `--gc copying|compacting|generational` still works: those collectors only exist under legacy persistence, so selecting one now implies `--legacy-persistence` (instead of crashing with an opaque moc error). `--gc incremental` or omitting `--gc` keeps measuring under EOP. ## Why the bug evaded detection #601 switched bench to EOP-by-default and smoke-tested with `--gc incremental`, but the `--gc` *default* stayed `copying`, so the unflagged `mops bench` invocation everyone runs was never exercised. More fundamentally, there was **no `mops bench` end-to-end test at all** — and this repo's own `mops.toml` pins `moc 0.14.14` (pre-EOP), so even the local bench never touched the EOP path. The crash only surfaced on a downstream project (motoko-core) running a newer moc. This PR adds an e2e test (`cli/tests/bench.test.ts` + `cli/tests/bench/` fixture) that pins `moc 1.3.0` + `pocket-ic 12.0.0` and runs `mops bench` for real, asserting exit 0 and no `Invalid compiler flag combination`. Verified it fails against the pre-fix state with the original moc error and passes with the fix. ## Migration Default benchmark numbers shift from `copying` (legacy) to `incremental` (EOP) — but 2.16.0 already shifted numbers by switching persistence modes, and `incremental` is the only GC consistent with EOP-by-default. To reproduce the old legacy-mode measurements: `mops bench --gc copying` (now auto-enables `--legacy-persistence`). --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mops benchcompiled benchmark canisters with--legacy-persistenceand ran every cell as an update call — neither reflects how a realquerycanister executes on the IC. This makes the instruction counts unrepresentative for read-only/queryworkloads (they include GC work a query never does, under the wrong persistence mode).Changes
getMocArgsno longer forces--legacy-persistence(it was gated only onmoc ≥ 0.15). The bench canister is already apersistent actor, so dropping the flag compiles it under enhanced orthogonal persistence — moc's default and the mode real canisters run.--query— measures each cell via the canister's existingrunCellQuery(aquerymethod) instead ofrunCellUpdateAwait. Queries run no GC on the IC, so the counts exclude GC work an update would incur. Synchronous runners only (async/inter-canister runners still need the update path).context(query/update) andpersistence.No change to the
mops-benchlibrary — the canister template already exposesrunCellQuery.Validation
npm run check(tsc) passes.tsxagainst a real project: verbose printscontext: query/persistence: enhanced/gc: incremental, and the measured numbers shift as expected — the real-runWithfloor drops sharply (no GC counted) while raw allocation rises slightly under 64-bit EOP.Docs / changelog
docs/docs/cli/4-dev/02-mops-bench.md: updated "How it works" + new### --querysection.cli/CHANGELOG.md: entries under## Next.--helpcarries a description for--query.mops-cliskill documents no individual bench flags, so nothing there to update.🤖 Generated with Claude Code