feat(core,mcp): binary cursor v2 — full-load ~2.7k chars (was ~5k) + maxCursorSeen knob#21
Merged
Merged
Conversation
…axCursorSeen knob
Cursor wire format v2: base64url of [magic Rk + version][page u32 LE][seen
u32 LE × N] instead of v1's JSON array of base36 hash strings — the seen keys
are packed as the raw fnv1a words (hash.ts gains fnv1a32; public fnv1a
unchanged). Cursors ride inside LLM tool outputs downstream, where hosts may
clamp long strings (4096-char defenses) and every char is replayed through
conversation history.
Contract preserved and hardened over v1:
- opaque, self-contained, no instance state; decode throws "invalid cursor"
on anything not produced by meta.nextCursor (magic + version + length +
page checks), including legacy v1 JSON cursors — no migration, cursors are
short-lived load-more state
- non-canonical base64url (tampered trailing bits) is rejected instead of
silently aliasing a valid cursor
- an out-of-uint32-range caller-supplied controls.page (negative, fractional,
NaN, ≥ 2^32) encodes as a poison cursor that fails loudly on the next call
— v1's zod schema did; naive binary packing would have silently wrapped
New createRefkit({ maxCursorSeen }) caps the remembered seen keys (default
still 500). Infinity disables the cap; effective floor is the batch just
returned, so a too-small cap can never make load-more repeat the batch it
just handed back (it would never converge). The zero-config MCP CLI reads
REFKIT_MAX_CURSOR_SEEN (strict decimal; invalid values warn on stderr and
fall back).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The load-more cursor is designed to ride inside LLM tool outputs (
@refkit/mcpreturnsnextCursorfromsearch_references), where some hosts clamp long tool-output strings (4096-char defenses are common) and every char gets replayed through conversation history. v1's JSON encoding hit ~5k chars at the 500-entry seen cap.What
Cursor wire format v2 (binary): base64url of
[magic "Rk" + version][page u32 LE][seen u32 LE × N]— the seen keys are packed as raw fnv1a uint32 words instead of v1's JSON array of base36 strings. Measured full-load size: 4765 → 2676 chars (56%), clearing a 4096-char clamp with ~35% headroom.hash.tsgains an internalfnv1a32; the publicfnv1ais byte-for-byte unchanged.createRefkit({ maxCursorSeen }): caps the remembered seen keys (default still 500, ~5.4 chars each).Infinitydisables the cap. Effective floor is the batch just returned — a too-small cap can never make load-more repeat the batch it just handed back (pagination would never converge).REFKIT_MAX_CURSOR_SEENenv var for the zero-config MCP CLI (strict decimal parse; invalid values warn on stderr and fall back to the default). Documented in both READMEs.Contract
invalid cursoron anything not produced bymeta.nextCursor— including legacy v1 JSON cursors (no migration: cursors are short-lived load-more state, noted in the changeset).controls.page(negative / fractional /NaN/ ≥ 2^32) encodes as a poison cursor that fails loudly on the next call, matching v1's zod behavior — naive packing would have silently wrapped to a different page.Tests
New
cursor.test.ts(roundtrip incl. Buffer cross-validation of the hand-rolled codec, size budget ≤ 2700, magic/version/length/page-0 rejection branches, trailing-bit tamper in both base64 remainder classes, poison-page range); client-level tests for bad-page loud failure, batch-floor non-eviction,Infinity, eviction order, legacy-v1 rejection; MCP env-parse tests incl.Number()-leniency garbage (0x10,1e3,1e100, …).Full suite: 376 passed / 22 skipped (live tests), typecheck + lint clean. All work done test-first.