[Pi] Cold start still dominated by eager 4.3MB shared chunk — ai-tokenizer (~2.8MB) can be lazy-loaded via createRequire (follow-up to #242)

[ElonJask]

Summary

Follow-up to #242. v0.33.0 fixed QuickJS (now a lazy chunk, entry 8.3MB → 1.25MB ✅), but the eager shared chunk is still ~4.3MB and continues to dominate Pi cold start. On 0.34.2 (macOS, Pi 0.84.1), PI_TIMING=1 shows module import: 365–582ms (warm) / ~1.2s (cold) for pi-magic-context alone — still ~52% of all extension load time.

Root cause

The eager chunk (index-fw42dv6t.js, 4.4MB) breaks down as:

• ai-tokenizer ≈ 2.8MB (claude encoding table) — the single biggest contributor
• zod ≈ 583KB, comment-json + esprima ≈ 485KB, typebox ≈ 464KB, subc-client ≈ 231KB

read-session-formatting.ts statically imports ai-tokenizer for estimateTokens(). The comment there says a previous eval("require") lazy pattern failed in Bun's ESM runtime — but that failure was the API choice, not the idea: eval("require") is unsupported in Bun bundles, while createRequire(import.meta.url) is the supported API (embedding-local.ts already uses it for onnxruntime-web).

Proposed fix (validated)

Lazy-load ai-tokenizer with createRequire + a non-literal specifier (same trick embedding-local.ts uses) so Bun does not re-inline it into the eager chunk:

import { createRequire } from "node:module";
import type TokenizerType from "ai-tokenizer";
import type * as ClaudeEncodingType from "ai-tokenizer/encoding/claude";

const lazyRequire = createRequire(import.meta.url);
let cachedTokenizer: InstanceType<typeof TokenizerType> | null = null;

function getTokenizer(): InstanceType<typeof TokenizerType> {
    if (!cachedTokenizer) {
        const mod = lazyRequire("ai-" + "tokenizer") as { default?: typeof TokenizerType; Tokenizer?: typeof TokenizerType };
        const TokenizerCls = mod.default ?? mod.Tokenizer!;
        const claudeEncoding = lazyRequire("ai-tokenizer/encoding/" + "claude") as typeof ClaudeEncodingType;
        cachedTokenizer = new TokenizerCls(claudeEncoding);
    }
    return cachedTokenizer;
}

export function estimateTokens(text: string): number {
    if (!text) return 0;
    return getTokenizer().encode(text, "all").length;
}

ai-tokenizer ships CJS entries (exports.default / exports.Tokenizer + named exports for encoding/claude), so require works. import type keeps type-safety with zero runtime cost.

Measured impact

• Bundle: eager chunk 4.63MB → 2.14MB
• module import (warm): 365–616ms → 164–272ms (−55%)
• Total Pi startup (warm): ~1.2–1.5s → ~0.65–1.0s (−40%)
• Behavior unchanged: same Tokenizer + claude encoding; first estimateTokens() pays a one-time ~100ms load (could be pre-warmed via the existing scheduleAfterBootQuiet hook)

Follow-ups

• typebox (~464KB, tools schemas) and subc-client (~231KB, RPC) are also in the eager path — worth evaluating after this lands.

[onedotmint]

Thanks for the detailed investigation and proposed approach here — it made the root cause and intended fix very clear.

I implemented the lazy-loading approach and validated it against the current master. The focused tests, typechecks, and builds all pass, and the Pi eager shared chunk dropped from ~4.63 MB to ~2.14 MB, with ai-tokenizer no longer appearing in the eager bundle metafile.

I've opened a PR with the implementation and validation details: #288

Thanks again for doing the initial investigation and benchmarking!

[alfonso-magic-context]

Fixed on master. ai-tokenizer (and its claude encoding table) now lazy-loads on first token estimation instead of at module import; measured effect on the Pi eager shared chunk: 4.63 MB → 2.14 MB. Combined with the v0.33.0 QuickJS fix, the eager cold-start surface is now ~a quarter of its original size. Thanks for the precise follow-up measurement and the pointer at the right mechanism — ships in the next release.