Testing and Debugging

Testing and Debugging

How the converter is tested, how to run the suite, the manifest corpus behind it, and the field protocol for debugging a converted extension running live in Safari.

Read this first, the suite is a local artifact, not part of the public repo. In .gitignore, under the comment "Tests, per-extension reports, and project notes, kept local, not pushed":

test/
reports/
CLAUDE.md

Only one test file is committed to the public repo: test/content-blocker.test.js: it predates the ignore rule and was never un-tracked. git ls-files test/ returns exactly that one path. The working tree holds ~25 test files and a reports/extension-tests/ folder, but those are local-only developer artifacts. This page documents the real local suite so a maintainer with the working tree can run and extend it, it does not describe what a fresh git clone gets. A clean checkout ships the single content-blocker test.

---

Running the tests

npm test          # build, then run every test in test/ under node:test
npm run typecheck # tsc --noEmit — types only, no build

npm test is defined as:

"test": "npm run build && node --test test/*.test.js"

Two things follow from that definition:

1. The build is mandatory and runs first. Every test imports from dist/, not from src/, e.g. content-blocker.test.js does import { scanExtension } from "../dist/analyze/analyze.js" and verify.test.js does import { parseEnabled } from "../dist/build/verify.js". Running the tests without a fresh build tests stale compiled output (or fails outright if dist/ is missing). npm test guards against this by always building; if you invoke the runner directly, build yourself first. This is the same "always build before running dist/ or the tests" rule that governs the CLI (see Architecture and Build and Install).
2. test/*.test.js is a shell glob, not a runner feature. Your shell expands it to the file list that node --test receives. The runner is Node's built-in node:test, no Jest, Mocha, Vitest, or other dependency. Tests use node:test's test() and node:assert/strict; several build a temp dir under os.tmpdir(), write a manifest.json, run the real pipeline function against it, and clean up in a finally.

npm run typecheck runs tsc --noEmit and is the fast "did I break the types" check, it does not emit dist/ and does not run any test.

---

The test suite (local-only)

The working tree carries ~25 test/*.test.js files, ~127 test(...) cases in total. Every one except content-blocker.test.js is gitignored and lives only on the maintainer's machine. Grouped by the subsystem each exercises:

Runtime shim, messaging, actions, injection

File	Committed?	Covers
action-click-bridge.test.js	local	chrome.action/browserAction click → popup/handler bridge
action-hotkey.test.js	local	keyboard-command → action dispatch
action-popup-autowire.test.js	local	auto-wiring a declared default_popup when the extension expects a click event
sendmessage-relay-scope.test.js	local	runtime.sendMessage relay targeting / scope correctness
sender-url-normalize.test.js	local	normalizing sender.url / sender.origin (the UUID-case-mismatch class of bug)
ancestor-origins-guard.test.js	local	guarding location.ancestorOrigins access on Safari
page-world-inject.test.js	local	MAIN-world vs isolated-world script injection
color-scheme-inject.test.js	local	prefers-color-scheme injection into pages
self-page-url-rewrite.test.js	local	rewriting the extension's own page URLs
offscreen-response.test.js	local	offscreen-document request/response emulation
native-messaging-handler.test.js	local	native-messaging host handshake / message handling
xhr-proxy.test.js	local	the XHR/fetch proxying path
mv2-background-shim.test.js	local	MV2 background-page shim behavior
webpack-chunk-preregister.test.js	local	pre-registering webpack chunk loaders so bundles resolve on Safari

Manifest transform & analysis

File	Committed?	Covers
content-blocker.test.js	public (only one)	scanExtension firing the content-blocker issue on uBO-shaped MV2 blocking-webRequest manifests
menus-ftp-pattern.test.js	local	menus/contextMenus URL-pattern handling incl. ftp:
command-shortcut-cap.test.js	local	capping/validating declared command shortcuts
dynamic-url-strip.test.js	local	stripping unsupported dynamic-URL constructs
scheme-rewrite.test.js	local	URL-scheme rewriting in the manifest
empty-content-scripts.test.js	local	handling an empty/absent content_scripts array
derive-app-name.test.js	local	deriving the Safari app name from the extension

Build / packaging / CDP

File	Committed?	Covers
verify.test.js	local	parseEnabled, reading pluginkit -mv election columns (+/-/!/=/?/blank) to tell if the .appex is enabled
cdp-manifest.test.js	local	Chrome DevTools Protocol manifest path
cdp-debugger.test.js	local	CDP debugger integration (largest suite, ~19 cases)

Parsing / codegen primitives

File	Committed?	Covers
balanced-paren.test.js	local	balanced-parenthesis / brace scanning used by the code transforms

The single public test, content-blocker.test.js, is representative of the house style: node:test + node:assert/strict, a temp-dir helper that writes a manifest and runs the real dist/ function, no mocking framework. Add a test for any non-trivial logic change, the suite is the regression net for the 50+ real extensions the project converts.

---

The corpus

The suite is backed by a manifest corpus: a large set of real Chrome extensions the conversion logic is stress-tested against:

npm run corpus   # build, then stress-test conversion against the local manifest corpus

Historically defined as npm run build && node scripts/corpus.mjs. Both the corpus and its inputs are local-only: the real extensions live in a test extensions/ directory that is gitignored ("Local test-extension corpus (large binaries; not source)"), and the scripts/ directory that held corpus.mjs was later removed from the repo (commit 7f971ad "chore: remove scripts/ and docs/ dirs"). So npm run corpus is a maintainer-machine workflow, not something a clone can run. The test/ suite is the durable, checked-in-locally regression net distilled from what the corpus runs surfaced.

Where the tests came from. The suite and CI gate were added in commit 0dd9519 "test: add test suite + CI gate; fix bugs found by corpus run", the tests exist to lock in bugs the corpus run had just exposed. Coverage was then grown file by file:

• 6080770, defaultBundleId / Safari-registration bundle-id path
• 78976ab, applyDnr error paths + Safari rule-limit warnings
• 5beb49b, pin deriveChromeId against Chrome's extension-id algorithm
• 748a9eb, extract.ts: CRX3 parsing, folder unwrap, zip-slip guard, error paths
• 2cc6c34, scanExtension: API/URL/file/icon/i18n detections + no-false-positive guard
• 1156251, analyzeManifest: CSP remote-script + MV3 host-misplacement detection
• 7e8477d, applyDnr: modifyHeaders stripping, missing-file notes, keyless-id fallback
• b8d8c31, icon PNG synthesis (was untested)

How the bugs were found, the multi-agent bug-hunt. The project's QA method is a repeated multi-agent audit: many agents read the codebase and cross-reference it against the real-world corpus, and each pass lands as a batch fix. The commit log is full of them, fix: 28 bugs from multi-agent bug hunt (shim semantics, manifest, OAuth, analyze), fix: 13 bugs from multi-agent bug hunt (iOS build, crashes, OAuth, codegen), fix: 22 issues from multi-agent bug hunt (runtime, CLI, analyze accuracy), fix: 8 more shim runtime gaps found via real-world corpus cross-reference, and several deep multi-agent audit rounds. That process is what produced both the bug fixes and the regression tests that pin them.

---

Live-Safari debug protocol

When a converted extension misbehaves in Safari, the failure is usually a Safari platform quirk the shim hasn't handled, not a plain logic bug, and Safari's error surface is misleading. Follow this protocol (it is the authoritative process from the project's historical CLAUDE.md):

1. Do not guess from pasted console errors. They are frequently stale or symptoms, not causes. Treat any console dump as a lead to verify, never as the diagnosis. Diagnose from live evidence instead.

2. Drive diagnostics from the background-page console. The background-page console is reliable; the popup/popover console relay is flaky and will mislead you. Open Safari → Develop → Web Extension Background Pages and work from there.

3. Use __C2S_DEBUG__ to capture structured state. The shim has a __C2S_DEBUG__ flag at the top of src/runtime/safari-compat-shim.js (var __C2S_DEBUG__ = false;), and diagnostic traces are gated behind it, the tracer reads DBG = (typeof __C2S_DEBUG__ !== "undefined") && __C2S_DEBUG__ and stays silent unless it's flipped on. For a diagnostic build, flip it on and have the shim write structured state to chrome.storage.local, then read that back from the background-page console. Flip __C2S_DEBUG__ OFF and strip the temporary traces before committing. See Runtime Shim for the flag and the traced sections.

4. Trace the failing flow link by link, not the end symptom. Chain it out: outcome → did the request reach the background port → did the background post a reply → is the privileged flag set → what are the exact string values being compared. Each link narrows the cause to a single fact instead of a vague symptom.

5. Reproduce the suspected Safari behavior in a Node vm context and prove the fix locally first. Safari's native namespaces (chrome/browser and members like chrome.scripting) are frozen / immutable / exotic in ways a plain object mock will not reveal. Recreate the real conditions in a vm context using Object.freeze, accessor/getter descriptors, live globals, and case-mismatched origins, and confirm the fix there before you reinstall. This is exactly what several shim tests do.

6. Reinstall cleanly, then verify the new shim actually shipped. A stale install is a common false "the fix didn't work."

   node dist/cli.js --uninstall <App>
   rm -rf <App>_Safari      # stale output dir → ENOTEMPTY on the next build
   node dist/cli.js <input> --install --force --clean

   Then quit and reopen Safari before retesting. Confirm the freshly built shim reached the installed bundle by grepping a marker in the appex's copy, grep <marker> …/Contents/Resources/safari-compat-shim.js. See CLI Reference for --uninstall / --install / --force / --clean (and --doctor for environment checks) and Build and Install for the reinstall/verify flow.

7. Distinguish shim-fixable from platform limits. Some failures are Safari or extension-internal, not converter bugs: WebAuthn / passkey RPID mismatches, WASM-SDK chunk loaders, and webfont CSP refusals. Document them and move on; don't chase them as if the shim could fix them.

8. After the session, write it up. Record a per-extension report and keep the README.md status table current (see below).

---

Per-extension reports (local-only)

After a debug session, the outcome is captured as a per-extension report under reports/extension-tests/, following a problem → process → solution → what's left structure. Like test/, reports/ is gitignored: these are local QA notes, not shipped to the public repo.

They were introduced in commit 7bd822d "docs: add per-extension Safari test reports + live-testing debug protocol", which added reports for real extensions, uBlock-Origin.md, Bitwarden.md, Grammarly.md, and an index README.md (e.g. "uBlock (working), Bitwarden (UI works; WASM SDK …)"). They record which extensions convert cleanly, which have platform-limited pieces, and what remains open.

---

See also

• Runtime Shim, safari-compat-shim.js, the __C2S_DEBUG__ flag, and the traced sections
• CLI Reference, --uninstall, --install, --force, --clean, --doctor
• Build and Install, the reinstall cycle and verifying the shim reached the installed .appex
• Architecture, the pipeline and the build-before-dist/ rule