Testing and Debugging

Testing and Debugging

How the converter is tested, how to run the suite, the manifest corpus behind it, and the field protocol for debugging a converted extension running live in Safari.

Read this first — the suite is a local artifact, not part of the public repo. In .gitignore, under the comment "Tests, per-extension reports, and project notes — kept local, not pushed":

test/
reports/
CLAUDE.md

Only one test file is committed to the public repo: test/content-blocker.test.js — it predates the ignore rule and was never un-tracked. git ls-files test/ returns exactly that one path. The working tree holds ~25 test files and a reports/extension-tests/ folder, but those are local-only developer artifacts. This page documents the real local suite so a maintainer with the working tree can run and extend it — it does not describe what a fresh git clone gets. A clean checkout ships the single content-blocker test.

---

Running the tests

npm test          # build, then run every test in test/ under node:test
npm run typecheck # tsc --noEmit — types only, no build

npm test is defined as:

"test": "npm run build && node --test test/*.test.js"

Two things follow from that definition:

1. The build is mandatory and runs first. Every test imports from dist/, not from src/ — e.g. content-blocker.test.js does import { scanExtension } from "../dist/analyze/analyze.js" and verify.test.js does import { parseEnabled } from "../dist/build/verify.js". Running the tests without a fresh build tests stale compiled output (or fails outright if dist/ is missing). npm test guards against this by always building; if you invoke the runner directly, build yourself first. This is the same "always build before running dist/ or the tests" rule that governs the CLI (see Architecture and Build and Install).
2. test/*.test.js is a shell glob, not a runner feature. Your shell expands it to the file list that node --test receives. The runner is Node's built-in node:test — no Jest, Mocha, Vitest, or other dependency. Tests use node:test's test() and node:assert/strict; several build a temp dir under os.tmpdir(), write a manifest.json, run the real pipeline function against it, and clean up in a finally.

npm run typecheck runs tsc --noEmit and is the fast "did I break the types" check — it does not emit dist/ and does not run any test.

---

The test suite (local-only)

The working tree carries ~25 test/*.test.js files, ~127 test(...) cases in total. Every one except content-blocker.test.js is gitignored and lives only on the maintainer's machine. Grouped by the subsystem each exercises:

Runtime shim — messaging, actions, injection

File	Committed?	Covers
action-click-bridge.test.js	local	chrome.action/browserAction click → popup/handler bridge
action-hotkey.test.js	local	keyboard-command → action dispatch
action-popup-autowire.test.js	local	auto-wiring a declared default_popup when the extension expects a click event
sendmessage-relay-scope.test.js	local	runtime.sendMessage relay targeting / scope correctness
sender-url-normalize.test.js	local	normalizing sender.url / sender.origin (the UUID-case-mismatch class of bug)
ancestor-origins-guard.test.js	local	guarding location.ancestorOrigins access on Safari
page-world-inject.test.js	local	MAIN-world vs isolated-world script injection
color-scheme-inject.test.js	local	prefers-color-scheme injection into pages
self-page-url-rewrite.test.js	local	rewriting the extension's own page URLs
offscreen-response.test.js	local	offscreen-document request/response emulation
native-messaging-handler.test.js	local	native-messaging host handshake / message handling
xhr-proxy.test.js	local	the XHR/fetch proxying path
mv2-background-shim.test.js	local	MV2 background-page shim behavior
webpack-chunk-preregister.test.js	local	pre-registering webpack chunk loaders so bundles resolve on Safari

Manifest transform & analysis

File	Committed?	Covers
content-blocker.test.js	public (only one)	scanExtension firing the content-blocker issue on uBO-shaped MV2 blocking-webRequest manifests
menus-ftp-pattern.test.js	local	menus/contextMenus URL-pattern handling incl. ftp:
command-shortcut-cap.test.js	local	capping/validating declared command shortcuts
dynamic-url-strip.test.js	local	stripping unsupported dynamic-URL constructs
scheme-rewrite.test.js	local	URL-scheme rewriting in the manifest
empty-content-scripts.test.js	local	handling an empty/absent content_scripts array
derive-app-name.test.js	local	deriving the Safari app name from the extension

Build / packaging / CDP

File	Committed?	Covers
verify.test.js	local	parseEnabled — reading pluginkit -mv election columns (+/-/!/=/?/blank) to tell if the .appex is enabled
cdp-manifest.test.js	local	Chrome DevTools Protocol manifest path
cdp-debugger.test.js	local	CDP debugger integration (largest suite, ~19 cases)

Parsing / codegen primitives

File	Committed?	Covers
balanced-paren.test.js	local	balanced-parenthesis / brace scanning used by the code transforms

The single public test, content-blocker.test.js, is representative of the house style: node:test + node:assert/strict, a temp-dir helper that writes a manifest and runs the real dist/ function, no mocking framework. Add a test for any non-trivial logic change — the suite is the regression net for the 50+ real extensions the project converts.

---

The corpus

The suite is backed by a manifest corpus — a large set of real Chrome extensions the conversion logic is stress-tested against:

npm run corpus   # build, then stress-test conversion against the local manifest corpus

Historically defined as npm run build && node scripts/corpus.mjs. Both the corpus and its inputs are local-only: the real extensions live in a test extensions/ directory that is gitignored ("Local test-extension corpus (large binaries; not source)"), and the scripts/ directory that held corpus.mjs was later removed from the repo (commit 7f971ad "chore: remove scripts/ and docs/ dirs"). So npm run corpus is a maintainer-machine workflow, not something a clone can run. The test/ suite is the durable, checked-in-locally regression net distilled from what the corpus runs surfaced.

Where the tests came from. The suite and CI gate were added in commit 0dd9519 "test: add test suite + CI gate; fix bugs found by corpus run" — the tests exist to lock in bugs the corpus run had just exposed. Coverage was then grown file by file:

• 6080770 — defaultBundleId / Safari-registration bundle-id path
• 78976ab — applyDnr error paths + Safari rule-limit warnings
• 5beb49b — pin deriveChromeId against Chrome's extension-id algorithm
• 748a9eb — extract.ts: CRX3 parsing, folder unwrap, zip-slip guard, error paths
• 2cc6c34 — scanExtension: API/URL/file/icon/i18n detections + no-false-positive guard
• 1156251 — analyzeManifest: CSP remote-script + MV3 host-misplacement detection
• 7e8477d — applyDnr: modifyHeaders stripping, missing-file notes, keyless-id fallback
• b8d8c31 — icon PNG synthesis (was untested)

How the bugs were found — the multi-agent bug-hunt. The project's QA method is a repeated multi-agent audit: many agents read the codebase and cross-reference it against the real-world corpus, and each pass lands as a batch fix. The commit log is full of them — fix: 28 bugs from multi-agent bug hunt (shim semantics, manifest, OAuth, analyze), fix: 13 bugs from multi-agent bug hunt (iOS build, crashes, OAuth, codegen), fix: 22 issues from multi-agent bug hunt (runtime, CLI, analyze accuracy), fix: 8 more shim runtime gaps found via real-world corpus cross-reference, and several deep multi-agent audit rounds. That process is what produced both the bug fixes and the regression tests that pin them.

---

Live-Safari debug protocol

When a converted extension misbehaves in Safari, the failure is usually a Safari platform quirk the shim hasn't handled, not a plain logic bug — and Safari's error surface is misleading. Follow this protocol (it is the authoritative process from the project's historical CLAUDE.md):

1. Do not guess from pasted console errors. They are frequently stale or symptoms, not causes. Treat any console dump as a lead to verify, never as the diagnosis. Diagnose from live evidence instead.

2. Drive diagnostics from the background-page console. The background-page console is reliable; the popup/popover console relay is flaky and will mislead you. Open Safari → Develop → Web Extension Background Pages and work from there.

3. Use __C2S_DEBUG__ to capture structured state. The shim has a __C2S_DEBUG__ flag at the top of src/runtime/safari-compat-shim.js (var __C2S_DEBUG__ = false;), and diagnostic traces are gated behind it — the tracer reads DBG = (typeof __C2S_DEBUG__ !== "undefined") && __C2S_DEBUG__ and stays silent unless it's flipped on. For a diagnostic build, flip it on and have the shim write structured state to chrome.storage.local, then read that back from the background-page console. Flip __C2S_DEBUG__ OFF and strip the temporary traces before committing. See Runtime Shim for the flag and the traced sections.

4. Trace the failing flow link by link — not the end symptom. Chain it out: outcome → did the request reach the background port → did the background post a reply → is the privileged flag set → what are the exact string values being compared. Each link narrows the cause to a single fact instead of a vague symptom.

5. Reproduce the suspected Safari behavior in a Node vm context and prove the fix locally first. Safari's native namespaces (chrome/browser and members like chrome.scripting) are frozen / immutable / exotic in ways a plain object mock will not reveal. Recreate the real conditions in a vm context using Object.freeze, accessor/getter descriptors, live globals, and case-mismatched origins, and confirm the fix there before you reinstall. This is exactly what several shim tests do.

6. Reinstall cleanly, then verify the new shim actually shipped. A stale install is a common false "the fix didn't work."

   node dist/cli.js --uninstall <App>
   rm -rf <App>_Safari      # stale output dir → ENOTEMPTY on the next build
   node dist/cli.js <input> --install --force --clean

   Then quit and reopen Safari before retesting. Confirm the freshly built shim reached the installed bundle by grepping a marker in the appex's copy — grep <marker> …/Contents/Resources/safari-compat-shim.js. See CLI Reference for --uninstall / --install / --force / --clean (and --doctor for environment checks) and Build and Install for the reinstall/verify flow.

7. Distinguish shim-fixable from platform limits. Some failures are Safari or extension-internal, not converter bugs — WebAuthn / passkey RPID mismatches, WASM-SDK chunk loaders, and webfont CSP refusals. Document them and move on; don't chase them as if the shim could fix them.

8. After the session, write it up. Record a per-extension report and keep the README.md status table current (see below).

---

Per-extension reports (local-only)

After a debug session, the outcome is captured as a per-extension report under reports/extension-tests/, following a problem → process → solution → what's left structure. Like test/, reports/ is gitignored — these are local QA notes, not shipped to the public repo.

They were introduced in commit 7bd822d "docs: add per-extension Safari test reports + live-testing debug protocol", which added reports for real extensions — uBlock-Origin.md, Bitwarden.md, Grammarly.md, and an index README.md (e.g. "uBlock (working), Bitwarden (UI works; WASM SDK …)"). They record which extensions convert cleanly, which have platform-limited pieces, and what remains open.

---

See also

• Runtime Shim — safari-compat-shim.js, the __C2S_DEBUG__ flag, and the traced sections
• CLI Reference — --uninstall, --install, --force, --clean, --doctor
• Build and Install — the reinstall cycle and verifying the shim reached the installed .appex
• Architecture — the pipeline and the build-before-dist/ rule