Testing and Debugging

Testing and Debugging

How the converter is tested, how to run the suite, the manifest corpus behind it, and the field protocol for debugging a converted extension running live in Safari.

Read this first, the suite is a local artifact, not part of the public repo. In .gitignore, under the comment "Tests, per-extension reports, and project notes, kept local, not pushed":

test/
reports/
CLAUDE.md

Only one test file is committed to the public repo: test/content-blocker.test.js: it predates the ignore rule and was never un-tracked. git ls-files test/ returns exactly that one path. The working tree holds ~25 test files and a reports/extension-tests/ folder, but those are local-only developer artifacts. This page documents the real local suite so a maintainer with the working tree can run and extend it, it does not describe what a fresh git clone gets. A clean checkout ships the single content-blocker test.

---

Running the tests

npm test          # build, then run every test in test/ under node:test
npm run typecheck # tsc --noEmit — types only, no build

npm test is defined as:

"test": "npm run build && node --test test/*.test.js"

Two things follow from that definition:

1. The build is mandatory and runs first. Every test imports from dist/, not from src/, e.g. content-blocker.test.js does import { scanExtension } from "../dist/analyze/analyze.js" and verify.test.js does import { parseEnabled } from "../dist/build/verify.js". Running the tests without a fresh build tests stale compiled output (or fails outright if dist/ is missing). npm test guards against this by always building; if you invoke the runner directly, build yourself first. This is the same "always build before running dist/ or the tests" rule that governs the CLI (see Architecture and Build and Install).
2. test/*.test.js is a shell glob, not a runner feature. Your shell expands it to the file list that node --test receives. The runner is Node's built-in node:test, no Jest, Mocha, Vitest, or other dependency. Tests use node:test's test() and node:assert/strict; several build a temp dir under os.tmpdir(), write a manifest.json, run the real pipeline function against it, and clean up in a finally.

npm run typecheck runs tsc --noEmit and is the fast "did I break the types" check, it does not emit dist/ and does not run any test.

---

The test suite (local-only)

The working tree carries ~25 test/*.test.js files, ~127 test(...) cases in total. Every one except content-blocker.test.js is gitignored and lives only on the maintainer's machine. Grouped by the subsystem each exercises:

Runtime shim, messaging, actions, injection

File	Committed?	Covers
action-click-bridge.test.js	local	chrome.action/browserAction click → popup/handler bridge
action-hotkey.test.js	local	keyboard-command → action dispatch
action-popup-autowire.test.js	local	auto-wiring a declared default_popup when the extension expects a click event
sendmessage-relay-scope.test.js	local	runtime.sendMessage relay targeting / scope correctness
sender-url-normalize.test.js	local	normalizing sender.url / sender.origin (the UUID-case-mismatch class of bug)
ancestor-origins-guard.test.js	local	guarding location.ancestorOrigins access on Safari
page-world-inject.test.js	local	MAIN-world vs isolated-world script injection
color-scheme-inject.test.js	local	prefers-color-scheme injection into pages
self-page-url-rewrite.test.js	local	rewriting the extension's own page URLs
offscreen-response.test.js	local	offscreen-document request/response emulation
native-messaging-handler.test.js	local	native-messaging host handshake / message handling
xhr-proxy.test.js	local	the XHR/fetch proxying path
mv2-background-shim.test.js	local	MV2 background-page shim behavior
webpack-chunk-preregister.test.js	local	pre-registering webpack chunk loaders so bundles resolve on Safari

Manifest transform & analysis

File	Committed?	Covers
content-blocker.test.js	public (only one)	scanExtension firing the content-blocker issue on uBO-shaped MV2 blocking-webRequest manifests
menus-ftp-pattern.test.js	local	menus/contextMenus URL-pattern handling incl. ftp:
command-shortcut-cap.test.js	local	capping/validating declared command shortcuts
dynamic-url-strip.test.js	local	stripping unsupported dynamic-URL constructs
scheme-rewrite.test.js	local	URL-scheme rewriting in the manifest
empty-content-scripts.test.js	local	handling an empty/absent content_scripts array
derive-app-name.test.js	local	deriving the Safari app name from the extension

Build / packaging / CDP

File	Committed?	Covers
verify.test.js	local	parseEnabled, reading pluginkit -mv election columns (+/-/!/=/?/blank) to tell if the .appex is enabled
detect-xcode-team.test.js	local	detectXcodeTeam against stub defaults / security / openssl and a temp HOME: every preference key/domain and profile layout, plus the rule that a profile alone never nominates a team (issue #15)
xcodebuild-signing-failure.test.js	local	xcodebuildDiagnostics over the real stdout/stderr split, and the ad-hoc retry: a detected team that cannot sign falls back, a team named with --team does not
cdp-manifest.test.js	local	Chrome DevTools Protocol manifest path
cdp-debugger.test.js	local	CDP debugger integration (largest suite, ~19 cases)

Parsing / codegen primitives

File	Committed?	Covers
balanced-paren.test.js	local	balanced-parenthesis / brace scanning used by the code transforms

The single public test, content-blocker.test.js, is representative of the house style: node:test + node:assert/strict, a temp-dir helper that writes a manifest and runs the real dist/ function, no mocking framework. Add a test for any non-trivial logic change, the suite is the regression net for the 50+ real extensions the project converts.

---

The corpus

The suite is backed by a manifest corpus: a large set of real Chrome extensions the conversion logic is stress-tested against:

npm run corpus   # build, then stress-test conversion against the local manifest corpus

Historically defined as npm run build && node scripts/corpus.mjs. Both the corpus and its inputs are local-only: the real extensions live in a test extensions/ directory that is gitignored ("Local test-extension corpus (large binaries; not source)"), and the scripts/ directory that held corpus.mjs was later removed from the repo (commit 7f971ad "chore: remove scripts/ and docs/ dirs"). So npm run corpus is a maintainer-machine workflow, not something a clone can run. The test/ suite is the durable, checked-in-locally regression net distilled from what the corpus runs surfaced.

Where the tests came from. The suite and CI gate were added in commit 0dd9519 "test: add test suite + CI gate; fix bugs found by corpus run", the tests exist to lock in bugs the corpus run had just exposed. Coverage was then grown file by file:

• 6080770, defaultBundleId / Safari-registration bundle-id path
• 78976ab, applyDnr error paths + Safari rule-limit warnings
• 5beb49b, pin deriveChromeId against Chrome's extension-id algorithm
• 748a9eb, extract.ts: CRX3 parsing, folder unwrap, zip-slip guard, error paths
• 2cc6c34, scanExtension: API/URL/file/icon/i18n detections + no-false-positive guard
• 1156251, analyzeManifest: CSP remote-script + MV3 host-misplacement detection
• 7e8477d, applyDnr: modifyHeaders stripping, missing-file notes, keyless-id fallback
• b8d8c31, icon PNG synthesis (was untested)

How the bugs were found, the multi-agent bug-hunt. The project's QA method is a repeated multi-agent audit: many agents read the codebase and cross-reference it against the real-world corpus, and each pass lands as a batch fix. The commit log is full of them, fix: 28 bugs from multi-agent bug hunt (shim semantics, manifest, OAuth, analyze), fix: 13 bugs from multi-agent bug hunt (iOS build, crashes, OAuth, codegen), fix: 22 issues from multi-agent bug hunt (runtime, CLI, analyze accuracy), fix: 8 more shim runtime gaps found via real-world corpus cross-reference, and several deep multi-agent audit rounds. That process is what produced both the bug fixes and the regression tests that pin them.

---

Live-Safari debug protocol

When a converted extension misbehaves in Safari, the failure is usually a Safari platform quirk the shim hasn't handled, not a plain logic bug, and Safari's error surface is misleading. Follow this protocol (it is the authoritative process from the project's historical CLAUDE.md):

1. Do not guess from pasted console errors. They are frequently stale or symptoms, not causes. Treat any console dump as a lead to verify, never as the diagnosis. Diagnose from live evidence instead.

2. Drive diagnostics from the background-page console. The background-page console is reliable; the popup/popover console relay is flaky and will mislead you. Open Safari → Develop → Web Extension Background Pages and work from there.

3. Convert with --debug to capture structured state. The shim's diagnostic traces are gated behind __C2S_DEBUG__ (top of src/runtime/safari-compat-shim.js, var __C2S_DEBUG__ = false;; the tracer reads DBG = (typeof __C2S_DEBUG__ !== "undefined") && __C2S_DEBUG__ and stays silent otherwise). viaduct <input> --debug flips that gate at staging time and splices in a persistent ring-buffer logger, so every gated trace reaches both the Inspector console and storage.local under __viaduct_debug_log__ — see The --debug build below for reading it back. No hand-edited source, no custom build; a plain re-convert without the flag ships a shim with no trace or ring-buffer path at all. See Runtime Shim for the gate and the traced sections.

4. Trace the failing flow link by link, not the end symptom. Chain it out: outcome → did the request reach the background port → did the background post a reply → is the privileged flag set → what are the exact string values being compared. Each link narrows the cause to a single fact instead of a vague symptom.

5. Reproduce the suspected Safari behavior in a Node vm context and prove the fix locally first. Safari's native namespaces (chrome/browser and members like chrome.scripting) are frozen / immutable / exotic in ways a plain object mock will not reveal. Recreate the real conditions in a vm context using Object.freeze, accessor/getter descriptors, live globals, and case-mismatched origins, and confirm the fix there before you reinstall. This is exactly what several shim tests do.

6. Reinstall cleanly, then verify the new shim actually shipped. A stale install is a common false "the fix didn't work."

   node dist/cli.js --uninstall <App>
   rm -rf <App>_Safari      # stale output dir → ENOTEMPTY on the next build
   node dist/cli.js <input> --install --force --clean

   Then quit and reopen Safari before retesting. Confirm the freshly built shim reached the installed bundle by grepping a marker in the appex's copy, grep <marker> …/Contents/Resources/safari-compat-shim.js. See CLI Reference for --uninstall / --install / --force / --clean (and --doctor for environment checks) and Build and Install for the reinstall/verify flow.

7. Distinguish shim-fixable from platform limits. Some failures are Safari or extension-internal, not converter bugs: WebAuthn / passkey RPID mismatches, WASM-SDK chunk loaders, and webfont CSP refusals. Document them and move on; don't chase them as if the shim could fix them.

8. After the session, write it up. Record a per-extension report and keep the README.md status table current (see below).

The --debug build and its persistent log

viaduct <input> --debug stages the shim with tracing on and a ring buffer that survives the very teardowns the console cannot: entries are { t, ctx, msg } (epoch-ms timestamp, a background/content/page context label, the trace line), batched behind a ~1 s timer so logging never hammers storage.local, and capped at the last 2000 under the reserved key __viaduct_debug_log__ (same naming convention as __viaduct_bookmarks__). The writer swallows every failure — a broken storage backend costs log entries, never the extension. The debug emit also sets the __C2S_DEBUG global, so the OAuth-bridge templates' call-time diagnostics (below) light up in every context the shim reaches without a console visit. (world:"MAIN" page scripts don't load the shim, so page-bridge.js still needs the manual window.__C2S_DEBUG = true.)

Two ways to read the log back:

From outside Safari — the same on-disk store the auth log lives in (see below):

viaduct --logs <app-or-bundle-id>     # e.g. viaduct --logs "TWP Translate Web Pages"

prints one line per entry (ISO time [context] message), matching <name> case- and punctuation-insensitively against the storage directories under ~/Library/Containers/com.apple.Safari/Data/Library/WebKit/WebExtensions/Default/. It snapshots the SQLite WAL trio to a scratch dir and queries read-only, so Safari can stay open.

From any extension console you can reach (including the popup's, when the background console is the thing you can't get at):

chrome.storage.local.get("__viaduct_debug_log__", r => {
  const log = r.__viaduct_debug_log__ || [];
  console.log(log.map(e => `${new Date(e.t).toISOString()} [${e.ctx}] ${e.msg}`).join("\n"));
});

A --debug build is a diagnostic artifact: it logs request URLs and trace state a shipped extension must not. Convert again without the flag before distributing anything.

When you can't get at the background console

Step 2 assumes Develop → Web Extension Background Content opens the page you want. Sometimes it doesn't, and a tester who is not sitting next to you will often end up pasting the popup's console instead without either of you noticing: the giveaway is filenames, popup bundles in the output when you asked for the background. Run location.href in any inspector to confirm which context you are in before trusting a word of it.

The way through is to stop needing the console. Prepend a probe to the extension's own background bundle in a copy of the unpacked input, buffer what it sees in an array, and write that array to chrome.storage.local after every entry. Convert that copy instead of the original. Then read the log from whatever console you can reach, including the popup's:

chrome.storage.local.get("__viaductBootLog", r => {
  const log = r.__viaductBootLog || [];
  const t0 = log.length ? log[0].t : 0;
  console.log(log.map(e => `+${e.t - t0}ms ${e.kind}: ${e.info}`).join("\n"));
});

Timestamps relative to the first entry are what make it worth the trouble, because they place the failure against the background's own boot. Honey's diagnosis turned on three readings of exactly that shape: the bundle ran to completion in 56 ms and threw nothing, so a suspected boot abort was dead; the popup's messages arrived 1.7 s later and were all rejected by one listener returning false, which named the guard; and the tab events the extension was waiting on were registered at 28 ms and never fired once, which named the quirk. Worth wrapping in the probe, all cheap: window.onerror and unhandledrejection, every runtime.onMessage listener the bundle registers along with what it returns and answers, and the addListener calls themselves so a silent event is distinguishable from a missing one.

This is __C2S_DEBUG__'s trick pointed at the extension rather than the shim, and the same rule applies: the probe lives in a scratch copy of the input and never in a commit.

The OAuth bridge reports its own hops

The page-to-background bridge is the one flow where every part can be healthy and the user still sees nothing happen, so its templates carry named diagnostics instead of failing quietly. All three read __C2S_DEBUG at call time, so self.__C2S_DEBUG = true in a background console that is already open, or window.__C2S_DEBUG = true on the page, takes effect on the next attempt. No rebuild, no flag flipped in a source file.

What each line tells you:

[bridge-cs] background answered no wake ping over either transport means the relay tried sendMessage three times and the storage mailbox three times and got nothing back. Delivery is no longer the suspect: the background is not running, or it threw while loading. Open its console. If Develop has no entry for the extension at all, that is the answer.

[bridge-cs] the page has no chrome.runtime even after injecting page-bridge.js means the page world never got the bridge. Safari runs world:"MAIN" content scripts from 18.4 only, and the <script> fallback was refused, almost always by the page's CSP. Confirm from the page console with chrome.runtime.id, which should return the extension's Chrome id.

[idpoly] bridge msg '<type>' accepted by N onMessageExternal listener(s) but none answered means the message arrived and the extension's own gate refused it. The line carries the origin the listeners were handed, which is usually the whole diagnosis: compare it against whatever the bundle's allow-list expects.

[idpoly] bridge msg but NO captured onMessageExternal listeners means the bundle never registered a handler, so look for a throw during its module evaluation.

Silence is also a reading. A tab that is going away suppresses all of the above on purpose, because Safari's "Tab not found." rejection means there is no page left to report to (see Safari Quirks E10).

Every auth flow leaves a record you can read afterwards

The failures in the silent re-auth path only happen once Safari has torn the background page down, and a Web Inspector attached to that page keeps it alive — so the console cannot observe them without preventing them. launchWebAuthFlow therefore appends one entry per attempt to __c2sAuthLog in storage.local (last 20), from every exit including a timeout:

chrome.storage.local.get("__c2sAuthLog", r => console.log(JSON.stringify(r.__c2sAuthLog, null, 1)));

Each entry carries silent, ok, ms, the reason it ended, and navs: the navigations the auth tab reported, as scheme + host + path. No query, no fragment, so the code and the token are never in it. That single field is usually the diagnosis — "navs": [] on a flow that timed out is Safari having reported no navigation at all (Safari Quirks E13a), where ["poll https://…/oauth/authorize", "poll https://<id>.chromiumapp.org/"] is a healthy silent refresh.

You can also read this from outside the browser entirely, which is the only way to watch a cold start: the store is a SQLite file at ~/Library/Containers/com.apple.Safari/Data/Library/WebKit/WebExtensions/Default/<bundle-id> (<team>)/LocalStorage.db, table extension_storage. Safari's own ~/Library/Safari/History.db is a useful second witness — an oauth/authorize visit carrying prompt=none is a silent attempt, and its absence after a relaunch is how you confirm no re-auth was needed.

Verifying a storage quirk without the extension that hit it

A quirk fix that lives in the shim can usually be proven against a purpose-built probe extension instead of the extension that surfaced it — which matters when the original needs a paid account to reach the broken state. The 1.11.7 session fixes (E14/E14a) were verified this way: a four-file MV3 probe whose background reads storage.session on every boot, writes a token on the first one, and opens its own extension page in a background tab so a non-background context reads the same key — the exact shape of a panel reading state the background owns. Both contexts POST what they saw to http://127.0.0.1:8765 with a text/plain body; a permissive CORS response is enough, no host_permissions needed, so the probe adds no permission prompts. Quit Safari, relaunch, and the second boot's report tells you whether the store survived.

Two pieces make the loop fully headless, useful when the machine is remote or the screen is locked so the Settings UI cannot be clicked:

• pluginkit -e use -i <bundle-id>.Extension records the election, but Safari's own on/off state lives in ~/Library/Containers/com.apple.Safari/Data/Library/Safari/WebExtensions/Extensions.plist, keyed "<bundle-id>.Extension (<team>)". With Safari quit, set Enabled to true there (plistlib; the file is binary and holds dates PlistBuddy chokes on), relaunch Safari, and the extension runs — permissions the manifest declares are already listed as granted.
• Sign with --team auto. An unsigned install needs the session-scoped unsigned toggle, which resets on exactly the Safari relaunch the test needs to perform.

---

When each attempt costs someone a reinstall

The protocol above is about finding the cause. This one is about not spending another person's afternoon doing it. Every live iteration costs a human a convert, an install, a Safari relaunch, a permission grant and a console copy-paste, so a guess shipped to them is far more expensive than the same guess tested locally.

These rules come from a Tampermonkey session that took roughly ten reinstall rounds to land fixes that one local test would have caught. They are worth following even when the next attempt feels obviously right.

Write the end-to-end test before the first reinstall. If a feature spans two halves, a background that produces and a page that consumes, test both halves in one test: publish from the real shim into a fake storage, then run the real generated injector against it in a page sandbox and assert the payload actually executed. Build it from data shapes captured live (a getScripts() dump, a real record from chrome.storage.local), not from what the API docs say the shape should be. The Tampermonkey injector was fixed four times against invented shapes before a test using the real ones settled it.

Never write catch {} in the path being debugged. A swallowed rejection is indistinguishable from a code path that never ran, and the difference is exactly what is being investigated. Two silent catches in the user-script wiring turned "the listener is attached to a dead stub" into three rounds of guessing.

Never gate a diagnostic on the condition under test. A trace that only logs when the registry is non-empty cannot tell an empty registry from a listener that never fired. Log the entry to the path, then the decision, then the outcome.

Make diagnostics visible from where the tester is standing. A content script runs in the isolated world, so asking someone to read window.__yourFlag from the page console returns undefined whether it ran or not. Log to the console instead, prefixed, and say which console to read. Safari's console also evaluates a multi-line paste as one program and prints only the last value, so hand over one snippet that logs a labeled object rather than several bare expressions.

Confirm host access before believing any negative result. Safari defaults broad host access to Ask, and withholds content scripts and tab/navigation events for sites that were never granted. Every --install reinstalls the app and resets the grant. A "nothing happened" from a build whose access was not re-granted is not evidence of anything, and a platform conclusion drawn from one is worse than no conclusion. See Safari Quirks.

Verify the artifact before theorizing about behavior. Before reasoning about why an injected file misbehaves, confirm it is in staged_extension/, declared in the written manifest, and present in the installed bundle. This costs one shell command and invalidates whole branches of speculation.

Do not build viaduct-internal channels on runtime.sendMessage. It broadcasts to every listener and the first sendResponse wins, so the converted extension's own background handler can consume the message and answer with its own shape. Tampermonkey did exactly that, on all twelve retries, and the reply came back as undefined rather than an error, which is indistinguishable from nobody listening. Use storage.local for anything the shim needs to hand to its own injected code: no race, and it survives the background being torn down.

A backfilled API is not distinguishable from a real one by truthiness. The shim fills missing namespaces with inert events so a module-eval read cannot throw, so chrome.webNavigation.onCommitted is always present and addListener always succeeds. Anything that depends on an event actually firing must check __c2sInert (see event() in the shim). Reporting a signal as wired because the object exists produces a log line that confidently lies.

Label evidence quality when recording a finding. "Safari delivers no navigation events" written into a commit message from tests run without host access granted becomes a fact the next person builds on. Say what was observed, under what conditions, and what remains unverified. A wrong platform claim in the wiki is more expensive than a gap.

---

Per-extension reports (local-only)

After a debug session, the outcome is captured as a per-extension report under reports/extension-tests/, following a problem → process → solution → what's left structure. Like test/, reports/ is gitignored: these are local QA notes, not shipped to the public repo.

They were introduced in commit 7bd822d "docs: add per-extension Safari test reports + live-testing debug protocol", which added reports for real extensions, uBlock-Origin.md, Bitwarden.md, Grammarly.md, and an index README.md (e.g. "uBlock (working), Bitwarden (UI works; WASM SDK …)"). They record which extensions convert cleanly, which have platform-limited pieces, and what remains open.

---

See also

• Runtime Shim, safari-compat-shim.js, the __C2S_DEBUG__ gate, the ring-buffer logger, and the traced sections
• CLI Reference, --debug, --logs, --uninstall, --install, --force, --clean, --doctor
• Build and Install, the reinstall cycle and verifying the shim reached the installed .appex
• Architecture, the pipeline and the build-before-dist/ rule