Input Handling

Input Handling

Before viaduct can transform a manifest or emit a Safari Web Extension, it has to get the extension onto disk and reduce it to a clean tree rooted at manifest.json. That is the job of the four modules under src/input/:

Stage	File	Entry point	Job
Download	input/download.ts	downloadExtension()	Turn a Chrome Web Store URL or a direct .crx/.zip link into a local archive file.
Extract	input/extract.ts	extractExtension()	Unpack a .zip/.crx/.xpi (or pass a directory through), find the real extension root, recover the Chrome id from the CRX.
Stage	input/stage.ts	stageExtension()	Copy the tree into a clean staged_extension/, dropping dev cruft while preserving every manifest-declared runtime asset.
Icons	input/icons.ts	synthesizePlaceholderIcons()	Synthesize placeholder PNGs when the extension ships no usable icons.

The pipeline that wires these together (input → stage → transform → build) is the Conversion Pipeline; how the manifest is rewritten afterward is Manifest Transform.

This tree hardening is a repeated security theme, untrusted archives, symlinks, path traversal, injected names. Each guard is called out in its own callout below; they are a deliberate selling point, not incidental (7395bec, 0ed107a).

---

Download (input/download.ts)

downloadExtension(url, scratchDir) accepts two URL shapes and returns the local path to the saved archive.

Exported functions: isUrl, extractStoreId, crxEndpoint, downloadExtension.

Accepted URL forms

1. Chrome Web Store detail URL: extractStoreId() parses the URL and, if the host is chromewebstore.google.com or the legacy chrome.google.com (/webstore/detail/…), pulls out the 32-char extension id. The id alphabet is a through p (EXT_ID_RE = /^[a-p]{32}$/), Chrome's base-16-ish encoding. The scan walks path segments last-to-first because store URLs put the id in the final segment.

2. Direct .crx / .zip URL: anything isUrl() accepts (http:// or https://) that is not a store URL is fetched verbatim.

downloadExtension() branches on extractStoreId():

• Store id found → download from the CRX endpoint.
• Looks like a store URL but no id found → hard error with the expected …/detail/<name>/<id> shape, rather than blindly fetching an HTML page.
• Otherwise → fetch the URL directly.

How the CRX download URL is built

crxEndpoint(id, prodversion?) constructs Google's clients2 update endpoint, which 302-redirects to the real CRX:

https://clients2.google.com/service/update2/crx
  ?response=redirect&acceptformat=crx2,crx3&prodversion=<CRX_PRODVERSION>
  &x=<urlencoded "id=<id>&installsource=ondemand&uc">

The endpoint gates on prodversion: if the extension's minimum_chrome_version is newer than the Chrome version viaduct claims, the answer is 204 with an empty body, not an error. Chrome majors ship every few weeks, so any hardcoded value eventually starts declining extensions.

storeEndpoints(id) is therefore a list, tried in order:

1. CRX_PRODVERSION, default "138.0", overridable with the VIADUCT_CRX_PRODVERSION env var.
2. CRX_PRODVERSION_MAX ("9999.0"), a version no extension can gate past.

downloadFromStore() walks the list and stops at the first non-empty body, so the second request only happens for an extension the first claim was too old for. Measured against the live endpoint for Enhancer for YouTube (ponfpcnoihfmfllpaingbgckeeldkhle): prodversion=120.0.0.0 returns 204 and 0 bytes, 130.0 and up return 200 and 498,490 bytes. This is issue #16, which surfaced through the Viaduct app's own Swift downloader, still pinned at 120.0.0.0, and the same retry now lives there in ChromeStore.downloadCRX.

Error handling (a hardened fetcher, not a bare fetch)

The internal httpGet() is deliberately careful because the endpoints are untrusted:

• Size cap: 100 MB (MAX_BYTES); the connection is destroyed the moment the running total exceeds it.
• Two independent timeouts: a 60 s socket idle timeout and a 5-minute wall-clock deadline. The comment records why they differ: equal values made the 100 MB cap unreachable below ~1.7 MB/s, and a slow-drip server that sends a few bytes every <TIMEOUT_MS would never trip the idle timeout alone. The wall-clock deadline is a single shared deadline threaded through the redirect recursion (httpGet takes a deadlineAt arg and arms each hop's timer to the remaining time), so it bounds the whole fetch — chain included. It used to be re-created per hop, so a 5-hop redirect chain could run ~6× the advertised limit (92fa2ee).
• Settle-exactly-once guard so a size-cap reject and a buffered end can't both fire.
• Protocol-matched agent: isUrl() permits http://, and https.get throws synchronously on it; the fetcher picks the matching node:http/node:https agent.

Security callout, redirect scheme pinning. Redirects are followed at most MAX_REDIRECTS = 5 times, and only https: redirect targets are honored. An untrusted endpoint cannot bounce the fetch down to http:// (or another scheme) to reach an internal host. Missing Location, too many hops, and unparseable Location values all fail loudly.

Two response-body pitfalls are turned into actionable errors:

• Empty / declined package: a body under 4 bytes cannot be a CRX or a ZIP, since both are longer than their magic header. For a store download that means the endpoint declined, so downloadFromStore() moves on to the next prodversion and only fails once the whole list is exhausted, saying the listing is likely delisted, region-blocked, or restricted to managed installs and to pass a local .crx instead. For a direct URL there is nothing to retry, so it fails immediately. Without this guard inferKind() would fall back to the URL suffix, write a 0-byte file, and the user would hit a bad-magic error deep inside the extractor.
• HTML error/captcha page: inferKind() sniffs CRX (Cr24) / ZIP (PK\x03\x04) magic; if neither matches and the response looks like HTML (content-type text/html or a leading <), it says so rather than trusting a misleading .crx/.zip suffix.

The saved file is written to scratchDir/download.<kind> and its path returned for extractExtension().

---

Extract (input/extract.ts)

extractExtension(inputPath, scratchDir) unpacks an archive, or passes a directory straight through, and returns the extension root (the folder that actually holds manifest.json).

Exported functions: cleanExtendedAttributes, extractExtension.

Magic-byte type detection

The archive kind is decided from magic bytes, not the file extension (sniffArchiveKind()):

Signature	Bytes	Kind
Cr24	43 72 32 34 (ASCII)	CRX
PK\x03\x04	50 4B 03 04	ZIP

This is why a CRX renamed to .zip (or a zip-based .xpi) still works, the content is authoritative. The file suffix is only a fallback when the bytes are inconclusive (.crx→crx, .zip/.xpi→zip).

CRX2 vs CRX3 header parsing

crxToZip() strips the CRX container to recover the embedded ZIP. After validating the Cr24 magic it reads the version at offset 4:

• CRX2: layout magic(4) version(4) pubKeyLen(4) sigLen(4); the ZIP starts at 16 + pubKeyLen + sigLen.
• CRX3: layout magic(4) version(4) headerLen(4); the ZIP starts at 12 + headerLen. The CRX3 header is a protobuf (CrxFileHeader).

Every length is bounds-checked against the buffer, and a header that fills the entire file (leaving a 0-byte payload) is rejected with a CRX-specific error instead of failing later with a vague unzip error.

Public-key extraction → Chrome-id derivation. CWS manifests ship without a key field, so the Chrome id can't be derived from the staged manifest alone, but the CRX carries the key. crxPublicKeyBase64() recovers it:

• CRX2: the public key is stored inline right after the header.
• CRX3: a minimal protobuf field-walker reads the CrxFileHeader. The signed_header_data (field 10000) holds the authoritative 16-byte crx_id (field 1); the header also carries several AsymmetricKeyProofs (fields 2/3). The code returns the proof key whose SHA-256, truncated to 16 bytes, matches the crx_id: there can be several proofs and only one matches.

After extraction, if the input was a CRX, extractExtension() injects the recovered key into the staged manifest.json (only when key is absent), so the rest of the pipeline (chrome-extension origin, runtime.id spoof, see Manifest Transform) has a real id. The manifest is read for this with the lenient parseJsonc (same as loadManifest), not strict JSON.parse — a Chrome bundle whose manifest.json carries a BOM, // comment, or trailing comma used to throw here, silently dropping the recovered key and losing the derived Chrome id downstream (92fa2ee). This id-from-CRX-key feature landed in 34d4724; its CRX3-parsing tests in 748a9eb.

Folder unwrap (single-manifest-subdir descent)

Archives rarely place manifest.json at the top level. resolveExtensionRoot() resolves the true root in three steps:

1. Root already has manifest.json → use it.
2. Exactly one subdirectory (ignoring __MACOSX/.DS_Store) that contains a manifest → descend into it. This handles the common "zip wraps everything in one top-level folder" case.
3. Otherwise, look for subdirs that carry a manifest. If exactly one does, descend into it, this covers a repo-style layout where the extension sits alongside non-extension siblings (host binaries, docs, build scripts), e.g. the Chrome native-messaging sample's extension/ + host/ + README.md.

Multi-variant packages are deliberately NOT unwrapped. If two or more subdirs each carry a manifest (a monorepo of extensions, or a sample bundle of variants), the resolver stays put rather than silently picking the wrong one, the caller then errors, and you point the CLI at the specific subdir. Introduced in e64f335.

Extraction mechanics & cleanup

unzipTo() uses macOS-native ditto -x -k --sequesterRsrc (which preserves structure and sequesters resource forks), falling back to unzip -q -o. Extended attributes that break code signing are stripped with xattr -cr via cleanExtendedAttributes(). ditto's --sequesterRsrc plus the __MACOSX/.DS_Store filtering keep resource-fork noise out of the tree. This zip/crx/dir extraction with xattr cleanup is the original 3846882.

xattr -cr is only run on extracted archives and the staged copy, never on a user's own source directory, since it is irreversible and that path is also hit by read-only --analyze.

Security callout, zip-slip / path-traversal guard. After every extraction, assertNoPathEscape() realpath-walks the whole tree. Both extractors already sanitize literal ../ entries, so this walk's real target is the symlink vector: any symbolic link is rejected outright: an in-tree symlink whose realpath happens to stay inside would pass a naive location check yet still let later cpSync staging follow it out of the package. Any entry whose realpath escapes the extraction root also fails, and the extraction dir is torn down on rejection. Added in 748a9eb; hardened further in 7395bec.

---

Stage (input/stage.ts)

stageExtension(sourceDir, stageDir, keep) recreates stageDir from scratch and copies the extension into it, dropping development cruft and store metadata while preserving runtime assets. It returns the list of manifest-referenced assets that could not be staged (so the caller can warn).

Exported functions: stageExtension, plus the source-rewrite passes that run over the staged tree, walkScripts, stripDanglingSourcemaps, inlineImmutableEnums, rewriteChromeSchemeLiterals, rewriteRuntimeIdUrlMatchers, guardAncestorOriginsAccess, rewriteSelfPageExtensionUrls, idempotentContentScriptGlobals. (The rewrite passes are Safari-compat surgery on the bundle source; they are documented under Manifest Transform / Safari Quirks. This page covers only the copy/keep behavior.)

idempotentContentScriptGlobals is the one rewrite pass that takes the transformed manifest (not just the staged dir): it only touches files an isolated-world content_scripts entry references, demoting their top-level const/let to var so a second Safari evaluation of the group into a shared world can't throw "Can't create duplicate variable". It normalizes each js path the same way collectReferencedPaths does (^\.?/, \→/). See Safari Quirks E4.

What gets dropped vs kept

Dropped as cruft (shouldExclude()):

• Exact names (EXCLUDE_EXACT): .DS_Store, __MACOSX, .git/.gitignore/.github, .svn, node_modules, _metadata (CWS signing metadata), package.json, package-lock.json, yarn.lock, pnpm-lock.yaml, tsconfig.json.
• Suffixes (EXCLUDE_SUFFIX): .map, .ts, .tsx, .md, .log.
• Doc files (EXCLUDE_DOC_RE): README/CHANGELOG/LICENSE, optionally with a doc extension (.txt, .md, .markdown, .rst, .adoc).
• Tool dotfiles (EXCLUDE_DOTFILE): names starting .eslint, .prettier.

Exclusion is checked against every path segment, not just the basename, so _metadata/junk.js is dropped because its parent is excluded even though its own name is clean.

The doc/lockfile regexes are intentionally not prefix matches: a runtime file that merely shares a doc name, license.js, changelog.html, LICENSE_KEY.js, READMExporter.js, must ship, or it 404s at runtime.

The key correctness behavior, manifest-asset preservation. The keep set is a set of manifest-relative paths (forward-slash) that the manifest declares as runtime assets. Any path in keep is copied even if its name matches an exclusion rule, and an excluded directory is still entered when a kept path lives inside it, otherwise cpSync wouldn't recurse into it and the kept child would be lost. This is why a web_accessible_resources-served LICENSE.txt, a content-script CSS file, or a .map a page actually fetches isn't 404'd in Safari.

The keep set is built by collectReferencedPaths() (manifest/manifest.ts, part of Manifest Transform), which sweeps every asset-bearing manifest slot: content-script js/css, background.scripts/service_worker/page, action/browser_action/page_action popups and icons, top-level icons, options_page/options_ui, devtools_page, chrome_url_overrides, sandbox.pages, declarative_net_request rule resources, side_panel/sidebar_action, storage.managed_schema, and web_accessible_resources in every form (MV2 string[], MV3 [{resources}], and the bare-string variant). Glob entries (*) are skipped and #fragment/?query suffixes are stripped so the concrete on-disk file is preserved.

Symlink handling (clean-copy that never ships a link)

stageExtension is always a clean copy: it never reproduces a symlink into the package. The cpSync filter drops every symlink: cpSync copies links verbatim, so a source link like evil.txt → /etc/hosts would ship a dangling/absolute link that both leaks the build host's layout and 404s in Safari.

Security callout, symlink leak in staging. Dropping all symlinks would also drop a legitimately symlinked runtime asset (common with pnpm/monorepo asset linking). So for kept paths only, a second pass dereferences the link and copies the actual target bytes, but only when the realpath stays inside the source tree; a target that escapes, vanishes, or isn't a regular file is recorded as dropped and reported, never followed. This fixed the symlink-leak-in-staging in b0c81a1.

The containment check compares against the realpath'd root (realRoot), not resolve(). On macOS, scratch dirs live under /tmp (a symlink to /private/tmp); without realpathing the root, an in-tree target would realpath to /private/…, fail a naive startsWith(root), and get silently dropped. That "stage symlink base" fix is 23aef5e. After copying, the staged tree gets its own xattr -cr pass.

--ci / --clean (symlink-vs-copy lives downstream, not here)

A common point of confusion: stageExtension does not implement a symlink-for-dev mode: it always clean-copies. The --ci flag controls how resources are wired into the Xcode project during packaging, not how this module stages:

• --ci sets copyResources: true, which runPackager() forwards as --copy-resources, the packager clean-copies resources into the project (CI/TestFlight-safe).
• The default (--ci off) symlinks resources into the project so live source edits are picked up without re-staging. The staged dir is written to a persistent location (not scratch) precisely so those dev-mode symlinks survive scratch cleanup.
• --clean wipes the output directory before staging, to drop stale leftovers.

See Build and Install for the packaging step and CLI Reference for the flags.

---

Icons (input/icons.ts)

synthesizePlaceholderIcons(stageDir, manifest, appName) is the only export. It synthesizes a placeholder icon set when the extension ships no usable icons, and returns the sizes it wrote (empty array = no-op).

When and why

Safari cannot render non-PNG toolbar icons, and an extension with no icons shows a blank toolbar glyph and gets rejected on App Store upload. The function is a no-op when the manifest already declares any usable icon:

• returns early if manifest.icons has any entry, and
• respects MV3 action-only icons: it also bails if action/browser_action's default_icon is a non-empty string or object.

Only when both checks find nothing does it generate a placeholder set. (The dedup guard that prevents double-writing when an action icon already exists is e9150d8; the synthesis path itself gained test coverage in b8d8c31.)

How a PNG is generated

The module writes real PNGs with no image dependency, just node:zlib:

1. Sizes 48, 128, 256, 512 (SIZES).
2. colorFor(appName) hashes the app name to a hue and converts HSL→RGB at mid-brightness, so the color is deterministic and stable across runs and the glyph stays visible on both light and dark chrome.
3. solidPng(size, r, g, b) hand-assembles a truecolor PNG: the 8-byte signature, an IHDR chunk (bit depth 8, color type 2), an IDAT chunk of the deflate-compressed raw scanlines (each row is a 0 filter byte + size RGB pixels), and IEND. Chunk CRCs are computed with a locally built CRC-32 table.
4. Each file is written as icon-<size>.png, and the manifest is mutated: manifest.icons is set, and if an action/browser_action exists its default_icon is wired to the same set.

Because it mutates manifest.icons, this runs as part of building the transformed manifest, see Manifest Transform and, for the Safari icon-format constraint, Safari Quirks.

---

Related pages

Conversion Pipeline · Manifest Transform · Build and Install · CLI Reference · Safari Quirks