v2.1.0

Adds local client packages with explicit launch permissions. Client-specific installation
and marketplace acceptance remain separate qualification steps.

• Add a local MCP profile with an explicit readable workspace, typed permission refusals, and process/network denial at launch. The full CLI/MCP remains available separately.
• Add optional exact-origin HTTP retrieval through launch flags, with a separate literal-loopback grant, bounded response size, pinned connection addresses and existing fetch receipts. Refuse redirects, ambient proxies, credentials and process execution.
• Add optional MCPB setup fields for public and local-service origin lists. Empty settings grant no access; malformed settings stop launch before serving tools.
• Add portable source plugins for Claude/Codex-compatible clients and Windows x64 ZIP/MCPB packages containing the runtime, licenses, source hashes and checksums.
• Check actual stdio behavior before packaging, reject untracked or credential-like release payloads, and attach checked client assets to the same product release.
• Keep process-backed intake and persistent-state operations on the full CLI/MCP. Local document reads and explicitly allowed HTTP retrieval are available in the client profile.

v2.0.0

Breaking changes

This release removes public names, so it is a major version. It also changes what video
intake does when YouTube asks for a bot check. What changed, and what to do:

• The stealth extra is gone. pip install 'gather-engine[stealth]' still installs Gather;
  pip warns that the extra does not exist and installs the core. Drop [stealth] from
  requirement files and install commands. gather-engine[all] now installs lxml and
  playwright only.
• The module gather.backends_stealth is gone, and importing it raises ImportError.
  Remove the import. Nothing replaces it.
• The constant gather.backends.CAP_STEALTH is gone, and importing it raises ImportError.
  gather caps no longer lists stealth, even where curl_cffi is installed. Remove any
  reference to it.
• A YouTube bot check ("confirm you're not a bot") is never retried. Builds from main after
  1.9.1 retried it with backoff like HTTP 429; 1.9.1 did not retry it either. Gather records
  the failure with the code bot-check, and the record starts "YouTube asked for a bot
  check, and gather stopped", then gives yt-dlp's own line. gather channel stops the pass
  at the first bot check, whatever --max-throttled says, and exits 1. It settles the entry
  that met the check, so a resumed run does not ask for it again. A bot check while listing
  a tab stops the run before any entry is gathered. What to do: when you choose to try a
  video again, ask for it by name with gather video URL --store DIR and the pass's flags.
  --retries and the backoff flags now cover HTTP 429 and YouTube's session rate limit
  only. gather.ytdlp.THROTTLE_CODES no longer holds bot-check; TERMINAL_CODES does.

Removed: the stealth extra and its backend

• The stealth extra, its curl_cffi dependency, and the stealth capability backend
  (gather.backends_stealth) are removed. That backend impersonated a browser's TLS
  fingerprint to get past bot detection. Gather no longer ships bot-detection evasion of any
  kind, and nothing replaces it. Gather's own HTTP requests go out with Gather's own
  User-Agent.
• pip install 'gather-engine[stealth]' still installs Gather. pip warns
  gather-engine 2.0.0 does not provide the extra 'stealth' and installs the core
  without it. Drop [stealth] from requirement files and install commands.
• gather-engine[all] now installs lxml and playwright only. An upgrade leaves an
  installed curl_cffi in place. Gather's own code no longer uses it, but a yt-dlp in
  the same environment can (see below), so run pip uninstall curl_cffi if nothing else
  needs it.
• Caption downloads no longer pose as a browser. yt-dlp marks every YouTube caption track
  for impersonation and keeps the mark in the info JSON Gather saves and hands back for the
  caption download. Where yt-dlp could import curl_cffi, the track went out with a
  browser's TLS fingerprint and headers. Gather now removes the mark first. A test on the
  saved info runs everywhere, and a test against real yt-dlp, run where yt-dlp is
  installed, checks that the caption request carries yt-dlp's own headers.
• What yt-dlp still decides: it sends its own default headers, including a desktop Chrome
  User-Agent whose version it picks each run, and for some sites other than YouTube its
  extractors ask for impersonation while they extract. Gather passes no flag that asks for
  either, and yt-dlp has no flag that turns off an extractor's request. That request takes
  effect only where yt-dlp can import curl_cffi.
• gather caps no longer lists stealth, including on a machine where curl_cffi is
  installed.
• Code that imports gather.backends_stealth or gather.backends.CAP_STEALTH now raises
  ImportError. Remove those imports.
• Tests fail if a dependency or extra names a known fingerprint-impersonation,
  patched-browser, or challenge-solving package, if a source file imports or looks one up
  or passes yt-dlp --impersonate, or if an installed curl_cffi registers a capability.
  The package list matches by name, so a new tool under another name still needs review.
• The credential strip on a same-host redirect from https to http has its own test for both
  redirect handlers, the one http_get uses and the one the accountable fetch uses. The
  removed stealth tests were the only ones that covered that branch.

Video intake pacing and channel runs

• gather channel URL --store DIR lists a channel's videos, shorts, and streams
  tabs (or one playlist) with --flat-playlist and gathers each entry into the corpus with
  bounded concurrency (default 2) and paced entry starts. A per-pass ledger under
  DIR/intake/ makes the run resumable, and summary-<pass>.json counts entries per tab,
  captions (manual, auto, missing by reason), comments, failures by reason, and retries.
  A run killed mid-write leaves an unfinished last row; the next run drops it and gathers
  that entry again. Any other unreadable row stops the run, before it calls yt-dlp, with
  the line number and exit status 1.
• Separate passes: --no-captions gathers metadata and comments without touching the
  caption endpoint; --captions-only stores only the transcript item.
• A gather run config or an MCP gather.run video job takes
  "captions": "with" | "skip" | "only" (default with). Any other value is a config error,
  raised before any job runs. On MCP the job still needs the video network grant,
  whichever pass it asks for.
• The run summary names its files relative to --store (a --summary outside the store by
  file name) and records the yt-dlp program by file name and a JS runtime without its
  path, so a summary you pass on carries no local path.
• --timeout must be above 0, and --sleep-requests and --sleep-subtitles must be 0 or
  more. Any other value exits 2 before yt-dlp starts.
• Caption intake downloads exactly one track per video, chosen from the info JSON: manual
  first, then the original-language auto-caption (en-orig). The old en.* pattern fetched
  every English variant and could pick a machine translation; a translation-only video is
  now recorded as missing with the reason translation-only.
• HTTP 429 and YouTube's session rate limit are retried with exponential backoff and
  jitter, bounded by attempts and by total wait. Every retry and final failure is logged and
  recorded. A channel run stops starting new entries once an entry spends its whole budget
  still throttled, and records the rest as stopped. A bot check is never retried and stops
  the run at once (see Breaking changes).
• The extraction runs with --ignore-no-formats-error, so a video whose formats are missing
  still yields its metadata and caption tracks. With that flag yt-dlp reports YouTube's
  playability reason as a warning and exits 0. When the extraction lists no formats, Gather
  reads that warning: a session rate limit is retried like an HTTP 429, a bot check ends
  the entry without a retry, and a private, members-only, age-restricted or removed video is
  recorded as failed with that reason and settled. A geo-blocked or upcoming video is
  recorded as failed and tried again on the next run. None of them stores a metadata item
  or a "no captions offered" outcome.
• yt-dlp runs with --js-runtimes node when it can start node (--js-runtime overrides),
  and --sleep-requests / --sleep-subtitles pass through. The check uses the same PATH
  lookup as every child Gather starts, so a node only the working folder holds does not
  count.
• Failure messages report yt-dlp's ERROR lines instead of the first 160 characters of
  stderr, which was often a version warning.
• A timeout, a missing yt-dlp binary, or a refused start is recorded as a failed call
  (timeout, tool-missing, tool-refused), not an exception.
• Every yt-dlp call (tab listing, extraction, caption download) starts the way every
  other tool does, from an absolute path in a private empty folder with an environment
  allowlist, and carries --ignore-config, so no yt-dlp.conf changes what it runs.

v1.9.1

Security: file sources and MCP path arguments refuse network and device paths

• On Windows, opening a path such as \\host\share\doc.md makes the SMB client connect to
  host and sign in as the user, which can send the user's NTLM response to whoever runs host.
  The docs, pdf, ocr and transcribe sources opened any path they were given, and so did
  every path argument on the MCP surface. None of these needed a launch grant. A model connected
  to gather mcp, or text it was asked to read, could name a share in gather.docs, in a
  gather.run job target, config path or store, or in the gather.context,
  gather.federation and gather.pilot path arguments. A store on a share also writes the
  gathered text there. Affected: 1.6.0 through 1.9.0, when gather mcp runs on Windows.
• gather.localpath checks the path text before anything opens it. It refuses text that starts
  with two separators of either kind (UNC, \\?\, \\.\, \\?\UNC\, and mixes such as
  /\host), text that starts with \??\, and any component with a reserved device name (CON,
  PRN, AUX, NUL, CONIN$, CONOUT$, COM1 to COM9 and LPT1 to LPT9, including
  the superscript 1, 2 and 3 forms), with or without an extension, trailing dots or spaces.
  COM0 and LPT0 are ordinary file names on Windows and still read. Windows rules apply on
  Windows and to Windows-style text on every platform. On Windows it then walks the path without
  following links and refuses a symbolic link or junction whose target is a network or device
  path, before anything opens through it. A relative path is refused when the working folder is
  a share.
• Where it applies: the four file sources on every surface, including each entry of a docs
  directory walk; a run config's file-source targets, before any job runs; every MCP path
  argument; a run config's store when the config comes through MCP; and a pilot manifest's
  local targets, fixtures and allowed_local_roots, where Windows rules apply on every platform
  so a manifest means the same on every machine.
• The MCP call returns isError: true with structuredContent {"code": "NON_LOCAL_PATH", "retryable": false, "kind": "network" | "device" | "link", "argument": "<name>", "detail": "<fixed sentence>"}. The CLI prints the reason and exits 1. The gather.docs path and the
  gather.run descriptions now say so.
• Changes you may notice: a \\?\C:\... long path is refused, so give the plain drive path. A
  share is refused as a source from the CLI too; copy the files to a local folder. A drive letter
  mapped to a share looks local to any check on the text, so Gather cannot see it. The CLI's own
  --store, --output and --state paths, and the store in a config run with gather run,
  are unchanged.
• tests/test_nonlocal_paths.py replaces the filesystem and child-process layer with a spy and
  hands each source and MCP tool one path of each class. On 1.9.0, 81 of its 82 tests failed on
  Windows and 76 on Linux, where POSIX-style names such as CON stay ordinary file names by
  design. tests/test_localpath.py covers the classifier, the working-folder case and the link
  walk against a fake tree on every platform, and real junctions and symlinks on Windows.
  tests/test_device_names.py asks Windows which bare names it opens as devices, so the
  reserved-name list cannot drift from the host. It also checks that COM0.md and LPT0.md
  read through the docs source, the MCP gather.docs tool and a pilot manifest.

Security: a PATH entry that reaches the working folder no longer starts a program there

• 1.9.0 skipped . and every other relative PATH entry when it looked up a child program. An
  absolute entry could still reach the working folder: one naming it or a folder below it (a
  project's node_modules/.bin, a venv other than the one running Gather), another spelling of
  it (a trailing separator, .., letter case), the same folder in quotes, or a junction or
  symlink to it. A program planted there under a tool's name (pdftotext, yt-dlp,
  tesseract, whisper, the browser, or a synthesizer or provenance command) then ran in
  place of the real one. The child also got those entries on its PATH, so a tool that starts
  its own helper by name, as yt-dlp starts ffmpeg, could start a copy planted there. On
  Windows a drive-relative command such as C:llm named a file in the working folder too.
  Affected: every release before 1.9.1. Releases before 1.9.0 started tools by bare name
  through the operating system's own search, which follows these entries as well; on Linux,
  1.8.3 ran the plant through each absolute-entry, link and child-lookup route. 1.9.0 closed the
  current-folder search and . entries (see 1.9.0) and left these routes open.
• Gather now vendors safe spawn 1.0.1 (SAFE_SPAWN_VERSION 1.0.1, hash-pinned in
  VENDORED.sha256). A PATH entry that reaches the working folder, by name or by file identity
  after links are resolved, leaves the lookup and the child's PATH. Each kept entry is searched,
  and handed to the child, as its real folder, so a link repointed after the check cannot change
  what starts. On Windows PATH is read as cmd.exe reads it, an entry whose folder name holds ;
  leaves, and a bare command name holding : is refused. On POSIX an entry written with quotes
  or a leading space counts as relative, and a child PATH the filter empties becomes
  /bin:/usr/bin, since an empty PATH means the current folder there. The folder of the Python
  that runs Gather and, on Windows, the Windows, System32 and SysWOW64 folders always stay.
• Where the guard narrows: when Gather runs from a filesystem root, or from the home folder or
  a folder above it, only an entry naming that folder itself leaves, because installed tools
  live below it. A working folder that is the folder of the Python that runs Gather, or on
  Windows the Windows, System32 or SysWOW64 folder, is not guarded, because Gather already
  runs code from there.
• Changes you may notice: a tool found only inside the working folder is no longer found by
  bare name. Give its absolute path in its GATHER_<TOOL> variable, or as the command itself
  for a synthesizer or provenance command. On Windows, a conda environment created inside
  the working folder keeps only its root folder: its Scripts and Library\bin folders leave,
  so set GATHER_PDFTOTEXT, GATHER_TESSERACT, GATHER_YT_DLP or GATHER_WHISPER to the
  tool's full path, or create the environment outside the project. On Windows, a PATH entry
  with an unmatched double quote hides every entry after it, as it does in cmd.exe; remove the
  stray quote or set the GATHER_<TOOL> variable. A drive-relative command is refused as not
  found.
  The child's PATH names real folders, so a version manager's current link reaches it
  resolved. Each tool start now reads every PATH entry and the folders above it. That took
  about 20 to 60 ms per start on Windows, and a median of about 1.4 s under WSL, where PATH
  inherits the Windows folders.
• tests/test_spawn_working_folder.py plants decoys in the working folder and a folder below
  it, puts the real tool later on PATH, and reaches the working folder by each route above. On
  1.9.0, 13 of its 16 tests failed on Windows and 9 on Linux, each because the planted program
  ran. On 1.8.3 on Linux, 7 route tests failed the same way. Its three controls keep a sibling
  folder whose name starts with the working folder's, a folder holding the working folder, and
  an override inside the working folder; they pass on 1.9.0 and 1.9.1. tests/test_vendored.py
  now names a 1.0.0 copy as superseded.
• CI now runs the child-spawn tests on Windows too, where the junction, letter-case and
  drive-relative routes live. Under CI, a Windows run that cannot build a real .exe decoy
  fails these tests instead of skipping them.

v1.9.0

Security: launch-only grants on the MCP surface

• gather.run took a config from tool arguments and ran whatever synthesizer or
  provenance command it named, fetched any network source it listed, and read any
  environment variable named as auth_env and sent its value as a bearer token to the host
  in the config. A model, or text a model was asked to read, could run commands and send a
  secret off the machine with one tool call. gather.pilot had the same exposure through a
  live manifest (auth_env, and a browser option naming any executable). Affected: every
  release with the MCP gather.run or gather.pilot tool, up to and including 1.8.3.
• These now need a grant set at launch. GATHER_ALLOW_EXEC (gather mcp --allow-exec) names
  the commands a config may run. GATHER_ALLOW_NETWORK (--allow-network) names the network
  sources. GATHER_AUTH_ENV_ALLOW (--auth-env NAME@HOST) binds each credential variable to
  the one host it may be sent to, over https only. A pilot manifest's browser option other
  than chromium, or no_sandbox, needs that browser named in GATHER_ALLOW_EXEC.
• Without the grant the call returns isError: true with structuredContent
{"code": "GRANT_REQUIRED", "retryable": false, "setup": "<VARIABLE>", "detail": "<fixed sentence>"} before anything runs, connects or reads a credential. The server reads grants
  once at startup; nothing in a config, manifest or tool call widens them. A pilot refresh
  checks the grants on the same read of the stored manifest it captures from.
• Tool descriptions changed: gather.run and gather.pilot now say which inputs need a launch
  grant, so a host sees why a call returns GRANT_REQUIRED.
• Breaking for MCP hosts that relied on the old behavior: add the grant to the server's launch
  configuration. The CLI and the Python API run the operator's own config and are unchanged.

Security: child programs start from an absolute path in a private folder

• pdftotext, yt-dlp, tesseract, whisper and the headless browser started by bare name
  from the server's working folder. On Windows a same-named .exe in that folder ran in place
  of the real tool, and on any platform a . entry on PATH did the same. yt-dlp also read a
  yt-dlp.conf from that folder, and a config can carry --exec. The synthesizer and
  provenance commands had the same lookup. Affected: every release up to and including 1.8.3.
• Every child now starts through gather.spawn, which calls the vendored safe spawn helper
  (SAFE_SPAWN_VERSION 1.0.0, hash-pinned in VENDORED.sha256). The program resolves to an
  absolute path (GATHER_<TOOL> overrides win; relative and empty PATH entries never count),
  runs in a new private empty folder, and sees an environment allowlist instead of Gather's
  whole environment. yt-dlp gets --ignore-config. On Windows the child also gets
  NoDefaultCurrentDirectoryInExePath=1, and a batch-file target refuses cmd.exe
  metacharacters. Output stays bytes, so receipts hash exactly what the tool wrote.
• Changes you may notice: a command given as a relative path is refused; a synthesizer that
  reads its API key from the environment needs GATHER_CHILD_ENV=KEY_NAME; yt-dlp no longer
  reads your user yt-dlp.conf; a python -m provenance command must be installed, not only
  present in the working folder.
• tests/test_spawn_children.py plants a decoy named like each child (a real .exe on
  Windows) in the caller's folder with . on PATH, a fake API key in the environment and a
  yt-dlp.conf. 18 of its 19 first tests failed on the unfixed code. With yt-dlp installed,
  a real-tool test shows a planted yt-dlp.conf taking effect without the fix and having no
  effect through VideoSource.

Release workflow

• The workflow grants nothing by default. build reads the repository, publish holds only
  id-token: write for trusted publishing, and a new github-release job holds only
  contents: write. Checkouts drop the token after cloning.
• The PyPI upload uses skip-existing: true, so a re-run after a partial upload completes.
• The build checks the tag against the package version and the vendored helper in the wheel
  against VENDORED.sha256, then writes SHA256SUMS.txt for the wheel and the sdist. The
  github-release job verifies those sums and creates the GitHub Release with the wheel, the
  sdist and SHA256SUMS.txt attached, or refreshes its files on a re-run. Before this, the
  release and its checksum file were made by hand.

gather-engine 1.8.3

Catalog line breaks

• Readable context (inspect_corpus, select_context and their MCP tools) now splits
  catalog.jsonl on LF, CRLF and CR only, the rule Corpus.rows reads by. It used
  str.splitlines, which also breaks on U+2028, U+2029 and U+0085. The catalog writer leaves
  those characters unescaped inside JSON strings, so one title such as a misdecoded
  Windows-1252 ellipsis made the context surface refuse a corpus that verify accepts.
• A regression test stores titles carrying each of those characters, rewrites the catalog
  with LF, CRLF and lone-CR row terminators, and checks that Corpus.rows and
  inspect_corpus read the same rows. The store newline test now also checks that the
  object bytes on disk hash to the catalog sha256.

Presentation parity

• README now exposes the current source version, operator commands, and the
  boundary between exact source-byte receipts and optional source adapters.

Install: pip install gather-engine==1.8.3

gather-engine 1.8.2

What's Changed

• Clarify Gather MCP scope filtering by @HarperZ9 in #27
• Release gather-engine 1.8.2 by @HarperZ9 in #28

Full Changelog: v1.8.1...v1.8.2

Gather 1.8.1: stable descriptor admission

Gather 1.8.1 corrects confined corpus reads on WSL Windows-drive 9p/v9fs mounts. Reads now refuse unsupported opened root, descendant-directory and catalog/body descriptors with UNSAFE_PATH, before reading content through unstable authority. Linux classification uses the opened descriptor's procfs mount ID, avoiding native statfs structure assumptions. Missing Linux mount information also refuses the read.

Native Windows and native WSL filesystem descriptor handoff remain available. Other POSIX platforms retain existing no-follow/type checks without a mount-detection claim. CLI and MCP still accept directory strings; retained descriptors remain a Python-only same-process interface.

Validation includes the full Windows suite (671 passed, 12 skipped), focused WSL controls (44 passed, 8 skipped), isolated Windows CLI/MCP/descriptor and tamper checks, native WSL reads, and actual WSL 9p rename/replacement refusal. Nested-directory and leaf refusal controls were instrumented. Live Linux execution covered WSL x86_64.

The release workflow passed. The attached wheel and sdist are the same bytes published to PyPI. Downloaded artifacts were checked directly against the release commit: all 64 wheel Python modules and 133 tracked sdist files matched exactly. Checksums are attached.

Install or upgrade with python -m pip install --upgrade gather-engine==1.8.1.

Gather 1.7.1: Corpus newline integrity

Gather 1.7.1

Gather 1.7.1 is a patch release for corpus newline integrity and readable-context provenance.

It resolves the inherited CR/LF and mixed-newline corpus integrity mismatch documented in 1.7.0 by writing new corpus objects as exact UTF-8 bytes and sealing a versioned storage witness into new catalog rows. Readable context now keeps the exact source-text identity separate from the LF-normalized readable view: sha256, verified_sha256, and source_sha256 bind the exact source text, while view_sha256 and view_codec describe the normalized view used for excerpt display and Python character offsets.

This patch also fixes a writer admission edge case found during independent review: adding a receipt now refuses a corrupt preexisting object at the content-addressed path before appending to the catalog. Valid legacy objects are still reused without rewriting when Gather can reconstruct exactly one matching source text from exact UTF-8 bytes or the old Windows LF-to-CRLF expansion.

Validation for this candidate used the full Gather pytest suite, ruff, mypy, source gather status/doctor, no-index wheel installs on Windows and WSL, CLI/MCP context selection, and tamper refusal checks.

Boundaries: legacy reconstruction verifies source text for an existing receipt, not historical raw object-byte integrity. Storage-witness stripping or codec changes are downgrade-detectable only when a consumer pins or independently verifies the prior corpus digest with --expect-digest / expected_corpus_digest. Readable context is acquired source context; it does not prove source truth, claim support, completeness, downstream model use, caller workspace-parent resolution, or absence of sensitive material in selected source text or URLs.

Gather 1.7.0

Gather 1.7.0

Gather 1.7.0 adds readable corpus context selection. Users can inspect bounded, re-hashed source/comment excerpts from a stored corpus and export selected private context payloads through the CLI, Python API, or MCP gather.context tool.

The new payload records row refs, source refs, full-body hashes, selected ranges, selected text, omissions, and a deterministic selection_digest. Bodies are decoded as UTF-8, CRLF/CR line endings normalize to LF, and selection ranges are Python character offsets over normalized text. verified_sha256 is the full normalized body hash, not a selected-slice hash.

This release also tightens corpus body reads around tamper, missing, unsafe path, oversized body, and read-budget cases; refreshes docs and examples; and keeps pilot fixture evidence portable with LF checkout rules and regenerated sample receipts.

Known inherited limitation: certain literal carriage-return/newline forms in corpus source strings, including CRLF/lone-CR/mixed-newline cases, can fail corpus integrity verification and be refused. This behavior is inherited from 1.6.1 and is not evidence of external tampering by itself. Preserve original corpora for diagnosis; a codec/provenance fix should be handled as follow-up work.

Validation for this candidate was rebuilt from commit d477b4ad5ebed6805ec88bc45eede54cf6f8160b using LF git blob authority. The archive matched tracked blobs byte-for-byte, package checks passed, the clean no-index wheel install exercised CLI and MCP context selection, and tamper selection was refused.

Boundary: readable context is acquired source context. It does not prove source truth, claim support, completeness, downstream model use, or absence of sensitive material in selected source text or URLs.

v1.6.1

What's Changed

• Style: spectrum banner + feature-first README header by @HarperZ9 in #9
• Release v1.6.1 by @HarperZ9 in #10

Full Changelog: v1.6.0...v1.6.1