v3.2.0

Release Notes

v3.2.0 — 2026-09-26

The resilience release: discovery becomes dual-homed, t.co shortlinks are
accepted, and the project ships as a PyPI package with a portfolio site.
The reconstruction truth layer is untouched — replying_to_status remains
the only chain signal, and no existing field changed meaning.

Added

• Second walker slot (dual-homed discovery). resolve_thread_ids now
  tries UnrollNow (primary) and falls back to ThreadReaderApp
  (threadreaderapp.com/thread/<id>) when the primary fails network-side
  or yields no candidates. The root guarantee, cap-at-50, dedup and
  first-seen ordering contracts are shared by both slots via
  _extract_candidate_ids. Politeness: at most one request per slot per
  run (≤2 walk requests total, vs 1 before).
• thread.walker_slot envelope field (unrollnow | threadreaderapp |
  none) — names the slot that served the walk; additive and optional in
  the schema (schema_version stays 3.0, old manifests stay valid).
  degraded_to_root_only is now defined as walker_slot == "none" (the
  previous definition keyed on error codes, which would have mis-flagged
  fallback successes).
• t.co shortlink inputs. expand_tco resolves one hop (HTTP redirect,
  or — for clients where t.co serves a 200 interstitial — the embedded
  <noscript> meta-refresh / location.replace target parsed from the
  body, never executed), validates the destination is a status URL on a
  supported host, and the CLI re-normalizes it. Non-status destinations are
  rejected with the real resolved host named. Verified live.
• PyPI packaging. pyproject.toml (PEP 621, setuptools, zero deps,
  console script xthread-agent, module invocation python -m xthread_agent); xthread_agent/__init__.py is a byte-identical copy of
  xthread-agent.py, regenerated by scripts/sync_package.py and guarded
  by tests/test_package_sync.py (drift = build failure). Build validated
  locally: sdist + wheel, twine check PASSED, clean-venv smoke OK.
• Release pipeline (.github/workflows/release.yml): tag v* →
  offline suite → sync/version checks → build + twine check → clean-venv
  smoke → GitHub Release artifacts → PyPI via Trusted Publishing (OIDC).
• Portfolio site (site/, deployed by .github/workflows/pages.yml):
  a Material 3 single-page home for the project — dark/light M3 color
  system, animated terminal, architecture stepper, MCP section, FAQ. Static,
  no build step, reduced-motion aware.
• Repo hygiene: GitHub issue templates, SECURITY.md (scope notes for
  the fetcher threat model), PUBLISHING.md checklist.

Changed

• resolve_thread_ids returns (candidates, slot_name) — internal only;
  the CLI/MCP envelopes are the contract and stay compatible (one additive
  optional field).
• Primary-walker failures now surface even when the fallback succeeds
  (errors[] records it, status becomes partial) — honest provenance
  over fake cleanliness.
• Docs: README (walker slots, t.co, install section, badge set, 135-test
  suite), endpoint matrix (ThreadReaderApp row), PROJECT_CONTEXT
  (§3 tiers, §10 roadmap, single-file note), agent.md (t.co input).

Fixed

• tests/test_walker.py: two test methods (test_cap_never_drops_root,
  test_empty_page_records_walker_empty) were accidentally placed after
  the if __name__ == "__main__" block and silently never ran; they are
  back in the class and running.

Total suite: 110 → 135 tests.

v3.1.0 — 2026-09-24

The integration release: an MCP wrapper for MCP-compatible agent hosts and
GitHub Actions CI. The core pipeline is untouched — no pipeline code changed,
no schema changed (schema_version stays 3.0), no CLI contract changed.

Added

• mcp_server.py — MCP wrapper (stdlib-only). Exposes the harvester as
  MCP tools over the standard stdio transport (newline-delimited JSON-RPC
  2.0; protocol versions 2024-11-05 / 2025-03-26 / 2025-06-18 — the server
  echoes a supported client version, else advertises its latest):
  extract_thread (full harvest), lookup_status (metadata-only),
  read_manifest (returns an existing manifest; refuses any other
  filename), get_schema (envelope JSON Schema). The wrapper shells out to
  the CLI as a subprocess (argument list, never a shell) with hard timeouts
  (120s lookups, 900s extracts, XTHREAD_MCP_EXTRACT_TIMEOUT to override)
  and implements no pipeline logic of its own — the CLI contract remains the
  single source of truth. Tool results distinguish invocation failures
  (isError: true) from honest negative results (isError: false with
  envelope.status == "empty").
• tests/test_mcp.py — 21 offline MCP tests: protocol handlers
  (initialize version echo/fallback, ping, tools/list shape, notifications
  stay silent, unknown methods), subprocess framing over stdio (parse
  errors, sequential request/response pairing), and tool error paths
  (invalid input fails before any network call, read_manifest filename
  guard, missing files). Total suite: 89 → 110 tests.
• .github/workflows/ci.yml — GitHub Actions CI. Python 3.9–3.13
  matrix on push/PR/manual: byte-compile all sources, validate the bundled
  JSON Schema, run the full offline suite, CLI smoke checks (usage-error
  exit 2, invalid-input E_INVALID_INPUT fail-closed), and an offline MCP
  handshake probe. CI deliberately never probes live endpoints — the
  politeness constraint outranks CI coverage, and determinism is the point
  (see PROJECT_CONTEXT.md §11).

Changed

• README.md: CI badge, MCP server section, docs table entry, testing
  section (root-based unittest discovery invocation, suite size).
• agent.md: "Via MCP" section — how MCP clients run the server and read
  tool results (the four tools, timeout/error semantics).
• PROJECT_CONTEXT.md: single-file constraint now documents the sanctioned
  MCP-wrapper exception; §11 debts updated (CI debt closed; new documented
  residual: MCP hosts must serialize heavy harvests themselves).
• docs/endpoint-matrix.md: all rows re-verified alive from a datacenter
  IP on 2026-09-24 (UTC) — FixTweet, vxtwitter, UnrollNow, twimg CDN, plus
  the fail-closed path for an unavailable root.

v3.0.0 — 2026-09-24

The reliability release: true thread reconstruction, a fallback decoder slot,
hardened data integrity, a machine-readable envelope, and the project's first
test suite (89 tests). Built on a full re-verification of every endpoint from
a datacenter IP.

⚠️ BREAKING — manifest schema

thread_manifest.json changed from a bare tweet array (v2) to an
enveloped document (schema_version: "3.0"): {schema_version, source, request, status, thread, posts[], errors[], metadata}. The v2 array shape is
preserved as the posts array. Formal JSON Schema:
schema/thread-result.schema.json. Agents pinned to v2 must read
.posts[] instead of the top-level array.

Fixed (live-verified defects)

• False thread members (v2 correctness bug): UnrollNow pages embed
  same-author recommendations alongside the conversation; v2 harvested them
  as thread members. v3 reconstructs the true self-reply chain from
  replying_to_status (walking up to the thread start and down through
  replies) and counts excluded recommendations (related_filtered).
• Root dropped for short legacy IDs (v2 correctness bug): the walker
  regex (\d{15,25}) could not see e.g. status/20, silently excluding the
  root from harvesting. v3 guarantees the root is always present.
• 404 retry storm: FixTweet returns real HTTP 404 statuses for
  unavailable tweets; urllib raises HTTPError, which v2 retried 3× with
  backoff before treating it as a filter. v3 treats HTTP 404/451 as instant
  filter signals (both response shapes).
• Non-atomic downloads (v2 integrity bug): a crash mid-write left a
  partial file that skip-if-exists treated as complete forever. v3 streams to
  .part, verifies against Content-Length, and atomically renames.
• write_text without explicit encoding (v2 mojibake risk): manifest now
  written UTF-8, atomically.
• Python 3.9 support was broken in v2 (PEP-604 annotations evaluated
  eagerly); v3 adds from __future__ import annotations — the advertised
  3.9+ is now true.

Added

• Input normalization: proper URL parsing (x.com/twitter.com, mobile./www.,
  /statuses/, /photo/<n>, /video/<n>, query strings, bare IDs) with
  E_INVALID_INPUT rejection for non-status URLs.
• Fallback decoder slot (vxtwitter): used automatically when FixTweet
  fails network-side; payloads normalized to an honest subset, provenance
  recorded per post (extraction_source).
• Enveloped manifest with status (ok/partial/empty), structured
  errors[] (stable codes: E_WALKER_UNAVAILABLE, E_WALKER_EMPTY,
  E_ROOT_UNAVAILABLE, E_DECODE_FAILED, E_DOWNLOAD_FAILED,
  E_MANIFEST_WRITE_FAILED, E_INVALID_INPUT), and extraction statistics.
• Full author mapping (name, id, verified, protected, followers, avatar,
  banner, joined, website, …), post metrics incl. quotes/bookmarks, lang,
  posting client, replying_to(_status), ISO-8601 UTC timestamps.
• Quoted posts recorded one level deep, including the quoted post's
  author, text, timestamps, and media URLs.
• Video formats[] variants (container/bitrate/codec) preserved; m3u8-only
  videos now fall back to the highest-bitrate mp4 variant instead of being
  skipped; non-downloadable videos explained (downloadable: false,
  reason: "hls_only" | "no_mp4_variant").
• Candidate cap (50, root kept), response body caps (20 MB walk / 5 MB
  decode), transfer deadline against slow-drip servers.
• Security hardening: media fetches restricted to https://*.twimg.com;
  remote tweet IDs validated (\d{1,25}) before use in filenames.
• demo.py — minimal end-to-end consumption example.
• tests/ — 89 offline tests (stdlib unittest, synthetic fixtures): URL
  normalization, ...

v3.1.0 — MCP wrapper + GitHub Actions CI

v3.1.0 — 2026-09-24

The integration release: an MCP wrapper for MCP-compatible agent hosts and
GitHub Actions CI. The core pipeline is untouched — no pipeline code changed,
no schema changed (schema_version stays 3.0), no CLI contract changed.

Added

• mcp_server.py — MCP wrapper (stdlib-only). Exposes the harvester as
  MCP tools over the standard stdio transport (newline-delimited JSON-RPC
  2.0; protocol versions 2024-11-05 / 2025-03-26 / 2025-06-18 — the server
  echoes a supported client version, else advertises its latest):
  extract_thread (full harvest), lookup_status (metadata-only),
  read_manifest (returns an existing manifest; refuses any other
  filename), get_schema (envelope JSON Schema). The wrapper shells out to
  the CLI as a subprocess (argument list, never a shell) with hard timeouts
  (120s lookups, 900s extracts, XTHREAD_MCP_EXTRACT_TIMEOUT to override)
  and implements no pipeline logic of its own — the CLI contract remains the
  single source of truth. Tool results distinguish invocation failures
  (isError: true) from honest negative results (isError: false with
  envelope.status == "empty").
• tests/test_mcp.py — 21 offline MCP tests: protocol handlers
  (initialize version echo/fallback, ping, tools/list shape, notifications
  stay silent, unknown methods), subprocess framing over stdio (parse
  errors, sequential request/response pairing), and tool error paths
  (invalid input fails before any network call, read_manifest filename
  guard, missing files). Total suite: 89 → 110 tests.
• .github/workflows/ci.yml — GitHub Actions CI. Python 3.9–3.13
  matrix on push/PR/manual: byte-compile all sources, validate the bundled
  JSON Schema, run the full offline suite, CLI smoke checks (usage-error
  exit 2, invalid-input E_INVALID_INPUT fail-closed), and an offline MCP
  handshake probe. CI deliberately never probes live endpoints — the
  politeness constraint outranks CI coverage, and determinism is the point
  (see PROJECT_CONTEXT.md §11).

Changed

• README.md: CI badge, MCP server section, docs table entry, testing
  section (root-based unittest discovery invocation, suite size).
• agent.md: "Via MCP" section — how MCP clients run the server and read
  tool results (the four tools, timeout/error semantics).
• PROJECT_CONTEXT.md: single-file constraint now documents the sanctioned
  MCP-wrapper exception; §11 debts updated (CI debt closed; new documented
  residual: MCP hosts must serialize heavy harvests themselves).
• docs/endpoint-matrix.md: all rows re-verified alive from a datacenter
  IP on 2026-09-24 (UTC) — FixTweet, vxtwitter, UnrollNow, twimg CDN, plus
  the fail-closed path for an unavailable root.