Skip to content

v3.2.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Sep 17:14
· 3 commits to main since this release
v3.2.0
e59b5d5

Release Notes

v3.2.0 — 2026-09-26

The resilience release: discovery becomes dual-homed, t.co shortlinks are
accepted, and the project ships as a PyPI package with a portfolio site.
The reconstruction truth layer is untouched — replying_to_status remains
the only chain signal, and no existing field changed meaning.

Added

  • Second walker slot (dual-homed discovery). resolve_thread_ids now
    tries UnrollNow (primary) and falls back to ThreadReaderApp
    (threadreaderapp.com/thread/<id>) when the primary fails network-side
    or yields no candidates. The root guarantee, cap-at-50, dedup and
    first-seen ordering contracts are shared by both slots via
    _extract_candidate_ids. Politeness: at most one request per slot per
    run (≤2 walk requests total, vs 1 before).
  • thread.walker_slot envelope field (unrollnow | threadreaderapp |
    none) — names the slot that served the walk; additive and optional in
    the schema (schema_version stays 3.0, old manifests stay valid).
    degraded_to_root_only is now defined as walker_slot == "none" (the
    previous definition keyed on error codes, which would have mis-flagged
    fallback successes).
  • t.co shortlink inputs. expand_tco resolves one hop (HTTP redirect,
    or — for clients where t.co serves a 200 interstitial — the embedded
    <noscript> meta-refresh / location.replace target parsed from the
    body, never executed), validates the destination is a status URL on a
    supported host, and the CLI re-normalizes it. Non-status destinations are
    rejected with the real resolved host named. Verified live.
  • PyPI packaging. pyproject.toml (PEP 621, setuptools, zero deps,
    console script xthread-agent, module invocation python -m xthread_agent); xthread_agent/__init__.py is a byte-identical copy of
    xthread-agent.py, regenerated by scripts/sync_package.py and guarded
    by tests/test_package_sync.py (drift = build failure). Build validated
    locally: sdist + wheel, twine check PASSED, clean-venv smoke OK.
  • Release pipeline (.github/workflows/release.yml): tag v* →
    offline suite → sync/version checks → build + twine check → clean-venv
    smoke → GitHub Release artifacts → PyPI via Trusted Publishing (OIDC).
  • Portfolio site (site/, deployed by .github/workflows/pages.yml):
    a Material 3 single-page home for the project — dark/light M3 color
    system, animated terminal, architecture stepper, MCP section, FAQ. Static,
    no build step, reduced-motion aware.
  • Repo hygiene: GitHub issue templates, SECURITY.md (scope notes for
    the fetcher threat model), PUBLISHING.md checklist.

Changed

  • resolve_thread_ids returns (candidates, slot_name) — internal only;
    the CLI/MCP envelopes are the contract and stay compatible (one additive
    optional field).
  • Primary-walker failures now surface even when the fallback succeeds
    (errors[] records it, status becomes partial) — honest provenance
    over fake cleanliness.
  • Docs: README (walker slots, t.co, install section, badge set, 135-test
    suite), endpoint matrix (ThreadReaderApp row), PROJECT_CONTEXT
    (§3 tiers, §10 roadmap, single-file note), agent.md (t.co input).

Fixed

  • tests/test_walker.py: two test methods (test_cap_never_drops_root,
    test_empty_page_records_walker_empty) were accidentally placed after
    the if __name__ == "__main__" block and silently never ran; they are
    back in the class and running.

Total suite: 110 → 135 tests.

v3.1.0 — 2026-09-24

The integration release: an MCP wrapper for MCP-compatible agent hosts and
GitHub Actions CI. The core pipeline is untouched — no pipeline code changed,
no schema changed (schema_version stays 3.0), no CLI contract changed.

Added

  • mcp_server.py — MCP wrapper (stdlib-only). Exposes the harvester as
    MCP tools over the standard stdio transport (newline-delimited JSON-RPC
    2.0; protocol versions 2024-11-05 / 2025-03-26 / 2025-06-18 — the server
    echoes a supported client version, else advertises its latest):
    extract_thread (full harvest), lookup_status (metadata-only),
    read_manifest (returns an existing manifest; refuses any other
    filename), get_schema (envelope JSON Schema). The wrapper shells out to
    the CLI as a subprocess (argument list, never a shell) with hard timeouts
    (120s lookups, 900s extracts, XTHREAD_MCP_EXTRACT_TIMEOUT to override)
    and implements no pipeline logic of its own — the CLI contract remains the
    single source of truth. Tool results distinguish invocation failures
    (isError: true) from honest negative results (isError: false with
    envelope.status == "empty").
  • tests/test_mcp.py — 21 offline MCP tests: protocol handlers
    (initialize version echo/fallback, ping, tools/list shape, notifications
    stay silent, unknown methods), subprocess framing over stdio (parse
    errors, sequential request/response pairing), and tool error paths
    (invalid input fails before any network call, read_manifest filename
    guard, missing files). Total suite: 89 → 110 tests.
  • .github/workflows/ci.yml — GitHub Actions CI. Python 3.9–3.13
    matrix on push/PR/manual: byte-compile all sources, validate the bundled
    JSON Schema, run the full offline suite, CLI smoke checks (usage-error
    exit 2, invalid-input E_INVALID_INPUT fail-closed), and an offline MCP
    handshake probe. CI deliberately never probes live endpoints — the
    politeness constraint outranks CI coverage, and determinism is the point
    (see PROJECT_CONTEXT.md §11).

Changed

  • README.md: CI badge, MCP server section, docs table entry, testing
    section (root-based unittest discovery invocation, suite size).
  • agent.md: "Via MCP" section — how MCP clients run the server and read
    tool results (the four tools, timeout/error semantics).
  • PROJECT_CONTEXT.md: single-file constraint now documents the sanctioned
    MCP-wrapper exception; §11 debts updated (CI debt closed; new documented
    residual: MCP hosts must serialize heavy harvests themselves).
  • docs/endpoint-matrix.md: all rows re-verified alive from a datacenter
    IP on 2026-09-24 (UTC) — FixTweet, vxtwitter, UnrollNow, twimg CDN, plus
    the fail-closed path for an unavailable root.

v3.0.0 — 2026-09-24

The reliability release: true thread reconstruction, a fallback decoder slot,
hardened data integrity, a machine-readable envelope, and the project's first
test suite (89 tests). Built on a full re-verification of every endpoint from
a datacenter IP.

⚠️ BREAKING — manifest schema

thread_manifest.json changed from a bare tweet array (v2) to an
enveloped document (schema_version: "3.0"): {schema_version, source, request, status, thread, posts[], errors[], metadata}. The v2 array shape is
preserved as the posts array. Formal JSON Schema:
schema/thread-result.schema.json. Agents pinned to v2 must read
.posts[] instead of the top-level array.

Fixed (live-verified defects)

  • False thread members (v2 correctness bug): UnrollNow pages embed
    same-author recommendations alongside the conversation; v2 harvested them
    as thread members. v3 reconstructs the true self-reply chain from
    replying_to_status (walking up to the thread start and down through
    replies) and counts excluded recommendations (related_filtered).
  • Root dropped for short legacy IDs (v2 correctness bug): the walker
    regex (\d{15,25}) could not see e.g. status/20, silently excluding the
    root from harvesting. v3 guarantees the root is always present.
  • 404 retry storm: FixTweet returns real HTTP 404 statuses for
    unavailable tweets; urllib raises HTTPError, which v2 retried 3× with
    backoff before treating it as a filter. v3 treats HTTP 404/451 as instant
    filter signals (both response shapes).
  • Non-atomic downloads (v2 integrity bug): a crash mid-write left a
    partial file that skip-if-exists treated as complete forever. v3 streams to
    .part, verifies against Content-Length, and atomically renames.
  • write_text without explicit encoding (v2 mojibake risk): manifest now
    written UTF-8, atomically.
  • Python 3.9 support was broken in v2 (PEP-604 annotations evaluated
    eagerly); v3 adds from __future__ import annotations — the advertised
    3.9+ is now true.

Added

  • Input normalization: proper URL parsing (x.com/twitter.com, mobile./www.,
    /statuses/, /photo/<n>, /video/<n>, query strings, bare IDs) with
    E_INVALID_INPUT rejection for non-status URLs.
  • Fallback decoder slot (vxtwitter): used automatically when FixTweet
    fails network-side; payloads normalized to an honest subset, provenance
    recorded per post (extraction_source).
  • Enveloped manifest with status (ok/partial/empty), structured
    errors[] (stable codes: E_WALKER_UNAVAILABLE, E_WALKER_EMPTY,
    E_ROOT_UNAVAILABLE, E_DECODE_FAILED, E_DOWNLOAD_FAILED,
    E_MANIFEST_WRITE_FAILED, E_INVALID_INPUT), and extraction statistics.
  • Full author mapping (name, id, verified, protected, followers, avatar,
    banner, joined, website, …), post metrics incl. quotes/bookmarks, lang,
    posting client, replying_to(_status), ISO-8601 UTC timestamps.
  • Quoted posts recorded one level deep, including the quoted post's
    author, text, timestamps, and media URLs.
  • Video formats[] variants (container/bitrate/codec) preserved; m3u8-only
    videos now fall back to the highest-bitrate mp4 variant     instead of being
    skipped; non-downloadable videos explained (downloadable: false,
    reason: "hls_only" | "no_mp4_variant").
  • Candidate cap (50, root kept), response body caps (20 MB walk / 5 MB
    decode), transfer deadline against slow-drip servers.
  • Security hardening: media fetches restricted to https://*.twimg.com;
    remote tweet IDs validated (\d{1,25}) before use in filenames.
  • demo.py — minimal end-to-end consumption example.
  • tests/ — 89 offline tests (stdlib unittest, synthetic fixtures): URL
    normalization, walker, decoders, fallback slot, chain reconstruction,
    mapping, atomic downloads, envelope contract, CLI behavior.

Changed

  • --json summary gains additive fields (status, canonical_url,
    downloaded, failed_downloads, errors); all v2 fields keep their
    meaning.
  • downloaded now means "file exists on disk" (v2 could report true under
    --no-download); with --no-download, file is null.
  • Human summary lines ([done], post-run warnings) moved to stderr —
    stdout is data-only in every mode.
  • Filter logging is honest: media-ID candidates log [skip] … not a tweet,
    real decode failures log [warn] … decode failed; the misleading v2
    "[warn] unavailable — skipping" per media-ID is gone.
  • harvest() signature extended (request_info, decode_sleep) and now
    returns the envelope dict (was: posts list) — see agents.md Role 4.
  • README, agent.md, agents.md, endpoint-matrix updated; endpoint matrix
    re-verified 2026-09-24 (vxtwitter row corrected: alive, not Cloudflare-
    blocked; UnrollNow row corrected: candidates ≠ conversation).

Removed

  • extract_status_id (loose regex that matched any 10–25 digit number
    anywhere in a string) — replaced by normalize_input.

v2.0.0 — 2026-09-20

The professional release: same three-tier pipeline, hardened contracts,
full documentation suite, and a machine-readable interface.

Added

  • --json flag: pipe-safe summary object on stdout (logs stay on stderr).
  • --quiet flag: silence stderr logs.
  • --version flag: semver output.
  • Honest exit codes: 0 = at least one tweet harvested, 1 = nothing.
  • downloaded boolean per media item in the manifest (works with
    --no-download, where file is null and url is always populated).
  • agent.md — complete self-sufficient operating manual for AI agents.
  • agents.md — perfection-based role prompts for the internal roster
    (Thread Walker, Metadata Decoder, Media Fetcher, Orchestrator).
  • docs/research-blog.md — full research chronicle of the X lockdown and
    the three-tier bypass architecture.
  • docs/endpoint-matrix.md — living endpoint status reference with
    maintenance protocol.

Changed

  • 404 from the decoder is now handled as an explicit filter signal
    (previously retried): media-ID candidates are skipped silently.
  • Root-tweet fallback when the thread walk fails is logged as a warning
    and reflected in the summary (tweets count).
  • README rewritten around the agent contract: quickstart, JSON schema,
    decision tree, constraints, documentation map.

Removed

  • All example targets and test artifacts from documentation. The tool's
    provenance is documented generically (scale, runtime, outcome) — no
    harvested content, account names, or status IDs appear anywhere in the
    repository.

v1.0.0 — 2026-09-20

Initial release: single-file harvester with UnrollNow thread walk, FixTweet
decoding, resumable twimg CDN downloads, and thread_manifest.json output.
Validated end-to-end on a real multi-video fan thread (11 videos, 4 tweets,
~33 MB, under two minutes, zero auth).

Full Changelog: v3.1.0...v3.2.0

Assets 4
Loading