X ingest: follow the whole thread + ingest links found across the thread

[yuanhao]

X ingest: follow the whole thread + ingest links found across the thread

Source: human feature request (yuanhao). "X 帖子：不只抓单条，会自动追踪整个帖子串，外链文章一并摄取."

Scope: single-file change — workers/x-ingest/index.ts only. No changes to the ingest endpoint, types, or deployment config.

---

Context

Currently the x-ingest cron worker processes each mention in isolation: it finds the direct parent tweet (referenced_tweets.type === "replied_to"), extracts its article text and external links, and ingests those. If the parent is tweet #5 in a 10-tweet thread, we only see tweet #5.

The fix: use the X API v2 conversation_id field (available in recent-search on all paid tiers) to retrieve the full thread, concatenate the original author's tweets into one document, and aggregate external links from all thread tweets.

---

Implementation Plan (step-by-step)

Step 1: Add conversation_id to the XTweet type and search fields

File: workers/x-ingest/index.ts

1a. XTweet interface (line 82): Add conversation_id?: string; field:

interface XTweet {
  id: string;
  author_id?: string;
  text?: string;
  conversation_id?: string;           // ← ADD
  referenced_tweets?: { type: string; id: string }[];
  article?: { ... };
  entities?: { ... };
  attachments?: { ... };
}

1b. searchUrl() function (line 170): Add conversation_id to the tweet.fields parameter:

tweet.fields=entities,author_id,referenced_tweets,article,attachments,created_at,conversation_id

Step 2: Add thread-fetching constants

File: workers/x-ingest/index.ts, near existing constants (after line 39)

/** Max tweets to retrieve from a thread. Bounds API cost + ingest size. */
const MAX_THREAD_TWEETS = 50;

/** Max external links to ingest per thread. Prevents runaway link-heavy threads. */
const MAX_THREAD_LINKS = 20;

Step 3: Add fetchThreadTweets() function

File: workers/x-ingest/index.ts, after searchUrl() (after line 174)

This function takes a parent tweet, fetches all tweets in its conversation by the same author, and returns them ordered by ID (chronological). It reuses searchUrl() for the API call.

/**
 * Fetch tweets in the same conversation by the same author, ordered oldest-newest.
 * Returns [parent] on failure (graceful degradation to single-tweet behavior).
 *
 * Uses: GET /2/tweets/search/recent?query=conversation_id:<id> from:<author>
 * This only covers the last 7 days (recent-search limitation), which is fine
 * because our since_id cursor + 48h window means we only process recent mentions.
 */
async function fetchThreadTweets(
  parent: XTweet,
  parentAuthor: string | undefined,
  xBearer: string,
): Promise<XTweet[]> {
  const convId = parent.conversation_id;
  if (!convId || !parentAuthor) return [parent];

  const query = `conversation_id:${convId} from:${parentAuthor}`;
  const bound = `&max_results=${Math.min(MAX_THREAD_TWEETS, 100)}`;
  const url = searchUrl(query, bound);

  try {
    const res = await fetch(url, {
      headers: { Authorization: `Bearer ${xBearer}` },
    });
    if (!res.ok) {
      console.warn(`Thread fetch failed (${res.status}) for conversation ${convId} — using parent only`);
      return [parent];
    }
    const payload: XSearchPayload = await res.json();
    const tweets = payload.data ?? [];
    if (tweets.length === 0) return [parent];

    // Sort by ID ascending (older tweets first — X IDs are monotonic snowflakes)
    tweets.sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0));
    return tweets;
  } catch (err) {
    console.warn(`Thread fetch error for conversation ${convId}:`, (err as Error)?.message ?? err);
    return [parent];
  }
}

Design decisions:

• Graceful degradation: Any failure returns [parent] — single-tweet behavior preserved.
• from:<author>: Only the thread author's tweets, not replies from others (avoids noise).
• Sort by ID: X tweet IDs are snowflake IDs (monotonically increasing), string comparison gives chronological order.
• No pagination: max_results up to 100 (API max). One API call per mention.

Step 4: Add thread aggregation helpers

File: workers/x-ingest/index.ts, after fetchThreadTweets()

/** Aggregate and dedup external links across all tweets in a thread. */
function aggregateThreadLinks(threadTweets: XTweet[]): string[] {
  const seen = new Set<string>();
  const links: string[] = [];
  for (const tweet of threadTweets) {
    for (const link of externalLinks(tweet)) {
      if (!seen.has(link)) {
        seen.add(link);
        links.push(link);
      }
    }
  }
  return links.slice(0, MAX_THREAD_LINKS);
}

/**
 * Build concatenated thread text for ingestion (only if >1 tweet).
 * Returns undefined for single-tweet threads (no extra text to ingest).
 */
function buildThreadText(threadTweets: XTweet[]): string | undefined {
  if (threadTweets.length <= 1) return undefined;
  const texts = threadTweets
    .map((t) => (t.text ?? "").trim())
    .filter((t) => t.length > 0);
  if (texts.length <= 1) return undefined;
  return texts.join("\n\n---\n\n");
}

Step 5: Modify the per-mention processing loop

File: workers/x-ingest/index.ts, lines 284–343

This is the core change. Three modifications inside the for (const mention of mentions) loop:

5a. Hoist parentAuthor and add thread fetch (after line 297, before current line 299):

Move parentAuthor resolution ABOVE the thread fetch (currently it's on line 302). The new order becomes:

    // (after parent is resolved and validated — line 297)
    
    const parentAuthor = parent.author_id ? userById[parent.author_id] : undefined;

    // Fetch the full thread (author's chain only). Falls back to [parent] on failure.
    const threadTweets = await fetchThreadTweets(parent, parentAuthor, X_BEARER);

    const sourceUrl = parentAuthor
      ? `https://x.com/${parentAuthor}/status/${parent.id}`
      : `https://x.com/i/status/${parent.id}`;

Delete the old const parentAuthor line (was line 302) since it's now earlier.

5b. Add thread text job — after the article block (after ~line 324), before link extraction:

    // If the thread has multiple tweets, ingest the concatenated thread text.
    const threadText = buildThreadText(threadTweets);
    if (threadText) {
      const threadTitle = parentAuthor
        ? `Thread by @${parentAuthor}`
        : `X Thread`;
      jobs.push({
        body: { text: threadText, title: threadTitle, sourceUrl },
        label: `thread (${threadTweets.length} tweets)`,
      });
    }

5c. Replace single-tweet link extraction with thread-wide aggregation:

Replace:

    for (const link of externalLinks(parent)) {
      jobs.push({ body: { url: link }, label: link });
    }

With:

    for (const link of aggregateThreadLinks(threadTweets)) {
      jobs.push({ body: { url: link }, label: link });
    }

Step 6: Update the probe() debug output

File: workers/x-ingest/index.ts, in probe() sample output (~line 416)

After the includesMedia line, add:

          parentConversationId: parent?.conversation_id ?? null,

---

Gotchas for the Build Agent

1. X_BEARER scope: X_BEARER is declared at line 177 inside run(). Pass it as a parameter to fetchThreadTweets() (shown in plan). Do NOT access env.X_BEARER_TOKEN inside the helper.

2. parentAuthor hoisting: Move the parentAuthor declaration from its current position (line 302) to before the thread fetch. DELETE the original line — do not duplicate it.

3. Rate limits: Each mention now costs one additional API call. X API recent-search allows 60 requests/15 min. If rate-limited, fetchThreadTweets degrades gracefully. Acceptable.

4. Thread search returns the parent too: conversation_id:<id> from:<author> results include all of the author's tweets including the parent. So aggregateThreadLinks() already covers the parent's links. The old externalLinks(parent) must be REPLACED, not kept alongside.

5. Article stays on parent only: The X Article handling (lines 308–324) must NOT change. Articles are a property of the specific tweet. Thread text is a separate job.

6. No test harness exists for this worker. The worker is standalone (excluded from app tsconfig/eslint). The pure functions (aggregateThreadLinks, buildThreadText) are unit-testable but the worker currently has no test setup. Acceptable for this change; file a follow-up if desired.

7. Do NOT modify wrangler.jsonc — no new bindings needed.

8. Do NOT modify any files in src/ — this is purely a worker change.

---

Acceptance Criteria (mechanical)

• XTweet interface has conversation_id?: string field
• searchUrl() includes conversation_id in tweet.fields
• MAX_THREAD_TWEETS constant exists (value 50)
• MAX_THREAD_LINKS constant exists (value 20)
• fetchThreadTweets() function exists, takes (parent, parentAuthor, xBearer), returns Promise<XTweet[]>, falls back to [parent] on any error
• aggregateThreadLinks() function exists, deduplicates URLs across thread tweets, respects MAX_THREAD_LINKS
• buildThreadText() function exists, returns undefined for single-tweet threads, returns joined text for multi-tweet
• Per-mention loop calls fetchThreadTweets() after parent is resolved
• Link extraction uses aggregateThreadLinks(threadTweets) instead of externalLinks(parent)
• Thread text is pushed as a separate job when buildThreadText() returns non-undefined
• Article handling (lines 308–324) is unchanged
• Probe output includes parentConversationId
• Single-tweet mentions behave identically to before (thread fetch returns [parent], buildThreadText returns undefined, aggregateThreadLinks([parent]) === externalLinks(parent))
• wrangler deploy --config workers/x-ingest/wrangler.jsonc --dry-run succeeds (TypeScript compiles)
• No changes outside workers/x-ingest/index.ts

[yoyo-evolve]

Designed implementation plan. Ready for build.

Summary: Atomic single-file change to workers/x-ingest/index.ts. Adds thread traversal via the X API v2 conversation_id field + recent-search conversation_id:<id> from:<author> operator. Four new functions (fetchThreadTweets, aggregateThreadLinks, buildThreadText, plus type/constant changes), three modifications to the existing per-mention loop. Graceful degradation: any thread-fetch failure falls back to current single-tweet behavior.

Key design decisions:

• Author-only thread chain (no noise from others' replies)
• Thread text as a separate ingest job (alongside, not replacing, article ingestion)
• Links aggregated and deduped across all thread tweets, capped at 20
• One extra API call per mention, bounded at 50 tweets max
• No changes to wrangler config, ingest endpoint, or any src/ files