XReporter

English | 中文

Ever run into this on X: the same original post keeps showing up because different followings retweet or quote it, and your timeline feels full of duplicates? Or you just cannot tell at a glance which posts are getting the most activity among people you follow. XReporter is built for exactly that workflow. It is a CLI-first pipeline that collects activity from a target X user's followings, normalizes it into SQLite, and generates a static HTML report. The idea is simple: merge and de-duplicate first, then group by original post and sort by activity count. You can scan hotspot posts quickly and drill into each following user's exact related actions (retweet, quote, reply) 🔍

Current version: 0.1.0 (MVI)

Note: This project has been primarily operated and implemented by Codex.

Before You Start

Official X API can be expensive at scale (pricing and quota can become a bottleneck).

Why XReporter

Keep collection reproducible: each run is tracked (runs, run_activities, run_warnings).
Keep reruns safe: upsert/idempotent persistence avoids duplicated core records.
Keep review efficient: HTML report has warning, grouped, user-grouped, and timeline views.
Keep operations practical: official/socialdata provider switch + fixture mode for offline testing.

3-Minute Quick Start

1) Setup environment

conda env create -f environment.yml
conda activate XReporter
pip install -e .[dev]

2) Configure credentials (env only)

# official provider
export X_BEARER_TOKEN="<your_token>"

# socialdata provider
export SOCIALDATA_API_KEY="<your_socialdata_api_key>"

Secrets are never persisted into project config files.

3) Initialize config

xreporter config init --username target_user --lang auto
# default provider for new config: official

4) Collect and render

xreporter collect --last 24h
xreporter render --latest

5) Health check

xreporter doctor

Example Reports

English sample: example/run_1_en.html
Chinese sample: example/run_1_zh.html
Source run: betterestli, --last 12h, run_id=1

English preview:

Chinese preview:

How The Report Is Organized

Generated report is static HTML and includes:

Warning section (provider/user/API path/raw error).
Grouped-by-original section (post/retweet/quote/reply).
Grouped-by-user section (all activities per actor).
Chronological full timeline.

Interaction and ordering rules:

Grouped-by-original, grouped-by-user, and timeline sections are collapsible (default collapsed).
Each grouped item card is also collapsible (default collapsed) to handle long content.
Timeline is sorted newest to oldest.
Grouped-by-original items are sorted by action count (desc), then latest action time (desc).
Grouped-by-user items are sorted by action count (desc), then latest action time (desc).
Report language follows config (en/zh; auto resolves by locale with English fallback).

Technical Route At A Glance

CLI (Typer + Rich)
  -> Config + i18n + logging bootstrap
  -> CollectorService
       -> provider adapter (XApiClient / SocialDataApiClient / FixtureXApiClient)
       -> normalizer
       -> SQLiteStorage
  -> HTML renderer

Detailed route: doc/tech_route.md | 中文

CLI Reference

xreporter config init --username <name> [--lang auto|en|zh] [--db-path <path>] [--report-dir <path>] [--following-cap <int>] [--include-replies/--no-include-replies] [--api-provider official|socialdata]
xreporter config show
xreporter collect [--username <name>] [--last 12h|24h | --since <ISO8601> --until <ISO8601>] [--following-cap <int>] [--include-replies/--no-include-replies] [--api-concurrency <int>] [--resume-run-id <id>]
xreporter render [--run-id <id> | --latest] [--output <html_path>]
xreporter doctor

Typical workflow

# 1) init once
xreporter config init --username jack --lang auto --following-cap 200

# 2) collect one window
xreporter collect --last 24h --api-concurrency 4

# 3) render latest run
xreporter render --latest

# 4) or render a specific run
xreporter render --run-id 3 --output ./reports/manual_run_3.html

# 5) resume an interrupted/failed run
xreporter collect --resume-run-id 3 --api-concurrency 4

Configuration

Default config path:

~/.xreporter/config.toml

Config fields:

username (string)
language (auto|en|zh)
db_path (string)
report_dir (string)
following_cap_default (int, default 200)
include_replies_default (bool, default true)
api_provider (official|socialdata; missing legacy field defaults to official)

Logging

Default log file: ~/.xreporter/logs/xreporter.log
If XREPORTER_HOME is set, log path becomes $XREPORTER_HOME/logs/xreporter.log.
Log includes command lifecycle, run-level collection progress, API request/retry/fallback status, and storage commit markers.
Retried API calls are both logged and printed to terminal (safe summary only, no secrets).
Optional env vars:
- XREPORTER_LOG_LEVEL (DEBUG|INFO|WARNING|ERROR, default INFO)
- XREPORTER_LOG_STDERR (1|true|yes|on) to mirror logs to stderr

Provider Notes

official
- Best aligned with canonical X API schema.
- Requires X_BEARER_TOKEN.
- Can face strong cost/rate-limit pressure depending on access tier.
socialdata
- Requires SOCIALDATA_API_KEY.
- Adapter aligns to documented endpoints/params and avoids unsupported filters.
- Referenced tweet backfill uses batch endpoint (tweets-by-ids) to reduce request count.
- Timeline 403 privacy responses are recorded as warnings and skipped.
Timeline page cap (anti-waste)
- Per-following timeline collection is capped at 5 pages by default (both official and socialdata providers).
- This is currently a code-level parameter (not a CLI flag).
- To modify it, edit:
  - src/xreporter/x_api.py -> XApiClient.__init__(..., max_timeline_pages=5)
  - src/xreporter/x_api.py -> SocialDataApiClient.__init__(..., max_timeline_pages=5)
fixture
- Set XREPORTER_FIXTURE_FILE to run offline demos/tests without real API calls.

Data Model

Core SQLite tables:

users, tweets, tweet_links, activities
runs, run_activities, run_warnings

Repository Map

src/xreporter/
  cli.py        # command interface and orchestration
  config.py     # config load/save/default paths
  i18n.py       # language resolution and message catalog
  logging_utils.py # runtime logging setup (file rotation + level control)
  models.py     # typed data contracts
  normalizer.py # payload -> normalized batch
  service.py    # collect workflow and warning handling
  storage.py    # SQLite schema, upsert, run metadata
  render.py     # static HTML generation
  time_range.py # last/since/until parsing
  x_api.py      # official/socialdata/fixture clients
tests/
doc/

Development And Testing

conda activate XReporter
pytest

Coverage focus:

unit: time range parsing, i18n fallback, activity classification, SQLite idempotency
integration: pagination, retry on 429/5xx, unresolved referenced tweet fetch
e2e: fixture collect -> render, rerun idempotency, bilingual CLI behavior

Troubleshooting

`xreporter doctor` fails on credentials

Confirm provider in config: xreporter config show
For official, check X_BEARER_TOKEN
For socialdata, check SOCIALDATA_API_KEY

Render output is empty

--latest may point to a failed run with 0 activities.
Use --run-id to render a known run with data:

xreporter render --run-id <id> --output ./reports/run_<id>.html

Need request/run diagnostics

Check ~/.xreporter/logs/xreporter.log (or $XREPORTER_HOME/logs/xreporter.log).
Set XREPORTER_LOG_LEVEL=DEBUG to capture per-request retry/fallback details.

Collect runs too long

New pagination safeguards stop repeated cursor/token loops automatically and write warning logs.
Per-following timeline pagination also has a hard cap of 5 pages by default (see max_timeline_pages in src/xreporter/x_api.py).
Increase API parallelism when your quota allows:
- xreporter collect --last 24h --api-concurrency 8
If runtime is still long, reduce collection scope temporarily:
- xreporter collect --last 12h --following-cap 100 --no-include-replies

Collection interrupted midway

Resume the same run without reprocessing completed followings:
- xreporter collect --resume-run-id <id> --api-concurrency 4

Language does not match expectation

Set explicit language in config (en or zh) instead of auto.

Contributing

Issues and PRs are welcome. Suggested contribution flow:

Create/focus an issue with expected behavior and scope.
Keep modules aligned with repository boundaries (x_api.py, normalizer.py, storage.py, render.py, cli.py).
Add tests for new behavior.
Keep English and Chinese docs in sync (*_cn.md).

Related Docs

Technical route: doc/tech_route.md / doc/tech_route_cn.md
Progress log: doc/progress.md / doc/progress_cn.md
Agent conventions: AGENTS.md / AGENTS_cn.md

Navigation Menu

Repository files navigation

XReporter

Before You Start

Why XReporter

Quick Navigation

3-Minute Quick Start

1) Setup environment

2) Configure credentials (env only)

3) Initialize config

4) Collect and render

5) Health check

Example Reports

How The Report Is Organized

Technical Route At A Glance

CLI Reference

Typical workflow

Configuration

Logging

Provider Notes

Data Model

Repository Map

Development And Testing

Troubleshooting

`xreporter doctor` fails on credentials

Render output is empty

Need request/run diagnostics

Collect runs too long

Collection interrupted midway

Language does not match expectation

Contributing

Related Docs

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
doc		doc
example		example
src/xreporter		src/xreporter
tests		tests
.gitignore		.gitignore
AGENTS.md		AGENTS.md
AGENTS_cn.md		AGENTS_cn.md
README.md		README.md
README_cn.md		README_cn.md
environment.yml		environment.yml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

XReporter

Before You Start

Why XReporter

Quick Navigation

3-Minute Quick Start

1) Setup environment

2) Configure credentials (env only)

3) Initialize config

4) Collect and render

5) Health check

Example Reports

How The Report Is Organized

Technical Route At A Glance

CLI Reference

Typical workflow

Configuration

Logging

Provider Notes

Data Model

Repository Map

Development And Testing

Troubleshooting

xreporter doctor fails on credentials

Render output is empty

Need request/run diagnostics

Collect runs too long

Collection interrupted midway

Language does not match expectation

Contributing

Related Docs

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`xreporter doctor` fails on credentials