XCrawler is a robust, stealth-oriented CLI tool built in TypeScript for scraping tweets from X (formerly Twitter). Designed strictly for research purposes, it avoids bot detection using human-like behavior emulation, Playwright stealth plugins, and GraphQL interception.
It features automatic tweet deduplication using SQLite and exports clean data directly to CSV.
- 🕵️ Stealth & Anti-Detection: Uses
playwright-extraandpuppeteer-extra-plugin-stealthto bypass basic bot mitigation. - 🧠 Human-like Behavior: Implements randomized scrolling, batching, and cooldowns to mimic real user patterns and prevent account flagging.
- 📡 GraphQL Interception: Intercepts raw GraphQL API responses to extract rich, accurate tweet data directly from the network layer.
- 💾 Smart Deduplication: Utilizes a local SQLite database to track seen tweets, ensuring your CSV outputs only contain unique records across multiple runs.
- 📊 CSV Export: Automatically formats and pipes extracted data into clean CSV files.
You can install xcrawler globally via npm (or pnpm/yarn):
npm install -g @rayhanhendra/xcrawlerOr run it directly using npx:
npx @rayhanhendra/xcrawler [command] [options]To scrape X effectively, you need an auth_token cookie from a logged-in X account.
- Log in to x.com on your browser.
- Open Developer Tools (F12) -> Application -> Cookies.
- Copy the value of the
auth_tokencookie. - Recommendation: Use a burner account for scraping to avoid risking your primary account.
The primary command is search, which accepts search queries using X's standard search operators.
xcrawler search --keywords "artificial intelligence OR AI" --auth-token "your_token_here"| Flag | Alias | Description | Default | Required |
|---|---|---|---|---|
--keywords <query> |
-k |
Search keywords (supports X search operators) | - | ✅ |
--auth-token <token> |
Your X/Twitter auth_token cookie value |
- | ✅ | |
--lang <code> |
-l |
Language filter (ISO 639-1 code) | en |
❌ |
--output <path> |
-o |
Output CSV file path | ./output.csv |
❌ |
--max <number> |
-m |
Maximum number of tweets to collect | 100 |
❌ |
--batch-size <num> |
Number of tweets per batch before cooldown | 100 |
❌ | |
--batch-delay <sec> |
Cooldown delay between batches in seconds | 60 |
❌ | |
--db <path> |
Path to the deduplication database file | ./xcrawler.db |
❌ | |
--no-headless |
Run browser in visible mode (useful for debugging) | false |
❌ |
1. Basic Search with custom output:
npx xcrawler search -k "Next.js 14" --auth-token "abc123xyz..." -o "./nextjs-tweets.csv"2. Large Scale Extraction with custom batching:
npx xcrawler search -k "from:elonmusk" --auth-token "abc123xyz..." -m 1000 --batch-size 200 --batch-delay 1203. Debugging Mode (Visible Browser):
npx xcrawler search -k "typescript" --auth-token "abc123xyz..." --no-headlessXCrawler creates a local SQLite database (default: xcrawler.db in your current working directory) during execution. This database remembers the IDs of tweets you have already scraped.
If you run the exact same search command twice, the second run will skip tweets already saved in the database, preventing duplicate rows in your CSV files.
You can specify a custom database location using the --db flag if you want to maintain separate memory banks for different projects.
This tool is created for academic and research purposes only. Scraping X/Twitter may violate their Terms of Service. The developers of this tool assume no liability for misuse, account bans, or legal repercussions resulting from the use of this software. Always respect platform limits and use responsibly.
Contributions, issues, and feature requests are welcome! Feel free to check the issues page.
This project is licensed under the ISC License.