A Node.js command-line tool to download a list of domains, scan their robots.txt files, identify disallowed user agents based on simple rules, and store the results in a local SQLite database for querying.
- Download Domain List: Fetches the Cisco Umbrella Top 1 Million domain list (
top-1m.csv.zip), extracts it, and savestop-1m.csv. - Scan Domains: Reads domains from a CSV file, concurrently fetches
robots.txtfiles (trying HTTPS then HTTP), performs basic parsing to find user agents listed before anyDisallow:directive. - Store Results: Saves the site URL and identified blocked user agents into a local SQLite database (
robots_data.db). - Query Data: Provides a basic report listing user agents and the count of sites where they were found blocked.
- Reset Database: Allows clearing all stored scan data.
- Clone the repository (if applicable):
git clone <repository_url> cd robots-txt
- Install dependencies:
Requires Node.js.
npm install
The tool is run from the command line using node index.js followed by a command and any required options.
-
download-listDownloads and extracts the Cisco Umbrella Top 1M list totop-1m.csvin the project directory.node index.js download-list
-
scanScans domains listed in a CSV file. The CSV file should have the domain name in the second column (e.g.,rank,domain).# Scan using the downloaded list node index.js scan -l top-1m.csv # Scan using a custom list node index.js scan --list path/to/your/domain_list.csv
-l,--list: (Required) Path to the CSV file containing domain names.
-
queryQueries the database for stored information. Currently supports one report type.node index.js query --report blocked-agents
-r,--report: (Required) Type of report. Currently onlyblocked-agentsis supported.
-
reset-dbDeletes all data from thesitesandblocked_agentstables in the database. Use with caution!node index.js reset-db
(Note: Currently lacks an interactive confirmation prompt.)
-
helpDisplays help information about commands and options.node index.js --help node index.js <command> --help
# 1. Download the list (only needed once initially or to update)
node index.js download-list
# 2. Scan the domains in the list
node index.js scan -l top-1m.csv
# 3. Query the results
node index.js query --report blocked-agents
# 4. (Optional) Reset the database if needed
# node index.js reset-db- Language: Node.js (ES Modules)
- Database: SQLite (
robots_data.db) - Key Libraries:
yargs,node-fetch,p-queue,better-sqlite3,adm-zip - Concurrency: Uses
p-queueto manage concurrentrobots.txtfetches (default: 50). - Parsing: Uses basic text parsing to identify user agents listed before any
Disallow:rule. It does not perform full, complexrobots.txtrule evaluation.