node-crawler

A reachable-node census of the bitcoin network, built for openbitcoin.com. It speaks the bitcoin P2P protocol itself: for every candidate address it opens a TCP connection, sends version, and waits for the peer's version and verack. A completed handshake is the definition of "reachable" here, and it yields the peer's user agent, protocol version, services and claimed height, all self-reported by the node. No third-party node API is involved at any point, which is the reason this exists.

How openbitcoin.com uses it

The /nodes page is fed entirely by the two tables this crawler fills: the reachable-node total and its history, the country breakdown, the user agent breakdown, and the counts of Tor and I2P addresses the census knows about but cannot reach. Each run inserts one dated row into nodes_census, so the history chart on that page is the run log itself.

Architecture

crawl.mjs runs in two phases. Phase one harvests the address space: it asks a sample of a few hundred peers for their address books with getaddr. Bitcoin Core does not answer getaddr inline; it queues the reply behind a Poisson timer averaging about thirty seconds, so harvesting peers stay on the line for that batch. Each answering peer returns up to a thousand addresses, and successive harvest rounds ask freshly discovered peers, which know different corners of the network. Phase two is the census: every dialable candidate gets a handshake-only probe that costs a second or two, run at high concurrency because most candidate addresses are stale and end in a timeout.

Seeding starts from your own node: getnodeaddresses (the node's address manager) plus getpeerinfo, topped up from the same DNS seeds Bitcoin Core queries on a cold start. The seeds only bootstrap the frontier; every figure in the census comes from the crawler's own handshakes.

Results land in PostgreSQL inside one transaction. nodes_seen holds the current roster and is replaced wholesale each run, so a node that stops answering leaves the census instead of lingering forever. nodes_census keeps one row per run, including a complete flag that is false when the run hit its time budget with candidates still unprobed, so a partial sweep can never pose as a total. A run that finds zero reachable nodes refuses to publish at all.

Geography is offline. geo.mjs loads a CSV of IP ranges (DB-IP lite country) into sorted arrays at startup and answers lookups by in-memory binary search; no lookup ever leaves the box. fetch-geo.sh refreshes the CSV monthly and refuses to install a truncated download.

Requirements

• Node.js 18 or newer (the crawler uses the built-in fetch and AbortSignal.timeout) and the pg package
• PostgreSQL, any recent version; the schema is two tables (schema.sql)
• A Bitcoin Core node reachable over RPC. Having getnodeaddresses on the RPC whitelist gives the crawler your node's own view of the network; without it the DNS seeds carry the bootstrap alone. No txindex, no coinstatsindex, no special flags.
• Outbound TCP to port 8333 and a file-descriptor limit above the configured concurrency (the shipped unit sets LimitNOFILE=8192 for 900 sockets)
• curl and gunzip for fetch-geo.sh

Setup

psql yourdb -f schema.sql   # grants reference a role named "openbitcoin"; edit to match yours
npm install pg
./fetch-geo.sh              # writes /var/lib/openbitcoin-crawler/ by default; edit DEST or set GEO_CSV
node crawl.mjs

Without the geo CSV the crawler still runs and publishes, just without country data.

Configuration is environment variables:

Variable	Default	Meaning
DB_HOST / DB_PORT / DB_NAME / DB_USER / DB_PASSWORD	127.0.0.1 / 5432 / openbitcoin / openbitcoin / none	PostgreSQL connection
RPC_URL (or RPC_PORT)	http://127.0.0.1:8332/	bitcoind RPC endpoint
OBWEB_RPC_USER / OBWEB_RPC_PASSWORD (or RPC_USER / RPC_PASS)	none	bitcoind RPC credentials
GEO_CSV	/var/lib/openbitcoin-crawler/dbip-country-lite.csv	offline IP-to-country CSV
CRAWL_CONCURRENCY	400	census sockets in flight
CRAWL_TIMEOUT_MS	8000	per-probe connect/handshake timeout
CRAWL_HARVEST_MS	75000	how long a harvesting peer is kept on the line
CRAWL_HARVEST_PEERS	600	peers asked for their address book per round
CRAWL_HARVEST_CONCURRENCY	600	harvest sockets in flight
CRAWL_HARVEST_ROUNDS	3	harvest rounds
CRAWL_MAX	250000	candidate address cap
CRAWL_BUDGET_MS	2400000	whole-run time budget

Production runs CRAWL_CONCURRENCY=900 with a 50-minute budget; at that setting a sweep of roughly 135k candidates takes about 20 minutes. Throughput is almost entirely concurrency, because most candidates are stale and cost a full timeout.

Running on a schedule

The systemd/ directory holds the units openbitcoin.com runs:

• openbitcoin-crawler.service + .timer: the census, every ten days
• openbitcoin-crawler-geo.service + .timer: the geo CSV refresh, monthly on the 5th

They bake in this box's layout: code in /opt/openbitcoin/node-crawler, env files under /etc/openbitcoin/, a service user named obapp, and the geo database under /var/lib/openbitcoin-crawler. Adjust those paths and the user to your box, then:

cp systemd/* /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now openbitcoin-crawler.timer openbitcoin-crawler-geo.timer

Data honesty

This census counts what a crawler can count, and says so:

• "Reachable" means this box completed a version handshake with that address during the run. Nodes that do not accept inbound connections (a large share of the network: anything behind NAT or a default home router) are invisible to any crawler, so the total is a lower bound, not the size of the network.
• Tor and I2P addresses are recorded as candidates but never dialed: the box runs no Tor or I2P daemon, and an .onion address carries no location by design. Their counts are published so the census can say how much of the network it cannot see, rather than pretending the visible part is all of it. CJDNS candidates are recognized and skipped the same way, without a published count.
• User agent, protocol version, services and height are self-reported by the peer and not verified.
• Country comes from a monthly IP-range database at country granularity. IP geolocation is approximate: a node behind a VPN or on a cloud range is located where the IP is registered, not where the operator sits. Nodes the database cannot place are published without a country, and each run row records how many were located.
• A run that exhausts its time budget publishes with complete = false so the page can label the count a partial sweep.

Geolocation data

fetch-geo.sh downloads the DB-IP lite country database, licensed CC-BY-4.0, which requires attribution. openbitcoin.com attributes it on its /sources page; if you run this, attribute it wherever you publish results. The database itself is not part of this repository.

License

MIT.