Bitcoin node network crawler (written in Go).
Clone or download
shazow Merge pull request #2 from fiksn/master
Make the code work with latest btcsuite repo changes
Latest commit 601a21d May 27, 2015

README.md

btc-crawl

Bitcoin node network crawler (written in golang).

This is a for-fun project to explore the Bitcoin protocol and network.

Current status:

  • JSON streaming is in place, and graceful shutdown.
  • It crawls with all kinds of nice parameters but stores everything in memory until dumping a giant JSON blob at the end.
  • It crawls from hard-coded values and spits a bunch of stuff to stdout.

Usage

$ go get github.com/shazow/btc-crawl
$ btc-crawl --help
...
$ btc-crawl \
  --concurrency=100 \
  --output="btc-crawl.json" \
  --peer-age="24h" \
  --user-agent="/batman:1.0/" \
  --verbose
...

Estimated crawl time: Unknown.

There should be under 10,000 active network nodes at any given time, according to bitnodes.io. Each node returns around ~2,500 known nodes, but usually only several have timestamps within the last hour.

Todo

(In approximate order of priority)

  • Namespace useful sub-packages properly (outside of main)
  • Fix go build -race race condition warnings. (Not sure if this is feasible. golog and other fundamental pieces seem to trigger warnings possibly erroneously.)
  • Tests would be nice.

License

MIT (see LICENSE file).