Ultra-massive Tor-based HA proxy. Build for anonymous scraping on large scale. Proof-of-concept!
Inspired by Tor, this project provides experimental framework / architecture and foundation for building infrastructure for massive anonymous scraping.
Starts 10 Tor deamons, 10 Privoxy instances and HAProxy, all pre-configured and ready to use!
NUMBER_OF_TORS=10 mina torrero:start
Point your client to server, port 3113...
curl --proxy torrero-main:3113 http://icanhazip.com/
mina setup # Setup infrastructure on your server
NUMBER_OF_TORS=10 mina torrero:start # Starts 10 Tors, 10 privoxys and 1 HAProxy
mina torrero:stop # Stop everything
mina tor:start_all # Start Tors
mina tor:stop_all # Stop Tors
mina tor:config # Dump configuration
mina tor:logs # Cat logs from Tors
mina haproxy:start # Start HAProxy
mina haproxy:stop # Stop HAProxy
mina haproxy:update # Update HAProxy configuration for Tors
mina haproxy:config # Dump configuration
- Tor - Anonymity network client
- HAProxy - High Performance TCP/HTTP Load Balancer
- Monit - Monit - utility for monitoring services on a Unix system
- Privoxy - Privacy enhancing HTTP Proxy
sudo apt-get install tor haproxy monit privoxy -y
I also suggest that you use Monit for monitoring all services. Configuration can be found in config folder.
ab -X torrero-main:3113 -n 1000 -c 100 -v 2 http://icanhazip.com/
ruby torrero_test.rb
This project is under LGPL you can do whatever you want with it. Contributions are more than welcome. Thanks!