Daemon that scrapes the DHT swarm and an express bootstrapped front-end connected to the same mongodb to create an automated magnet db with search.
This can index over 1 million magnets per 24/hr on 2GB of RAM and around 2MB/s connection. It is a RAM/CPU hog though, it will consume 100% of the CPU and the RAM if allowed and can be controlled via the 'ecosystem.json' file. On 2GB RAM it is suggested to use 8 instances of the daemon and 2 of the webserver; all limited at 175MB.
apt install mongodb
apt install redis-server
npm install -g pm2
npm install
pm2 start ecosystem.json
pm2 monit
You will need to have a port open on the outside and defined, by default it is 6881.
Some minor configuration and tweaks can be done, depending on the hardware and connection you are connected to; you may want to experiment with some of these settings. In the bin/daemon.js file you can find a block of code that looks like this below.
const p2p = P2PSpider({
nodesMaxSize: 250,
maxConnections: 500,
timeout: 1000
});
It isn't reccomended changing the nodesMaxSize or the maxConnections, but timeout seems to be a possibility to get your indexing to go faster. It may require a little bit more RAM with a higher timeout though, the highest we would reccomend at the time is 5000ms.
Would like to keep the information private easily?
I have added a section of code to bin/webserver.js that will add basic authentication with mutliple users possible by default. It looks like this below. If you would like to remove this feature, you can either change challenge to false, or just comment out the entire block.
app.use(basicAuth({
users: {
'username': 'password',
'username': 'password',
},
challenge: true,
realm: 'Secret Place'
}));
bep_0005, bep_0003, bep_0010, bep_0009
Cluster mode will not and will never work on Windows... With other OSs we can get multiple instances listening on the same UDP port. For the more technical folks, Windows doesn't allow us to get down to the lower levels of our system like Linux and various others do. We simply cannot touch things at the hardware level while using that operating system.
Please don't share the data p2pspider crawled to the internet. Because sometimes it crawls sensitive/copyrighted/porn data.
MIT