Will multiple magneticod of processes help?

[lozbrown]

Left overnight magnetico managed to add around 100 items, so not going to imminently replace centralised sites. It's not even using much bandwidth.

But I have multiple cores, and several other raspberry Pi's sitting doing very low utilisation jobs that could be put to further work.

Would multiple magneticod processes help or would they just interfere with each other? Would they just repeat the same work?

If not could they be made to work well together?

[lozbrown]

From what i can work out in theory becuase the DHT bootstrap node node would not return the same peers for a second request the crawling would take a completely different path

[boramalper]

Hey @lozbrown, sorry for the belated reply.

Honestly speaking, I don't know the answer and you should try and see (and please inform us as well!). The thing is, you will probably be limited by the network anyway so I don't think it will be much helpful.

Lastly, one running magneticod instance should be sufficient by design (i.e. that is what we aim for), so if you see an increase in performance, let us know, and we will try to improve its performance by doing necessary modifications! =)

[itdaniher]

I was disappointed by only scraping 30-50 torrents a minute, so modified my mana project to support multiple public ports and use a different UID for each, and launch one instance per CPU core. Gave me a clean N-fold speedup.

[boramalper]

@itdaniher That's interesting! Maybe we can modify the networking logic.

I'll surely think about it and I would love some suggestions to make it possible in a single running instance.

[itdaniher]

It only helps if you're CPU-bound, but the python multiprocessing library would probably be a good place to start.

The obvious issue you'll run into is sharing a list of already-seen torrents amongst multiple threads. Mana uses Redis to great effect.

[boramalper]

This is supported in the new written-in-Go versions. You can simply pass multiple --indexer-addr to spawn multiple indexers. =)

Let me know if you have any other/similar questions with the recent version of magnetico.

[Glandos]

I ran a test with 2 scenarios:

1. One indexer with 3500 neighbours
2. Two indexers with 1500 neighbours

This test runs with an Intel(R) Atom(TM) CPU D2550 @ 1.86GHz. There is enough RAM. I have a 1Gb/s down and 600Mb/s up connection.

The result is that scenario one use less CPU: ~87% vs ~107%.
It also uses a bit less network: 970kb/s up - 1600kb/s down vs 1000kb/s up - 1550kb/s down. 6800 packets/s up - 4900 packets/s down vs 7100 packets/s up - 5000 packets/s down.
It discovers at the same rate, around 0.25 new torrent per second.

So I will stick with one indexer for now, since I need some CPU for something else.