Scan top one million webpages

[ChargingBulle]

In #64 I proposed to offering statistics on how private the internet currently is. This of course would would become better when more pages are scanned.

Would be it be cool for you to scan the top one million webpages in the world?
A list of top million webpages by traffic can be downloaded for free here: https://majestic.com/reports/majestic-million

Scanning all these pages within a year would result in about 2700 new queries per day. I assume these could be automatically put into queue during time of low usage?

[hprid]

Thank you for your suggestion. Currently we are rewriting our scanning engine, since the old one has several (stability and scalability) issues. The new engine have already been used for research and capable of scanning one million pages in about 4-5 days on one of our servers (without TLS checks, those take a little longer). Until we have our new engine up and running, we would like to avoid larger scans. We hopefully will deploy it within the next one or two months and start scanning larger amounts of sites.

[ChargingBulle]

that's very cool

[ChargingBulle]

What's the status?