Web: Check a site can be indexed before completing submission process

[m-i-l]

A surprising number of sites (including verified sites) have been submitted with a robots.txt containing:

User-agent: *
Disallow: /

Which means searchmysite doesn't index them. It would be good to check robots.txt at the point of Quick Add or Verified Add so feedback can be given immediately if it isn't possible to index the site.

Note also that there are cases where robots.txt initially allows indexing, but is subsequently changed - see #11 for details of the handling of these.

[m-i-l]

Also #46 Indexing of some sites is blocked by Cloudflare and some of the rarer issues described in https://blog.searchmysite.net/posts/some-of-the-challenges-of-building-an-internet-search/.

I think it may be possible to run an indexing check on the web servers via something like:

docker exec -it src_indexing_1 scrapy shell -s ROBOTSTXT_OBEY=True -s USER_AGENT='Mozilla/5.0 (compatible; SearchMySiteBot/1.0; +https://searchmysite.net)' --nolog '<site>' -c 'response'

[m-i-l]

I've installed scrapy into the web container (this required the installation of quite a few additional dependencies given the web container is based on the stripped down Apache httpd container), and execute the scrapy shell command above to check a site can be indexed at step 3, i.e. the same step that site ownership if verified.

There's a message on the step 3 page advising users their site will get a request and that it might take a moment. It seems to usually complete within 2-3 seconds which is okay, but I guess it will depend on the site and may be something to keep an eye on.

Responses I've seen so far are 200 for success None for a fail, so I'm checking for a 200. This could be an issue if there are other success responses, e.g. 302. Unfortunately I can't seem to get details on why it has failed, e.g. robots.txt, so the failure message presented to users is fairly generic.