RobotsTxtMiddleware doesn't support wildcards in Disallow rules

[mattfullerton]

This is because, as I understand, Python's robotparser module doesn't support them either. There is an alternative, drop-in module, Robotexclusionrulesparser:

http://nikitathespider.com/python/rerp/

This line would need to be changed:
https://github.com/scrapy/scrapy/blob/master/scrapy/contrib/downloadermiddleware/robotstxt.py#L7

[mattfullerton]

It's not that simple :) Robotexclusionrulesparser doesn't work any better for me, but reppy does (https://pypi.python.org/pypi/reppy), and looks like its taking care of caching as well. I could submit a pull request with the alternative code or instead add a new middleware to contrib... what would be best?

[pablohoffman]

I don't think there'd be room for two robots.txts built-in implementation on Scrapy, so I'd favor replacing the builtin one with one bases on reppy, if we agree it's better. Does reppy work well with an asynchronous API?

[mattfullerton]

I can't give a good answer to that. Its a fairly straightforward module: https://github.com/seomoz/reppy/tree/master/reppy

And I am using it right now with the normal concurrent requests settings with good results.

[kmike]

I don't like some parts of reppy - for example, it has "install_requires: python-dateutil==1.5" in setup.py (which could downgrade user's python-dateutil), it doesn't have a CI setup, it doesn't provide an official list of Pythons supported. On the other hand, reppy code is OK: the parser is separated from downloader and from the cache, and there is a good testing suite.

[kmike]

reppy now has a CI setup, and it removed stupid python-dateutil==1.5 from install_requires. It provides other features that could be useful, see #892. I'm fine with depending on it.

[graingerkid]

Just wondering if there were any more developments on this?

[mattfullerton]

I'll see if I can change the middleware and requirements accordingly

[mattfullerton]

I want to test before I make a pull request. On both my local test and running my repo on Travis I get a connection error to http://site.local, which is used by the test:
https://github.com/scrapy/scrapy/blob/master/tests/test_downloadermiddleware_robotstxt.py

My guess is that the python robotparser doesn't throw an exception if the site doesn't exist but reppy does. Although that wouldn't explain how the test is able to pass. I see there is a test website in Scrapy but it doesn't include those paths. And I see nothing in any of the test config about starting a webserver and nothing in the docs about starting one.

Any ideas?

[kmike]

@mattfullerton if some parts of reppy do HTTP requests to check if a site is available we shouldn't use these parts. For Scrapy it is fine to use reppy.parser; reppy.cache is not OK.

The test doesn't make a real HTTP request, the response is mocked - see

scrapy/tests/test_downloadermiddleware_robotstxt.py

Line 69 in ed84231

crawler.engine.download.side_effect = return_response

. I agree that the code is not straightforward, so refactoring it to be easier to follow (if that's possible) is a welcome addition.

Making a PR with failing tests is fine if you want to discuss the changes; it won't be merged as-is, but it'd be easier for other people to check.

I don't know why tests are failing for you - are they failing with unmodified Scrapy or after some changes you've done? Travis tests pass for Scrapy master: https://travis-ci.org/scrapy/scrapy/builds/40556422

[mattfullerton]

Thanks, now I understand. Indeed it was my code that was failing on Travis. Although there are a set of failures I get locally and on Travis involving the ProxyServer, but that has nothing to do with me.

I now made the changes allowing Scrapy to do the downloading and Reppy only to do the parsing and allowing/disallowing. I've made a pull request.

[redapple]

I believe this is the one: #949
@mattfullerton , are you still working on it?

[mattfullerton]

No 😊