Adding Round Robin Queue

[tianhuil]

This queue (with tests) is to solve the issues raised scrapy/scrapy#2474 and scrapy/scrapy#1802

I would like a domain scheduler implemented here which scrapes in a domain-smart way: by round-robin cycling through the domains. This has two benefits:

1. Spreading out load on the target server instead of hitting the server with many requests at once
2. Reducing delays caused by server-overloaded errors or CONCURRENT_REQUESTS_PER_IP type restrictions.

This implements the proposed solution in scrapy/scrapy#1802. I would like to merge the round-robin queue first, and then merge in the changes from in the domain scheduler into scrapy.

[codecov]

Codecov Report

Merging #21 into master will decrease coverage by 0.99%.
The diff coverage is 92.3%.

@@           Coverage Diff           @@
##           master      #21   +/-   ##
=======================================
- Coverage   98.52%   97.53%   -1%     
=======================================
  Files           3        4    +1     
  Lines         204      243   +39     
  Branches       26       34    +8     
=======================================
+ Hits          201      237   +36     
- Misses          1        2    +1     
- Partials        2        4    +2

Impacted Files	Coverage Δ
queuelib/__init__.py	100% <100%> (ø)	⬆️
queuelib/rrqueue.py	92.1% <92.1%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06439b7...b608deb. Read the comment docs.

[dangra]

Please, remove this file. This kind of files are part of developer environment, if we have something project specific we add it here.

[dangra]

I like the idea and implementation. Let's merge as soon as my .gitignore comment is sorted out.

[tianhuil]

Thanks @dangra: I removed the .gitignore. Let me know if you'd like me to do anything else!

[dangra]

@cathalgarvey LGTM and release but we need to fix travis-ci build which seems broken due to missing pypy binary.

[dangra]

broken travis-ci builds addressed by #22