Allow remote items store

[mgregson]

Sometimes it's convenient to allow storage of item feeds somewhere other than local disk (ie, S3). Scrapy allows this but the current scrapyd handling of FEED_URI blocks the feature. This should allow passing non-file URI schemes through to scrapy without losing existing functionality when provided a file-scheme URI or a raw path.

[pablohoffman]

Thanks for the useful contribution @mgregson, and sorry this took so long to get merged!

[pablohoffman]

@mgregson this broke tests, could you take a look?
https://travis-ci.org/scrapy/scrapyd/builds/28011871

[mgregson]

@pablohoffman I'm not particularly familiar with travis-ci, but it looks to me (from reading the log at https://travis-ci.org/scrapy/scrapyd/jobs/28011872) like this failure is entirely unrelated to my patch. Nothing in the stack trace indicates that scrapyd was even running - the errors were emitted by pip. The messages of note seem to be:

error: [Errno 104] Connection reset by peer
Storing debug log for failure in /home/travis/.pip/pip.log
The command "pip install --use-mirrors -r .travis/requirements-$BUILDENV.txt" failed and exited with 2 during .

and

$ pip install --use-mirrors -r .travis/requirements-$BUILDENV.txt
--use-mirrors has been deprecated and will be removed in the future. Explicit uses of --index-url and/or --extra-index-url is suggested.

Since none of the other tests failed and the project dependencies presumably did not change between tests, I suspect that this was some random network error with either travis-ci or whatever host pip was trying to connect to. However, it seems appropriate to update your .travis.yml to remove the deprecated --use-mirrors option anyway, so perhaps do that and re-run the test suite?

[pablohoffman]

@mgregson you were right, the Travis failure was completely unrelated to your change. Sorry for the noise!