-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] S3 Storage #477
Comments
What would be the expected behavior? Scrapy can write lines to log files, items files, etc. at a very high frequency, and over a very long period of time - it would not make sense to store these on S3 while the files are "open". It's perhaps possible (once the file is closed) to transfer these files to S3... but that's something you can do as a separate job (like a backup script) - it's not clear that it should be something Scrapyd is responsible for. I assume this need arises from attempting to run Scrapyd on a host with only temporary storage (like Heroku). To get it working on such a platform:
|
I've made a commit to clarify the above in the docs: e92edd6 If you implement a new egg storage option, please feel free to open a pull request. |
@jpmckinney Thanks for the information! I wasn't aware of some of those features. You are correct that I am running scrapyd on temporary storage. I created my own separate script that runs afterward that sends the logs to S3. I think you are correct that it isn't scrapyd's job to do this. Thanks again for the insight. |
It would be great if you could specify an S3 bucket URI in the
scrapyd.conf
file for the eggs_dir, logs_dir, items_dir, etc...The text was updated successfully, but these errors were encountered: