New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MRG+2] Fixes #685 FilesPipeline support for Google Cloud Storage. #2923
Conversation
rhoboro
commented
Sep 13, 2017
•
edited
edited
- Fixes Google Cloud Storage Support (Storage backends) #685
- FilesPipeline support for Google Cloud Storage.
- This feature use two setting values.
- FILES_STORE = 'gs://your_bucket/'
- GCS_PROJECT_ID = 'your_project_id'
- Using google-cloud-storage as optional dependency. For information about authenticating, see this page.
- Added some tests and passed. (the last line)
Codecov Report
@@ Coverage Diff @@
## master #2923 +/- ##
=========================================
- Coverage 84.83% 84.6% -0.24%
=========================================
Files 164 164
Lines 9192 9241 +49
Branches 1370 1375 +5
=========================================
+ Hits 7798 7818 +20
- Misses 1138 1166 +28
- Partials 256 257 +1
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good work. It looks clean and good to merge.
@kmike any pushback?
It looks good to me. @rhoboro would you mind adding some docs? See https://doc.scrapy.org/en/latest/topics/media-pipeline.html#supported-storage and https://doc.scrapy.org/en/latest/topics/media-pipeline.html#amazon-s3-storage. |
👍 Thanks @rhoboro! FTR: I haven't tried to run tests or use this code, but it looks good to merge. |
can you please point me to the docs for GCP as I need this feature asap. With S3 the authentication params need to be entered into settings.py. Couldn't find a similar code for GCP |
@zinyosrim In another way, google-cloud-storage gets credential information from GOOGLE_APPLICATION_CREDENTIALS. Please see this page. |
@rhoboro I hope you can stand by your code in case of issues with this storage backend. But not least, thanks for contributing it. |
Sorry, I'm still using my branch. So, I leave it as it is released. |
I have the issue #3044 ZIn |
@zinyosrim |