[MRG+2] Fixes #685 FilesPipeline support for Google Cloud Storage.

[rhoboro]

• Fixes #685
• FilesPipeline support for Google Cloud Storage.
• This feature use two setting values.
  • FILES_STORE = 'gs://your_bucket/'
  • GCS_PROJECT_ID = 'your_project_id'
• Using google-cloud-storage as optional dependency. For information about authenticating, see this page.
• Added some tests and passed. (the last line)

(venv) ~/github/scrapy/scrapy % GCS_PROJECT_ID="myproject" GCS_TEST_FILE_URI="gs://my-bucket/" tox -- tests/test_pipeline_files.py -v

...

tests/test_pipeline_files.py::FilesPipelineTestCase::test_file_expired <- .tox/py27/lib/python2.7/site-packages/twisted/internet/defer.py PASSED
tests/test_pipeline_files.py::FilesPipelineTestCase::test_file_not_expired <- .tox/py27/lib/python2.7/site-packages/twisted/internet/defer.py PASSED
tests/test_pipeline_files.py::FilesPipelineTestCase::test_file_path PASSED
tests/test_pipeline_files.py::FilesPipelineTestCase::test_fs_store PASSED
tests/test_pipeline_files.py::DeprecatedFilesPipelineTestCase::test_default_file_key_method PASSED
tests/test_pipeline_files.py::DeprecatedFilesPipelineTestCase::test_overridden_file_key_method PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseFields::test_item_fields_default PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseFields::test_item_fields_override_settings PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_cls_attrs_with_DEFAULT_prefix PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_custom_settings_and_class_attrs_for_subclasses PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_custom_settings_for_subclasses PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_different_settings_for_different_instances PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_no_custom_settings_for_subclasses PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_subclass_attributes_preserved_if_no_settings PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_subclass_attrs_preserved_custom_settings PASSED
tests/test_pipeline_files.py::FilesPipelineTestCaseCustomSettings::test_user_defined_subclass_default_key_names PASSED
tests/test_pipeline_files.py::TestS3FilesStore::test_persist <- .tox/py27/lib/python2.7/site-packages/twisted/internet/defer.py SKIPPED
tests/test_pipeline_files.py::TestGCSFilesStore::test_persist <- .tox/py27/lib/python2.7/site-packages/twisted/internet/defer.py PASSED

========================================================================================== 17 passed, 1 skipped in 11.54 seconds ===========================================================================================
_________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________
  py27: commands succeeded
  congratulations :)

[codecov]

Codecov Report

Merging #2923 into master will decrease coverage by 0.23%.
The diff coverage is 34.14%.

@@            Coverage Diff            @@
##           master   #2923      +/-   ##
=========================================
- Coverage   84.83%   84.6%   -0.24%     
=========================================
  Files         164     164              
  Lines        9192    9241      +49     
  Branches     1370    1375       +5     
=========================================
+ Hits         7798    7818      +20     
- Misses       1138    1166      +28     
- Partials      256     257       +1

Impacted Files	Coverage Δ
scrapy/pipelines/images.py	90.37% <100%> (+0.14%)	⬆️
scrapy/utils/test.py	55.55% <27.27%> (-7.24%)	⬇️
scrapy/pipelines/files.py	67.04% <32.14%> (-4.15%)	⬇️
scrapy/utils/versions.py	78.57% <0%> (-4.04%)	⬇️
scrapy/commands/version.py	100% <0%> (ø)	⬆️

[dangra]

Good work. It looks clean and good to merge.

@kmike any pushback?

[kmike]

It looks good to me. @rhoboro would you mind adding some docs? See https://doc.scrapy.org/en/latest/topics/media-pipeline.html#supported-storage and https://doc.scrapy.org/en/latest/topics/media-pipeline.html#amazon-s3-storage.

[rhoboro]

@dangra @kmike
Thank you for review.
I modified some docs. Please check my changes.

[kmike]

👍 Thanks @rhoboro!

FTR: I haven't tried to run tests or use this code, but it looks good to merge.

[zinyosrim]

can you please point me to the docs for GCP as I need this feature asap. With S3 the authentication params need to be entered into settings.py. Couldn't find a similar code for GCP

[rhoboro]

@zinyosrim
If locally, gcloud auth login command(in cloud sdk) is the easiest way to check this feature.

In another way, google-cloud-storage gets credential information from GOOGLE_APPLICATION_CREDENTIALS. Please see this page.

[dangra]

@rhoboro I hope you can stand by your code in case of issues with this storage backend. But not least, thanks for contributing it.

[rhoboro]

Sorry, I'm still using my branch. So, I leave it as it is released.

[zinyosrim]

I have the issue #3044
Any idea what's going wrong here? I am using the splash docker image, because I am crawling a site using JS. I first tried with Twisted-17.5.0 then upgraded to Twisted-17.9.0 with no effect.

ZIn

[rhoboro]

@zinyosrim
hmm... I tried to store images to GCS bucket with splash docker image, and it worked fine.
I pushed my source code to following repository, so please check it.
https://github.com/rhoboro/splash_gcs