[MRG+1] Add from_crawler constructor for feed exporters and storages #3348
Conversation
| storage = S3FeedStorage(uri) | ||
| access_key = os.environ.get('AWS_ACCESS_KEY_ID') | ||
| secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY') | ||
| storage = S3FeedStorage(uri, access_key, secret_key) |
There was a problem hiding this comment.
@kmike I believe this test is skipped, that's why passing access_key=Settings() didn't break. Is S3_TEST_FILE_URI available here?
There was a problem hiding this comment.
It is supposed to be set in the environment which runs tox - see
Line 20 in 258121d
AFAIK we're not running s3 tests on travis if they require real s3 objects, i.e. this env variable is not set on Travis.
| category=ScrapyDeprecationWarning, | ||
| stacklevel=2 | ||
| ) | ||
| access_key = settings['AWS_ACCESS_KEY_ID'] |
There was a problem hiding this comment.
Just checking: given that the operator here is an or, a KeyError might raise if one of the keys is missing, I guess that's expected? User code will break with a clear message I think.
There was a problem hiding this comment.
Settings object returns None if a key is missing; see
scrapy/scrapy/settings/__init__.py
Line 92 in 258121d
There was a problem hiding this comment.
In that case, I think this might still break in the boto side if one of access_key/secret_key is not set either in the settings or in uri, the error message there will probably be clear enough.
elacuesta
changed the title
Codecov Report
@@ Coverage Diff @@
## master #3348 +/- ##
==========================================
+ Coverage 84.23% 84.35% +0.11%
==========================================
Files 167 167
Lines 9340 9359 +19
Branches 1384 1388 +4
==========================================
+ Hits 7868 7895 +27
+ Misses 1219 1209 -10
- Partials 253 255 +2
|
kmike
changed the title
|
Looks good to me, thanks for picking it up @elacuesta, and thanks @jdemaeyer for the initial PR! |
Picking up #1605 to add some changes requested by @kmike