[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings #1847
Conversation
Current coverage is
|
if os.path.isdir(path): | ||
folder = path | ||
else: | ||
raise NotADirectoryError(path) |
redapple
Mar 17, 2016
Contributor
NotADirectoryError
doesnt seem to be defined under Python 2
NotADirectoryError
doesnt seem to be defined under Python 2
@aron-bordin , could you add tests for this feature and setting to |
f96ab1e
to
e5ed4b3
Hi @redapple , I updated the PR. Please, take a look when possible. |
@aron-bordin , could you add a test for the default case to make Codecov happy? :) |
@@ -47,7 +47,16 @@ def store(file): | |||
class BlockingFeedStorage(object): | |||
|
|||
def open(self, spider): | |||
return TemporaryFile(prefix='feed-') | |||
folder = None | |||
path = spider.crawler.settings['BLOCKING_FEED_STORAGE_PATH'] |
redapple
Mar 18, 2016
Contributor
you can probably use path
all along, and not use folder
at all,
something like
path = spider.crawler.settings.get('BLOCKING_FEED_STORAGE_PATH')
if path:
if not os.path.isdir(path):
raise OSError('Not a Directory: ' + str(path))
return NamedTemporaryFile(prefix='feed-', dir=path)
you can probably use path
all along, and not use folder
at all,
something like
path = spider.crawler.settings.get('BLOCKING_FEED_STORAGE_PATH')
if path:
if not os.path.isdir(path):
raise OSError('Not a Directory: ' + str(path))
return NamedTemporaryFile(prefix='feed-', dir=path)
ce85442
to
ac96a23
@redapple updated |
ac96a23
to
377f798
looks good to me. Thanks @aron-bordin ! |
The PR looks good But the |
377f798
to
80f0e12
80f0e12
to
9250a5b
HI @kmike I updated the name to Thx |
@redapple are you OK with the updated option name? |
Yup! |
PR Overview
I'm adding a new setting BLOCKING_FEED_STORAGE_PATH to configure a folder used to save temporary files before uploading them.
This setting defaults to None and will pick a system temporary folder. If a value is set, will raise an exception if it's not a valid directory and use it otherwise.
See #1779 for more details