[MRG+1] Per-key priorities for dict-like settings by promoting dicts to Settings instances #1149
Conversation
Implementing per-key priorities like this would render #1110 obsolete |
I've resolved issue 1 by providing a |
@@ -13,7 +13,7 @@ class DownloadHandlers(object): | |||
def __init__(self, crawler): | |||
self._handlers = {} | |||
self._notconfigured = {} | |||
handlers = crawler.settings.get('DOWNLOAD_HANDLERS_BASE') | |||
handlers = crawler.settings.get('DOWNLOAD_HANDLERS_BASE', {}) |
curita
May 27, 2015
Member
We could use crawler.settings.getdict()
(which has {}
as default already) to get both 'DOWNLOAD_HANDLERS_BASE' and 'DOWNLOAD_HANDLERS'.
We could use crawler.settings.getdict()
(which has {}
as default already) to get both 'DOWNLOAD_HANDLERS_BASE' and 'DOWNLOAD_HANDLERS'.
nramirezuy
May 27, 2015
Contributor
Not just that it also allows you to pass DOWNLOAD_HANDLERS
from command line.
Not just that it also allows you to pass DOWNLOAD_HANDLERS
from command line.
@@ -195,8 +195,7 @@ def item_scraped(self, item, spider): | |||
return item | |||
|
|||
def _load_components(self, setting_prefix): | |||
conf = dict(self.settings['%s_BASE' % setting_prefix]) | |||
conf.update(self.settings[setting_prefix]) | |||
conf = dict(self.settings[setting_prefix]) |
curita
May 27, 2015
Member
Shouldn't we keep '_BASE' loading an updating here for backward compatibility as the other cases?
Shouldn't we keep '_BASE' loading an updating here for backward compatibility as the other cases?
curita
May 27, 2015
Member
Also it's probably better to replace that call for conf = self.settings.getdict(settings_prefix)
.
Also it's probably better to replace that call for conf = self.settings.getdict(settings_prefix)
.
@@ -19,6 +19,12 @@ | |||
'cmdline': 40, | |||
} | |||
|
|||
def get_settings_priority(priority): |
curita
May 27, 2015
Member
👍
# that does not map keys to values, e.g.a list). I included | ||
# this b/c the test_cmdline/__init__.py test uses a deprecated | ||
# configuration API (EXTENSIONS is a list) and fails otherwise. | ||
# Should it stay in? |
curita
May 27, 2015
Member
Let's remove and isinstance(value, Mapping)
, build_component_list
already handles the case of custom
being a list or a tuple by returning it right away, without updating '_BASE'. Other settings that don't use build_component_list
should break since passing lists wasn't an option before.
Let's remove and isinstance(value, Mapping)
, build_component_list
already handles the case of custom
being a list or a tuple by returning it right away, without updating '_BASE'. Other settings that don't use build_component_list
should break since passing lists wasn't an option before.
jdemaeyer
May 28, 2015
Author
Contributor
The thing is that the EXTENSIONS
test was only an example of what developers might actually want to do. What if they have a settings that is a BaseSettings
instance, but for some reason they decide to replace it with a different kind of variable, say None
. The isinstance(value, Mapping)
allows for this.
I guess the alternative is to make developers actively acknowledge that they're losing the BaseSettings
instance and all its values/priorites by deleting it, then inserting a new setting with the type they want. While that is more explicit, it's probably not expected behaviour, given that "type replacement" for all other types of settings does not yield exceptions.
The thing is that the EXTENSIONS
test was only an example of what developers might actually want to do. What if they have a settings that is a BaseSettings
instance, but for some reason they decide to replace it with a different kind of variable, say None
. The isinstance(value, Mapping)
allows for this.
I guess the alternative is to make developers actively acknowledge that they're losing the BaseSettings
instance and all its values/priorites by deleting it, then inserting a new setting with the type they want. While that is more explicit, it's probably not expected behaviour, given that "type replacement" for all other types of settings does not yield exceptions.
jdemaeyer
Jun 2, 2015
Author
Contributor
On second thought, now that only the default dicts are promoted, and given that these should never be replaced by any incompatible type, I will remove the and isinstance(value, Mapping
). It also causes problems with passing json-encoded strings.
On second thought, now that only the default dicts are promoted, and given that these should never be replaced by any incompatible type, I will remove the and isinstance(value, Mapping
). It also causes problems with passing json-encoded strings.
# configuration API (EXTENSIONS is a list) and fails otherwise. | ||
# Should it stay in? | ||
self.value.update(value, priority) | ||
self.priority = max((priority, self.priority)) |
curita
May 27, 2015
Member
Idea here is that self.priority
is the highest priority within the internal BaseSettings
? In that case, I think we should lookup inside the provided value
instead of relying in the passed priority
. Examples like this one won't get the expected priority otherwise:
> s = Settings()
> s.set('option', BaseSettings({'key': 'value'}, priority='spider'))
> s.getpriority('option')
20 # priority 'project' instead of 'spider'
Idea here is that self.priority
is the highest priority within the internal BaseSettings
? In that case, I think we should lookup inside the provided value
instead of relying in the passed priority
. Examples like this one won't get the expected priority otherwise:
> s = Settings()
> s.set('option', BaseSettings({'key': 'value'}, priority='spider'))
> s.getpriority('option')
20 # priority 'project' instead of 'spider'
jdemaeyer
May 28, 2015
Author
Contributor
Yep, that was the idea. I've updated the code so it works as expected, thereby introducing a maxpriority()
method to the BaseSettings
class since there are two places in the code where I need to find the maximum priority. It's probably not very useful for anything else though, does it clutter up the API too much?
Yep, that was the idea. I've updated the code so it works as expected, thereby introducing a maxpriority()
method to the BaseSettings
class since there are two places in the code where I need to find the maximum priority. It's probably not very useful for anything else though, does it clutter up the API too much?
@@ -45,14 +62,12 @@ def __str__(self): | |||
__repr__ = __str__ | |||
|
|||
|
|||
class Settings(object): | |||
class BaseSettings(MutableMapping): |
curita
May 27, 2015
Member
I really like this change 👍
I really like this change
@@ -88,19 +106,30 @@ def getdict(self, name, default=None): | |||
value = json.loads(value) | |||
return dict(value) | |||
|
|||
def getpriority(self, name): |
curita
May 27, 2015
Member
Needed method 👍
Needed method
else: | ||
if isinstance(value, dict): | ||
value = BaseSettings(value, priority) | ||
self.attributes[name] = SettingsAttribute(value, priority) |
curita
May 27, 2015
Member
After giving it some more thought, I think we should deal with BaseSettings instances explicitly instead of promoting dictionaries. That way users won't get unexpected or backward incompatible behaviors in their own dict settings, and we can choose specifically which settings are going to be BaseSettings by just declaring their values to be that.
We can still do some sort of implicit propagation only in the Settings
class while loading the default values, checking if a setting is a dict and replacing or setting it (depending when it's done) to a BaseSettings object instead. Another option if that's not possible could it be to just import BaseSettings into default_settings.py
and use it there for the dict values. For that matter maybe we could move Settings
to another module within scrapy/settings/
so we get rid of the default_settings import in scrapy/settings/init.py.
After giving it some more thought, I think we should deal with BaseSettings instances explicitly instead of promoting dictionaries. That way users won't get unexpected or backward incompatible behaviors in their own dict settings, and we can choose specifically which settings are going to be BaseSettings by just declaring their values to be that.
We can still do some sort of implicit propagation only in the Settings
class while loading the default values, checking if a setting is a dict and replacing or setting it (depending when it's done) to a BaseSettings object instead. Another option if that's not possible could it be to just import BaseSettings into default_settings.py
and use it there for the dict values. For that matter maybe we could move Settings
to another module within scrapy/settings/
so we get rid of the default_settings import in scrapy/settings/init.py.
jdemaeyer
May 28, 2015
Author
Contributor
I agree. Now that we split Settings
and BaseSettings
explicit promotion of only the default settings seems a more sane choice.
I agree. Now that we split Settings
and BaseSettings
explicit promotion of only the default settings seems a more sane choice.
del self.attributes[name] | ||
|
||
def __delitem__(self, name): | ||
self.delete(name) |
curita
May 27, 2015
Member
I'm not sure I like the new delete()
method, seems like users could get confused about their usage and their differences and similarities with set()
. For keeping the dict-like interface I think we could keep __delitem__
with a simpler implementation, such as deleting the mapping in self.attributes
.
I'm not sure I like the new delete()
method, seems like users could get confused about their usage and their differences and similarities with set()
. For keeping the dict-like interface I think we could keep __delitem__
with a simpler implementation, such as deleting the mapping in self.attributes
.
jdemaeyer
May 28, 2015
Author
Contributor
I like having the option of "Delete this setting unless someone has updated it with a higher priority than what I'm doing", but would yield to the majority if that is considered useless. ;)
But I think you're right in that __delitem__
should ignore priorities and just delete from attributes
I like having the option of "Delete this setting unless someone has updated it with a higher priority than what I'm doing", but would yield to the majority if that is considered useless. ;)
But I think you're right in that __delitem__
should ignore priorities and just delete from attributes
@@ -13,7 +15,7 @@ def build_component_list(base, custom): | |||
""" | |||
if isinstance(custom, (list, tuple)): | |||
return custom | |||
compdict = base.copy() | |||
compdict = BaseSettings(base, priority = 'default') |
curita
May 27, 2015
Member
A note about coding style in this line and in general: though we're not enforcing it we try to comply with pep8 for new added code to scrapy as closely as we can (only rule we agreed to change it's the 80 chars max line width rule, we settled on a more flexible 100 chars). This pull request follows pep8 mostly, but I'd try to stick to the rule of "There shouldn't be spaces around default arguments nor immediately after parenthesis, brackets or braces".
A note about coding style in this line and in general: though we're not enforcing it we try to comply with pep8 for new added code to scrapy as closely as we can (only rule we agreed to change it's the 80 chars max line width rule, we settled on a more flexible 100 chars). This pull request follows pep8 mostly, but I'd try to stick to the rule of "There shouldn't be spaces around default arguments nor immediately after parenthesis, brackets or braces".
jdemaeyer
May 28, 2015
Author
Contributor
I definitely try to stick to PEP8, but I have to admit that I frequently break the "no spaces for keyword arguments" rule b/c it feels so unnatural to me. Will fix :)
I definitely try to stick to PEP8, but I have to admit that I frequently break the "no spaces for keyword arguments" rule b/c it feels so unnatural to me. Will fix :)
FEED_STORAGES and FEED_EXPORTERS are dictionaries with paths that don't use ordering too, and those should handle *BASE settings (DEFAULT_REQUEST_HEADERS is a dict as well, but it's used differently). I like the idea of outsourcing the common code from FEED* and DOWNLOAD_HANDLERS loading and
Don't worry, just rebase |
Thank you guys for the feedback! I now have:
To do:
|
Alright, this PR is now at a stage where I'm fairly happy with it and would remove the WIP tag as soon as I've written/updated the documentation and rebased. Still very open for feedback of course :) I've updated the first post as an overview if you haven't followed this PR. @curita I made some changes to the |
self.settings_module = settings_module | ||
Settings.__init__(self, **kw) |
curita
Jun 11, 2015
Member
There is a reason for swapping these two lines?
There is a reason for swapping these two lines?
jdemaeyer
Jun 17, 2015
Author
Contributor
Yes, took me quite a while to figure out though ;D It became necessary after I moved the dict promotion into __init__()
. Settings.__init__()
uses __getitem__()
during the promotion of default dictionaries to BaseSettings
instances, which in turn accesses self.settings_module
, so it needs to be defined before calling __init__()
Yes, took me quite a while to figure out though ;D It became necessary after I moved the dict promotion into __init__()
. Settings.__init__()
uses __getitem__()
during the promotion of default dictionaries to BaseSettings
instances, which in turn accesses self.settings_module
, so it needs to be defined before calling __init__()
# It's for internal use in the transition away from the _BASE settings and | ||
# will be removed along with _BASE support in a future release | ||
basename = name + "_BASE" | ||
if basename in self.attributes: |
curita
Jun 11, 2015
Member
nitpick: This line should be if basename in self:
and other similar references could be replaced too.
nitpick: This line should be if basename in self:
and other similar references could be replaced too.
warnings.warn('_BASE settings are deprecated.', | ||
category=ScrapyDeprecationWarning) | ||
compsett = BaseSettings(self[name + "_BASE"], priority='default') | ||
compsett.update(self.get(name)) |
curita
Jun 11, 2015
Member
nitpick: self[name]
instead of self.get(name)
nitpick: self[name]
instead of self.get(name)
jdemaeyer
Jun 17, 2015
Author
Contributor
True. I forgot that __getitem__()
doesn't throw KeyError
s either. I like all your nitpicks ;)
True. I forgot that __getitem__()
doesn't throw KeyError
s either. I like all your nitpicks ;)
|
||
def maxpriority(self): | ||
if len(self) > 0: | ||
return max(self.getpriority(name) for name in self.attributes) |
curita
Jun 11, 2015
Member
nitpick: for name in self
instead of for name in self.attributes
. I think there are other similar calls that could be replaced.
nitpick: for name in self
instead of for name in self.attributes
. I think there are other similar calls that could be replaced.
if basename in self.attributes: | ||
warnings.warn('_BASE settings are deprecated.', | ||
category=ScrapyDeprecationWarning) | ||
compsett = BaseSettings(self[name + "_BASE"], priority='default') |
curita
Jun 11, 2015
Member
I think getting this setting should be self.getdict(name + "_BASE")
instead, the old deprecated _BASE could be a json dict.
I think getting this setting should be self.getdict(name + "_BASE")
instead, the old deprecated _BASE could be a json dict.
jdemaeyer
Jun 17, 2015
Author
Contributor
Since __init__()
calls update()
, it can deal with JSON strings
Since __init__()
calls update()
, it can deal with JSON strings
return type(custom)(convert(c) for c in custom) | ||
compdict = BaseSettings(_map_keys(compdict), priority='default') | ||
compdict.update(_map_keys(custom)) | ||
# End backwards support |
curita
Jun 11, 2015
Member
I think we should remove support for the old signature, build_component_list
is an internal helper for scrapy, there's no need to maintain backward compatibility for it and it makes this function slightly more difficult to understand.
I think we should remove support for the old signature, build_component_list
is an internal helper for scrapy, there's no need to maintain backward compatibility for it and it makes this function slightly more difficult to understand.
jdemaeyer
Jun 19, 2015
Author
Contributor
I'll have to update a couple of tests but I like dropping this, too
I'll have to update a couple of tests but I like dropping this, too
for k, v in six.iteritems(compdict): | ||
prio = compdict.getpriority(k) | ||
compbs.set(convert(k), v, priority=prio) | ||
return compbs |
curita
Jun 11, 2015
Member
It took me a bit to understand this, I think this implementation is halfway between the old behavior and the new one, though I'm probably missing something.
If you call _check_components in a compdict
that is a BaseSettings instance, you're making sure that no keys convert to the same path/value, regardless of their priority. In that case adding them converted into a new BaseSettings shouldn't make any difference since there aren't any clashes, we can stick to {convert(k): v for k, v in six.iteritems(compdict)}
(there's no need to return a BaseSettings too).
Another possibility should be to not call _check_components in a BaseSettings, and check manually if there are any two keys that convert to the same path and also have the same priority so we can't tell which should be overridden first (commented in the description of #1267).
It's true that there weren't per-keys priorities before, users working with old paths should get an error if any two keys clash if they switch to scrapy 1.0, but it's probably a nicer deprecation for the old paths to users skipping 1.0 and updating directly to 1.1. If possible I would prefer the second option of checking if any keys that clash have the same priority or not though I think it's not that critical, we can implement the first one and delete this if
otherwise.
It took me a bit to understand this, I think this implementation is halfway between the old behavior and the new one, though I'm probably missing something.
If you call _check_components in a compdict
that is a BaseSettings instance, you're making sure that no keys convert to the same path/value, regardless of their priority. In that case adding them converted into a new BaseSettings shouldn't make any difference since there aren't any clashes, we can stick to {convert(k): v for k, v in six.iteritems(compdict)}
(there's no need to return a BaseSettings too).
Another possibility should be to not call _check_components in a BaseSettings, and check manually if there are any two keys that convert to the same path and also have the same priority so we can't tell which should be overridden first (commented in the description of #1267).
It's true that there weren't per-keys priorities before, users working with old paths should get an error if any two keys clash if they switch to scrapy 1.0, but it's probably a nicer deprecation for the old paths to users skipping 1.0 and updating directly to 1.1. If possible I would prefer the second option of checking if any keys that clash have the same priority or not though I think it's not that critical, we can implement the first one and delete this if
otherwise.
jdemaeyer
Jun 19, 2015
Author
Contributor
In that case adding them converted into a new BaseSettings shouldn't make any difference since there aren't any clashes, we can stick to {convert(k): v for k, v in six.iteritems(compdict)}
(there's no need to return a BaseSettings too).
Keeping track of the priorities (i.e. returning BaseSettings
) was necessary when there were base
and custom
BaseSettings objects with different priorities. But when we drop support for the (base, custom)
signature you're right, returning a simple dict should be fine.
I'll update this and include the edge case of different paths with different priorities
In that case adding them converted into a new BaseSettings shouldn't make any difference since there aren't any clashes, we can stick to
{convert(k): v for k, v in six.iteritems(compdict)}
(there's no need to return a BaseSettings too).
Keeping track of the priorities (i.e. returning BaseSettings
) was necessary when there were base
and custom
BaseSettings objects with different priorities. But when we drop support for the (base, custom)
signature you're right, returning a simple dict should be fine.
I'll update this and include the edge case of different paths with different priorities
@@ -21,7 +21,8 @@ def _get_mwlist_from_settings(cls, settings): | |||
category=ScrapyDeprecationWarning, stacklevel=1) | |||
# convert old ITEM_PIPELINE list to a dict with order 500 | |||
item_pipelines = dict(zip(item_pipelines, range(500, 500+len(item_pipelines)))) | |||
return build_component_list(settings['ITEM_PIPELINES_BASE'], item_pipelines) | |||
settings.set('ITEM_PIPELINES', item_pipelines) |
curita
Jun 11, 2015
Member
There are a couple of issues here:
- There's no guarantee that
settings.set('ITEM_PIPELINES', item_pipelines)
overrides the stored ITEM_PIPELINES, that settings.set()
call should be made with a priority higher than the stored one for that key.
- We didn't modify the settings before, and not too long ago the settings were frozen at this point, this would have thrown an error. A solution for it could be to make a copy and set ITEM_PIPELINES there.
- This is incompatible with the goal of allowing addons to modify settings, addons won't be able to add pipelines easily if ITEM_PIPELINES is a list.
My vote is to remove all this deprecation support for lists, it was introduced way too long ago (0.20 release: fc388f4).
There are a couple of issues here:
- There's no guarantee that
settings.set('ITEM_PIPELINES', item_pipelines)
overrides the stored ITEM_PIPELINES, thatsettings.set()
call should be made with a priority higher than the stored one for that key. - We didn't modify the settings before, and not too long ago the settings were frozen at this point, this would have thrown an error. A solution for it could be to make a copy and set ITEM_PIPELINES there.
- This is incompatible with the goal of allowing addons to modify settings, addons won't be able to add pipelines easily if ITEM_PIPELINES is a list.
My vote is to remove all this deprecation support for lists, it was introduced way too long ago (0.20 release: fc388f4).
curita
Jun 11, 2015
Member
I recalled that ITEM_PIPELINES can't be a list with the changes in SettingsAttribute.set()
, ITEM_PIPELINES has as value a BaseSettings instance, if a user tries to set a list here it's going to throw an error when trying to merge it into those BaseSettings.
I recalled that ITEM_PIPELINES can't be a list with the changes in SettingsAttribute.set()
, ITEM_PIPELINES has as value a BaseSettings instance, if a user tries to set a list here it's going to throw an error when trying to merge it into those BaseSettings.
jdemaeyer
Jun 19, 2015
Author
Contributor
Good point. Providing a list still worked when the and isinstance(value, Mapping)
was still present in SettingsAttribute.set()
, but I also prefer dropping support for lists, especially as it will lead to unexpected behaviour for add-on developers.
Good point. Providing a list still worked when the and isinstance(value, Mapping)
was still present in SettingsAttribute.set()
, but I also prefer dropping support for lists, especially as it will lead to unexpected behaviour for add-on developers.
I love the unification of the code, really like the design decisions you took there. I pointed out a couple of remaining details but I think the overall functionality is well defined, you should be able to start with the documentation. |
2f288fa
to
4192d8b
Alright, I think this PR is ready for final review. I've incorporated your recent feedback (nitpicks, |
+1 to merge, It needs a note about backwards incompatibilities introduced by this PR and how to update users code if possible. |
I'm happy to add a note, would that go into |
Removing item pipelines list support is fine and it is not a backward The change that worries me a bit is the new behaviour for dictionary El ago. 20, 2015 7:05, "Jakob de Maeyer" notifications@github.com
|
compsett = BaseSettings(self[name + "_BASE"], priority='default') | ||
compsett.update(self[name]) | ||
return compsett | ||
else: |
kmike
Aug 20, 2015
Member
else is not necessary, let's drop it
else is not necessary, let's drop it
jdemaeyer
Aug 24, 2015
Author
Contributor
Now that I've seen it with a little more distance, I think this whole function doesn't really achieve what I had intended.
When users override XY_BASE
, they explicitly don't want any of Scrapy's defaults for that component setting. But line 206 pulls Scrapy's defaults (which now live in XY
) back in, and even worse overwrites the users XY_BASE
settings where they have the same keys (say when the user simply changed some orders).
I guess either line 206 could be changed so that only those keys from XY
that have a priority higher than default
are considered, or support for _BASE
settings could be dropped altogether with this PR. After all, they have always been marked as "never edit this", the behaviour of the dict-like settings changes with this PR anyways, and we could get rid of this non-public helper function.
Now that I've seen it with a little more distance, I think this whole function doesn't really achieve what I had intended.
When users override XY_BASE
, they explicitly don't want any of Scrapy's defaults for that component setting. But line 206 pulls Scrapy's defaults (which now live in XY
) back in, and even worse overwrites the users XY_BASE
settings where they have the same keys (say when the user simply changed some orders).
I guess either line 206 could be changed so that only those keys from XY
that have a priority higher than default
are considered, or support for _BASE
settings could be dropped altogether with this PR. After all, they have always been marked as "never edit this", the behaviour of the dict-like settings changes with this PR anyways, and we could get rid of this non-public helper function.
def __str__(self): | ||
return str(self.attributes) | ||
|
||
__repr__ = __str__ |
kmike
Aug 20, 2015
Member
I think repr should be something like %s(%s) % (self.__class__, self.attributes)
I think repr should be something like %s(%s) % (self.__class__, self.attributes)
kmike
Aug 20, 2015
Member
because otherwise it'd be hard to distinguish Settings objects from regular dicts in console
because otherwise it'd be hard to distinguish Settings objects from regular dicts in console
jdemaeyer
Aug 24, 2015
Author
Contributor
Agreed, I'll change that
Agreed, I'll change that
compdict.update(_map_keys(custom)) | ||
items = (x for x in six.iteritems(compdict) if x[1] is not None) | ||
return [x[0] for x in sorted(items, key=itemgetter(1))] | ||
def remove_none_values(compdict): |
kmike
Aug 20, 2015
Member
This utility function looks generic enough for scrapy.utils.conf - what about moving it to scrapy.utils.python?
I expect remove_none_values
it to remove values inplace. without_none_values
? We can also extend it to handle iterables.
This utility function looks generic enough for scrapy.utils.conf - what about moving it to scrapy.utils.python?
I expect remove_none_values
it to remove values inplace. without_none_values
? We can also extend it to handle iterables.
jdemaeyer
Aug 24, 2015
Author
Contributor
+1 will change/extend
+1 will change/extend
Here's a proposed release note:
|
03349ff
to
d9577da
d9577da
to
03f1720
Rebased onto current master and updated the function that handles backwards-compatibility for users who explicitly set They will now find the expected behaviour that they get none of Scrapy's defaults for each setting where they manually have set a
will now result in no downloader middlewares being enabled (and a warning that Not sure about the codecov test failing. It says only 95 % of this diff are hit because there are a couple of places that weren't covered before but where I switched the syntax to use the new helpers, e.g. like this: - valid_output_formats = (
- list(self.settings.getdict('FEED_EXPORTERS').keys()) +
- list(self.settings.getdict('FEED_EXPORTERS_BASE').keys())
- )
+ feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS'))
+ valid_output_formats = feed_exporters.keys() These should definitely be covered but I don't think it belongs into this PR. |
…priorities [MRG+1] Per-key priorities for dict-like settings by promoting dicts to Settings instances
Removed "backward-incompatible" tag after #1586 merged |
Expand settings priorities by assigning per-key priorities for the dict-like settings (e.g.
DOWNLOADER_MIDDLEWARES
), instead of just a single priority for the whole dictionary. This allows updating these settings from multiple locations without having to care (too much) about order. It is a prerequisite for the add-on system (#1272, #591).There are two main updates:
BaseSettings
class (formerlySettings
). They behave just like dictionaries, but honour per-key priorities when being written to.X_BASE
settings are deprecated, with default entries now living in theX
setting.And several smaller updates:
Settings
is a subclass ofBaseSettings
. It loads the default settings and promotes dictionaries within them toBaseSettings
instances.BaseSettings
has a complete dictionary-like interface.None
. A new helper,scrapy.util.without_none_values()
was introduced for this. This was previously not supported byFEED_STORAGES
,FEED_EXPORTERS
, andDEFAULT_REQUEST_HEADERS
.scrapy.util.build_component_list()
helper has been updated according to the deprecation of_BASE
settings, as the(base, custom)
call signature does not make much sense anymore.It's still backwards-compatible.ITEM_PIPELINES
can no longer be provided as listComes with many new/updated tests and documentation.