[MRG+1] Backwards compatible per key priorities #1586

jdemaeyer · 2015-11-06T15:17:04Z

This is a partial reversal of #1149 that provides per-key priorities in a backwards compatible fashion.

The backwards incompatible API change in #1149 was that when writing to dictionary settings, their contents were updated, not completely overwritten. This PR brings back the old behaviour of completely discarding the previous contents of the dictionary. If the dictionary setting written to is a BaseSettings instance, it is not overwritten but cleared and then updated (resulting in the same contents, but conserving per-key priorities).

The price for this is:

we cannot get rid of the _BASE settings
it will again be much harder to disable single components from the command line or from a spider's custom_settings (because disabling a single component requires writing down the complete dictionary without that component). @kmike had the idea that we could add some helper magic for updating (not overwriting) dictionaries, e.g. allow people to do scrapy ... -s update:EXTENSIONS={"blah": null}, or custom_settings: { "update:EXTENSIONS": {"blah": None}}. I'll implement that in a follow-up PR to discuss it and see if it makes unforeseen trouble.

As a bonus, I added backwards compatibility to the build_component_list() utility function, albeit it not being public. @rolando maybe you could check whether it is indeed backwards compatible with how you use it? Also take a look at the now-public BaseSettings.getcomposite() method. It helps in separating a utility function that's meant to convert a dictionary into a sorted list from how that dictionary is created.

kmike · 2015-11-06T15:49:34Z

@jdemaeyer there is a failure in docs check, could you please fix it?

codecov-io · 2015-11-06T15:58:39Z

Current coverage is `82.89%`

Merging #1586 into master will not affect coverage as of 5049915

Powered by Codecov. Updated on successful CI builds.

jdemaeyer · 2015-11-06T15:58:54Z

ugh damnit, thanks, fixed.

rmax · 2015-11-06T21:02:44Z

@jdemaeyer yes, our code works as expected on this PR. Thanks.

kmike · 2015-11-09T15:07:25Z

I'm checking https://github.com/scrapy/scrapy/compare/4a9f76ebd3242b2f91e238c26ff38874e9ee1960...jdemaeyer:fix/backwards-compatible-per-key-priorities?name=fix%2Fbackwards-compatible-per-key-priorities&short_path=7d64b15#diff-7d64b15bbaac9444c9da295924bf42bf:

How to disable download handlers? Is this feature supported? It looks like relevant code and docs are deleted.
Are lists in ITEM_PIPELINES supported again after this PR?
getcomposite public function: I think it is better to extract "If only one argument is given, a second argument with _BASE appended will automatically be added" part into another function.
I think getpriority can be written a bit simpler, one option is:

if name not in self:
    return None
return self.attributes[name].priority

jdemaeyer · 2015-11-09T17:37:11Z

How to disable download handlers? Is this feature supported? It looks like relevant code and docs are deleted.

Yep, it's supported and got lost while I reverted the docs to "with _BASE", will put it back in. They're disabled by setting their class to None in the DOWNLOAD_HANDLERS dict (much like all other components can be disabled).

Are lists in ITEM_PIPELINES supported again after this PR?

Nope. Though with this PR we could support it again (although it has been deprecated for over two years).

getcomposite public function: I think it is better to extract "If only one argument is given, a second argument with _BASE appended will automatically be added" part into another function.

Hm, calling getcomposite with only one argument is useless then. But I'm fine with adding a new method if you think it's more consistent.

I think getpriority can be written a bit simpler, one option is:

Looks good, we should update __getitem__() then as well.

kmike · 2015-11-10T13:14:42Z

Hm, calling getcomposite with only one argument is useless then. But I'm fine with adding a new method if you think it's more consistent.

Do we have use cases for a public .getcomposite method with 1+ arguments, i.e. is it used in scrapy? If not then we can drop it for now and implement only _BASE support. Do you need it for addons PR?

kmike · 2015-11-10T13:17:18Z

Nope. Though with this PR we could support it again (although it has been deprecated for over two years).

Ok, I'm fine with dropping support for ITEM_PIPELINE as a list, but we should remember to add it to the release notes. It is easy to miss because this change was not in a separate PR.

jdemaeyer · 2015-11-10T21:58:49Z

All done. Instead of getwithbase() (getcomposite() before) we could also go back to merging the settings in build_component_list(), as it was before #1149. But I really think that function is not where the merging should take place.

kmike · 2015-11-10T22:05:05Z

+1 to merge. //cc @curita @dangra

curita · 2015-11-11T06:05:11Z

docs/topics/spider-middleware.rst

+is the one closer to the spider.
+
+To decide which order to assign to your middleware see the
+:setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where
 middleware performs a different action and your middleware could depend on some


Line 34 from the original version got deleted (https://github.com/scrapy/scrapy/pull/1586/files#diff-17a393aa4b57b33782da9b32c3422e73L34). It should be "<...> and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs <...>".

i also noticed that, but the sentence reads fine without this line :)

curita · 2015-11-11T06:09:26Z

There's a remaining _getcomposite() in scrapy/commands/runspider.py.

curita · 2015-11-11T06:14:51Z

Not sure what's the long term plan for *_BASE settings now, but if we plan to keep them for good we could implement DEFAULT_REQUEST_HEADERS_BASE as Paul suggested.

curita · 2015-11-11T06:34:51Z

scrapy/settings/__init__.py

+        :type name: string
+        """
+        compdict = self.get(name + '_BASE', {}).copy()
+        compdict.update(self.get(name, {}))


Wanted to point out here that since we're promoting <name>_BASE settings to BaseSettings instances, .copy() does a deep copy of <name>_BASE, but we're passing <name> as it is when updating that copy. We could be consistent and use self.get(name, {}).copy() but doesn't matter too much since we don't have mutable values in default dict settings.

curita · 2015-11-11T06:49:53Z

scrapy/settings/__init__.py

+        if priority >= self.priority:
+            if isinstance(self.value, BaseSettings):
+                self.value.clear()
+                self.value.update(value, priority=priority)


It could be a little odd having a reference for value (gotten from .get() for example) and having it changed when setting something new, as set() was supposed to override the value, not change it. Maybe we could create a new BaseSettings instance?

good catch, +1

curita · 2015-11-11T07:53:07Z

Pointed out some minor issues, but after reviewing these changes I still kind of prefer the approach in #1149. Here are some of my (truly subjective) thoughts about it:

It has better design. *_BASE settings are redundant having per-key priorities.
It has better usability. We usually want to update a set of values in those dict settings (not just *_BASE, I often want to update project-defined components with spider ones). I'd rather prefer updating the dictionaries by default and having a helper to occasionally override them.
We could improve its backward compatibility a little, by defining the old *_BASE settings with copies of the new values, but without using them.

Well, having explained myself, I don't think it's worth of a discussion really, this PR seems the way to go. Probably we should try to follow any backward compatible option if there is one, and the magic helper for updating dictionaries you were talking about is just what I'm looking for. So +1 on my side once the nitpicks I commented before get fixed 👍

jdemaeyer · 2015-11-11T15:41:31Z

Damnit! :/ thanks alot for the catches/feedback everyone! I've implemented Julia's feedback. There is now no deep-copying at all in getwithbase(): If we decide to allow Python objects (instead of paths) in the component settings (6f91110, part of #1272), users will expect that these exact objects are used in Scrapy, not deep copies of them.

I'll squash commits once you've looked at the changes.

curita · 2015-11-11T17:08:36Z

Perfect! Thank you both @jdemaeyer and @kmike :) I'll be merging it now.

…key-priorities [MRG+1] Backwards compatible per key priorities

jdemaeyer force-pushed the jdemaeyer:fix/backwards-compatible-per-key-priorities branch from 1a35a0c to 0c97a75 Nov 6, 2015

jdemaeyer mentioned this pull request Nov 9, 2015

Add-ons #1272

Open

37 of 37 tasks complete

kmike added this to the Scrapy 1.1 milestone Nov 10, 2015

kmike changed the title ~~Backwards compatible per key priorities~~ [MRG+1] Backwards compatible per key priorities Nov 10, 2015

curita reviewed Nov 11, 2015
View changes

jdemaeyer added 5 commits Nov 6, 2015

Bring back _BASE settings

e66f649

Add backwards compatibility for build_component_list

b6a023c

Replace BaseSettings._getcomposite() with public .getwithbase() method

52ecee6

Overwrite, not update, dictionary-like settings

44f6ada

Simplify BaseSettings.__get__(), .getpriority()

Loading status checks…

4f36476

jdemaeyer force-pushed the jdemaeyer:fix/backwards-compatible-per-key-priorities branch from 9176af4 to 4f36476 Nov 11, 2015

curita added a commit that referenced this pull request Nov 11, 2015

Merge pull request #1586 from jdemaeyer/fix/backwards-compatible-per-…

Loading status checks…

54216d7

…key-priorities [MRG+1] Backwards compatible per key priorities

curita merged commit 54216d7 into scrapy:master Nov 11, 2015
1 of 2 checks passed

1 of 2 checks passed

codecov/patch 93.00% of diff hit (target 100.00%)
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

jdemaeyer mentioned this pull request Nov 12, 2015

[MRG+1] Allow updating dictionary-like settings with magic prefix #1591

Open

redapple mentioned this pull request Jan 25, 2016

[MRG+1] Per-key priorities for dict-like settings by promoting dicts to Settings instances #1149

Merged

redapple mentioned this pull request Jul 5, 2016

The setting ITEM_PIPELINE can't be overridden from terminal #2018

Closed

nyov mentioned this pull request Dec 13, 2020

Replace `os.path` with `pathlib`? #4916

Open

scrapy / scrapy

[MRG+1] Backwards compatible per key priorities #1586

[MRG+1] Backwards compatible per key priorities #1586

jdemaeyer commented Nov 6, 2015

kmike commented Nov 6, 2015

codecov-io commented Nov 6, 2015

jdemaeyer commented Nov 6, 2015

rmax commented Nov 6, 2015

kmike commented Nov 9, 2015

jdemaeyer commented Nov 9, 2015

kmike commented Nov 10, 2015

kmike commented Nov 10, 2015

jdemaeyer commented Nov 10, 2015

kmike commented Nov 10, 2015

This comment has been minimized.

This comment has been minimized.

curita commented Nov 11, 2015

curita commented Nov 11, 2015

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

curita commented Nov 11, 2015

jdemaeyer commented Nov 11, 2015

curita commented Nov 11, 2015

scrapy / scrapy

[MRG+1] Backwards compatible per key priorities #1586

[MRG+1] Backwards compatible per key priorities #1586

Conversation

jdemaeyer commented Nov 6, 2015

kmike commented Nov 6, 2015

codecov-io commented Nov 6, 2015

Current coverage is 82.89%

jdemaeyer commented Nov 6, 2015

rmax commented Nov 6, 2015

kmike commented Nov 9, 2015

jdemaeyer commented Nov 9, 2015

kmike commented Nov 10, 2015

kmike commented Nov 10, 2015

jdemaeyer commented Nov 10, 2015

kmike commented Nov 10, 2015

This comment has been minimized.

curita Nov 11, 2015 Member

This comment has been minimized.

kmike Nov 11, 2015 Member

curita commented Nov 11, 2015

curita commented Nov 11, 2015

This comment has been minimized.

curita Nov 11, 2015 Member

This comment has been minimized.

curita Nov 11, 2015 Member

This comment has been minimized.

kmike Nov 11, 2015 Member

curita commented Nov 11, 2015

jdemaeyer commented Nov 11, 2015

curita commented Nov 11, 2015

Current coverage is `82.89%`

curita Nov 11, 2015
Member

kmike Nov 11, 2015
Member

curita Nov 11, 2015
Member

curita Nov 11, 2015
Member

kmike Nov 11, 2015
Member