Description
Description
Calling setdefault method of class BaseSettings does nothing.
Steps to Reproduce
from scrapy.settings import BaseSettings
settings = BaseSettings()
stored = settings.setdefault('key', 'value')
print(stored) # prints None
print(settings.copy_to_dict()) # prints empty dictionaryExpected behavior:
settings.setdefault(key, default) must work as described in MutableMapping interface: set default to settings[key] and return default if key is not present, otherwise return settings[key].
Actual behavior:
settings.setdefault(key, default) does nothing regardless of holding key or not.
Reproduces how often: 100%
Versions
Scrapy : 2.7.1
lxml : 4.8.0.0
libxml2 : 2.9.12
cssselect : 1.1.0
parsel : 1.6.0
w3lib : 1.22.0
Twisted : 22.4.0
Python : 3.10.7 (tags/v3.10.7:6cc6b13, Sep 5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)]
pyOpenSSL : 22.0.0 (OpenSSL 3.0.3 3 May 2022)
cryptography : 37.0.2
Platform : Windows-10-10.0.19044-SP0
Additional context
BaseSettings explicitly inherits from MutableMapping and does not redefine setdefault method. Thus, it uses base implementation:
def setdefault(self, key, default=None):
'D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D'
try:
return self[key]
except KeyError:
self[key] = default
return defaultBase implementation refers to self[key] which is in fact self.__getitem__[key]. BaseSettings has own __getitem__ implementation:
def __getitem__(self, opt_name):
if opt_name not in self:
return None
return self.attributes[opt_name].valueAnd here is the root of the problem: when passed key is not present, __getitem__ returns None, and setdefault follows.
Solution
Implement own setdefault method. An example with matching signature:
def setdefault(self, opt_name, default=None):
if opt_name not in self:
self.set(opt_name, default)
return default
return self.attributes[opt_name].valuepriority='project' argument can be added although this changes signature.
Other way is to inherit from Mapping instead of MutableMapping if this method and other base methods are redundant.
Current workaround
Convert BaseSettings object to a dictionary and only then use setdefault.