get_project_settings of doesn't reflect per-spider settings

[mohmad-null]

It's possible I'm using this wrong. But I'm doing the following:

class MySpider (Spider)
  def __init__(self):
		custom_settings = {
			'AUTOTHROTTLE_ENABLED': False
		}

  a = scrapy.utils.project.get_project_settings()
  print(a.copy_to_dict())

My settings.py has AUTOTHROTTLE_ENABLED: True.

Based on my reading of https://doc.scrapy.org/en/latest/topics/settings.html#populating-the-settings - the custom_settings value should override the value of settings.py. But the result of the above is:

....
AUTOTHROTTLE_ENABLED: True
...

It's using the value from settings.py - am I misunderstanding this or is it a bug? I was hoping to load and alter settings dynamically during the init.

[IAlwaysBeCoding]

You need to put customs_settings as a class attribute, outside your __init__() method because you are setting it up as an instance attribute. The way to use get_project_settings() is so you can load up your settings from settings.py from anywhere in your Scrapy project. When you call that method you are getting all of your settings from your project. In other words, that method does not override any settings at all, it only retrieves your settings from settings.py.

[mohmad-null]

@IAlwaysBeCoding - Thanks for the correction on custom_settings - I can now see that, looking again at the settings page.

Looking again at the documentation, I can see that the "project settings" is what actually the name for the settings.py settings. Given that, your information does make sense, though the documentation could benefit from spelling that out explicitly.

[redapple]

@mohmad-null , what would you change to https://doc.scrapy.org/en/latest/topics/settings.html#project-settings-module ?

[mohmad-null]

@redapple
https://doc.scrapy.org/en/latest/topics/settings.html#populating-the-settings - on that under the "3. Project settings module" bit, I'd probably add a line "get_project_settings can be used to access these settings" - which links to the documentation for get_project_settings

https://doc.scrapy.org/en/latest/topics/practices.html (which seems to be the documentation for "get_project_settings" - it's not documented on either the settings page or the api page) - I'd mention that settings loaded this way are not updated globally (so it can't be used in conjunction with os.environ['SCRAPY_SETTINGS_MODULE'] = 'myproject.settings' within the spider).