Easy to break installing the custom reactor

[wRAR]

I created a spider that imports scrapy.mail and got "wrong reactor installed" after setting TWISTED_REACTOR. This is because scrapy.mail imports twisted.internet.reactor at the top level and because scrapy crawl imports all spider modules.

So I think we should kill all top-level twisted.internet.reactor imports (it was partially done when adding the feature, so that importing CrawlerProcess doesn't install the reactor) and add documentation that the user code shouldn't do similar top-level imports.

[adityaa30]

I am working on this issue.

[elacuesta]

True, this is what happened to me in scrapinghub/testspiders#10, I didn't realize there were still conflicting imports within Scrapy itself.

[adityaa30]

@wRAR I have tried a lot to get "wrong reactor installed" error but cannot get it. Can you provide an example?

[wRAR]

@adityaa30 set TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor' in settings.py and add import scrapy.mail in your spider.

[adityaa30]

@wRAR I have found out that, even if don't do import scrapy.mail and execute scrapy crawl quotes with TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor' set in my settings.py. I still get the error:

I feel this is because twisted.internet.reactor does:

from __future__ import division, absolute_import
import sys
del sys.modules['twisted.internet.reactor']
from twisted.internet import default
default.install()

I want to know If I am going in correct direction?

[wRAR]

Then it looks like some other code imports twisted.internet.reactor in your test.

[adityaa30]

@wRAR Okay thanks! I think I have a good idea now what the exact issue is. I will soon send a pr fixing this issue. Also, shall I add in documentation that the user code shouldn't do similar top-level imports in the same pr? If yes, where in docs should this be added?

[wRAR]

The docs should say what can happen, why, and how to prevent it. It's also not just about user code but also about 3rd part libraries that are imported in user code. I think a good place for this is in https://docs.scrapy.org/en/latest/topics/settings.html#std:setting-TWISTED_REACTOR

[adityaa30]

Okay, working on it.

[adityaa30]

@wRAR I have sent a pr #4406 fixing this issue.

[nyov]

Should/could we perhaps wrap and replace the twisted.internet.reactor function with a custom lazy-loading reactor-loading function instead? I'm not sure how feasible that is, but it seems there are quite a few twisted.internet.reactor imports still necessary, and maybe not all of them can always be moved around into a method call.

[adityaa30]

@nyov I think this is possible. A custom class which takes a library/package name as string with customized attribute accessing. We can override __getattr__ method to check if the target library/package is loaded. If it is not loaded, we will use importlib.import_module to load the library/package and get the attribute we need.

For example:

import importlib

class LazyLoader:
    def __init__(self, lib_name):
        self.lib_name = lib_name
        self._mod = None

    def __getattr__(self, name):
        if self._mod is None:
            self._mod = importlib.import_module(self.lib_name)
        return getattr(self._mod, name)

Now, LazyLoader can be used to lazy-load twisted.internet.reactor module by declaring at top level:

reactor = LazyLoader('twisted.internet.reactor')

Upd: @wRAR this may be a good idea as we can declare reactor at top level.