Problem with Twisted AttributeError: 'EPollReactor' object has no attribute '_handleSignals'

[RobertoHigor]

Description

Since the release of Twisted 23.8.0 recently, I've had problems running Scrapy. I had to fix to the previous version, Twisted==22.10.0

Steps to Reproduce

1. Install Scrapy in a new environment
2. Run .start() from an CrawlerProcess object

Expected behavior: Scrapy to crawl

Actual behavior:

Reproduces how often: Always in the new version

Versions

Scrapy>=2.9.0

Additional context

I've seen this fix on stackoverflow

[mrtornado]

I'm having the same issue, but if you revert back to pip install Twisted==22.10.0 it's working.

[GeorgeA92]

I assume that changes from twisted side that caused this issue came from this issue twisted/twisted#11751 and this pull request twisted/twisted#11752

[wRAR]

This affects both scrapy crawl and direct usage of CrawlerProcess.

[wRAR]

Looks like we are calling reactor._handleSignals() in scrapy.utils.ossignal.install_shutdown_handlers() so that we can override the Twisted handlers. Twisted itself calls reactor._handleSignals() in _SignalReactorMixin._reallyStartRunning(), I assume this happens later.

It seems that the equivalent code in new Twisted is in twisted.internet._signals._WithSignalHandling.install() but this is even more private and even more tied to internal logic so I hesitate to use it. It may be possible to redo our logic instead.

Unfortunately, this isn't covered by tests and deleting reactor._handleSignals() doesn't fail them :)

Overall, this looks like releasing a version with the Twisted version restriction is much more viable than quickly fixing this.

[wRAR]

After looking at the code for some more I think we need to check the order of handler installation in Scrapy with older Twisted in order to understand how it works. For example, I'm not sure how do other handlers work if Twisted resets them on reactor.run() (as the comments in the new logic suggest).

[glyph]

Just popping my head in here to say that this might be something to discuss on the Twisted mailing list, if there are requirements that Scrapy has for signal handling that we've missed!

https://mail.python.org/mailman3/lists/twisted.python.org/

[GeorgeA92]

Prepared pull request with removed failing call of _handleSignals - with this change scrapy works on my local environment as expected (both earlier 22.10.0 and new 23.8.0 version of twisted checked locally). At least it looks like expected from my first view.

Also it is noteable fact that the most of scrapy code parts related to handling these signals(/scrapy/utils/ossignal.py, /scrapy/crawler.py) looks nearly the same as 10+ years ago when It appeared.

So I assume that this call of _handleSignals was needed for.. much earlier versions of twisted however I am not aware how twisted looked like 10+ years ago (I am not aware of deep internals of recent versions of twisted as well)