Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests broken with Twisted 22.1.0 #5400

Closed
wRAR opened this issue Feb 7, 2022 · 2 comments · Fixed by #5405
Closed

Tests broken with Twisted 22.1.0 #5400

wRAR opened this issue Feb 7, 2022 · 2 comments · Fixed by #5405
Labels

Comments

@wRAR
Copy link
Member

wRAR commented Feb 7, 2022

ImportError: cannot import name 'PayloadResource' from 'twisted.web.test.test_webclient'

ImportError: cannot import name 'ForeverTakingResource' from 'twisted.web.test.test_webclient'

@wRAR
Copy link
Member Author

wRAR commented Feb 7, 2022

The direct failures are caused by removal of twisted.web.test.test_webclient. The only things we used from there are several Resource classes used in tests/mockserver.py, tests/test_webclient.py and tests/test_downloader_handlers.py. Not a problem, but related: tests/test_webclient.py has a modified copy of twisted.web.client.getPage removed in the same release.

I'm going to run tests with these class definitions copied from the old Twisted code.

@wRAR
Copy link
Member Author

wRAR commented Feb 7, 2022

The tests worked, so Scrapy itself is unlikely to be impacted and the tests can be fixed by copying that code. Still, I wonder if there is something better in Twisted that would allow us to have the same tests without relying on this code.

twisted.web.test.test_webclient contained "Tests for the old twisted.web.client APIs, getPage and friends.". getPage was deprecated in 17.1.0 "in favour of treq and twisted.web.client.Agent". While we use the latter for the HTTP/1.1 download handler, we keep some old Twisted code in scrapy.core.downloader.webclient.ScrapyHTTPClientFactory which is used by the HTTP/1.0 handler. In any case, both tests/test_webclient.py and tests/test_downloader_handlers.py test various Scrapy HTTP clients by providing these resource classes inside a twisted.web.static.File object and this doesn't look like something affected by these deprecations at all. I've checked twisted/web/test/test_agent.py and I think it just uses a different approach, not actually running a HTTP server, but I'm not sure.

So maybe the easiest way is still just copying the resource classes and not doing anything else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant