Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set WindowsSelectorEventLoopPolicy on Windows. #5315

Merged
merged 9 commits into from
Dec 31, 2021

Conversation

wRAR
Copy link
Member

@wRAR wRAR commented Nov 11, 2021

Closes: #4976

Needs fixing tests that expect a default loop to be used, and adding docs, including manual setting for CrawlerProcess/Runner in https://docs.scrapy.org/en/latest/topics/asyncio.html#installing-the-asyncio-reactor

@codecov
Copy link

codecov bot commented Nov 11, 2021

Codecov Report

Merging #5315 (9b3dacf) into master (57dc581) will decrease coverage by 0.03%.
The diff coverage is 20.00%.

@@            Coverage Diff             @@
##           master    #5315      +/-   ##
==========================================
- Coverage   88.59%   88.56%   -0.04%     
==========================================
  Files         163      163              
  Lines       10619    10624       +5     
  Branches     1807     1809       +2     
==========================================
+ Hits         9408     9409       +1     
- Misses        936      939       +3     
- Partials      275      276       +1     
Impacted Files Coverage Δ
scrapy/utils/reactor.py 76.92% <20.00%> (-4.75%) ⬇️

@wRAR
Copy link
Member Author

wRAR commented Nov 12, 2021

Tests that hang on old Twisted because of https://twistedmatrix.com/trac/ticket/9390 also hang on Windows with these changes, I need to investigate the reason.

@wRAR wRAR marked this pull request as ready for review November 23, 2021 12:36
@wRAR wRAR changed the title [WIP] Set WindowsSelectorEventLoopPolicy on Windows. Set WindowsSelectorEventLoopPolicy on Windows. Nov 23, 2021
@wRAR
Copy link
Member Author

wRAR commented Nov 23, 2021

Looks like there are several tests getting 100% reproducible unclean reactor problems only on windows-3.10-asyncio.

@wRAR
Copy link
Member Author

wRAR commented Nov 24, 2021

I've filed https://twistedmatrix.com/trac/ticket/10279 and skipped those tests.

docs/topics/asyncio.rst Outdated Show resolved Hide resolved
scrapy/utils/reactor.py Show resolved Hide resolved
tests/test_downloader_handlers.py Show resolved Hide resolved
@Gallaecio Gallaecio merged commit c5ab580 into scrapy:master Dec 31, 2021
@wRAR wRAR deleted the asyncio-windows-loop branch May 8, 2023 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix and document asyncio reactor problems on Windows
2 participants