Skip to content
This repository has been archived by the owner on Mar 12, 2024. It is now read-only.

fix event loop is closed in win10 #69

Merged
merged 3 commits into from
Jan 16, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,7 @@
## 4.1.0(2021-01-11)
* 功能
- aio http session使用自定义timeout,可以和cache兼容使用(默认的timeout为什么不行,原因不明……);

## 4.1.1(2021-01-16)
* bugfix
- 修复(workaround)win下结束爬取后报错的问题
4 changes: 4 additions & 0 deletions src/__main__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import sys
import datetime

from src.crawl import item_crawler
Expand All @@ -6,6 +7,9 @@

if __name__ == '__main__':

if sys.version_info[:2] == (3, 7):
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())

# start
start = datetime.datetime.now()
log.info("Start Time: {}".format(start))
Expand Down
5 changes: 2 additions & 3 deletions src/crawl/item_crawler.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import re
import asyncio
import math
import asyncio

# from tqdm import tqdm

Expand Down Expand Up @@ -59,8 +59,7 @@ def csgo_all_categories():
def enrich_item_with_price_history(csgo_items, crawl_steam_async=True):
# crawl price for all items
if crawl_steam_async:
# asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

直接取消这行注释不就可以了吗?我之前做这个的时候这里是没注释的, @puppylpg 注释掉的吧
然后现在新版本不推荐使用run_until_complete了,我之前测试的时候先set_event_loop_policy再run没报错,建议没有特殊问题的话还是还是用asyncio.run

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

之前注释掉是因为去掉跑着并没有什么问题,我就把windows specific code删了 😢

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

问题不大,能用就行
我也是测试没发现问题,完全没注意到这里改了

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aio-libs/aiohttp#4324 (comment)
应该是Win下面aiohttp的一个Bug,他说3.7已经解决了,然而我这也是3.7,错误情况和他3.6的一模一样,而且同样的workaround可以绕过去..

asyncio.run(history_price_crawler.async_crawl_history_price(csgo_items))
asyncio.get_event_loop().run_until_complete(history_price_crawler.async_crawl_history_price(csgo_items))
else:
history_price_crawler.crawl_history_price(csgo_items)
return csgo_items
Expand Down