Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

【错误回报】动态推送出现验证码,且高频率复现 #366

Closed
forchannot opened this issue May 11, 2023 · 39 comments
Closed

【错误回报】动态推送出现验证码,且高频率复现 #366

forchannot opened this issue May 11, 2023 · 39 comments
Labels
bug Something isn't working
Milestone

Comments

@forchannot
Copy link

forchannot commented May 11, 2023

运行环境

win server 2022

HarukaBot 版本

1.5.4

go-cqhttp 版本

1.0.1

错误描述

动态推送出现验证码
Screenshot_2023-05-11-12-12-14-728_com tencent mobileqq
Screenshot_2023-05-11-12-12-19-689_com tencent mobileqq

报错日志(如果有)

05-11 12:02:47 [ERROR] haruka_bot | 截取动态时发生错误:https://t.bilibili.com/794307828895449208
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\Scripts\hb.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\cli\__init__.py", line 16, in run
    run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\cli\bot.py", line 35, in run
    nonebot.run(app="haruka_bot.cli.bot:app")
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\nonebot\__init__.py", line 309, in run
    get_driver().run(*args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\nonebot\drivers\fastapi.py", line 198, in run
    uvicorn.run(
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\uvicorn\main.py", line 568, in run
    server.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\uvicorn\server.py", line 59, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 628, in run_until_complete
    self.run_forever()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\windows_events.py", line 316, in run_forever
    super().run_forever()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 595, in run_forever
    self._run_once()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 1881, in _run_once
    handle._run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\apscheduler\executors\base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\plugins\pusher\dynamic_pusher.py", line 74, in dy_sched
    image = await get_dynamic_screenshot(dynamic_id)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\utils\browser.py", line 42, in get_dynamic_screenshot
    return await get_dynamic_screenshot_pc(dynamic_id)
> File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\utils\browser.py", line 142, in get_dynamic_screenshot_pc
    assert card
AssertionError
@boxie123
Copy link
Contributor

#364

@boxie123
Copy link
Contributor

目前小样本实验中,直接访问 https://m.bilibili.com/opus/{dynamic_id} 不会出现验证码,而通过 https://m.bilibili.com/dynamic/{dynamic_id}https://t.bilibili.com/{dynamic_id} 跳转到前者会大概率出现验证码。

需更多样本测试

可将移动端截图功能中的URL添加opus/暂时避免此问题。

@forchannot
Copy link
Author

那电脑端有解决办法吗

@boxie123
Copy link
Contributor

电脑端同样存在 https://www.bilibili.com/opus/{dynamic_id} 链接,但我没测试过

(莫不是阿b终于发现动态翻译成dynamic太蠢了)

@forchannot
Copy link
Author

async def get_dynamic_screenshot_mobile(dynamic_id):
    """移动端动态截图"""
    url = f"https://m.bilibili.com/dynamic/{dynamic_id}"

async def get_dynamic_screenshot_pc(dynamic_id):
    """电脑端动态截图"""
    url = f"https://t.bilibili.com/{dynamic_id}"

请问下是把上述两个部分分别改为
https://m.bilibili.com/opus/{dynamic_id}https://t.bilibili.com/opus/{dynamic_id}

@boxie123
Copy link
Contributor

移动端可以直接改,我刚看了一下电脑端改链接的话样式就变了,更接近移动端样式,不存在.card,所以仍然会截全图

@boxie123
Copy link
Contributor

如果一定要用电脑端的话,或许可以添加:

if await page.query_selector(".geetest_panel"):
    return None

来避免截图,并在下一次爬取时重试。

@forchannot
Copy link
Author

好的,感谢,我试一下,我发现好像www.bilibili.com/opus/会直接跳转到t.bilibili.com/,似乎并没有出现样式改变

@boxie123
Copy link
Contributor

好的,感谢,我试一下,我发现好像www.bilibili.com/opus/会直接跳转到t.bilibili.com/,似乎并没有出现样式改变

群友测试反馈改成/opus/会有很多问题😫,例如:

  • 跳转到404页面重定向不用301,改用302,hb没有识别,所以会截到404页面
  • 除了纯文字和图文动态(存疑),其他还会跳转回/dynamic/链接

所以还是建议用刷新重试的方法😢:

if await page.query_selector(".geetest_panel"):
    return None

@boxie123
Copy link
Contributor

另外,修改UA可能会降低出验证码的概率

@forchannot
Copy link
Author

这行代码加在哪里呢

@boxie123
Copy link
Contributor

这行代码加在哪里呢

我改了一些地方所以不确定原本应该是第几行,但你应该能在get_dynamic_screenshot_pcget_dynamic_screenshot_mobile函数内部找到判断404页面的两行代码,例如:

# 动态被删除或者进审核了
if page.url == "https://m.bilibili.com/404":
     return None

加到这两行代码的前面或后面都可以。

@forchannot
Copy link
Author

好的感谢

@boxie123
Copy link
Contributor

期待反馈

@forchannot
Copy link
Author

目前关注的主播都还没有更新动态,等动态更新样本多了再来反馈

@forchannot
Copy link
Author

今天一天都还挺稳定的,不知道是因为我改了ua还是加了

if page.url == "https://m.bilibili.com/404":
     return None

的原因

@Sevenyine
Copy link

今天一天都还挺稳定的,不知道是因为我改了ua还是加了

if page.url == "https://m.bilibili.com/404":
     return None

的原因

今天不是某个二字游戏前瞻,我这边仍然有验证码
看来是lz的解决方法有效果了
5FB7D396-165E-4660-8D40-2201F58BA79A

@forchannot
Copy link
Author

今天前瞻十几个都没遇到验证码,暂时稳定,ua我是直接去抓的物理手机上的真实ua,然后也加上了那段代码

@zszcq
Copy link

zszcq commented May 15, 2023

我这边使用的pc端截图,但是我发现源代码没有PC端的UA设置,加上之后验证码消失了,目前恢复正常,需要再观察几天.
ps:我试过了部署到其他ip的服务器,确认ip地址不同,依然无法解决这个问题,确认不是ip引起的风控.
pps:我修改了移动端的地址,也就是改为opus,配置文件改为使用移动端后,发现样式和pc端一样,目前不知道是配置文件未生效还是其它问题,待验证

1684166266393

@boxie123
Copy link
Contributor

pps:我修改了移动端的地址,也就是改为opus,配置文件改为使用移动端后,发现样式和pc端一样,目前不知道是配置文件未生效还是其它问题,待验证

  • 不建议修改地址,会截到404页面
  • hb不支持fastapi_reload,修改配置文件后需手动重启

@AyasakiAtre
Copy link

mobile加上刷新重试的代码的话会无限获取动态,然而一直发不出来。换成pc后成功发出并且无验证码。
ps:mobile这边加不加/opus/都会无限获取

@zszcq
Copy link

zszcq commented May 16, 2023

我一直没用移动端,也只是改了代码,一直用的电脑端截图,反正是电脑端添加上UA就正常了

@boxie123
Copy link
Contributor

mobile加上刷新重试的代码的话会无限获取动态,然而一直发不出来。换成pc后成功发出并且无验证码。 ps:mobile这边加不加/opus/都会无限获取

修改UA可降低出现验证码概率

@boxie123
Copy link
Contributor

#372

@SK-415 SK-415 added this to the v1.5.5 milestone May 21, 2023
@SK-415 SK-415 added the bug Something isn't working label May 21, 2023
@SunshineboyZj
Copy link
Contributor

手机样式,尝试更改UA,可以解决,一两天后也会重新出现验证码。目前我的解决办法就是一天改一次UA。

@KutuyLovch
Copy link

手机样式,尝试更换了opus和修改了AU,还添加了代码,目前动态截图稳定,就是视频推送动态依旧是验证码

@QwindF
Copy link

QwindF commented Jun 5, 2023

image

@QwindF
Copy link

QwindF commented Jun 5, 2023

image

06-05 12:12:22 [ERROR] haruka_bot | 截取动态时发生错误https://m.bilibili.com/dynamic/803588020310638661
Traceback (most recent call last):
  File "E:\IDEAyes\BOT\nonebot2\bot.py", line 60, in <module>
    nonebot.run(app="__mp_main__:app")
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\nonebot\__init__.py", line 309, in run
    get_driver().run(*args, **kwargs)
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\nonebot\drivers\fastapi.py", line 198, in run
    uvicorn.run(
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\uvicorn\main.py", line 569, in run
    server.run()
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\uvicorn\server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 600, in run_forever
    self._run_once()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 1896, in _run_once
    handle._run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\apscheduler\executors\base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\haruka_bot\plugins\pusher\dynamic_pusher.py", line 74, in dy_sched
    image = await get_dynamic_screenshot(dynamic_id)
  File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\haruka_bot\utils\browser.py", line 40, in get_dynamic_screenshot
    return await get_dynamic_screenshot_mobile(dynamic_id)
> File "E:\IDEAyes\BOT\nonebot2\venv\lib\site-packages\haruka_bot\utils\browser.py", line 102, in get_dynamic_screenshot_mobile
    assert card
AssertionError

@Sevenyine
Copy link

修改了UA之后,bot现在是在晚上经常会出现空图或者动态不存在的截图情况…
不知道是ip问题还是UA问题,如果是UA问题,求助一下各位有没有什么办法能够添加多个UA啊(?)谢谢!

@SunshineboyZj
Copy link
Contributor

SunshineboyZj commented Jun 13, 2023 via email

@Sevenyine
Copy link

修改了UA之后,bot现在是在晚上经常会出现空图或者动态不存在的截图情况…
不知道是ip问题还是UA问题,如果是UA问题,求助一下各位有没有什么办法能够添加多个UA啊(?)谢谢!

自己解决了 换了个百度蜘蛛的UA()
用魔法对抗魔法

@SunshineboyZj
Copy link
Contributor

SunshineboyZj commented Jun 19, 2023 via email

@KayaiJia
Copy link

确实,把移动端的UA改成Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko)Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html) 可解

@forchannot
Copy link
Author

如果一定要用电脑端的话,或许可以添加:

if await page.query_selector(".geetest_panel"):
    return None

来避免截图,并在下一次爬取时重试。

加了这玩意好像有概率死循环

06-23 21:05:17 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:05:24 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:05:31 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:05:37 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:05:44 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:05:51 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:05:57 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:03 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:10 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:16 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:22 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:28 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:34 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)
06-23 21:06:40 [INFO] haruka_bot | 检测到新动态(810386176411172886):原神(401742377)

今天前瞻,就一直这样

@djkcyl
Copy link
Collaborator

djkcyl commented Jun 27, 2023

fix 分支修复了该问题

@djkcyl djkcyl closed this as completed Jun 28, 2023
@SasoriXizixu
Copy link

SasoriXizixu commented Jun 28, 2023

fix 分支修复了该问题

尝试了删除原文件并使用fix分支重新安装bot,仍然弹出验证码。不过此前该issue中所有解决方法都尝试过均未能解决问题

@djkcyl
Copy link
Collaborator

djkcyl commented Jun 28, 2023

@SasoriXizixu

新分支理论上100%解决了 mobile 的截图验证码问题,你这个失败请发下日志,不然我给你抽张塔罗牌算算命?

@djkcyl
Copy link
Collaborator

djkcyl commented Jun 28, 2023

fix 分支修复了该问题

尝试了删除原文件并使用fix分支重新安装bot,仍然弹出验证码。不过此前该issue中所有解决方法都尝试过均未能解决问题

另外,fix分支一直有新更新,可以再拉下新代码试试

@SasoriXizixu
Copy link

SasoriXizixu commented Jun 28, 2023

@SasoriXizixu

新分支理论上100%解决了 mobile 的截图验证码问题,你这个失败请发下日志,不然我给你抽张塔罗牌算算命?

实在不好意思,以下是日志

06-28 12:15:21 [ERROR] haruka_bot | 截取动态时发生错误:https://m.bilibili.com/dynamic/812123812542808089
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\Scripts\hb.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\cli\__init__.py", line 16, in run
    run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\cli\bot.py", line 35, in run
    nonebot.run(app="haruka_bot.cli.bot:app")
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\nonebot\__init__.py", line 309, in run
    get_driver().run(*args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\nonebot\drivers\fastapi.py", line 198, in run
    uvicorn.run(
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\uvicorn\main.py", line 578, in run
    server.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\uvicorn\server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 600, in run_forever
    self._run_once()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 1896, in _run_once
    handle._run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\apscheduler\executors\base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\plugins\pusher\dynamic_pusher.py", line 72, in dy_sched
    image = await get_dynamic_screenshot(dynamic_id)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\utils\browser.py", line 38, in get_dynamic_screenshot
    return await get_dynamic_screenshot_mobile(dynamic_id)
> File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\haruka_bot\utils\browser.py", line 100, in get_dynamic_screenshot_mobile
    assert card
AssertionError

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet