feat(doubao): capture conversation_url and classify captcha blocks - #63
Conversation
- after a successful ask, best-effort `doubao status` to grab the active conversation URL (https://www.doubao.com/chat/<id>); status failure never fails the collect - classify captcha/verification-wall errors as error_type "captcha_challenge" so runners can apply cooldown/retry policy - keep default site_session=ephemeral (fresh conversation per ask); add capture_conversation_url config flag (default true) - unit tests: conversation_url parsing (id / root / garbage), status captured + tolerated, captcha classified vs generic
|
⏳ Repowise has not indexed this repository yet No analysis on this PR because there is no index to compare against. Indexing usually runs automatically after install; if this persists, start it from the dashboard. |
📝 WalkthroughSummary by CodeRabbit
WalkthroughDoubao research collection now detects CAPTCHA challenges, retrieves conversation URLs from status output, and stores them in successful results. Status failures remain non-fatal. Tests cover URL parsing, error classification, collection behavior, and source configuration. ChangesDoubao collection updates
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Collector
participant DoubaoAsk
participant DoubaoStatus
Collector->>DoubaoAsk: submit research question
DoubaoAsk-->>Collector: answer or command error
alt successful answer
Collector->>DoubaoStatus: request status as JSON
DoubaoStatus-->>Collector: conversation URL or status failure
Collector-->>Collector: store conversation_url
else CAPTCHA marker detected
Collector-->>Collector: classify as captcha_challenge
end
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/channels/doubao_research_channel.py`:
- Line 166: Update the tuple unpacking in the command execution flow around
_run_doubao_command to discard the unused status-error value instead of binding
it to se, resolving Ruff RUF059 while preserving rc and so handling.
- Around line 57-59: Update the URL extraction logic around the visible Url/url
lookup so it only returns a valid HTTPS Doubao conversation URL matching
www.doubao.com/chat/<id>. Reject HTTP URLs, non-Doubao hosts, and paths with no
non-empty conversation ID before assigning the result to conversation_url.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 144ddca8-ab5a-4e4c-8fb7-8dd65c8c8ea2
📒 Files selected for processing (2)
backend/channels/doubao_research_channel.pytests/unit/channels/test_doubao_research_channel.py
| url = str(row.get("Url", row.get("url", "")) or "").strip() | ||
| if "/chat/" in url: | ||
| return url |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Validate the extracted conversation URL.
The predicate accepts http URLs, non-Doubao hosts, and /chat/ with no conversation ID. The method then stores these values in conversation_url, although its contract requires an HTTPS www.doubao.com/chat/<id> URL.
Proposed fix
+from urllib.parse import urlsplit
+
- if "/chat/" in url:
+ parsed = urlsplit(url)
+ chat_id = parsed.path.removeprefix("/chat/").strip("/")
+ if (
+ parsed.scheme == "https"
+ and parsed.hostname == "www.doubao.com"
+ and chat_id
+ and "/" not in chat_id
+ ):
return url📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| url = str(row.get("Url", row.get("url", "")) or "").strip() | |
| if "/chat/" in url: | |
| return url | |
| from urllib.parse import urlsplit | |
| url = str(row.get("Url", row.get("url", "")) or "").strip() | |
| parsed = urlsplit(url) | |
| chat_id = parsed.path.removeprefix("/chat/").strip("/") | |
| if ( | |
| parsed.scheme == "https" | |
| and parsed.hostname == "www.doubao.com" | |
| and chat_id | |
| and "/" not in chat_id | |
| ): | |
| return url |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/channels/doubao_research_channel.py` around lines 57 - 59, Update the
URL extraction logic around the visible Url/url lookup so it only returns a
valid HTTPS Doubao conversation URL matching www.doubao.com/chat/<id>. Reject
HTTP URLs, non-Doubao hosts, and paths with no non-empty conversation ID before
assigning the result to conversation_url.
| str(config.get("site_session", "ephemeral")), | ||
| ] | ||
| try: | ||
| rc, so, se = await _run_doubao_command(status_command) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove the unused status-error binding.
Line 166 binds se but does not use it. Ruff reports RUF059.
Proposed fix
- rc, so, se = await _run_doubao_command(status_command)
+ rc, so, _ = await _run_doubao_command(status_command)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| rc, so, se = await _run_doubao_command(status_command) | |
| rc, so, _ = await _run_doubao_command(status_command) |
🧰 Tools
🪛 Ruff (0.16.1)
[warning] 166-166: Unpacked variable se is never used
Prefix it with an underscore or any other dummy variable pattern
(RUF059)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/channels/doubao_research_channel.py` at line 166, Update the tuple
unpacking in the command execution flow around _run_doubao_command to discard
the unused status-error value instead of binding it to se, resolving Ruff RUF059
while preserving rc and so handling.
Source: Linters/SAST tools
2 participants
关联
Fixes / 部分实现 #62（doubao_research 批量采集实测反馈）
改动
backend/channels/doubao_research_channel.py：捕获会话 URL（P0）：ask 成功后，best-effort 执行一次
opencli doubao status -f json，提取https://www.doubao.com/chat/<id>写入输出conversation_url字段。capture_conversation_url（默认 true）可关闭site_session=ephemeral（opencli 1.8.6 的 ephemeral = 每次全新浏览器会话，天然每词新对话，无需显式 new）验证码错误分类（P1）：ask 非零退出且 stderr 含验证码特征（
verification challenge/captcha/blocked the request/中文标记）时，error_type="captcha_challenge"——runner 可据此做冷却重试或人工介入策略，而不是当作永久失败。普通错误仍返回error_type=None。为什么
实测 240 词批量采集（见 #62 评论）确认：
测试
tests/unit/channels/test_doubao_research_channel.py新增 8 个用例（原有 4 个全保留）：_conversation_url：提取 chat id / 忽略根 /chat / 容错垃圾输入captcha_challenge；普通错误error_type=None