feat: fetch multiple proxies at once via /random?count=N (#194)#232
Closed
Germey wants to merge 1 commit into
Closed
feat: fetch multiple proxies at once via /random?count=N (#194)#232Germey wants to merge 1 commit into
Germey wants to merge 1 commit into
Conversation
- redis: add RedisClient.randoms(count, ...) which prefers max-score proxies and falls back to rank, returning a de-duplicated random sample - server: /random accepts a count parameter and returns N random proxies (one per line); count<=1 keeps the original single-proxy behavior; works with key too - README: document the new count parameter Closes #194
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
Issue 收敛计划 Phase 4a(增强:一次获取多个代理),对应 #194。
需求(#194)
多线程爬虫希望一次请求拿到多个代理,减少对
/random的调用次数。改动
storages/redis.py:新增RedisClient.randoms(count, ...),优先返回满分代理,不足时按分数从高到低回退,最终用random.sample去重随机抽取,返回List[Proxy]。processors/server.py:/random支持count参数,一次返回多个随机代理(每行一个)。count不传或 ≤1 时行为完全不变(返回单个代理);可与key组合,子池不足时沿用PROXY_RAND_KEY_DEGRADED的降级逻辑。README.md:补充count参数说明。用法
验证
server.py/redis.py编译通过。count > 可用数量时返回全部可用(min(count, len)封顶);count<=1走原单代理分支。关联 issue