为何每次抓取的信息很少，而且不按时间要求抓取

[bobopu]

1.已经要求抓取发布日期在2025年3月2日以后的，抓来的信息还是2022和2024年的。
2.每次抓取的信息很少就只有几条而且内容还是重复的。

[Tshenhua]

您这个.env里边的配置能给一份么，主要是不知道LLM_API_KEY=""
LLM_API_BASE="https://api.siliconflow.cn/v1"

for the search tool

ZHIPU_API_KEY=""
PRIMARY_MODEL="Qwen/Qwen2.5-32B-Instruct"
SECONDARY_MODEL="Qwen/Qwen2.5-14B-Instruct"
这几个参数 怎么配置

[bigbrother666sh]

@bobopu 你用的主模型是什么

[bobopu]

@bobopu你用的模型是什么

windows10下Dokcer部署，用的硅基流动的api，

LLM_API_KEY="sk-vywaipqeexxluqxxvaajunzgvajmodroulbcwtee******"
LLM_API_BASE="https://api.siliconflow.cn/v1"
ZHIPU_API_KEY="ec2879ef43614b46868fb147b7272c8d.jVkk3XQ******"
PRIMARY_MODEL="Qwen/Qwen2.5-32B-Instruct"
SECONDARY_MODEL="Qwen/Qwen2.5-14B-Instruct"
VL_MODEL="Pro/Qwen/Qwen2-VL-7B-Instruct"
PB_SUPERUSER_EMAIL="test@163.com"
PB_SUPERUSER_PASSWORD="1234567890" #no '&' in the password and at least 10 characters
PROJECT_DIR="./core/work_dir"

[bobopu]

@bobopu 你用的主模型是什么
我用0.39版本后现在可以持续爬了，并且把VL_MODEL切换成了
VL_MODEL="deepseek-ai/deepseek-vl2"
但我发现重复爬取的问题依然存在
例如第4条和最后一条是同一个url
https://caifuhao.eastmoney.com/news/20250303190320124121260?from=guba&name=5Zu65oCB55S15rGg5ZCn&gubaurl=aHR0cHM6Ly9ndWJhLmVhc3Rtb25leS5jb20vbGlzdCxCSzA5NjguaHRtbA%3D%3D

第二个问题是我如果不设置目标源，例如第一个ponits“A股金银河的相关信息”则不会抓取，infos里全是第二个ponits的信息。

此外，还有个问题。抓取停止是系统自行判断目标源已经没有多余内容可抓后自动停还是什么机制？如果我设置每小时抓取一次，一个小时还没抓取完，第二小时的任务又开始了会怎样？

[bigbrother666sh]

1、你试试把focuspoint 改为 “最新发布的固态电池信息”，explanation 维持不变；
2、是的，0.3.7之后都是，一定要为 focuspoint 单独指定信源，只会从指定的信源中挖取（不指定就不会执行）；
3、自动判定，如果前一小时的没执行完，第二小时就跳过了。

[bobopu]

1.发现新问题，focuspoint 改为 “最新发布的固态电池信息”后从指定源抓取的是2025年的信息，但是自主搜索的信息不受explanation控制，有24年之前的。

2.从抓取回来的内容来看，似乎遗漏了原文的主要关键内容。原文链接 https://news.bjx.com.cn/html/20241126/1412950.shtml

3.另外，如果设定每天定时抓取这个源且focuspoint和explanation都不变，每次抓取都要把全站重新扫一遍吗？前一天已经抓取过的URL还会去扫描抓取吗？或者还是有类似增量更新的机制？

[bigbrother666sh]

1、近期会推送改进代码；
2、你的关注点设定就是“最新发布的固态电池信息”，从原文来看，ai总结了题目和主要内容，并无出入啊
你指的遗漏的重要内容，具体是哪些啊？
3、不会，已经收录数据库的会自动跳过，没有收录但是已经爬过的会用缓存。但是 sites（也就是入口页面，每天都会重新扫描）

[bigbrother666sh]

pls update the code(0.3.9-patch1) and try again
look forward to here the feedback!

Good Luck @bobopu

[bobopu]

pls update the code(0.3.9-patch1) and try again look forward to here the feedback!
Good Luck @bobopu

更新到0.3.9-patch1版本后不爬了，看起来像是提取相关url的时候提取出来的格式不正确url被拼接了多次，就停了，什么也没抓到。

2025-03-08 23:38:08 core-1 | Installing collected packages: snowballstemmer, sgmllib3k, pyperclip, faust-cchardet, zipp, xxhash, urllib3, typing-extensions, tqdm, soupsieve, sniffio, rpds-py, regex, pyyaml, python-dotenv, pygments, pycparser, psutil, propcache, pillow, packaging, numpy, mdurl, MarkupSafe, lxml, loguru, joblib, jiter, idna, humanize, h11, greenlet, fsspec, frozenlist, feedparser, fake-useragent, fake-http-header, exceptiongroup, distro, cssselect, colorama, click, charset-normalizer, certifi, attrs, async-timeout, annotated-types, aiohappyeyeballs, aiofiles, requests, referencing, rank-bm25, pyee, pydantic-core, nltk, multidict, markdown-it-py, jinja2, importlib-metadata, httpcore, cffi, beautifulsoup4, anyio, aiosqlite, aiosignal, yarl, tiktoken, rich, pydantic, playwright, jsonschema-specifications, huggingface-hub, httpx, cryptography, tokenizers, tf-playwright-stealth, pyOpenSSL, pocketbase, openai, jsonschema, aiohttp, litellm, crawl4ai 2025-03-08 23:38:17 core-1 | Successfully installed MarkupSafe-3.0.2 aiofiles-24.1.0 aiohappyeyeballs-2.5.0 aiohttp-3.11.13 aiosignal-1.3.2 aiosqlite-0.21.0 annotated-types-0.7.0 anyio-4.8.0 async-timeout-5.0.1 attrs-25.1.0 beautifulsoup4-4.13.3 certifi-2025.1.31 cffi-1.17.1 charset-normalizer-3.4.1 click-8.1.8 colorama-0.4.6 crawl4ai-0.5.0.post4 cryptography-44.0.2 cssselect-1.2.0 distro-1.9.0 exceptiongroup-1.2.2 fake-http-header-0.3.5 fake-useragent-2.0.3 faust-cchardet-2.1.19 feedparser-6.0.11 frozenlist-1.5.0 fsspec-2025.3.0 greenlet-3.1.1 h11-0.14.0 httpcore-1.0.7 httpx-0.28.1 huggingface-hub-0.29.2 humanize-4.12.1 idna-3.10 importlib-metadata-8.6.1 jinja2-3.1.6 jiter-0.8.2 joblib-1.4.2 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 litellm-1.63.3 loguru-0.7.3 lxml-5.3.1 markdown-it-py-3.0.0 mdurl-0.1.2 multidict-6.1.0 nltk-3.9.1 numpy-2.2.3 openai-1.65.4 packaging-24.2 pillow-10.4.0 playwright-1.50.0 pocketbase-0.15.0 propcache-0.3.0 psutil-7.0.0 pyOpenSSL-25.0.0 pycparser-2.22 pydantic-2.10.6 pydantic-core-2.27.2 pyee-12.1.1 pygments-2.19.1 pyperclip-1.9.0 python-dotenv-1.0.1 pyyaml-6.0.2 rank-bm25-0.2.2 referencing-0.36.2 regex-2024.11.6 requests-2.32.3 rich-13.9.4 rpds-py-0.23.1 sgmllib3k-1.0.0 sniffio-1.3.1 snowballstemmer-2.2.0 soupsieve-2.6 tf-playwright-stealth-1.1.2 tiktoken-0.9.0 tokenizers-0.21.0 tqdm-4.67.1 typing-extensions-4.12.2 urllib3-2.3.0 xxhash-3.5.0 yarl-1.18.3 zipp-3.21.0 2025-03-08 23:38:17 core-1 | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. 2025-03-08 23:38:17 core-1 | 2025-03-08 23:38:17 core-1 | [notice] A new release of pip is available: 25.0 -> 25.0.1 2025-03-08 23:38:17 core-1 | [notice] To update, run: python -m pip install --upgrade pip 2025-03-08 23:38:19 core-1 | 2025-03-08 15:38:19.079 | DEBUG | utils.pb_api:init:12 - initializing pocketbase client: http://pocketbase:8090 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT _superusers.* FROM _superusers WHERE email='p@1.com' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [1.00ms] SELECT _authOrigins.* FROM _authOrigins WHERE collectionRef='pbc_3142635823' AND recordRef='m4k7w479e84whob' ORDER BY created DESC 2025-03-08 23:38:19 pocketbase-1 | [1.00ms] SELECT _collections.* FROM _collections WHERE id='pbc_3142635823' 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT (1) FROM _superusers WHERE id='m4k7w479e84whob' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [10.00ms] INSERT INTO _authOrigins (collectionRef, created, fingerprint, id, recordRef, updated) VALUES ('pbc_3142635823', '2025-03-08 15:38:19.178Z', '48178c710d1a0aeddc02e95bd81fd0f8', '0kiw7978ix03g67', 'm4k7w479e84whob', '2025-03-08 15:38:19.178Z') 2025-03-08 23:38:19 pocketbase-1 | INFO POST /api/collections/_superusers/auth-with-password 2025-03-08 23:38:19 core-1 | 2025-03-08 15:38:19.192 | INFO | utils.pb_api:init:22 - pocketbase ready authenticated as admin - p@1.com 2025-03-08 23:38:19 core-1 | 2025-03-08 15:38:19.193 | INFO | main:schedule_task:19 - task execute loop 1 2025-03-08 23:38:19 pocketbase-1 | [3.00ms] SELECT _superusers.* FROM _superusers WHERE _superusers.id='m4k7w479e84whob' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT focus_points.* FROM focus_points WHERE focus_points.activated = 1 LIMIT 500 2025-03-08 23:38:19 pocketbase-1 | [15.00ms] SELECT COUNT(DISTINCT focus_points.id) FROM focus_points WHERE focus_points.activated = 1 2025-03-08 23:38:19 pocketbase-1 | INFO GET /api/collections/focus_points/records?filter=activated=True&fields=&expand=&skiptotal=true&page=1&perPage=500 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT _superusers.* FROM _superusers WHERE _superusers.id='m4k7w479e84whob' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT COUNT(DISTINCT focus_points.id) FROM focus_points WHERE focus_points.activated = 1 2025-03-08 23:38:19 pocketbase-1 | [1.00ms] SELECT focus_points.* FROM focus_points WHERE focus_points.activated = 1 LIMIT 500 OFFSET 500 2025-03-08 23:38:19 pocketbase-1 | INFO GET /api/collections/focus_points/records?filter=activated=True&fields=&expand=&skiptotal=true&page=2&perPage=500 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT _superusers.* FROM _superusers WHERE _superusers.id='m4k7w479e84whob' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT sites.* FROM sites LIMIT 500 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT COUNT(DISTINCT sites.id) FROM sites 2025-03-08 23:38:19 pocketbase-1 | INFO GET /api/collections/sites/records?filter=&fields=&expand=&skiptotal=true&page=1&perPage=500 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT _superusers.* FROM _superusers WHERE _superusers.id='m4k7w479e84whob' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT COUNT(DISTINCT sites.id) FROM sites 2025-03-08 23:38:19 pocketbase-1 | [1.00ms] SELECT sites.* FROM sites LIMIT 500 OFFSET 500 2025-03-08 23:38:19 pocketbase-1 | INFO GET /api/collections/sites/records?filter=&fields=&expand=&skiptotal=true&page=2&perPage=500 2025-03-08 23:38:19 core-1 | 2025-03-08 15:38:19.232 | DEBUG | general_process:main_process:53 - new task initializing... 2025-03-08 23:38:19 core-1 | 2025-03-08 15:38:19.232 | DEBUG | general_process:main_process:57 - focus_id: 540j74b625k68rf, focus_point: 最新发布的锂电池信息, explanation: 2025年发布的内容, search_engine: False 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT _superusers.* FROM _superusers WHERE _superusers.id='m4k7w479e84whob' LIMIT 1 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT infos.* FROM infos WHERE infos.tag = '540j74b625k68rf' LIMIT 500 2025-03-08 23:38:19 pocketbase-1 | [0.00ms] SELECT COUNT(DISTINCT infos.id) FROM infos WHERE infos.tag = '540j74b625k68rf' 2025-03-08 23:38:19 pocketbase-1 | INFO GET /api/collections/infos/records?filter=tag='540j74b625k68rf'&fields=url&expand=&skiptotal=true&page=1&perPage=500 2025-03-08 23:38:19 core-1 | [INIT].... → Crawl4AI 0.5.0.post4 2025-03-08 23:38:19 core-1 | 2025-03-08 15:38:19.828 | DEBUG | general_process:main_process:140 - process new url, still 0 urls in working list 2025-03-08 23:38:20 core-1 | [COMPLETE] ● Database backup created at: /root/.crawl4ai/crawl4ai.db.backup_20250308_153819 2025-03-08 23:38:20 core-1 | [INIT].... → Starting database migration... 2025-03-08 23:38:20 core-1 | [COMPLETE] ● Migration completed. 0 records processed. 2025-03-08 23:38:25 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/... | Status: True | Time: 5.00s 2025-03-08 23:38:25 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/... | Time: 0.111s 2025-03-08 23:38:25 core-1 | [COMPLETE] ● https://news.bjx.com.cn/... | Status: True | Total: 6.16s 2025-03-08 23:38:44 core-1 | 2025-03-08 15:38:44.731 | DEBUG | general_process:main_process:209 - get 6 more related urls, will add to working list 2025-03-08 23:38:44 core-1 | 2025-03-08 15:38:44.732 | DEBUG | general_process:main_process:140 - process new url, still 5 urls in working list 2025-03-08 23:38:47 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Time: 2.52s 2025-03-08 23:38:47 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Time: 0.01s 2025-03-08 23:38:47 core-1 | [COMPLETE] ● https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Total: 2.53s 2025-03-08 23:38:47 core-1 | 2025-03-08 15:38:47.583 | DEBUG | general_process:main_process:140 - process new url, still 4 urls in working list 2025-03-08 23:38:49 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Time: 2.29s 2025-03-08 23:38:49 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Time: 0.019s 2025-03-08 23:38:49 core-1 | [COMPLETE] ● https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Total: 2.31s 2025-03-08 23:38:50 core-1 | 2025-03-08 15:38:50.964 | DEBUG | general_process:main_process:140 - process new url, still 3 urls in working list 2025-03-08 23:38:53 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Time: 2.29s 2025-03-08 23:38:53 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Time: 0.018s 2025-03-08 23:38:53 core-1 | [COMPLETE] ● https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Total: 2.31s 2025-03-08 23:38:53 core-1 | 2025-03-08 15:38:53.632 | DEBUG | general_process:main_process:140 - process new url, still 2 urls in working list 2025-03-08 23:38:55 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Time: 2.28s 2025-03-08 23:38:55 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Time: 0.01s 2025-03-08 23:38:55 core-1 | [COMPLETE] ● https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Total: 2.30s 2025-03-08 23:38:56 core-1 | 2025-03-08 15:38:56.463 | DEBUG | general_process:main_process:140 - process new url, still 1 urls in working list 2025-03-08 23:38:58 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Time: 2.24s 2025-03-08 23:38:58 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Time: 0.01s 2025-03-08 23:38:58 core-1 | [COMPLETE] ● https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Total: 2.26s 2025-03-08 23:38:59 core-1 | 2025-03-08 15:38:59.384 | DEBUG | general_process:main_process:140 - process new url, still 0 urls in working list 2025-03-08 23:39:01 core-1 | [FETCH]... ↓ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Time: 2.25s 2025-03-08 23:39:01 core-1 | [SCRAPE].. ◆ https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Time: 0.009s 2025-03-08 23:39:01 core-1 | [COMPLETE] ● https://news.bjx.com.cn/<https:/news.bjx.com.cn/ht... | Status: True | Total: 2.26s 2025-03-08 23:39:02 core-1 | 2025-03-08 15:39:02.259 | DEBUG | general_process:main_process:229 - task finished, focus_id: 540j74b625k68rf 2025-03-08 23:39:02 core-1 | 2025-03-08 15:39:02.259 | INFO | main:schedule_task:33 - task execute loop finished, work after 3600 seconds

2.另外我想问个问题，对于一个源，我想爬多个方面的内容，例如既要锂电池、又要电网、还要光伏。我是建立多个focus_point呢，还是一个focus_point就支持与或非？

3.另外，我之前发现Qwen/Qwen2.5-14B-Instruct消耗较高，爬回20条数据，消耗了5毛多，有成本更低的模型吗？或者可以用gemini2.0flash-lite吗？

[bigbrother666sh]

1、请先把库里面已有的 info 删除，这是因为它发现这些 info 已经存在，所以对应的 url 就不爬了；
2、可以写在同一个focuspoint里面，xx或xx 或 xx 方面的最新信息；
3、gemini 是支持 openaiSDK 的，可以试一下

[bobopu]

1、请先把库里面已有的 info 删除，这是因为它发现这些 info 已经存在，所以对应的 url 就不爬了； 2、可以写在同一个focuspoint里面，xx或xx 或 xx 方面的最新信息； 3、gemini 是支持 openaiSDK 的，可以试一下

1.可以抓取了，我发现替换成Qwen2-VL-7B-Instruct后抓取速度似乎比deepseek-vl2慢了不少。
2.能否将每个模型的API和KEY分别单独设置，例如主模型调用gemini或者本地自建，视觉模型调用硅基流动等。
3.搜索除了智普外，还有其他替代方案吗？例如博查，或者searxng可否替代？
4.有无测试过智普的GLM-4-Flash？如果使用此模型建议用于PRIMARY_MODEL还是SECONDARY_MODEL？

[bigbrother666sh]

1、理论上这两个速度应该差不多，而且 vl-model 其实不是卡点；2~3后续会考虑；4、GLM-4-Flash建议还是先用在SECONDARY_MODEL