Skip to content

Error: 知识库中上传的是剑来小说的txt,报Chunk token length 1414 exceeds chunk_token_size 1200. #554

@wysstartgo

Description

@wysstartgo

1️⃣ 描述一下问题

知识库中上传的是剑来小说的txt,报Chunk token length 1414 exceeds chunk_token_size 1200.

2️⃣ 报错日志

请运行以下命令,并提供部分相关日志:

# macOS / Linux
make logs

# Windows
docker logs --tail=100 api-dev
git rev-parse HEAD
make logs 的输出:
api-dev          | 03-11 19:27:49 WARNING __init__.py:1028: 2026-03-11 19:27:49,441 - lightrag - WARNING - Chunk split_by_character exceeds token limit: len=1414 limit=1200
api-dev          | 03-11 19:27:49 ERROR __init__.py:1028: 2026-03-11 19:27:49,449 - lightrag - ERROR - Traceback (most recent call last):
api-dev          |   File "/usr/local/lib/python3.12/site-packages/lightrag/lightrag.py", line 1848, in process_document
api-dev          |     chunking_result = self.chunking_func(
api-dev          |                       ^^^^^^^^^^^^^^^^^^^
api-dev          |   File "/usr/local/lib/python3.12/site-packages/lightrag/operate.py", line 121, in chunking_by_token_size
api-dev          |     raise ChunkTokenLimitExceededError(
api-dev          | lightrag.exceptions.ChunkTokenLimitExceededError: Chunk token length 1414 exceeds chunk_token_size 1200. Preview: '“我曾下山游历三年,知道天时有变,顺带着地利人和,皆有极大变化,天下多出了许多前所未有的神异怪事。”
api-dev          | “但是这些年来,我不曾遇到任何一位来自外乡的谪仙人。”
api-dev          | 陈'
api-dev          | 
api-dev          | 03-11 19:27:49 ERROR __init__.py:1028: 2026-03-11 19:27:49,450 - lightrag - ERROR - Failed to extract document 1/1: http://localhost:9000/ref-kb-3360ab47af15bc3598c3473c761c0/《剑 来 》(精校版全本)_1773228413234.txt



3️⃣ 相关截图

Image

#️⃣ 其他相关信息

✅ 如果问题与模型调用相关,请尝试切换到其他在线模型

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions