Indexing 1.7MB file causes CPU to crash in memory

[wenxinji]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

docker-worker-1         | 2025-03-07 08:52:07.520 INFO [MainThread] [strategy.py:161] - Task tasks.recover_document_indexing_task.recover_document_indexing_task[cae7eb7f-1ab9-408b-9213-2bf105affc71] received
docker-worker-1         | 2025-03-07 08:52:07.521 INFO [Dummy-2] [recover_document_indexing_task.py:22] - Recover document: 5fe427c8-14f1-407a-b3c5-adc7c69fb018
docker-worker-1 exited with code 0
docker-worker-1         | 2025-03-07 08:52:47.349 INFO [MainThread] [utils.py:162] - NumExpr defaulting to 4 threads.
docker-worker-1         | 2025-03-07 08:52:49.648 INFO [MainThread] [_client.py:1038] - HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json "HTTP/1.1 200 OK"
docker-worker-1         | Preparing database migration...
docker-worker-1         | Starting database migration.
docker-worker-1         | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
docker-worker-1         | INFO  [alembic.runtime.migration] Will assume transactional DDL.
docker-worker-1         | Database migration successful!
docker-worker-1         | 2025-03-07 08:52:56.586 INFO [MainThread] [utils.py:162] - NumExpr defaulting to 4 threads.
docker-worker-1         | 2025-03-07 08:52:58.855 INFO [MainThread] [_client.py:1038] - HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json "HTTP/1.1 200 OK"
docker-worker-1         | /app/api/.venv/lib/python3.12/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
docker-worker-1         | absolutely not recommended!
docker-worker-1         | 
docker-worker-1         | 
docker-worker-1         | Please specify a different user using the --uid option.
docker-worker-1         | 
docker-worker-1         | 
docker-worker-1         | User information: uid=0 euid=0 gid=0 egid=0
docker-worker-1         | 
docker-worker-1         | 
docker-worker-1         |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
docker-worker-1         |  
docker-worker-1         |  -------------- celery@d97f561e5694 v5.4.0 (opalescent)
docker-worker-1         | --- ***** ----- 
docker-worker-1         | -- ******* ---- Linux-6.8.0-1024-aws-x86_64-with-glibc2.36 2025-03-07 08:53:01
docker-worker-1         | - *** --- * --- 
docker-worker-1         | - ** ---------- [config]
docker-worker-1         | - ** ---------- .> app:         app_factory:0x752e9d8e2a20
docker-worker-1         | - ** ---------- .> transport:   redis://:**@redis:6379/1
docker-worker-1         | - ** ---------- .> results:     postgresql://postgres:**@db:5432/dify

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d97f561e5694 docker-worker-1 100.87% 2.752GiB / 3GiB 91.74% 90.4kB / 60.6kB 8.32MB / 1.78MB 5

✔️ Expected Behavior

Index successful

❌ Actual Behavior

No response

[wenxinji]

[wenxinji]

8GB of memory is limited to 3GB, otherwise it may cause the server to crash. It has already collapsed 4-5 times

[dosubot]

I found a similar closed issue where a solution was suggested to address service lag during document indexing. The solution involves deploying the worker service related to document indexing separately on other machines to distribute the load and prevent the main service from freezing when uploading large documents. You can start the Worker service on a different machine using the following commands:

For Linux or macOS:

celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail,ops_trace

For Windows:

celery -A app.celery worker -P solo --without-gossip --without-mingle -Q dataset,generation,mail,ops_trace --loglevel INFO

Additionally, ensure that your Docker setup meets the necessary requirements, such as configuring Docker Desktop to use at least 2 virtual CPUs and 8 GB of initial memory on macOS, and ensuring Docker 19.03 or later and Docker Compose 1.25.1 or later are installed on Linux. On Windows with WSL 2 enabled, it is recommended to store source code and other data in the Linux filesystem rather than the Windows filesystem ^[1].

Regarding the segmentation rules, ensure they are properly configured. If the segmentation identifier does not match any part of the document, or if the document is empty or only contains elements that are filtered out during preprocessing, the resulting segments could be empty, leading to a segment length and paragraph count of zero ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[crazywoola]

This might be related to a memory leak in our plugin daemon. langgenius/dify-plugin-daemon#55