Help

[Synchronzied]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I uploaded a large number of documents to a knowledge base in Dify. Before the documents finished parsing, I deleted the knowledge base.

After that, I created a new knowledge base and tried to upload documents again. However, all newly uploaded documents remain stuck in the “Queued” state. It seems that the documents from the previously deleted knowledge base are still being parsed in the background.

I have tried restarting Dify, but the issue persists and the queue does not move forward.

Expected behavior

After deleting a knowledge base, all related document parsing tasks should be canceled or cleaned up. New knowledge bases should be able to process documents normally.

Actual behavior

Document parsing tasks appear to remain in the queue even after the knowledge base is deleted, blocking new uploads.

Additional context

The issue occurred when the original documents were still being parsed.

Restarting the Dify service did not resolve the problem.

2. Additional context or comments

No response

[dosubot[bot]]

This is a known issue in Dify: when you delete a knowledge base while documents are still being parsed, the related document parsing tasks remain in the tenant-isolated Redis queue. These orphaned tasks block the queue, so any new uploads to new knowledge bases get stuck in the "Queued" state. Restarting Dify doesn't help because the tasks persist in Redis and will only be cleared as workers attempt to process them—if the dataset no longer exists, the task just logs "Dataset is not found" and exits, but the queue slot remains blocked until all such tasks are consumed. There is currently no built-in admin tool or documented workaround to manually clear these stuck tasks or reset the queue for deleted knowledge bases.

For advanced users, you could consider manually clearing the relevant Redis keys for your tenant's document indexing queue, but this should be done with caution and only if you are comfortable with Redis and understand the risks (such as potentially affecting other queued tasks).

It would be beneficial for maintainers to add a cleanup mechanism that removes orphaned tasks from the queue when a knowledge base is deleted to prevent this issue in the future. If you need more technical details or guidance on identifying the Redis keys, let me know and I can provide more specifics.

Reference: code analysis and queue management

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[dosubot[bot]]

The document parsing queue in Dify uses Redis keys like tenant_self_document_indexing_task_queue:<tenant_id> and tenant_document_indexing_task:<tenant_id> for task management. If you don't see these keys, it's likely because either the Celery worker isn't listening on the required queues (dataset and priority_dataset), or you're inspecting the wrong Redis database (check your REDIS_DB and CELERY_BROKER_URL settings) source.

To resolve this, make sure your Celery worker is started with the following queue list:
dataset,priority_dataset,priority_pipeline,pipeline,mail,ops_trace,app_deletion,plugin,workflow_storage,conversation
You can set this via the CELERY_QUEUES or CELERY_WORKER_QUEUES environment variable, or by updating the worker startup arguments source.

Also, confirm you're inspecting the correct Redis database (default is 0, but your Celery broker might use another, like 1). Once the worker and Redis DB are configured correctly, document indexing tasks should process normally and the expected Redis queue keys will appear source.

If the issue persists, check your backend and worker logs for errors related to task dispatch or Redis connectivity.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[Synchronzied]

Does Dify use Redis as the Celery broker? If yes, what is the recommended way to clear stale or orphaned tasks from previous document parsing jobs?
@dosu

[dosubot[bot]]

Yes, Dify uses Redis as the default Celery broker and result backend for managing document parsing and other background tasks details.

There’s no built-in admin tool or UI for clearing stale or orphaned document parsing jobs. For self-hosted setups, the recommended way is to restart the Celery worker (for example, with docker compose restart worker). This often flushes the worker’s state and allows Redis-backed tasks to be reprocessed, which can clear stuck queues discussion issue.

If that doesn’t resolve it, you can manually delete the relevant Redis keys for your tenant’s document parsing queue. The keys follow the pattern:

• tenant_self_document_indexing_task_queue:<tenant_id>
• tenant_document_indexing_task:<tenant_id>

Use a Redis client or script to remove these keys. This will flush all pending tasks for that tenant and may require you to re-upload any stuck documents. Always make sure you have backups or copies of important files before clearing queues, as this action cannot be undone discussion.

If you don’t see these keys, double-check you’re inspecting the correct Redis database (matching your REDIS_DB setting), and that your Celery worker is listening on the required queues (dataset, priority_dataset, etc.) issue.

Let me know if you need help with the exact Redis commands or identifying your tenant ID.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[Synchronzied]

According to the logs, there is a task currently being executed, but its corresponding dataset no longer exists. How can I clean up or cancel the indexing documents associated with this task?
@dosu

[dosubot[bot]]

Dify doesn't have a built-in way to cancel or clean up document indexing tasks if the dataset has already been deleted. When this happens, the tasks remain in the queue and will keep blocking new uploads until they're consumed by a worker, which just logs "Dataset is not found" and exits—so the queue stays stuck reference.

To manually clear these orphaned tasks, you need to delete the Redis keys for your tenant's document indexing queue. The keys are:

• tenant_self_document_indexing_task_queue:<tenant_id>
• tenant_document_indexing_task:<tenant_id>

Before deleting, stop your Celery workers to avoid race conditions. Use a Redis client (like redis-cli) to run:

DEL tenant_self_document_indexing_task_queue:<tenant_id>
DEL tenant_document_indexing_task:<tenant_id>

Replace <tenant_id> with your actual tenant ID. This will clear all pending document indexing tasks for that tenant. Make sure you have backups, as this can't be undone and may require you to re-upload any stuck documents reference.

After cleanup, restart your Celery workers and verify that new uploads process as expected. If you don't see these keys, double-check you're using the correct Redis database and that your worker is listening on the right queues reference.

Let me know if you need help finding your tenant ID or with the Redis commands.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.