How to clean the embeddings table data?

[xiaopangalf]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Content

What is the correspondence between table documents_segments and table embeddings? I need to clean up some data in the document_degments table. How can I also clean up the corresponding embeddings table data? The embeddings table is already 60GB and needs to be cleaned up.

[dosubot[bot]]

There’s no direct foreign key or automatic cleanup between documents_segments and embeddings in Dify—the link is indirect, based on a hash of the segment content, model name, and provider name. The embeddings table acts as a persistent cache and will accumulate orphaned data over time if segments are deleted or changed reference.

To clean up embeddings, you need to identify which embeddings are no longer referenced by any current segment. Since document_segments doesn’t store the hash, you’ll need to recompute the hash for each segment’s content (using the same logic as Dify’s embedding cache), then compare those hashes to what’s in the embeddings table. Any embedding whose hash/model/provider combo isn’t referenced by any current segment can be safely deleted reference.

There’s no built-in script for this, so you’ll need to write a cleanup script or SQL query. Always back up your database before running deletes. For large tables, do this in batches and consider locking or running during low-traffic periods to avoid performance issues.

Best practice: schedule regular maintenance to remove unreferenced embeddings, and monitor the size of your embeddings table to prevent future bloat reference.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[xiaopangalf]

You did not provide a clear data correspondence method or hash value calculation method. Please provide me with an SQL statement to determine the corresponding data. I did not adjust the model and provider. So there's no need to consider the difference between these two. @dosu

[dosubot[bot]]

Here’s how the correspondence works: the embeddings table’s hash column is generated as sha256(segment_content || 'None')—that is, the SHA-256 hash of the segment’s content plus the string 'None' (no normalization or transformation). Since your model and provider are constant, you can ignore those fields for matching reference.

To clean up orphaned embeddings in PostgreSQL, you need to recompute the hash for each current segment and compare. Here’s how you can do it:

-- List all valid hashes (referenced by current segments)
SELECT encode(digest(content || 'None', 'sha256'), 'hex') AS hash
FROM document_segments;

-- Delete orphaned embeddings (not referenced by any segment)
DELETE FROM embeddings
WHERE hash NOT IN (
  SELECT encode(digest(content || 'None', 'sha256'), 'hex') FROM document_segments
);

If your embeddings table is very large, do this in batches and always back up your data first. This approach matches Dify’s internal logic reference reference.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[zy30651]

What impacts will there be if the Embedding table is directly deleted?