Lorenze/fix duplicating doc ids for knowledge#3840
Merged
lorenzejay merged 9 commits intomainfrom Nov 6, 2025
Merged
Conversation
… SHA-256 hashing and include index for uniqueness
|
You have run out of free Bugbot PR reviews for this billing cycle. This will reset on November 28. To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial. |
lorenzejay
changed the title
…cating-doc-ids-for-knowledge
lucasgomide
approved these changes
Nov 6, 2025
…m:crewAIInc/crewAI into lorenze/duplicating-doc-ids-for-knowledge
…s to check for doc_id at the top level of the document
…deduplicate documents and ensure unique hash-based IDs without suffixes
greysonlalonde
requested changes
Nov 6, 2025
…ility functions to ensure robust processing and uniqueness
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes:
Failed to upsert documents: Expected IDs to be unique, found 11 duplicated IDs: d4240b0365c2c5924168f3face32bd58a10fa1b4b0b35dec09c31a9536bafc27, 0d1e108d762cd32e466a0ceaaf0f6b47192f87ac0b4709655cecd17fe6c8e847, 810450747867acb256ccb33449c2efa36b79325be8b73cdb47fe0418b13bba9c, cdf4eac9d7909a5b3e0df2a02f313d9b7903efd71d3334a884ec1390a84a1061, 90edd08867753148c96eacdce31edd6fd4d96041f3b9af962d4baf3b548366da, ..., 2121a2f1f7e15e946a9c5aa1d2c1c29f07613cad15cfc589ca047753039e347c, dc2458769785b945e26aa21e9be883cf2cf300b713086077d7365d76644b3f69, cf7b761e9be0a4c2cab21735ec588df02bd394820f54e08d22fa71773e2a1017, af28a4b2ee7a0e113893ddbb3c51c50a46f6cee70ba1427e3a7bd179c618e39c, 312f6b99bc7f9601213576d81e3883de424d7dc2bf1c4cbfe63499732c59fca7 in upsert.