Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to md5 for deduplication in neo4j integrations #18846

Merged
merged 3 commits into from
Mar 9, 2024

Conversation

tomasonjo
Copy link
Contributor

Deduplicate documents using MD5 of the page_content. Also allows for custom deduplication with graph ingestion method by providing metadata id attribute

@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Mar 9, 2024
Copy link

vercel bot commented Mar 9, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
langchain ⬜️ Ignored (Inspect) Visit Preview Mar 9, 2024 7:38pm

@dosubot dosubot bot added Ɑ: vector store Related to vector store module 🔌: neo4j Primarily related to Neo4j integrations 🤖:improvement Medium size change to existing code to handle new use-cases labels Mar 9, 2024
@tomasonjo
Copy link
Contributor Author

Linter error seems unrelated:

tests/unit_tests/agents/test_sql.py:14: error: Any? not callable [misc]

@hwchase17 hwchase17 merged commit a28be31 into langchain-ai:master Mar 9, 2024
59 checks passed
bechbd pushed a commit to bechbd/langchain that referenced this pull request Mar 29, 2024
…8846)

Deduplicate documents using MD5 of the page_content. Also allows for
custom deduplication with graph ingestion method by providing metadata
id attribute

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
gkorland pushed a commit to FalkorDB/langchain that referenced this pull request Mar 30, 2024
…8846)

Deduplicate documents using MD5 of the page_content. Also allows for
custom deduplication with graph ingestion method by providing metadata
id attribute

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
hinthornw pushed a commit that referenced this pull request Apr 26, 2024
Deduplicate documents using MD5 of the page_content. Also allows for
custom deduplication with graph ingestion method by providing metadata
id attribute

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🤖:improvement Medium size change to existing code to handle new use-cases 🔌: neo4j Primarily related to Neo4j integrations size:S This PR changes 10-29 lines, ignoring generated files. Ɑ: vector store Related to vector store module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants