[Feature Request] Collaboration with local-deep-research open source project

[LearningCircuit]

Description

Hi paperless-ngx team,

highly appreciate your great tool and wanted to connect it to our https://github.com/LearningCircuit/local-deep-research by adding it as one of our search engines for local document research.

Please mostly consider the request by us as a collaborative outreach from another open source project. Maybe there is some potential how these open source projects could be helpful for each other.

Current Situation

I've successfully integrated Paperless-ngx's REST API into LDR (in development branch), and while the basic integration works, we've encountered limitations with the current search capabilities that prevent it from being most effective for research-oriented queries.

We will try to work around the current limitations. A first prototype achieves reasonable results under very limited testing. However, this will not provide the same capabilities that could be achieved with other search options and is also much slower.

Core Issue

Keyword-only search misses relevant documents

• Query: "climate impact on agriculture"
• Misses: documents about "global warming effects on farming"

Things needed

Semantic search via embeddings for retrieval augmented generation (RAG)

• Semantic/embedding-based search using sentence-transformers
• Hybrid search combining keywords + semantic similarity
• API endpoint for similarity search: /api/documents/<id>/similar

Use Case

When researchers query complex topics, they need conceptually related documents, not just keyword matches. This would make Paperless-ngx valuable for RAG systems and AI research tools.

We Can Help

• We have RAG implemented for local documents
• Happy to discuss on implementation
• Can share our integration code
• Will test and provide feedback

Related: #4059, #3673

Would you be interested in exploring this together?

Other

No response

[shamoon]

I'm not sure I entirely understand what you're asking from us but generally happy to collaborate in the OSS space. #10319 implements RAG, thought not (yet) for search

[LearningCircuit]

Thanks for your fast response.

Basically, we come from different direction and implement capabilities that the other project already has implementations for, e.g. we are implementing PDF download and viewing, which you already do much better
LearningCircuit/local-deep-research#793 .

As open source project we have the chance that instead of reinventing the wheel we just collaborate.

[LearningCircuit]

For example, we could send PDFs via API directly to Paperless-ngx . (Our users want to download source PDFs from their researches because than they can search them deeper with our RAG)

[github-actions[bot]]

This discussion has been automatically closed due to lack of community support. Please see our contributing guidelines for more details.

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. See our contributing guidelines for more details.