Skip to content
Anjana Tiha edited this page Mar 28, 2026 · 2 revisions

FAQ

General

Does this send my documents to any external server?

No. When running locally, all processing happens on your machine. The LLM and embedding models run via Ollama. ChromaDB stores vectors on disk locally. No data is transmitted externally.

The Hugging Face demo runs on Hugging Face infrastructure — documents uploaded there are processed on HF servers.


What languages are supported?

Document ingestion and retrieval work with any language, but answer quality depends on the underlying LLaMA 3.2 model's multilingual capability. The translate tool supports any language pair that LLaMA 3.2 handles (strong coverage for major European languages, moderate for others).


How many documents can I index?

There is no hard limit. Performance degrades as the index grows because reranking makes one LLM call per candidate chunk. Practical limits depend on your hardware:

RAM Recommended index size
8 GB Up to ~5,000 chunks
16 GB Up to ~20,000 chunks
32 GB+ Larger collections

Use TOP_RETRIEVE and TOP_RERANK in config.py to tune the retrieval budget.


Can I use a different LLM?

Yes. Change LANGUAGE_MODEL in src/rag/config.py to any model available via Ollama:

LANGUAGE_MODEL = 'llama3.1:8b'   # for higher quality
LANGUAGE_MODEL = 'phi3:mini'      # for lower resource usage

Pull the model first: ollama pull <model-name>


Can I use a different embedding model?

Yes. Change EMBEDDING_MODEL in config.py. After changing, delete ./chroma_db/ and restart so the index is rebuilt with the new model's dimensions.


Retrieval

Why does the system sometimes say "I could not find relevant information" even though I have the document?

The confidence gate rejects answers when the best retrieved chunk scores below SIMILARITY_THRESHOLD (default 0.55). This prevents hallucination when the query does not match the document content closely enough.

Possible causes:

  • The query uses different terminology than the document
  • The document has not been indexed yet (check sidebar chunk count)
  • The document format was not parsed correctly (check terminal for errors)

Try rephrasing the query to use words that appear in the document.


What is hybrid search?

Hybrid search combines two retrieval methods:

  • BM25 (lexical): exact keyword matching — reliable for specific terms, names, and numbers
  • Dense vector (semantic): embedding similarity — reliable for paraphrased and conceptual queries

The scores are fused with equal weight (alpha=0.5). This gives higher recall than either method alone, particularly for structured documents where exact terms matter.


Agent Mode

When should I use agent mode vs chat mode?

Use case Recommended mode
Simple factual questions Chat
Document summarisation Chat
Multi-step reasoning Agent
Math on document data Agent
Sentiment analysis Agent
Translation Agent
Comparing values across documents Agent

Why does the agent sometimes give a different answer than chat mode?

Agent mode uses a different synthesis path. It makes targeted searches for specific sub-topics and synthesises from accumulated context. Chat mode runs the full pipeline in a single pass. For complex queries, agent mode typically produces more thorough answers.


Development

How do I run the tests?

source rag_env_311/bin/activate
pytest                          # all local tests (859)
pytest --cov=src               # with coverage
cd huggingface && pytest       # HF Space tests (385)

How do I contribute?

See the Contributing page.

Clone this wiki locally