International Student Services RAG Assistant

Model Evaluation

The RAG assistant was evaluated using a benchmark designed to measure answer quality, document grounding, scope adherence, refusal behavior, source usage, and inference latency across multiple locally hosted LLMs.

Evaluation Setup

I Ecvaluated Four local instruction-tuned models:

• Qwen 2.5 7B
• Gemma 2 9B
• Llama 3.1 8B
• DeepSeek-R1 7B

The benchmark contains 10 test questions across three categories:

Category	Description
In-Scope	Questions that should be answerable using the provided university/international-student documents
Out-of-Scope	Questions unrelated to the document collection
Not in Documents	Questions related to the domain but requiring information that is not contained in the retrieved documents

The evaluation tracks:

• Response latency
• Faithfulness to retrieved context
• Relevance
• Refusal behavior
• Expected refusal compliance
• Source attribution
• Scope adherence
• Potential hallucinations

Results

Model	Avg. Latency	Overall Assessment	Key Observation
Qwen 2.5 7B	~9.5s	Strong	Best balance of response quality and latency
Gemma 2 9B	~10.8s	Strong	Conservative and well-grounded, but sometimes under-answers
Llama 3.1 8B	~19.7s	Strong	High-quality responses with higher latency
DeepSeek-R1 7B	~48.5s	Mixed	Detailed responses but significantly higher latency and hallucination risk

Key Findings

Qwen 2.5 7B provided the strongest overall trade-off between response quality and latency, making it the preferred candidate for interactive local deployment.

Gemma 2 9B demonstrated conservative behavior, frequently declining to answer when information was unavailable. While this improves safety and grounding, some responses were less complete than those generated by Qwen or Llama.

Llama 3.1 8B produced strong document-grounded answers, but its average latency was approximately twice that of Qwen and Gemma in this benchmark.

DeepSeek-R1 7B produced substantially higher latency and occasionally generated plausible but unsupported information. For example, when asked for CPT application instructions that were not explicitly available in the retrieved documents, it generated a detailed procedure rather than acknowledging the information gap.

Grounding Failure Analysis

One of the most important findings was that a model can retrieve documents that are semantically related to a question without those documents actually containing the requested answer.

For example:

Question: "Exactly how many days does an OPT application take to process this year?"

The retrieved documents contained general OPT processing information but did not establish an exact current processing time. Several models nevertheless generated processing estimates.

This demonstrates an important limitation of basic RAG pipelines:

Semantic relevance does not guarantee answerability.

A production system should distinguish between:

1. Relevant documents being retrieved
2. Retrieved documents actually supporting the answer
3. The model having sufficient evidence to answer confidently

Evaluation Conclusion

The benchmark demonstrates that model selection alone is not sufficient to prevent hallucinations in a RAG system. Stronger grounding, answerability detection, retrieval thresholds, and citation validation are required to ensure that responses remain within the knowledge boundaries of the document collection.

For the current implementation, Qwen 2.5 7B provides the best starting point for further optimization because it combines relatively low latency with strong performance on the evaluated tasks.

---

Potential Improvements

The current implementation provides a functional local RAG pipeline, but several improvements could increase reliability, retrieval quality, and production readiness.

1. Improve Retrieval Quality

The current system can retrieve documents that are related to a question without necessarily containing the required answer.

Potential improvements include:

• Implement hybrid search combining dense vector search with BM25/keyword retrieval
• Add a cross-encoder reranker after initial retrieval
• Tune the number of retrieved chunks (top-k)
• Experiment with different chunk sizes and overlap
• Add metadata filtering by document type or topic
• Evaluate multiple embedding models

2. Add Answerability Detection

Before generating an answer, the system could determine whether the retrieved context actually contains sufficient evidence.

User Question
      ↓
Document Retrieval
      ↓
Re-ranking
      ↓
Answerability Check
      ↓
 ┌───────────────┐
 │               │
Answerable    Insufficient Evidence
 │               │
 ↓               ↓
Generate       Refuse / Redirect
Answer