RAGAL — companion scripts

Generic, self-contained scripts from the paper "RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency" (Dan Mușetoiu, AFIR — Agency for Financing Rural Investments, Romania).

📄 Paper: arXiv:2607.18756

RAGAL is an on-premise, Romanian-language RAG assistant for a government technical-support team, built under three hard constraints: zero data egress, a read-only mandate (the assistant drafts, humans execute), and one consumer laptop with 8 GB of VRAM as the only development and training machine.

These scripts contain no institutional data, schemas, endpoints or connectors — bring your own corpus as JSONL. They reproduce the parts of the pipeline that generalize: local synthetic-query generation, hard-negative mining, the 8 GB fine-tuning recipe, and per-domain IR evaluation.

Pipeline

your corpus (JSONL)
   │
   ├─ scripts/01_genq_local.py          # synthetic queries for domains without real ones
   │                                    #   (zero-egress: a local LLM via Ollama)
   ├─ scripts/02_mine_hard_negatives.py # (query, positive) → + K hard negatives
   ├─ scripts/03_train_embedder_8gb.py  # full fine-tune of bge-m3 in 8 GB VRAM
   └─ scripts/04_ir_eval.py             # recall@k + MRR, held-out queries vs pool

Data formats

• Pairs (pairs_train.jsonl, one JSON object per line): {"query": "...", "positive": "...", "negatives": ["...", "..."]} (negatives optional — added by script 02).
• IR eval set (ir_eval.jsonl): {"qid": "...", "query": "...", "gold_id": "..."}
• IR corpus pool (ir_corpus.jsonl): {"id": "...", "text": "..."}

Requirements

pip install "sentence-transformers>=4.1" datasets accelerate bitsandbytes requests numpy

A GPU with ≥8 GB VRAM (the whole point) and CUDA torch ≥2.6. Script 01 expects a local Ollama instance.

Results reported in the paper

Fine-tuning bge-m3 on 4,835 real ticket pairs, 72 minutes on an RTX 2000 Ada (8 GB):

Eval set	Model	R@10	MRR@20
Tickets	bge-m3 (stock)	0.663	0.489
Tickets	fine-tuned (v2)	0.850	0.677
Documents	bge-m3 (stock)	0.881	0.613
Documents	fine-tuned (v1, tickets only)	0.849 ↓	0.562 ↓
Documents	fine-tuned (v2, + local GenQ)	0.881	0.632

The v1 row is the point: a fine-tune that wins big on the trained domain can silently push a sibling domain below the stock model.

The 8 GB lessons baked into these scripts (§6.2 of the paper)

1. On Windows, the NVIDIA driver spills to system RAM silently instead of raising OOM — training runs, but each step gets slower. Watch the per-process "GPU Process Memory / Shared Usage" performance counter, not nvidia-smi.
2. --optim adamw_bnb_8bit shrinks AdamW states ~4.5 GB → ~0.6 GB.
3. --grad-checkpoint (non-reentrant) trades ~30% compute to eliminate the spill — a 3× net win on PCIe-bound consumer hardware.
4. On Windows, import datasets before import torch (pyarrow/torch DLL conflict → access violation on the reverse order).
5. Train in the same text space you serve: if production strips diacritics before embedding, strip them in training too.

Evaluation discipline (§6.3 — the silent regression)

If your corpus has multiple domains (e.g. tickets and documents), build one IR eval set per domain before you fine-tune. A fine-tune that wins big on the trained domain can silently push a sibling domain below the stock model — invisible in training curves and single-domain evaluations.

Citation

@misc{musetoiu2026ragal,
  title  = {RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant
            for Technical Support at a Government Agency},
  author = {Mu\c{s}etoiu, Dan},
  year   = {2026},
  eprint = {2607.18756},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR}
}

License

Code: MIT. The paper text and figures are licensed separately (CC BY 4.0 on arXiv).