Osuda — a length-calibrated psychology assistant

Fine-tuning LLaMA 3.1 8B with QLoRA so it answers like a counselor — short, warm, and directive — instead of like a chatbot dispensing a numbered listicle. Served over Telegram.

The headline result is not accuracy. It is style control: the base model answered a 44-word reference with 193 words. After fine-tuning, 42.

---

Results

Evaluated on a held-out sample of 100 examples from samhog/psychology-10k (random_state=42). Both models decoded identically and with no system prompt.

Metric	Base LLaMA 3.1 8B	Osuda (QLoRA)	Change
ROUGE-1	0.2316	0.4385	+89%
ROUGE-2	0.0679	0.1820	+168%
ROUGE-L	0.1443	0.3133	+117%
BERTScore F1	0.8587	0.9139	+0.055
Avg response length (words)	193.2	42.3	reference: 43.6
Avg latency (s)	89.32	13.53	6.6× faster

Response length distribution — the result the other metrics mostly follow from:

Source	Mean	Median	Std
Ground truth	43.6	41.0	14.8
Base LLaMA 3.1 8B	193.2	199.0	20.4
Osuda (fine-tuned)	42.3	40.0	14.7

The fine-tuned model matches the reference length distribution almost exactly — mean within 1.3 words, standard deviation within 0.1. Read the Limitations before citing the ROUGE deltas: a large share of that gain is length calibration, not new knowledge.

Demo

Unedited conversation with the deployed bot. Note the replies run 40–50 words — the length calibration in the table above, visible in practice — and that the assistant holds its scope when asked "Can you do anything else?"

Qualitative example

User: I'm having trouble with my anger. What can I do?

Reference: Anger can be a difficult emotion to manage. Let's discuss anger management techniques and therapy options to help you address your anger.

Base: Managing anger can be a challenging but rewarding process. Here are some strategies that may help: 1. Identify your triggers: Take some time to reflect on what triggers your anger. Is it a specif… (truncated at 256 tokens)

Osuda: It's important to identify the source of your anger and find healthy ways to express it. We can work together to develop coping mechanisms such as deep breathing, mindfulness, and communication skills.

---

How it works

• Base model — unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit, loaded in 4-bit (NF4).
• Adaptation — QLoRA via Unsloth: the base weights stay frozen and quantized; only low-rank adapter matrices train. This is what makes an 8B fine-tune fit in a single free-tier Colab GPU.
• Data — samhog/psychology-10k (instruction / input / output), rendered with the llama-3.1 chat template and filtered to a 2048-token budget.
• Serving — the merged model runs under Ollama; a small aiogram bot bridges Telegram to Ollama's /api/chat.

LoRA configuration

Read from the published adapter_config.json:

Setting	Value
Rank r	16
lora_alpha	16
lora_dropout	0
bias	none
use_rslora	true
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable params	~42M of 8B (~0.5%)

Two choices here explain a lot of the behavioural shift:

• All seven projections are adapted, not just attention. Including the MLP block (gate/up/down_proj) is what lets the model change how it writes, not merely what it attends to — style lives largely in the feed-forward layers.
• use_rslora=true switches the update scaling from alpha/r to alpha/√r. With alpha=16, r=16 that is 16/4 = 4.0 rather than 16/16 = 1.0 — a 4× stronger effective adapter contribution than the raw alpha == r pairing suggests. If you are investigating why the length distribution moved so decisively, this is the first knob to look at.

Telegram user ──► run.py (aiogram) ──► handler.py ──► Ollama /api/chat ──► Osuda model
                                          │
                                    per-chat history
                                    (bounded, isolated)

---

Quickstart

Requires Python 3.9+ and a running Ollama instance.

# 1. Install
pip install -r requirements.txt

# 2. Configure
cp .env.example .env
#    Then edit .env: set TOKEN (from @BotFather) and LLAMA_MODEL.

# 3. Verify the setup before running anything live
python verify.py        # 14 known-answer checks; exits 0 on success

# 4. Run
python run.py

run.py prints its resolved configuration at startup (token redacted) so a wrong model tag or a stale proxy is visible immediately rather than inferred from odd behaviour later.

Configuration

All settings come from the environment via config.py — the single source of truth. Missing required values raise ConfigError at import with the variable named; nothing silently defaults to None.

Variable	Required	Purpose
TOKEN	yes	Telegram bot token from @BotFather
OLLAMA_API_URL	yes	Ollama chat endpoint, e.g. http://localhost:11434/api/chat
LLAMA_MODEL	yes	Model tag as registered in Ollama
TELEGRAM_PROXY	no	Outbound HTTP proxy, only where api.telegram.org is blocked
SYSTEM_PROMPT	no	Persona override
MAX_HISTORY_MESSAGES	no	Messages retained per chat (default 20)
REQUEST_TIMEOUT_SECONDS	no	Per-request timeout (default 120)

Conversation history is stored per chat id and bounded. An earlier version kept one global list shared by every user — so one user's context leaked into another's. verify.py section [1] is a regression test for exactly that.

---

Evaluation

Reproduce with Osuda_testing.ipynb (Colab, GPU runtime).

Protocol

1. Load samhog/psychology-10k; filter to examples fitting 2048 tokens under the llama-3.1 chat template; hold out a test split.
2. Sample N = min(100, len(test)) with random_state=42.
3. Generate from the fine-tuned model, free the GPU, then generate from the base model on the same sampled rows.
4. Score.

Decoding — identical for both models: max_new_tokens=256, temperature=1.0, min_p=0.1, do_sample=True. Neither model receives a system prompt.

Metrics

• ROUGE-1/2/L — rouge_score.RougeScorer, use_stemmer=True, F-measure, averaged per example.
• BERTScore F1 — bert_score, lang="en" (which selects roberta-large), mean over examples. The pooler.dense "missing key" warning on load is benign: BERTScore reads hidden states, never the pooler.
• Length — whitespace token count (len(x.split())).
• Latency — wall-clock per generate() call on the Colab GPU. Comparable between the two models here, but not a hardware-independent benchmark.

Saving results — use eval/save_artifacts.py rather than a bare to_csv. It writes to both the local runtime and Drive, round-trips the CSV to confirm the row count survived, and prints a checksum. A plain to_csv in Colab is lost when the runtime disconnects, and plt.savefig() called after plt.show() silently writes a blank image — both mistakes are avoided there.

fig, axes = plt.subplots(1, 2, figsize=(12, 5))
# ... build plots ...
save_all(test_sample, summary, fig=fig)   # BEFORE plt.show()
plt.show()

---

Limitations

Stated plainly, because the headline numbers are easy to over-read.

• ROUGE rewards length matching. The base model wrote ~4.5× the reference length, which depresses its ROUGE F-measure through precision regardless of content quality. A meaningful share of the +89% ROUGE-1 is the model learning the target length distribution, not new psychological knowledge. BERTScore (+0.055) is the less length-sensitive signal, and its gain is real but far more modest.
• The baseline is unprompted. Neither model got a system prompt. The base model was never asked to be brief or counselor-like, so this measures fine-tuning against a zero-shot default — not against a prompt-engineered baseline. A base model given "answer in 2–3 sentences as a counselor" would close much of this gap. That control is not yet run, and it is the single most important missing comparison.
• Base responses are truncated. At max_new_tokens=256, a 193-word mean response is at or near the cap. The true base length is censored — 193.2 is a floor, not a measurement, and the "4.5×" ratio is correspondingly a lower bound.
• n = 100, single run, no confidence intervals. With do_sample=True and temperature=1.0 and no generation seed, a re-run will not reproduce these figures exactly. Treat differences as directional. No significance testing was performed.
• Reference-based metrics on a synthetic dataset. psychology-10k responses are the "ground truth"; agreeing with them is not the same as being clinically sound.
• Not a clinical tool. No safety evaluation, no crisis-handling evaluation, no human clinician rating. Do not deploy this to real users in distress.
• History is in-memory. Restarting the bot clears every conversation, and it will not scale beyond a single process.

---

License

Code in this repository: MIT (see LICENSE).

Model weights are a derivative of Meta's Llama 3.1 and are governed by the Llama 3.1 Community License, not by this repository's license. Redistributing them requires, per §1.b:

• the model name to begin with Llama;
• "Built with Llama" displayed prominently;
• a copy of the Agreement, and a Notice file containing: Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Status: the published HuggingFace repos do not currently meet these terms — see LLAMA_LICENSE_ACTIONS.md for the specific gaps and fixes.