SLM Experiments

Evaluate whether inference-time interventions make small language models (0.5B–3.8B) produce simpler English for beginner learners. Primary binary outcome is CEFR-SP document level A1 (meets_a1_criteria when cefr_sp_level == "A1").

Quick Start

Thesis path = Phase 2 sweeps (weights, prompting, guided, kvl_beam). Formal claims use --prompts all (25 prompts); the CLI default n=3 is a smoke-test guardrail only. Phase 1 (factorial) remains in the codebase but is out of thesis scope.

# 1. Create virtual environment
python3.11 -m venv venv
source venv/bin/activate

# 2. Install dependencies (editable install adds src/ to Python path)
pip install -e .
pip install -r requirements-dev.txt

# 3. GGUF models — auto-resolved from sibling thesis repo by default:
#    ../SLMs-master-thesis/Tesis/Codigo/models/gguf/
#    Override with SLM_GGUF_DIR or copy files into models/gguf/

# 4. Run a Phase 2 sweep (default: 3 prompts = smoke test)
python -m slm_experiments phase2 weights

# Formal thesis run (25 prompts)
python -m slm_experiments phase2 weights --prompts all

CLI Overview

Run python -m slm_experiments --help for a quick-start guide with examples.

# Phase 2 — thesis path: hyperparameter sweeps (all 4 models)
python -m slm_experiments phase2 weights   [--weights 1.0,1.5,2.0,4.0] [--prompts N|all]
python -m slm_experiments phase2 prompting [--shots 0,1,3]              [--prompts N|all]
python -m slm_experiments phase2 guided    [--top-k-pools 5,10,20]      [--prompts N|all]
python -m slm_experiments phase2 kvl_beam  [--widths 4,8]               [--prompts N|all]
# phase2 beam is deprecated (hard-fails at temperature=0)

# Phase 1 — optional / out of thesis scope (2×2 factorial)
python -m slm_experiments phase1 [--prompts N|all] [--models all|Qwen3,...] [--seed 42]

# Post-run utilities
python -m slm_experiments plot --run-id <id>
python -m slm_experiments runs list
python -m slm_experiments runs show <id>

# Human evaluation
python -m slm_experiments human export --run-id <id> [--sample 60]
python -m slm_experiments human import --run-id <id> --tags <csv>

Results

Every run writes a self-contained bundle to results/runs/{run_id}/:

File	Description
manifest.json	Run metadata, CLI args, observation counts
specification.csv	Reduced columns, European decimals (paper-compatible)
full.csv	All fields including guided / KVL metadata
summary.json	Aggregated stats (overall + by_config + by_model; Phase 2 adds sweep sections)
plots/	Boxplots (after plot --run-id)

Run ID format: {YYYYMMDD_HHMMSS}_{phase}_{experiment}

ClusterUY (HPC)

Run Phase 2 sweeps on Uruguay's national cluster via Singularity (not Docker on-cluster):

# On cluster login node
cd ~/SLMs-experiments
sbatch scripts/clusteruy/smoke_test.sh          # quick Phi3 check
sbatch scripts/clusteruy/run_phase2_weights.sh  # full sweep

Full workflow (SSH, image pull, results download): docs/clusteruy.md

Dockerfile lives in the sibling thesis repo: ../SLMs-master-thesis/Tesis/Codigo/scripts/clusteruy/Dockerfile

Documentation

Document	Purpose
AGENTS.md	Agent entry point — structure, CLI, rules
ExperimentDesign.md	Formal experiment specification
docs/clusteruy.md	ClusterUY SSH, Singularity, batch jobs
docs/metrics.md	Readability metrics and proxy thresholds
docs/models.md	GGUF files, chat templates, GPU setup
docs/interventions.md	Weighting, prompting, guided, KVL (beam deprecated)
docs/guided-decoding.md	Top-k A1-guided decoding (phase2 guided)
docs/kvl_beamsearch.md	KVL-scored beam (phase2 kvl_beam)

Testing

pytest

Most tests mock the pipeline — no GGUF files required.

Relationship to Thesis Repo

This repo is a clean replacement for Tesis/Codigo/ in the SLMs-master-thesis repository. The thesis paper is frozen; this repo carries the experiment framework forward with run-centric results and a single CLI entry point.