OpenLoRA

Corpus-to-hotload LoRA adapter lifecycle for local llama.cpp servers.

OpenLoRA is a small, four-stage CLI toolkit that takes you from a folder of your own docs/code to a specialized adapter hot-swapped into a running local LLM with zero downtime:

  ┌────────────┐   ┌──────────┐   ┌───────────┐   ┌──────────────────────────┐
  │ 1 data-gen │ → │ 2 train  │ → │ 3 convert │ → │ 4 manage (live /lora)    │
  │ corpus→JSONL│   │ QLoRA    │   │ PEFT→GGUF │   │ hot-load, NO restart     │
  └────────────┘   └──────────┘   └───────────┘   └──────────────────────────┘
        manifest.json is the spine that ties the four stages together

The standout idea: once your llama-server is up, swapping a LoRA adapter in or out is an HTTP call to the server's /lora endpoint. The multi-gigabyte base weights never reload, the server never restarts, and the swap is effectively instant. That makes a "train nightly on your own work, hot-load it in the morning" flywheel practical on a single workstation GPU.

Why this matters

A general local model trades depth for breadth. Instead of paying for a bigger model, you can keep one fast base resident and specialize at the adapter layer: a small adapter for your codebase's conventions, another for your docs voice, etc. QLoRA makes training one of these feasible on a single 24 GB card; llama.cpp's /lora endpoint makes serving them feasible without an ops team.

OpenLoRA is the glue: the scrape framework, the QLoRA harness with the sharp edges already filed off (see Earned pain), the GGUF hand-off, and a dependency-free manager that drives the live endpoint.

Install

# base package (data-gen + manager stages — pure Python)
pip install -e .

# training stage (on a CUDA-enabled machine — pulls torch/peft/trl/bitsandbytes)
pip install -e ".[train]"

# dev (tests)
pip install -e ".[dev]"

Requires Python ≥ 3.10. The data-gen and manager stages are stdlib-only (plus PyYAML for --config); only the training stage needs the heavy CUDA stack.

Quickstart (no GPU needed for stages 1, 3-manager, 4-list)

# 1. Scrape the bundled sample corpus into train/eval JSONL + a manifest entry
openlora-data-gen --adapter code-docstrings --source examples/sample_docs

# 2. Validate the corpus shape without torch (CI-friendly)
openlora-train --adapter code-docstrings --dry-run

# 3. Check the manifest (schema + adapter-composition rules)
openlora-validate check

# 4. List adapter lifecycle state
openlora-manager list

To run the whole network-free smoke test (and the unit tests):

./openlora-check.sh

Train for real (CUDA)

# fp16/bf16 base; the harness 4-bit-quantizes it at load (QLoRA)
openlora-train --adapter code-docstrings \
  --base Qwen/Qwen2.5-7B-Instruct --rank 16 --epochs 3

A 7B QLoRA (NF4 + double-quant + gradient checkpointing) fits in roughly 10–12 GB free VRAM. Stop any inference server holding the card first, or point CUDA_VISIBLE_DEVICES at a free GPU.

Convert PEFT → GGUF

OpenLoRA hands off to llama.cpp's own converter (clone llama.cpp):

python3 /path/to/llama.cpp/convert_lora_to_gguf.py \
  --base <base_model.gguf> \
  --lora openlora-workspace/lora/code-docstrings-qwen2.5-7b-instruct \
  --outfile openlora-workspace/lora/code-docstrings.gguf

Hot-load into a running server (zero restart)

Start llama-server with the adapter file registered, then drive it live:

# launch (adapter registered at index 0, scale 0 = inactive)
llama-server -m base.gguf --lora-init-without-apply \
  --lora openlora-workspace/lora/code-docstrings.gguf --port 8080 &

# hot-load it — no restart, base weights stay resident
openlora-manager load code-docstrings --port 8080 --scale 1.0
openlora-manager status --port 8080
openlora-manager unload --port 8080   # scale → 0, base model restored

The four stages

Stage	Command	What it does
1. data-gen	openlora-data-gen	Walks a corpus folder; extracts (signature → docstring) pairs from .py and (section title → body) pairs from .md; builds chat-format records; 80/20 train/eval split; writes JSONL + a data-ready manifest entry.
2. train	openlora-train	QLoRA fine-tune (4-bit frozen base, PEFT/TRL). --dry-run validates data shape with no torch. Saves a PEFT adapter; marks the manifest trained.
3. convert	(external)	llama.cpp's convert_lora_to_gguf.py turns the PEFT adapter into a .gguf llama-server can load.
4. manage	openlora-manager	load / unload / status / list over llama.cpp's /lora HTTP API — live, no restart.
(aux) validate	openlora-validate	Manifest schema check + layer-range composition check (see below).

Adding your own adapter

The extraction primitives are generic. To add an adapter, write a function that returns a list[dict] of make_pair(system, user, assistant) records and register it in ADAPTERS in src/openlora/data_gen.py. Or just point --source at your own docs/code folder and reuse the bundled code-docstrings generator.

Configuration / KNOBS

Nothing is hardcoded to a machine, user, or model. Resolution order (highest wins): CLI flags → --config YAML → OPENLORA_* env vars → built-in defaults (self-contained under ./openlora-workspace/). See .env.example and examples/openlora.example.yaml.

Knob	CLI flag	Env var	Default	Meaning
Corpus dir	--source	OPENLORA_CORPUS_DIR	examples/sample_docs	Folder scraped by data-gen
Training data dir	--out	OPENLORA_TRAINING_DATA	openlora-workspace/training-data	Where JSONL splits are written
LoRA dir	—	OPENLORA_LORA_DIR	openlora-workspace/lora	PEFT + .gguf adapters + manifest.json
Base model	--base	OPENLORA_BASE_MODEL	Qwen/Qwen2.5-7B-Instruct	HF id or local safetensors dir
LoRA rank	--rank	—	16	Adapter rank (alpha = 2×rank)
Epochs	--epochs	—	3	Training epochs
Min VRAM	--min-vram-gb	—	10.0	Abort training below this free VRAM
MoE base	--moe	—	off	Target attention-only (see Earned pain)
Server host	--host	OPENLORA_SERVER_HOST	localhost	llama-server host
Server port	--port	OPENLORA_SERVER_PORT	8080	llama-server port
Config file	--config	OPENLORA_CONFIG	—	YAML overriding env + defaults

Earned pain

These are the sharp edges this toolkit has already paid for. They live as comments in src/openlora/train.py and are summarized here because they're the whole point of shipping the code rather than a tutorial.

• device_map="auto" will sabotage a single-GPU QLoRA run. On one card where the 4-bit model fits, "auto" can still conservatively offload a sliver to CPU. That either hard-errors (offload forbidden) or — if you permit CPU offload via llm_int8_enable_fp32_cpu_offload — trips accelerate's meta-device dispatch and crashes inside bitsandbytes with Params4bit.__new__() ... _is_hf_initialized. Fix: on a single GPU use device_map={"": 0} and do not set max_memory or enable CPU offload. Only enable offload/max_memory when there is genuinely more than one GPU to split across.

• Eval will OOM at the epoch boundary without prediction_loss_only=True. The default eval loop materializes the full-vocab logits tensor (gigabytes for a large vocab) right when VRAM is already tight. Setting prediction_loss_only computes only the loss and skips that allocation.

• Don't LoRA a fused-MoE's expert FFNs by name. gate_proj/up_proj/down_proj live inside every routed expert; on an unpruned N-expert model, targeting them by bare name explodes into N_experts × 3 × N_layers adapters. For an unpruned MoE, target attention only (--moe flag). Only LoRA the experts' FFN linears on a pruned MoE where the remaining count is tractable.

• Layer-range collision when stacking adapters. A LoRA (weight-space) and a soft-prompt / P-Tuning adapter (activation-space) can occupy orthogonal parameter spaces yet still collide in activation space when they target overlapping layers — the soft prompt shifts the activation distribution before the LoRA- modified weights compute, pushing activations outside the subspace the LoRA was trained on. openlora-validate composition enforces disjoint target_layer_range values (e.g. one adapter on layers 0–15, another on 16–31) to avoid this.

• LoRA on a quantized base degrades quality slightly. Train on an fp16/bf16 base when you can; apply the resulting adapter at INT4 for serving.

How the live swap works

llama.cpp's server exposes a /lora endpoint. Adapters are registered at launch via --lora <file> (each gets an index in launch order). At runtime you POST a list of {"id": index, "scale": s} to change which adapters are active and how strongly. OpenLoRA's manager maps your friendly adapter name → its index via the manifest, so you say load code-docstrings instead of bookkeeping indices. unload sets every known adapter's scale to 0, restoring the pure base model.

Project layout

src/openlora/
  config.py      # CLI/env/YAML config resolution (no hardcoded paths)
  manifest.py    # pure helpers: make_pair, JSONL I/O, split, manifest state
  data_gen.py    # stage 1 — corpus → JSONL (+ extraction primitives)
  train.py       # stage 2 — QLoRA harness (lazy heavy imports; --dry-run)
  manager.py     # stage 4 — /lora hot-load client (stdlib only)
  validator.py   # manifest schema + layer-range composition checks
examples/
  sample_docs/   # tiny demo corpus for the code-docstrings adapter
  openlora.example.yaml
tests/           # network-free unit tests on the pure functions
openlora-check.sh  # end-to-end network-free smoke test

Tests

pip install -e ".[dev]"
pytest -q
# or the full smoke test:
./openlora-check.sh

The unit tests cover the pure functions only (chat-pair construction, JSONL I/O, train/eval split, manifest state transitions, markdown/docstring extraction, and the manifest validator including the layer-range overlap rule) — no network, no GPU, no model downloads.

License

MIT — see LICENSE.