Runtime Integration

Runtime Integration

Guidance on choosing quantization formats and connecting BitForge outputs to inference runtimes.

Decision matrix

Runtime	Preferred formats	BitForge path
llama.cpp	GGUF (Q4_K_M, Q8_0, …)	GGUFConverter → quantize GGUF
Ollama	GGUF Modelfile	Same as llama.cpp; import via ollama create
Super-Ollama	GGUF	Super-Ollama consumes BitForge GGUF artifacts
vLLM	AWQ, GPTQ, FP16	Export HF-compatible quant checkpoint
Hugging Face Transformers	GPTQ, AWQ, bitsandbytes	Direct from_pretrained with quant config
ExLlamaV2 / TabbyAPI	EXL2 (external)	Convert outside BitForge today

When in doubt: GGUF for local CPU/GPU via llama.cpp family; GPTQ/AWQ for GPU server stacks (vLLM, TGI).

llama.cpp / Ollama / Super-Ollama

Workflow

1. Quantize or fine-tune in Hugging Face format
2. Convert to FP16 GGUF via GGUFConverter.convert_to_f16()
3. Apply llama.cpp quant preset via GGUFConverter.quantize_gguf()

from bitforge.core.gguf import GGUFConverter

converter = GGUFConverter("./hf-model-dir")
converter.validate_metadata()
f16 = converter.convert_to_f16("./model-f16.gguf")
converter.quantize_gguf(f16, "./model-q4_k_m.gguf", quant_type="Q4_K_M")

Common GGUF quant types

Type	Bits (approx)	Quality / speed
Q8_0	8	Highest GGUF quality, larger files
Q4_K_M	~4.5	Recommended default for 7B–13B
Q4_K_S	~4	Smaller, slightly lower quality
Q2_K	~2	Aggressive; use with caution

Super-Ollama

Super-Ollama is Inference Foundry's inference layer for running quantized models locally. BitForge produces the artifacts; Super-Ollama handles loading, scheduling, and serving.

Typical handoff:

BitForge (quantize → GGUF) → Super-Ollama (serve)

Cross-link experiment results (perplexity, latency) with Super-Ollama bench configs to validate end-to-end quality.

vLLM

vLLM supports AWQ and GPTQ checkpoints with compatible config JSON. After bitforge quantize --method gptq|awq:

from vllm import LLM

llm = LLM(model="./out/mistral-7b-gptq", quantization="gptq")

Check vLLM docs for supported model architectures and minimum GPU capability.

Hugging Face Transformers

GPTQ

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "./out/model-gptq",
    device_map="auto",
)

Requires auto-gptq and a model with embedded quant metadata.

AWQ

from awq import AutoAWQForCausalLM  # or transformers native AWQ loader

model = AutoAWQForCausalLM.from_quantized("./out/model-awq")

bitsandbytes (4-bit load)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

Best for training / QLoRA; not a portable single-file artifact like GGUF.

QLoRA merge considerations

When fine-tuning with QLoRA on a 4-bit base and merging adapters for deployment:

• Merged weights may re-quantize differently than base + adapter at runtime
• Experiment 03 measures this rounding error
• For production: prefer serving adapter separately or re-run GPTQ/AWQ post-merge

See Experiments.

Validation checklist

Before deploying a quantized artifact:

• Perplexity within acceptable delta vs. FP16 baseline
• Latency and VRAM meet hardware budget
• Spot-check downstream tasks (reasoning, code, factual QA)
• Runtime loads checkpoint without fallback to FP16
• Tokenizer and chat template unchanged from base model

BitForge evaluate and experiment scripts automate the first two; task-specific eval is still manual today.