Experiments

Experiments

Reproducible scripts for measuring quantization quality and performance. Each lives in experiments/ and can be run standalone.

Overview

| # | Script | Measures | |--------|-------| | 01 | 01_perplexity_benchmarks.py | Perplexity across FP16, INT8, INT4, INT2 | | 02 | 02_latency_memory.py | TTFT, tokens/sec, peak VRAM | | 03 | 03_qlora_merge.py | Error from merging LoRA into 4-bit base | | 04 | 04_outlier_detection.py | Activation outliers and suppression |

Experiment 01 — Perplexity benchmarks

Goal: Reproducible perplexity sweeps across bit widths for reference models (e.g. Mistral-7B) on Wikitext-2 and C4.

Run:

python experiments/01_perplexity_benchmarks.py

Uses: QuantizationBenchmarkSuite from bitforge.evaluation.benchmarks

Output: Table with columns model_id, bits, perplexity, vram_gb, tokens_per_sec

Customize: Edit model_id and datasets in the script's run() function, or import the suite in your own notebook.

Experiment 02 — Latency and memory

Goal: Measure inference throughput and memory on consumer hardware (CPU/GPU).

Run:

python experiments/02_latency_memory.py

Metrics:

• peak_vram_gb — maximum GPU memory during generation
• tokens_per_sec — sustained decode throughput
• time_to_first_token_sec — prefill + first decode latency

Uses: latency and memory_footprint from bitforge.evaluation.metrics

Experiment 03 — QLoRA merge rounding error

Goal: Quantify precision loss when merging a high-precision LoRA adapter into a 4-bit quantized base.

Workflow (planned full pipeline):

1. Load 4-bit NF4/FP4 base model
2. Attach FP16 LoRA adapter
3. Dequantize base, merge: W_merged = dequantize(W_base) + (A × B) × scale
4. Compare output divergence vs. FP16 base + LoRA

Run:

python experiments/03_qlora_merge.py

Output: MSE and max weight divergence metrics.

Experiment 04 — Outlier detection

Goal: Find activation channels with extreme spikes that break naive INT8 quantization, and simulate suppression (SmoothQuant).

Run:

python experiments/04_outlier_detection.py

Uses: detect_outliers from bitforge.evaluation.metrics

Output: Outlier channel count, indices, max activation value.

The script injects synthetic outliers at fixed channels to validate detection before running on real model activations.

Extending experiments

When adding a new experiment:

1. Create experiments/NN_descriptive_name.py with a run() entry point
2. Import from bitforge rather than duplicating metric logic
3. Log configuration at startup (model ID, bit width, device, dataset)
4. Print or save structured results (JSON/CSV) for comparison across runs

CI and regression testing

One of BitForge's research goals is a lightweight benchmark that catches quantization regressions in CI. Experiment 01 is the starting point—once mock data is replaced with real pipelines, a subset (e.g. Wikitext-2 perplexity on a tiny model) could gate PRs.

See Contributing if you want to help wire this up.