Getting Started

Getting Started

This page covers installation, environment setup, and your first BitForge commands.

Requirements

Requirement	Version
Python	≥ 3.8
PyTorch	≥ 2.1
GPU (recommended)	CUDA-capable for quantization and eval

BitForge targets consumer and workstation hardware. CPU-only runs are possible for small models and metric utilities, but GPTQ/AWQ calibration is GPU-bound in practice.

Installation

From source (recommended)

git clone https://github.com/Inference-Foundry/BitForge.git
cd BitForge
pip install -e .

Dependencies only

pip install -r requirements.txt

Core dependencies include torch, transformers, accelerate, bitsandbytes, auto-gptq, autoawq, gguf, datasets, and click.

Verify the install

bitforge --help

Expected subcommands:

• bitforge quantize — run quantization on a Hugging Face model
• bitforge evaluate — measure perplexity, latency, and memory

Quick start

Quantize a model

bitforge quantize \
  --model-id meta-llama/Llama-2-7b-hf \
  --bits 4 \
  --method gptq \
  --output-path ./artifacts/llama-2-7b-gptq-4bit

Supported methods: ptq, gptq, awq, bitsandbytes. Supported bit widths: 2, 4, 8.

Evaluate a quantized model

bitforge evaluate \
  --model-id ./artifacts/llama-2-7b-gptq-4bit \
  --dataset wikitext2 \
  --device cuda

Run an experiment script

python experiments/01_perplexity_benchmarks.py

See Experiments for the full list.

Open the interactive lab

The site/ folder contains a static dashboard for exploring quantization trade-offs—no build step required.

cd site
python -m http.server 8080
# open http://localhost:8080

Or open site/index.html directly in a browser. See Interactive Lab for details.

Hardware notes

• VRAM: 4-bit quantization of a 7B model typically needs ~6–8 GB GPU memory during calibration; inference can run lower depending on runtime.
• Apple Silicon: use --device mps for evaluation where supported; some quant backends remain CUDA-only.
• Disk: quantized artifacts and calibration caches can be several GB per model.

Next steps

• Architecture — understand the package layout
• Quantization Theory — choose a method and bit width
• Runtime Integration — pick the right export format for your inference stack