Alignify

An end-to-end LLM alignment pipeline — instruction tuning, reward modeling, PPO, and DPO — built as a real, runnable Flask app instead of a set of standalone notebooks.

All four stages are live in this release: Instruction Tuning, Reward Modeling, PPO (RLHF), and DPO. Each one is a real technique trained on a real dataset, wired into one consistent UI with a shared design system and a shared live/demo inference pattern.

Why this exists

Fine-tuning a base language model and aligning it with human preference is usually shown as isolated notebook exercises. Alignify turns that into one coherent product: a base model gets instruction-tuned, a reward model learns to judge its outputs from human preference pairs, then two different alignment strategies — PPO and DPO — use that same idea of "preferred vs. rejected" to steer generation, each with its own working demo and real training metrics.

Stage 01 — Instruction Tuning

• Base model: facebook/opt-350m
• Dataset: CodeAlpaca-20k
• Method: LoRA (r=16, alpha=32, target modules q_proj/v_proj, dropout 0.1)
• Eval metric: SacreBLEU against reference responses
• What the app does: takes an instruction, shows the base model's response next to the LoRA-tuned model's response, side by side.

Stage 02 — Reward Modeling

• Base model: gpt2 (GPT2ForSequenceClassification, num_labels=1)
• Dataset: Dahoas/synthetic-instruct-gptj-pairwise (chosen / rejected response pairs per prompt)
• Method: LoRA (r=8, task_type=SEQ_CLS) via TRL's RewardTrainer
• What the app does: takes a prompt and two candidate responses, scores both, and shows which one the reward model prefers.

Stage 03 — PPO (RLHF)

• Base model: lvwerra/gpt2-imdb
• Dataset: IMDB movie reviews
• Reward signal: a sentiment classifier (lvwerra/distilbert-imdb) scores generations; PPO (trl.PPOTrainer, lr=1.41e-5) optimizes the policy against that reward with a KL penalty against the reference model
• What the app does: takes the start of a review-style sentence and shows how the PPO-tuned model continues it, alongside the sentiment reward score.

Stage 04 — DPO

• Base model: gpt2
• Dataset: BarraHome/ultrafeedback_binarized (chosen / rejected pairs)
• Method: LoRA (r=4, target modules c_proj/c_attn, alpha=8) via TRL's DPOTrainer (beta=0.1) — optimizes directly on preference pairs, no separate reward model or RL rollouts needed
• What the app does: takes a prompt and shows the base model's response next to the DPO-tuned model's response.

Live inference vs. demo mode

Training a 350M-parameter model and a GPT-2 classifier isn't something that runs on a CPU-only machine in reasonable time, so the app is built to run in two modes and tells you which one is active:

• Live mode — if a trained checkpoint exists under the matching models/local/* path (sft_adapter/, reward_adapter/, ppo_model/, dpo_adapter/), the app loads it and runs real inference for that stage.
• Demo mode — otherwise, the app falls back to the example outputs and training logs shipped in data/, generated from the same prompt templates and dataset structure the real training scripts use. Every page clearly labels which mode you're looking at.

This means the repo is fully explorable and reviewable with zero setup, and becomes a real inference app the moment you (or a reviewer with a GPU) run the training scripts.

Project structure

alignify/
├── app.py                      # Flask entry point
├── config.py                   # Paths, model names, env-driven config
├── alignify/
│   ├── __init__.py              # App factory
│   ├── routes/
│   │   ├── main.py              # HTML page routes
│   │   └── api.py               # JSON API (sft/reward/ppo/dpo generate+score endpoints)
│   ├── services/
│   │   ├── model_registry.py    # Lazily loads real checkpoints if present
│   │   ├── sft_service.py       # Instruction-tuning inference / demo fallback
│   │   ├── reward_service.py    # Reward scoring inference / demo fallback
│   │   ├── ppo_service.py       # PPO generation + sentiment scoring / demo fallback
│   │   ├── dpo_service.py       # DPO generation / demo fallback
│   │   └── data_loader.py
│   ├── templates/               # Jinja2 templates
│   └── static/{css,js}/         # Hand-written CSS, dependency-free canvas charts
├── training/
│   ├── train_sft.py             # Real, GPU-runnable SFT training script
│   ├── train_reward_model.py    # Real, GPU-runnable reward model training script
│   ├── train_ppo.py             # Real, GPU-runnable PPO training script
│   ├── train_dpo.py             # Real, GPU-runnable DPO training script
│   └── requirements-{sft,reward,ppo,dpo}.txt   # Separate pinned deps per script
├── data/
│   ├── sample_outputs/          # Shipped demo-mode example outputs
│   └── training_logs/           # Shipped demo-mode training curves
├── tests/
│   └── test_app.py
├── requirements.txt
├── Dockerfile
└── .env.example

Running locally

git clone https://github.com/tanishcode-12/alignify.git
cd alignify
python -m venv venv && source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
python app.py

Visit http://localhost:5000. This runs in demo mode out of the box — no GPU, no downloads required.

Running with Docker

docker build -t alignify .
docker run -p 5000:5000 alignify

Running tests

pip install pytest
pytest tests/

Every push to main and every pull request also runs this same test suite (plus a syntax lint and a Docker build check) via GitHub Actions — see .github/workflows/ci.yml. The badge at the top of this file reflects the latest run once you push this repo to GitHub.

Running with your own checkpoints (live mode)

Each training script pins its own dependency versions (they were built against different trl/transformers releases) — install each in its own virtual environment on a CUDA machine:

# Stage 01 — download CodeAlpaca-20k.json first (see training/train_sft.py docstring)
pip install -r training/requirements-sft.txt
python training/train_sft.py --data-path data/raw/CodeAlpaca-20k.json

# Stage 02
pip install -r training/requirements-reward.txt
python training/train_reward_model.py

# Stage 03
pip install -r training/requirements-ppo.txt
python training/train_ppo.py

# Stage 04
pip install -r training/requirements-dpo.txt
python training/train_dpo.py

Each script saves its checkpoint to the matching models/local/* path and overwrites the matching file in data/training_logs/ with the real run's metrics. Restart the Flask app and that stage's page switches to live mode automatically — no code changes needed.

Roadmap

• Stage 01 — Instruction Tuning (SFT + LoRA)
• Stage 02 — Reward Modeling
• Stage 03 — PPO (RLHF), steering generation using a sentiment reward model
• Stage 04 — DPO, direct preference optimization without a separate reward model

Acknowledgments

Training pipelines adapted from labs in IBM's Generative AI Engineering Professional Certificate (Coursera), rebuilt as standalone production scripts and wired into a real web application rather than notebook exercises.

License

MIT — see LICENSE.

Built as part of a portfolio project — see the roadmap above for what's next.