diff-qwen

A lightweight tool for layer-by-layer comparison of two safetensor models. Compare any two safetensor model directories side-by-side with interactive visualizations in the browser.

Note: Large models may take a moment to load individual layers for analysis — only one tensor is loaded into memory at a time to keep RAM usage low.

Quick Start

1. Download the models

pip install huggingface_hub

# Download both models into models/ directory
huggingface-cli download Qwen/Qwen3-4B --local-dir models/Qwen3-4B
huggingface-cli download refortifai/Qwen3-4B-obfuscated --local-dir models/Qwen3-4B-obfuscated

2. Install and run

pip install -e .
python -m diff_qwen models/Qwen3-4B models/Qwen3-4B-obfuscated

Then open http://localhost:8787 in your browser.

Options

Flag	Default	Description
--port	8787	Server port
--host	0.0.0.0	Server host

Features

• Memory-safe: Only loads one tensor at a time. Never loads full model into RAM.
• Lazy header parsing: Instant startup - reads safetensor headers only (~KB).
• Sharded model support: Auto-detects model.safetensors.index.json for multi-shard models.
• BF16/FP16/FP32: Handles all dtypes natively via PyTorch backend.
• No build step: Vanilla JS + Plotly.js from CDN. Just pip install and run.

Per-Layer Statistics

• Min, max, mean, std, norm, zero count
• Value distribution histogram
• Box plot (quartiles, whiskers)
• Weight heatmap (subsampled for large tensors)
• Sparsity pattern
• Row/column statistics (means, stds, norms)

Comparison View

• Cosine similarity, max/mean absolute difference, norm ratio
• Overlaid distributions (A vs B on same axes)
• Q-Q plot (quantile comparison)
• Difference heatmap
• Per-row cosine similarity
• Overlaid row/column norms

Overview

• Progressive SSE stream computing cosine similarity across all shared layers
• Color-coded layer list (green = identical, red = divergent)

Requirements

• Python 3.10+
• PyTorch >= 2.0
• safetensors >= 0.4
• FastAPI >= 0.100
• uvicorn >= 0.20

Screenshot

Compare view for model.layers.0.mlp.gate_proj.weight — showing overlaid distributions, Q-Q plot, difference histogram, abs difference heatmap, and per-row cosine similarity between the original and obfuscated Qwen3-4B models:

License

RefortifAI 2026. Free to use, modify, and distribute — this is an open source tool built for the RefortifAI challenge.