Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docs/measurements/v0.9.0-corpus-baseline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# v0.9.0 Corpus Baseline -- LOOM vs wasm-opt -O3

_Generated by `scripts/measure_corpus.sh` at `2026-05-14T16:46:17Z`._

- LOOM commit: `723f6a5844019b6f43c888cbecb1e0132cd74f53`
- LOOM branch: `release/v0.9.0-pr-p-corpus-harness`
- LOOM version: `loom 0.8.0`
- wasm-opt: `wasm-opt version 116 (version_116)` (used)
- wasm-tools: `wasm-tools 1.243.0`

## Headline

On this corpus (only workloads where both LOOM and wasm-opt produced valid output): LOOM produced a **smaller** output than wasm-opt on: gale.

Missing fixtures (skipped, marked `n/a`):
- `httparse`
- `nom_numbers`
- `state_machine`
- `json_lite`
- `loom`
- `calculator`

## Results

| Workload | Baseline | LOOM | wasm-opt -O3 | wasm-opt -> LOOM | LOOM Δ% vs base | wasm-opt Δ% vs base | Note |
|---|---:|---:|---:|---:|---:|---:|---|
| gale | 1941 | 1846 | 1925 | 1846 | -4,9 | -0,8 | kernel-FFI fixture |
| httparse | n/a | n/a | n/a | n/a | n/a | n/a | HTTP parser |
| nom_numbers | n/a | n/a | n/a | n/a | n/a | n/a | parser-combinator primitives |
| state_machine | n/a | n/a | n/a | n/a | n/a | n/a | FSM kernel |
| json_lite | n/a | n/a | n/a | n/a | n/a | n/a | minimal JSON tokenizer |
| loom | n/a | n/a | n/a | n/a | n/a | n/a | LOOM self-build (dogfood target) |
| calculator | n/a | n/a | n/a | n/a | n/a | n/a | component-shaped fixture |
| calculator_root | 2337724 | 2327794 | error | n/a | -0,4 | -100,0 | 2.3 MB component (root, large) |
| simple_component | 261 | 212 | error | n/a | -18,8 | -100,0 | tiny component (adapter-heavy) |
| calc_component | 442 | 392 | error | n/a | -11,3 | -100,0 | small component (adapter-heavy) |

## Methodology

For each workload (fixture path is relative to repo root):
1. Record baseline byte count via `wc -c` and code-section size via `wasm-tools dump`.
2. Run `loom optimize <fixture> -o <name>.loom.wasm`.
3. Run `wasm-opt -O3 <fixture> -o <name>.wopt.wasm` (skipped if wasm-opt unavailable).
4. Re-run LOOM on the wasm-opt output (`wasm-opt -> LOOM` column).
5. Validate every output via `wasm-tools validate`. **A validation failure is a HARD ERROR** -- the harness aborts with exit code 2.

Conventions:
- Δ% is `(out - base) / base * 100`. Negative means smaller (better).
- A row is flagged :red_circle: if LOOM grew the file vs. baseline, or if wasm-opt beats LOOM by more than 1% of baseline.
- Outputs of every run are in `/tmp/loom-measure-corpus` for forensic inspection.

## Reproducing

```bash
# Build LOOM first (Z3 verification enabled)
Z3_SYS_Z3_HEADER=/opt/homebrew/include/z3.h \
LIBRARY_PATH=/opt/homebrew/lib cargo build --release

# Run the harness
bash scripts/measure_corpus.sh
```
Loading
Loading