Feature Performance Validation

Feature/performance validation toolkit

beellama/main includes a manifest-driven developer toolkit for comparing a baseline and candidate without losing source identity, command parity, profiler provenance, or cleanup evidence. It is test infrastructure, not a production inference feature.

What it provides

The runner validates registered build provenance, exact source trees, binary and input hashes, controlled command deltas, output exactness, staged performance gates, GPU lock ownership, interruption cleanup, and optional Nsight Systems/Compute artifacts. It separates fast screening, production confirmation, and long-context acceptance so an early microbenchmark is not reported as final proof.

It also includes an MTP exactness workflow that compares like-for-like MTP geometry across placement policies. A target-only completion is not treated as the correctness oracle for a speculative run.

There is no runtime flag and no build dependency. The core runner uses the Python standard library; NVIDIA profilers are optional external tools for CUDA studies.

Implementation and documentation

• Runner: scripts/feature-performance-validation.py
• Modules: scripts/feature_validation
• Manifest schema: manifest.schema.json
• Full operating guide: docs/feature-performance-validation.md
• MTP exactness runner: scripts/mtp-exactness.py

The toolkit is currently published only on beellama/main. It is not inherited by the llama/main branch family.

Review history: PR 13, PR 18, PR 22, and PR 24.

Feature index - BeeLlama overview