ncp-olmo-eval 0.1.1

ncp-olmo-eval 0.1.1

This maintenance release moves the canonical project home to
LUMIA-Group/ncp_olmo_eval.

Changes

• Update project metadata, documentation, OCI source labels, and release checks
  for the new GitHub organization.
• Publish the 0.1.1 runtime and scorer images under ghcr.io/lumia-group.
• Preserve the benchmark, inference, scoring, and artifact contracts from
  0.1.0; this release introduces no evaluation-protocol change.
• Keep the immutable 0.1.0 image coordinates documented for historical
  reproducibility.

The release workflow publishes signed-by-OIDC Python distributions to PyPI and
attaches the wheel, source distribution, and immutable public image manifest to
this release.

ncp-olmo-eval 0.1.0

ncp-olmo-eval 0.1.0

First stable release of the scheduler-neutral vLLM evaluation toolkit for
stock OLMo and NCP-ArchPreview checkpoints.

Included

• Reproducible GSM8K, SciQ, Core88, RULER, and HELMET workflows.
• Fail-closed model registration, inference planning, scoring, validation, and
  result finalization.
• Core88 execution sandboxes and stable 88-column / 30-column CSV output.
• Optional, separately registered NCP DFlash speculative decoding for GSM8K
  and Core88, including single-file and sharded SafeTensors draft checkpoints.
• Portable local, Slurm, Kubernetes, and Apptainer adapters without
  site-specific scheduler or storage assumptions.
• Five public GHCR image recipes and immutable digest manifests.

Install

python -m pip install 'ncp-olmo-eval[vllm,helmet,scoring]==0.1.0'

This release promotes the validated 0.1.0a16 protocol and implementation
without changing benchmark semantics. See docs/PROTOCOLS.md,
docs/IMAGES.md, and docs/RELEASE_CHECKLIST.md for exact contracts and
operational boundaries.

ncp-olmo-eval 0.1.0a16

PyPI-publishable follow-up to v0.1.0a15.

• Uses the official lm-eval==0.4.13 release instead of a Git direct dependency while retaining strict runtime identity checks.
• Rejects direct URL requirements in release-contract tests.
• Fixes checkout-free GitHub Release asset attachment for Python distributions and OCI digest manifests.

The GSM8K task files are unchanged between the previously validated development commit and the pinned v0.4.13 upstream tag.

ncp-olmo-eval 0.1.0a15

What's Changed

• Prepare v0.1.0a15 public release by @LuckySJTU in https://github.com/LuckySJTU/ncp_olmo_eval/pull/7

Full Changelog: https://github.com/LuckySJTU/ncp_olmo_eval/compare/v0.1.0a14...v0.1.0a15

v0.1.0a13

Seal and replay the complete NCP DFlash operating point across A/B verification, registration, Core88, and GSM8K. Preserve the verified batch/queue/adaptive-width policy, exercise continuous admission with a full scheduler queue, and reject legacy unsealed comparison artifacts.

v0.1.0a12

v0.1.0a12 adds an opt-in NCP DFlash speculative-decoding integration for the pinned vLLM 0.13.0 runtime.

Highlights:

• packaged DFlash proposer and vLLM model-runner adapter without patching installed vLLM files
• immutable target, draft, runtime, and comparison-artifact binding
• standalone target/spec A/B benchmark with GPU-memory and proposal/rejection/rollback telemetry
• continuous scheduler queue and request-local state handling
• verified speculative registrations routed only through GSM8K and Core88; SciQ, RULER, and HELMET remain target-only
• exact-labelled modes fail closed unless every generated token matches; the tested stateful NCP target did not establish full exact parity
• segmented_kv_approx requires explicit opt-in, a separate evaluation identity, positive benchmark speed evidence, and matched downstream quality A/B

Validation:

• 98 host contract tests passed; 25 GPU-runtime tests were skipped locally and covered by the separate H200 smoke where applicable
• GitHub host and formal scorer-Docker contract jobs passed on the merged revision
• clean installed-wheel CLI and Core88 source-identity checks passed
• H200 target/draft loading plus approximate proposal, rejection, rollback, and telemetry completed end to end

No private checkpoint, mount, registry, scheduler endpoint, or credential is included in the release.

Artifact SHA-256:

• wheel: 28c2054d3f6cd3c0665f576e0f6cf7c65161cf54d32f2481d987767f8c3dc236
• source distribution: afd1cca426d80f8778b2b62adb1a01ad29f0b731ba3c5b1ce2ee9bdad2cb55fc

v0.1.0a14

Final alpha validation release for the sealed NCP DFlash continuous-batching path.

• Keeps the 0.1.0a13 inference and benchmark protocols unchanged.
• Publishes the fixed 1:8,2:8,4:4,8:2 batch/width sweep and a mixed-length 32-request continuous-queue result (1.43x steady-state throughput for the tested pair).
• Records the 14.82-second draft cold-start increment, 97.25% observed acceptance lower bound, and matched GSM8K/HumanEval quality deltas.
• Adds release-version consistency coverage and sanitized machine-readable validation evidence.

The throughput and quality figures are pair-specific. segmented_kv_approx remains explicitly approximate, requires a fresh identity-bound comparison artifact and matched downstream A/B, and is never reported as target-exact.

v0.1.0a11

v0.1.0a11 adds SciQ to the scheduler-neutral vLLM evaluation workflow.

SciQ is sealed to the official OLMo export at all_supported_local task 348:

• 1,000 zero-shot Question:/Answer: examples
• four continuation likelihoods per example
• seed 42, batch size 8, one 8-GPU inference task
• official raw accuracy (acc), never acc_norm
• exact source SHA-256 and complete unique-prediction coverage checks
• CPU rescoring and final score.json / sciq-score.csv

The release also adds subset-aware Core-native planning and aggregation needed to evaluate the sealed SciQ task without relaxing full-profile order validation. SciQ remains standalone and does not change Core88's 88-task or 30-column contracts.

CI, clean-wheel installation, hcluster real-asset loading, one-machine dispatch planning, and a synthetic 1,000/1,000 full rescore were validated before release.

v0.1.0a9

Fix Core88 final 88/30-column aggregation from installed release packages by forwarding the recorded evaluator source-tree SHA-256 into the workflow validator. Adds unit coverage and a clean-wheel CI smoke with different workflow and site-packages roots. Inference, scoring data, and benchmark protocols are unchanged.

v0.1.0a10

Fix cached Core88 finalization across release upgrades. The evaluator commit/tree that produced the prediction-score snapshot remains immutable and is validated against the workflow; the newer clean finalizer is recorded separately instead of overwriting scorer identity. Adds cached-report and installed-package path regressions. Inference, scoring data, and benchmark protocols are unchanged.