Repository navigation
v0.1.1
M3 completes all 38 Level 1 LLM experiment rules and finalizes the seven built-in
profiles: core, inference, evaluation, llm_judge, finetuning, privacy,
and safety.
An intentionally incomplete OpenAI judge manifest now reports the missing judge
sampling contract directly:
CRITICAL (1)
X judge.params_declared Missing fields: evaluation.judge.params.max_tokens, evaluation.judge.params.temperature
evaluation.judge.params.max_tokens (absent)
evaluation.judge.params.temperature (absent)
fix: Set evaluation.judge.params.max_tokens, evaluation.judge.params.temperature in reprollm.yaml.
28 passed · 0 suppressed · 11 skipped
Result: FAIL (1 critical, 2 warning)
Added (M3, second half)
- Generated rule and profile reference pages with CI freshness checks, a
documented Level 1 coverage overview, and an enforced 85% CI coverage floor
(M3-T09, spec §§6, 12, 22–23). - Complete and intentionally incomplete Level 1 manifests for the HF/vLLM,
OpenAI judge, and privacy fixtures, with six reviewed JSON snapshots and
variant-scoped dependency/secret overlays (M3-T08, spec §§3, 6.1, 12, 22). - Apply
audit.ignoreby canonical rule ID or alias, preserving the original
severity and mandatory reason in INFO findings. Honor config defaults for
fail_onandshow_passed, with explicit CLI flags taking precedence;
suppressed results remain visible and do not affect exit thresholds
(M3-T07, spec §§1.1, 8, 11, 21). - Complete built-in profile rule selections and severity overrides, with
most-derived override origins in audit findings and profile inspection.
Add 15 explicitly skipped Level 2 placeholders and preserve project-owned
severity metadata without implementing lock or runtime checks (M3-T06,
spec §§6.1, 11–12, 18.2).
Fixed (M3, second half)
- Treat either explicit boolean value for
models.<role>.trust_remote_codeas
satisfying the repository-wide detection rule, matching the selected
execution path without encouraging an inaccuratetruedeclaration
(#1, M3-T10, spec §12.4). - Compare detected profiles with the resolved inheritance closure, so a profile
inherited by the selected experiment is not reported as undeclared. Clarify
that unrelated repository capabilities do not need to be added to the
manifest (M3-T10, spec §§6, 12.3). - Allow
reprollm profiles show corewhile continuing to reject explicitly
selecting core in a manifest oraudit --profiles(M3-T06).
Documentation (M3-T10)
- Mark statically detected model IDs as candidates whose experiment role must
be verified, add a practical manifest guide, and record Project A/B Level 1
dogfooding findings and their disposition.
Added (M3, first half)
- Five training and four privacy rules for training hyperparameters, complete
LoRA adapters, threat models, nonempty mechanism parameters, privacy metrics
and attack budgets (M3-T05, spec §§12.10–12.11). - Six evaluation and four judge rules for metrics and implementation references,
aggregation, repetitions, safety definitions/budgets, and independently declared
judge roles and sampling settings (M3-T04, spec §§12.8–12.9). - Five generation/backend rules and three prompt rules, including conditional
sampling seeds, vLLM fields, explicit stop/few-shot declarations and prompt
file existence without reading prompt contents (M3-T03, spec §§12.6–12.7). - Five Level 1 dataset rules for role declarations, Hugging Face split/subset,
preprocessing and sampling seeds, with zero-valued seeds preserved (M3-T02,
spec §12.5). - Six Level 1 model declaration rules, including per-role API exclusions,
inference dtype/quantization fallback evidence, adapter detection, and explicit
remote-code trust declarations (M3-T01, spec §12.4). The trust rule is selected
by core per the maintainer-approved correction to spec §6.1.
Fixed (M2-fix corrective sprint, 2026-09-10)
- Audit JSON no longer persists the raw invocation path (absolute paths
leaked/home/...); the target is normalized to.or a repository-
relative POSIX path, andgenerated_atis second-precision UTC (F-01,
F-02). - CLI error contract enforced: output-write failures are exit 2 with an
actionable message; unexpected exceptions exit 3 without a traceback
(traceback only with-v) (F-03). - A
pyproject.tomlwithout[project]/[tool.poetry]no longer counts
as a dependency manifest (F-04). - Dependency evidence points at physical declaration lines instead of
array/mapping indexes; no moreline: 0(F-05). - Separator-equivalent keyword spellings (
red team/red-team,
dp-sgd/dp_sgd, …) count as one detection concept (F-06). - Exact
eval/evaluationdirectory segments are an evaluation signal
(F-07). - Scan diagnostics (truncation, syntax errors, unparsed declarations)
surface on stderr under-v; stdout stays one JSON document (F-08).
Added
- Normative secret fixture corpus (§16.5): positive/negative/env/file
cases for the M5 redaction classifiers (F-09).
Changed
- init/doctor domain logic extracted into
core.manifest_scaffoldand
core.diagnostics; CLI commands only parse/prompt/render/exit (F-10). - Detection signals now come from the profile YAML definitions (user
overrides honored); only schema-inexpressible semantics stay in Python
(F-11).
Added
- Authoritative five-project gold-answer validation matrix (
val.md), milestone-gated real
training/inference/API checks, and the M2 corrective sprint plan (M2F-T12).