Skip to content

[codex] add AI stochastic evidence commands#162

Merged
rizumita merged 1 commit into
mainfrom
codex/ai-stochastic-evidence
Jul 9, 2026
Merged

[codex] add AI stochastic evidence commands#162
rizumita merged 1 commit into
mainfrom
codex/ai-stochastic-evidence

Conversation

@rizumita

@rizumita rizumita commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the implemented fsl-ai stochastic evidence surface: project-level AI declarations, precomputed eval checks, migration regression comparison, runtime drift checks, and AI compatibility profile generation. Removes proposal docs whose content is now reflected in implementation/design docs.

Why

  • Goal: make docs/fsl_ai_stochastic_proposal.md real rather than a stale proposal.
  • Reason: stochastic AI quality, AI migration, drift, and compatibility need explicit external-evidence commands that never claim kernel proof.
  • Alternatives: putting probability semantics into fslc verify was rejected; the implemented boundary keeps the kernel deterministic and reports formal_result:"not_run" for statistical and observed evidence.

What Changed

  • Added src/fslc/ai_project.py and src/fslc/ai_stochastic.py for project-level parsing and evidence evaluation.
  • Added CLI commands: fslc ai eval, ai regress, ai compare, ai drift, ai compat, and compat check --include-ai integration.
  • Extended ai replay to select a component from project files and detect artifact metadata mismatches plus tool_calls[] events.
  • Updated schemas, examples, snapshots, docs, website pages, and skills/fsl/reference.md.
  • Renamed AI/domain v0 result identifiers away from *-mvp.v0 wording and deleted superseded proposal documents.

Blast Radius

  • Affected: fsl-ai parsing/check/replay CLI, AI result schemas, docs/site, skill reference, corpus snapshot, and domain finding version labels.
  • Compatibility: AI result fsl identifiers now use fsl-ai-hard.v0, fsl-ai-agent.v0, and fsl-stochastic.v0. Consumers pinned to the older *-mvp.v0 labels need to accept the new v0 names.

Invariants

Condition Evidence Symptom if broken
Statistical and observed AI evidence must not be reported as kernel proof formal_result:"not_run", schema updates, tests Users could confuse eval/drift evidence with proved
AI hard-contract kernel checks remain deterministic existing and extended AI hard-contract tests fslc verify semantics would change unexpectedly
Proposal content must live in design/reference docs after proposal deletion docs/DESIGN-ai-hard.md, docs/DESIGN-stochastic.md, docs/LANGUAGE.md, skills/fsl/reference.md Deleted proposal would remove required user-facing guidance

Failure Modes

Type Pattern Detection
Likely project-level block scanner accepts or drops the wrong proposal construct tests/test_ai_stochastic.py, fslc ai check examples/ai/support_answer_quality.fsl
Severe unsupported statistical result is treated as success ai eval/regress/drift tests and CLI smoke commands
Subtle schema/result version drift between code, examples, and schemas schema tests, corpus snapshot, full pytest

Evidence

  • .venv/bin/python -m py_compile src/fslc/ai_check.py src/fslc/cli.py src/fslc/ai_expand.py src/fslc/ai_agent.py src/fslc/ai_stochastic.py src/fslc/ai_project.py
  • .venv/bin/python -m pytest tests/test_ai_stochastic.py tests/test_ai_hard_contract.py tests/test_ai_recursive_agent.py -q
  • .venv/bin/python -m pytest tests/test_corpus_snapshot.py tests/test_ai_stochastic.py tests/test_ai_hard_contract.py tests/test_ai_recursive_agent.py tests/test_db_dialect.py tests/test_domain_dialect.py tests/test_analysis_schema.py tests/test_lsp_index.py -q
  • .venv/bin/python -m pytest -q => 1035 passed, 78 skipped
  • git diff --check
  • CLI smoke: ai check, ai eval, ai replay, ai compat, ai regress, ai drift, ai compare, compat check --include-ai

Rollout & Rollback

  • Rollout: merge normally; commands are additive except for documented result identifier cleanup and proposal doc deletion.
  • Rollback: revert this commit if downstream consumers require the old labels or proposal files.
  • Cannot rollback if: no known data migration or external state change is required.

Review Focus

  • src/fslc/ai_project.py: lenient project-level parser boundary.
  • src/fslc/ai_stochastic.py: statistical/regression/drift status semantics.
  • src/fslc/ai_check.py and src/fslc/cli.py: replay/CLI integration and exit behavior.
  • docs/DESIGN-stochastic.md, docs/DESIGN-ai-hard.md, skills/fsl/reference.md: proof vs external-evidence wording.

Unknowns

Item Resolution Blocker?
Full evaluator execution and provider sampling are still external to fslc documented as outside this layer No
Paired migration statistics are stricter than aggregate comparison documented as optional external evidence No

Related

@rizumita rizumita merged commit 68134ff into main Jul 9, 2026
6 checks passed
@rizumita rizumita deleted the codex/ai-stochastic-evidence branch July 9, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant