Skip to content

SYM-20 Fix libcrypto canonical evaluation contract and add repo-local Codex skill#2

Open
iskindar wants to merge 1 commit into
masterfrom
sym-20-canonical-libcrypto-eval
Open

SYM-20 Fix libcrypto canonical evaluation contract and add repo-local Codex skill#2
iskindar wants to merge 1 commit into
masterfrom
sym-20-canonical-libcrypto-eval

Conversation

@iskindar
Copy link
Copy Markdown
Collaborator

Summary

  • add a repo-local canonical libcrypto evaluation workflow for SYM-20
  • distinguish authoritative gtBlock.pb compare from historical sidecar-union / old-binary results
  • record fresh canonical baseline and optimized metrics plus caveats

What Changed

  • add .codex/skills/runnable-libcrypto-canonical-eval/
  • add scripts/libcrypto_bench_paths.py and scripts/validate_libcrypto_ground_truth.py
  • add unit coverage for the new path resolvers and compare guards
  • add docs/exp/2026-05-10-libcrypto-canonical-eval-contract.md and index it from docs/exp/topics.md
  • add repo-local Symphony issue/follow-up skill scaffolding used to manage this workflow cleanly

Canonical Result

  • baseline: precision=0.9523329079813952, recall=0.7765067563788988
  • optimized: precision=0.9302727358917905, recall=0.7797547041527227
  • verdict: recall improves slightly, precision regresses materially, so the canonical result is mixed

Validation

  • python3 -m unittest tests.test_libcrypto_bench_paths tests.test_validate_libcrypto_ground_truth
  • python3 scripts/libcrypto_bench_paths.py binary --must-exist
  • python3 scripts/libcrypto_bench_paths.py groundtruth-pb --must-exist
  • python3 scripts/libcrypto_bench_paths.py text-start
  • python3 scripts/validate_libcrypto_ground_truth.py gap-audit --out-dir runs/groundtruth_validation/canonical_gap_audit
  • python3 scripts/validate_libcrypto_ground_truth.py cmp --ll runs/runnable-dev-2026-0429-textstart-serial/libcrypto.so.3.entry_0x500cef80.ll --out-dir runs/groundtruth_validation/textstart_serial_cmp --examples 10
  • python3 scripts/validate_libcrypto_ground_truth.py cmp --ll runs/runnable-dev-2026-0501-textstart-dynamic-reapfix/libcrypto.so.3.entry_0x500cef80.dynamic.ll --out-dir runs/groundtruth_validation/textstart_dynamic_cmp --examples 10
  • python3 scripts/validate_libcrypto_ground_truth.py cmp --ll runs/validate-libcrypto-llm-agent-current/docker_exec/baseline/libcrypto.so.3.entry_0x500cf000.ll --out-dir runs/groundtruth_validation/canonical_baseline_cmp --examples 10 --allow-low-metrics

Refs SYM-20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant