Skip to content

Make debug artifacts ATIF-compatible#380

Merged
mmprotest merged 1 commit into
mmprotest/add-full-transcript-and-atif-v1.7-artifactsfrom
mmprotest/patch-villani-runner-for-terminal-bench-compatibility
Jun 5, 2026
Merged

Make debug artifacts ATIF-compatible#380
mmprotest merged 1 commit into
mmprotest/add-full-transcript-and-atif-v1.7-artifactsfrom
mmprotest/patch-villani-runner-for-terminal-bench-compatibility

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Ensure Villani’s generic debug artifacts contain package and model/provider metadata sufficient for later conversion to an external ATIF/packaging format without adding any benchmark-specific paths or logic.
  • Provide a stable separation between API compatibility (OpenAI-compatible request format) and the actual inference provider (e.g., LM Studio) in run metadata.
  • Emit a generic ATIF-compatible trajectory artifact at run start (and preserve partial trajectories on failure) using Villani’s existing artifact directory and filenames.

Description

  • Add small metadata helpers and populate run/session metadata with the package version and structured model metadata by introducing _agent_metadata, _model_metadata, _api_compatibility, and _inference_provider, and use villani_code.__version__ for the agent version. (files changed: villani_code/debug_recorder.py, villani_code/atif_export.py, villani_code/trace_summary.py)
  • Include model_metadata in the generic transcript artifact and add inference_provider + api_compatibility into the ATIF trajectory.json agent extra fields while preserving existing schema fields. (files changed: villani_code/atif_export.py)
  • Ensure a Villani-owned generic ATIF-compatible trajectory.json is written for every run that starts successfully by writing a trajectory at run startup and again on model-request failure; this reuses existing debug artifact locations and filenames and does not add any benchmark-specific path or naming. (files changed: villani_code/state.py, villani_code/debug_recorder.py)
  • Surface the new agent and model_metadata into summary aggregation by loading session_meta.json when building the run summary, without changing runner behavior. (files changed: villani_code/trace_summary.py)
  • Tests adjusted/added to cover the new metadata and trajectory behavior; modified test expectations only (no behavior changes). (files changed: tests/test_regular_trace_artifacts.py, tests/test_debug_mode_and_recorder.py)

Files changed: villani_code/debug_recorder.py, villani_code/atif_export.py, villani_code/state.py, villani_code/trace_summary.py, tests/test_regular_trace_artifacts.py, tests/test_debug_mode_and_recorder.py.

Testing

  • Ran pytest tests/test_regular_trace_artifacts.py tests/test_debug_mode_and_recorder.py and the suite passed (24 passed).
  • Ran the focused test selection: pytest tests/test_regular_trace_artifacts.py::test_openai_compatible_metadata_uses_lmstudio_inference_provider tests/test_regular_trace_artifacts.py::test_regular_trajectory_exists_after_model_request_failure tests/test_regular_trace_artifacts.py::test_normal_debug_writes_generic_trajectory_only tests/test_debug_mode_and_recorder.py::test_debug_metadata_separates_api_compatibility_from_inference_provider and they passed.
  • Verified compiled modules with python -m compileall and checked git diff --check; both succeeded without warnings.

Sample artifact snippets produced by the patch (examples):

  • session_meta.json / run-start payload (generic run metadata):

    {
    "agent": { "name": "villani-code", "version": "0.1.0rc1" },
    "model_metadata": { "identifier": "local-model", "inference_provider": "lmstudio", "api_compatibility": "openai" }
    }

  • ATIF trajectory.json agent block (extra fields):

    {
    "schema_version": "ATIF-v1.7",
    "agent": {
    "name": "villani-code",
    "version": "0.1.0rc1",
    "model_name": "local-model",
    "extra": {
    "provider": "openai",
    "inference_provider": "lmstudio",
    "api_compatibility": "openai",
    "mode": "regular"
    }
    },
    "extra": { "status": "completed", "termination_reason": "completed", "transcript_artifact": "transcript.full.json" }
    }

  • Failed run (partial trajectory) trajectory.json extra fields show failure:

    {
    "schema_version": "ATIF-v1.7",
    "extra": { "status": "failed", "termination_reason": "model_request_failed", "transcript_artifact": "transcript.full.json" },
    "steps": [ { "source": "system", ... }, { "source": "user", ... } ]
    }

Confirmations

  • No runner task-solving, planning, repair, validation, tool policy, retry, or execution behavior was changed; only metadata and artifact emission were added or enriched.
  • No Terminal-Bench-specific paths, filenames, directory layouts, benchmark branching, or submission-mode logic were introduced; artifacts are written to Villani’s existing debug artifact directory and filenames.
  • Patch is intentionally minimal and restricted to metadata enrichment and writing a generic ATIF-compatible trajectory artifact that external packagers can later adapt for submission.

Codex Task

@mmprotest mmprotest merged commit 0b4d11b into mmprotest/add-full-transcript-and-atif-v1.7-artifacts Jun 5, 2026
8 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant