workspace.repos checkout.base_commit fails for valid raw commit SHAs from git sources

[christso]

Summary

When workspace.repos[].checkout.base_commit is set to a valid raw commit SHA for a source.type: git repo, AgentV repo materialization fails with Ref <sha> not found on remote, even though the commit exists and can be checked out locally after clone.

AgentV version

4.27.0

Minimal repro

Use a public repo and a known commit SHA:

description: repro for raw sha checkout
workspace:
  path: /tmp/agentv-sha-repro
  mode: static
  repos:
    - path: ./repo
      source:
        type: git
        url: https://github.com/harbor-framework/harbor.git
      checkout:
        base_commit: df199aa02f2bdaecd04d157ce9f2e2c0b65ded9f

tests:
  - id: noop
    input: say hi
    assertions:
      - type: contains
        value: hi

Run:

agentv eval repro.eval.yaml --target <any target>

Actual behavior

AgentV fails during workspace setup with an error like:

Failed to materialize repos: Ref 'df199aa02f2bdaecd04d157ce9f2e2c0b65ded9f' not found on remote https://github.com/harbor-framework/harbor.git

Expected behavior

AgentV should clone the repo and then git checkout <raw-sha> locally. A valid commit SHA should not need to be discoverable through git ls-remote <url> <sha>.

What we verified

• The failing SHA is real and reachable:
  • local repo: git rev-parse --verify <sha>^{commit} succeeds
  • GitHub commit API resolves the SHA
• But git ls-remote <repo-url> <raw-sha> returns nothing.
• AgentV then fails during repo materialization with Ref <sha> not found on remote.

Important comparison inside AgentV itself

There is already a runnable AgentV example that pins a prior commit from a git source:

examples/showcase/bug-fix-benchmark/evals/bug-fixes.eval.yaml

It uses:

checkout:
  base_commit: "6e446b722627e9df017b22e391fa63320362d8c7"
  resolve: local

That strongly suggests the failing path is specifically:

• source.type: git
• checkout.base_commit: <raw commit sha>
• default resolution path (no resolve: local)

Likely root cause

AgentV appears to treat checkout.base_commit like a remote-resolvable ref by default, instead of allowing a raw commit object and resolving it after clone.

That works for branch/tag refs, but raw SHAs are not discoverable via git ls-remote <url> <sha>.

Suggested fix

One of these would make the UX sane:

1. If checkout.base_commit looks like a full commit SHA, resolve locally by default after clone.
2. Or: try remote resolution first, then fall back to local checkout for SHA-like values.
3. At minimum: improve the error and document that raw SHAs require resolve: local.

Why this matters

resolve: local feels too low-level for eval authors. For a field named base_commit, the natural expectation is that a valid commit SHA just works.

Harbor/Terminal-Bench style harnesses generally materialize the repo first and then run local git checkout <base_commit>, which avoids this trap entirely.