Skip to content

v0.5.0 — doctor --probe

Choose a tag to compare

View all tags
@makedirectory makedirectory released this 11 Aug 05:14
· 2 commits to main since this release
v0.5.0
6faa0a4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Verify the read-only contract against the CLIs you actually have installed,
rather than against Model Peer's own flags.

Install

curl -fsSL https://raw.githubusercontent.com/makedirectory/ModelPeer/v0.5.0/install.sh | bash

Docs: https://modelpeer.app

model-peer doctor --probe

model-peer doctor --probe [--models LIST] [--timeout S]

The smoke suite runs against stub CLIs. That is right for CI — fast, and it
consumes no usage — but it verifies the flags Model Peer passes, not what the
vendors do with them. Every wrong assumption this project has shipped was of
the second kind.

--probe creates a throwaway Git repository containing a token file and a
sentinel, then asks each installed CLI to read the token and to attempt to
modify the sentinel and create a new file.

The judgement comes from the filesystem, never from the reply. A model
claiming it could not write proves nothing; an unchanged file does. The prompt
asks the peer to attempt a write precisely so the disk can answer.

Claude    2.1.227
  ok        read the workspace (quoted the probe token)
  ok        sentinel.txt unchanged
  ok        created no files

Codex     0.147.0
  ok        read the workspace (quoted the probe token)
  ok        sentinel.txt unchanged
  ok        created no files

Gemini    0.46.0
  timeout   no answer within 600s — nothing verified

Verified read-only: Claude, Codex
Not verified:       Gemini — no usable answer, so the contract is
                    untested for them, not confirmed.

A peer that times out or returns nothing is reported as unverified, not as a
pass. Exit 1 covers both an inconclusive run and a safety failure; a safety
failure — meaning the workspace actually changed — is called out separately and
loudly.

This also separates "Model Peer broke it" from "the vendor is broken here". In
the run above Gemini did not answer, and a direct gemini -p call in the same
directory hung identically. If a CLI hangs during a review, the probe tells you
which side the fault is on.

It consumes real usage — one model call per installed CLI, against your own
vendor account. It is opt-in and never part of plain doctor. The natural moment
to run it is after upgrading a vendor CLI, when the assumptions underneath Model
Peer are most likely to have moved.

Also

model-peer doctor now reports each CLI's version alongside its path, so a bug
report carries the versions of the three tools whose behaviour all of this
depends on.

Claude    installed  2.1.227    /Users/you/.local/bin/claude
Codex     installed  0.147.0    /opt/homebrew/bin/codex
Gemini    installed  0.46.0     /opt/homebrew/bin/gemini

Full notes in CHANGELOG.md.

Full Changelog: v0.4.0...v0.5.0

Assets 2
Loading