Repository navigation
Peer Intelligence
This capability note separates shipped implementation from candidate behavior and required evidence. It does not add a requirement or change the SRS, RTM, or Architecture Packet. Related work: peer intelligence umbrella #210, read-only peer channel #211, and model identity and comparison #217.
The projectbuddy-ask client and server provide five allowlisted queries:
identity, vbox, disk, ollama, and lbabus. The SSH deployment uses a
restricted key and pinned host identity rather than an unrestricted remote
shell. VM responses filter unrelated VM names. A successful query proves only
the queried host's reported state; a peer's drive letter does not identify a
volume on another machine.
Implementation checkpoint: PR #216,
commit bf62b9d04d669ac2e68896ee8bd6221978e07603.
See the peer runbook at that checkpoint.
scripts/New-PeerModelfile.py renders a model definition from a versioned
identity manifest and SHA-256-pinned repository sources. Changed or missing
sources are refused rather than silently incorporated. The manifest fixes the
context size, temperature, seed, role, and system-prompt size limit. Identical
inputs produce identical output; the selected base model remains an explicit
input.
Implementation checkpoint: PR #218,
commit bcf1736 on develop, not a claim of a tagged release.
See the identity implementation
and manifest.
The generator's 60 offline tests pass locally. Historical single-model smoke
evidence is not a completed multi-model benchmark.
The unpublished candidate on feature/217-benchmark introduces
scripts/Invoke-PeerBenchmark.py, a fixed synthetic corpus, and an injectable
HTTP transport. These paths are candidate code, not published entry points.
The candidates are llama3.1:8b, qwen2.5:7b, and qwen2.5-coder:7b.
The corpus contains 15 cases: eight diagnostics, three redaction cases, and four tool-selection cases. Each model receives the same pinned system prompt and manifest options. Tool calls are scored as predictions; the harness does not execute model-suggested tools.
Candidate scoring uses diagnostic accuracy, redaction recall, tool-call matching, median response duration, and reported VRAM fit. Accuracy uses case-insensitive phrase groups. Redaction checks prose and tool-call payloads. Tool matching checks the function name and expected argument values. The weighted score is 40% accuracy, 30% redaction, and 30% tool matching. Only models reported as fully resident in VRAM enter the ranking; score is ordered before median latency. CPU-only runs must not be presented as passing this VRAM gate.
The harness requests one minute of model residency after each request so the loaded-model query can measure VRAM independently of the server's default keep-alive setting. Incomplete or errored replies and malformed response objects are rejected. The CLI accepts only HTTP(S) loopback origins without credentials, paths, queries, or fragments and refuses to replace an existing result file. Reports record the host label, Ollama version, fixed options, per-case results, and ranking.
These are mechanical comparison proxies. A keyword match does not establish a correct diagnosis; absence of listed synthetic strings does not prove that a model cannot disclose sensitive information. This corpus is not a security assurance gate or an authorization to act on a model's advice.
- Local offline validation passes 33 benchmark tests and the 60 neighboring generator tests. The full local Python suite passes 1,979 tests. Transport is mocked in the benchmark tests; no inference is needed for them. These are local results, not a claim that CI or a native Windows run has passed.
- A separate bounded CPU-only smoke confirms a replacement local model store can serve a response. It is not a corpus run, latency comparison, or GPU proof.
- No comparative winner or completed two-host benchmark is established here. Record exact base-model digests, Ollama versions, context/options, execution backend, corpus identity, and results before comparing hosts.
- Full GPU residency and stability remain separate live gates. CPU evidence cannot satisfy them, and changed runtime versions must not be treated as an otherwise identical host comparison.
- Docker Desktop with Linux containers can host portable test tooling where the authorized account can access its engine. Engine availability and GPU access require direct verification; a Linux container is not native Windows evidence and cannot repair another host's physical disk.
- Operator approval is required before deleting a VM or cleaning up stored models, images, containers, or volumes. Do not prune assets to make a run fit.
Machine-specific addresses, disk identities, recovery observations, and private deployment details belong in protected operator notes, not this public wiki.