Skip to content

fix: resolve image paths before the model ever sees them - #20

Open
carstenlucke wants to merge 1 commit into
getpipher:mainfrom
carstenlucke:fix/tilde-path-expansion
Open

fix: resolve image paths before the model ever sees them#20
carstenlucke wants to merge 1 commit into
getpipher:mainfrom
carstenlucke:fix/tilde-path-expansion

Conversation

@carstenlucke

@carstenlucke carstenlucke commented Aug 6, 2026

Copy link
Copy Markdown

Problem

A ~/x.png reference never reached the vision tool as a usable path, and the paths the model is shown were the raw tokens the user typed rather than the resolved ones. Both come from the same gap: path resolution happening too late, or not at all.

1. Tilde expansion was broken in three places

Location ~/x.png resolved to
lib/image.ts:151replace(/^~/, "") /x.png (stripped tilde wins the resolve() against cwd)
extensions/paste.ts:73 <cwd>/~/x.png (tilde as a literal directory name)
updateComposePreview() <cwd>/~/x.png (isAbsolute("~/x") is false)

Neither path exists, existsSync fails, and the token is treated as unresolvable. So a home-relative path pasted into the editor produced no [Image-#N] marker, no hint line, no compose preview and no auto-delegation.

This went unnoticed because the LLM expands the tilde itself before calling describe_image — the extension never saw a tilde on the tool path. The paste path, which runs before the model, had no such luck.

2. The hint listed tokens the model cannot act on

buildHintLine and buildDescriptionsBlock rendered token while LoadedImage.abs sat in the same object. A ~/x.png or ./x.png in the hint is not actionable without $HOME or the cwd, which defeats the stated purpose of the v0.4.0 hint (SPEC-4 §3.4): list the paths "so the model can actually call describe_image".

Observed in a real GLM-5.2 session — the model shelled out purely to resolve the tilde before it could use the tool:

[4] TOOLCALL bash            → { "command": "echo ~/vision-tilde-test.png" }
[6] TOOLCALL describe_image  → { "image_path": "/Users/carsten/vision-tilde-test.png" }

Fix

  • A shared, pure expandTilde() in lib/image.ts, used by both the tool path and the paste path so they expand identically. ~user/… is deliberately not expanded — that needs a passwd lookup, and mapping it onto the current user's home would be wrong; it falls through and fails existsSync.
  • resolveImageFile is exported so the resolution rules are unit-testable without a pi runtime.
  • buildHintLine now takes paths: string[], and buildDescriptionsBlock takes path instead of token. Both call sites collapse to loaded.map((l) => l.abs). index was never read by buildHintLine, and token only fed a fallback no call site could reach.

Verification

End to end against GLM 5.2 with a text-only primary. Same input, before and after:

before after
hint lists ~/vision-tilde-test.png /Users/carsten/vision-tilde-test.png
bash echo … detour yes gone
tool calls 2 1

The delegated description was correct in both runs (the test image carries a checksum string), so this changes only how the path reaches the model.

Tests

+11 (364 total, was 353). Every behavioural test was verified to fail against the pre-fix code, so they are real regression guards rather than passengers.

The integration tests (T52b hint mode, T32b auto mode) drive the full input event with $HOME on a temp dir and cwd deliberately elsewhere — that combination catches an l.token-instead-of-l.abs slip in paste.ts that unit tests on marker.ts alone did not.

tsc --noEmit clean.

Note

Independent of #21 — different cause, different scope, no shared commits. Either can merge first.

A `~/x.png` reference never reached the vision tool as a usable path,
and the paths the model was shown were the raw tokens the user typed
rather than the resolved ones. Both stem from the same gap: path
resolution happening too late, or not at all.

1. Tilde expansion was broken in three places:

   - lib/image.ts stripped the tilde (`replace(/^~/, "")`), turning
     `~/x.png` into the absolute `/x.png`, which then won the
     resolve() against cwd.
   - extensions/paste.ts resolved `~/x.png` against cwd, yielding
     `<cwd>/~/x.png` (tilde as a literal directory name).
   - updateComposePreview() had the same bug in a third variant
     (isAbsolute("~/x") is false → `<cwd>/~/x.png`).

   Effect: a home-relative path pasted into the editor produced no
   [Image-#N] marker, no hint line, no compose preview and no
   auto-delegation. It only appeared to work because the LLM expanded
   the tilde itself before calling describe_image, so the extension
   never saw a tilde on the tool path.

   Adds a shared, pure `expandTilde()` in lib/image.ts so the tool
   path and the paste path expand identically. `~user/…` is
   deliberately not expanded (that needs a passwd lookup; mapping it
   onto the current user's home would be wrong) — it falls through and
   fails existsSync. resolveImageFile is exported so the resolution
   rules are unit-testable without a pi runtime.

2. buildHintLine and buildDescriptionsBlock rendered the raw token
   while LoadedImage.abs sat in the same object. A `~/x.png` or
   `./x.png` in the hint is not actionable without $HOME or the cwd,
   which defeats the stated purpose of the v0.4.0 hint (SPEC-4 §3.4):
   list the paths "so the model can actually call describe_image".
   Observed in a real GLM-5.2 session: the model shelled out to
   `bash echo ~/vision-tilde-test.png` purely to resolve the tilde
   before it could call the tool. Both now take resolved paths only —
   buildHintLine as `paths: string[]`, since `index` was never read
   and `token` only fed a fallback that no call site could reach.

Verified end to end against GLM 5.2 with a text-only primary: the
hint now lists the absolute path, and the model calls describe_image
directly — one tool call instead of two.

Tests: +11 (364 total, was 353), including integration tests that
drive the full input event with $HOME on a temp dir and cwd
deliberately elsewhere. All were verified to fail against the
pre-fix code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant