Skip to content

fix(media): preview host files and stream videos across clients - #251

Merged
rynfar merged 2 commits into
pylonfrom
upstream/2026-09-02-media-preview
Sep 2, 2026
Merged

fix(media): preview host files and stream videos across clients#251
rynfar merged 2 commits into
pylonfrom
upstream/2026-09-02-media-preview

Conversation

@rynfar

@rynfar rynfar commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Adopts #9023, plus the small piece of #8803 Pylon had left unfinished.

Images and videos anywhere the environment can read now preview inline across
web, desktop, and mobile, with byte-range streaming so a video plays without
downloading in full first. Agent "Read file" entries that name an image render a
thumbnail in the expanded work-log row.

On the access model

I previously flagged this one for a maintainer decision, on the grounds that any
orchestration:read client can mint a signed URL for any host image or video.
That was my mistake — the same commit adds 34 lines to
docs/internals/environment-auth.md reasoning about exactly that and choosing
the environment-wide authorization model over per-project filesystem permissions,
which is the model Pylon already runs on. The URL is scoped to one canonical path
plus device/inode, expires in an hour, and re-validates identity on serve.

Dependency Pylon was missing

#9023 expects FileBrowserPanel to take a workspaceMutationId, which comes
from #8803. Pylon had adopted the rest of #8803 — the hook, its tests,
ChatView, DiffPanel, FilePreviewPanel — and had even improved the hook with
burst coalescing that upstream lacks, but never plumbed the prop into the tree.
So the preview pane refreshed on agent edits while the tree beside it did not.
First commit completes that; cherry-picking #8803 wholesale would have reverted
Pylon's coalescing.

Carve-outs

  • packages/shared/src/video.ts keeps Pylon's extension-first videoMimeType
    again; upstream's rewrite of that file would have restored MIME-first ordering
    and the .tsvideo/mp2t bug. Upstream's new VIDEO_FILE_EXTENSIONS export
    is taken.
  • The desktop CSP test keeps pylon-code: rather than t3code:, while taking
    upstream's new http:/https: media sources that host video streaming needs.
  • ChatMarkdown's menu keeps Pylon's lazy getOpenInEditorMenuLabel() getter
    rather than upstream's eager openInEditorMenuLabel value, with the new
    "Preview media" entry added alongside.
  • ChatView keeps Pylon's sessionInteraction helpers and
    mergeFailedComposerSend; loadVideoPreviewUrl is dropped because #9023
    removes the function.
  • MessagesTimeline's work-log row is Pylon's, with upstream's viewedImage
    derivation wired into it — upstream's version of that component reads a
    context Pylon's row does not use.
  • docs/user/composer.md arrived with three "T3 Code" strings; they say Pylon.

ChatMarkdownWorkspaceImage is replaced by upstream's ChatMarkdownAssetImage.
That is the feature rather than a regression: the successor handles media-file
resources, video, retry, and a failure fallback.

Pylon had lost work-log/presentation.test.ts in an old rebase reconciliation and
had no coverage for that module; upstream's version restores it. One of its
assertions then failed, which surfaced a hunk that had silently not applied:
#9023 loosens the dynamic_tool_call match from exact "Read File" to a
trimmed case-insensitive compare, so Pylon would have kept missing read entries
whose title differs only in case.

Verification

@t3tools/web, @t3tools/mobile, t3, @t3tools/client-runtime,
@t3tools/shared typecheck clean. Lint clean. 3398 web, 1177 mobile, 869
client-runtime, 614 desktop, 56 server asset tests passing.

Pre-existing and unrelated: src/terminal/ghostty/runtimeAbi.test.ts fails to
parse a .wasm?inline import locally on pylon too.

Model: Claude Opus 5 in Claude Code.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

rynfar and others added 2 commits September 2, 2026 13:55
Pylon adopted #8803's useWorkspaceMutationRefresh hook and improved it with
burst coalescing, but never plumbed workspaceMutationId into FileBrowserPanel.
The preview pane and diff panel refresh on agent edits; the tree beside them did
not, so a file the agent created stayed missing until a manual refresh.

Completes that adoption. #9023 also depends on the prop existing.
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.6 KiB 13.6 KiB −3 B (−0.0%) 15.1 KiB
Codex Thread snapshot wire 6.9 KiB 6.9 KiB +9 B (+0.1%) 7.3 KiB
Codex Live turn WebSocket wire 6.6 KiB 6.6 KiB −12 B (−0.2%) 7.8 KiB
Codex Live turn WebSocket decoded 57.2 KiB 57.2 KiB 0 B (0.0%) 66.4 KiB
Codex Live turn messages 10 10 0 (0.0%) 21
Claude Total thread wire 13.8 KiB 13.5 KiB −218 B (−1.5%) 15.1 KiB
Claude Thread snapshot wire 6.9 KiB 6.9 KiB +7 B (+0.1%) 7.3 KiB
Claude Live turn WebSocket wire 6.8 KiB 6.6 KiB −225 B (−3.2%) 7.8 KiB
Claude Live turn WebSocket decoded 59.5 KiB 58.0 KiB −1.5 KiB (−2.5%) 66.4 KiB
Claude Live turn messages 11 9 −2 (−18.2%) 21

Baseline: 9378c6e · PR result: 8e98cef · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 109.5 KiB
  • Claude decoded thread snapshot: 110.2 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@rynfar
rynfar merged commit 5ac5778 into pylon Sep 2, 2026
17 checks passed
@rynfar
rynfar deleted the upstream/2026-09-02-media-preview branch September 2, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants