Skip to content

feat(mobile): adopt upstream video, image, and PDF media chain - #241

Merged
rynfar merged 4 commits into
pylonfrom
upstream/2026-09-02-mobile-media
Sep 2, 2026
Merged

feat(mobile): adopt upstream video, image, and PDF media chain#241
rynfar merged 4 commits into
pylonfrom
upstream/2026-09-02-mobile-media

Conversation

@rynfar

@rynfar rynfar commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Adopts three upstream mobile media commits as a chain, hand-merged to keep Pylon's composer.

  • #8919 video playback with native iOS controls
  • #8959 native image and PDF previews
  • #8978 upload attachments while composing

Together these give mobile inline video playback, a native image/PDF viewer, and
uploads that start as soon as you attach rather than at send — so queued
messages no longer hold their bytes hostage until delivery.

Carve-outs

Upstream moved videoMimeType into a new packages/shared/src/video.ts, but the
version they moved is the MIME-first one Pylon corrected in f9eb4230d. Hosts map
a TypeScript .ts source to video/mp2t, which turned source files into unnamed
black play tiles. The move is adopted; the body is Pylon's extension-first
implementation, and upstream's test that asserted the opposite ordering was
rewritten to Pylon's contract with the .ts regression cases added.

Pylon's composer diverges from upstream's and was preserved:

  • ControlPill and ComposerToolbarButton call sites kept — upstream's hunks
    are written against ComposerActionButton, which Pylon no longer uses here.
  • Stop stays outside the dictation ternary. Upstream's #8978 folds it into an
    else-if chain with Send; Pylon keeps an agent stoppable for the whole recording
    and transcription window.
  • Send stays gated on voicePresentation.showsSend, so the dictation error phase
    does not strand the user without a send button.
  • QuickQuestionModal, SessionResourcesModal, SessionAgentLiveActivityModal,
    and the agent-message sheet all sit in a block upstream replaces wholesale with
    a single VideoPreviewModal. Both sides are kept.
  • Upstream's new upload-gating block redeclared three consts Pylon already has and
    read voiceInput about 800 lines before it exists. It was trimmed to the new
    uploadStates/attachmentBlockReason and reseated next to Pylon's canSend.
  • NewTaskDraftScreen keeps Pylon's stricter workspaceRoot start condition and
    its richer unavailability messages, with the upload block reason folded in.
  • useThreadOutboxDrain's dep array keeps Pylon's list plus serverConfigs;
    upstream's three extra deps name symbols Pylon's refactor removed.

Branding: NSPhotoLibraryAddUsageDescription and the composer docs arrived saying
"T3 Code" and "T3 Connect" and now say Pylon.

Behavior change worth naming

The collapsed send pill moves from disabled={!hasContent} to
disabled={!canSend}, matching upstream and the expanded toolbar. This is wider
than "attachment uploads": the pill previously rendered enabled and then no-opped
when projectCwd === null or provider admission was unavailable, because
handleSend early-returns on !canSend. It now reflects that state instead of
lying about it.

Verification

@t3tools/mobile typecheck clean, @t3tools/shared and @t3tools/web typecheck
clean, lint clean, 1163 mobile tests passing. No user-visible T3 Code string in
any added line.

Native rebuild and device pass still to come — this chain adds expo-video,
expo-document-picker, and three new Swift files, so the JS-only checks above do
not cover it.

Model: Claude Opus 5 in Claude Code.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.6 KiB 13.6 KiB −6 B (−0.0%) 15.1 KiB
Codex Thread snapshot wire 6.9 KiB 6.9 KiB +4 B (+0.1%) 7.3 KiB
Codex Live turn WebSocket wire 6.6 KiB 6.6 KiB −10 B (−0.1%) 7.8 KiB
Codex Live turn WebSocket decoded 57.2 KiB 57.2 KiB 0 B (0.0%) 66.4 KiB
Codex Live turn messages 10 10 0 (0.0%) 21
Claude Total thread wire 13.6 KiB 13.6 KiB +21 B (+0.2%) 15.1 KiB
Claude Thread snapshot wire 6.9 KiB 6.9 KiB +12 B (+0.2%) 7.3 KiB
Claude Live turn WebSocket wire 6.6 KiB 6.7 KiB +9 B (+0.1%) 7.8 KiB
Claude Live turn WebSocket decoded 58.1 KiB 58.1 KiB 0 B (0.0%) 66.4 KiB
Claude Live turn messages 10 10 0 (0.0%) 21

Baseline: a509639 · PR result: d9a3bdc · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 109.5 KiB
  • Claude decoded thread snapshot: 110.2 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

juliusmarminge and others added 4 commits September 2, 2026 11:55
Co-authored-by: Julius Marminge <julius@mac.lan>
Co-authored-by: Julius Marminge <julius@mac.lan>
Co-authored-by: Julius Marminge <julius@mac.lan>
Review follow-ups on the media chain.

onSendMessage grew an upload block guard from #8978 but onQueueFollowUp did
not, and upstream has no onQueueFollowUp to carry the change. Both UI entry
points gate on canSend today, so this is currently unreachable — but a future
caller would queue a follow-up mid-upload and start a second transfer of the
same bytes alongside the background worker's.

The collapsed queue pill also hardcoded its accessibility label while its send
sibling and the toolbar button both surface the block reason, so a failed
upload disabled the pill with nothing for VoiceOver to read.
@rynfar
rynfar force-pushed the upstream/2026-09-02-mobile-media branch from 5ba0143 to d9a3bdc Compare September 2, 2026 17:55
@rynfar
rynfar merged commit d7d58cf into pylon Sep 2, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants