Skip to content

feat: render HLS/HEIC inline + compress uploads + transcode GIFs#465

Merged
barrydeen merged 2 commits intomainfrom
feat/media-compression-and-formats
Apr 17, 2026
Merged

feat: render HLS/HEIC inline + compress uploads + transcode GIFs#465
barrydeen merged 2 commits intomainfrom
feat/media-compression-and-formats

Conversation

@barrydeen
Copy link
Copy Markdown
Owner

Summary

  • Renders .m3u8 (HLS) and .heic/.heif URLs as inline video/image inside notes — the HLS module and Coil3 ImageDecoder support were already on the classpath; only the extension/MIME sets in the content parser needed updating.
  • Adds a shared preprocessing pipeline on every Blossom upload path (profile picture, compose attachment, DM attachment):
    • Images are decoded via ImageDecoder (handles HEIC natively on API 28+), EXIF-rotated when needed, proportionally scaled, and JPEG re-encoded. Profile pictures cap at 400px / quality 75; compose and DM content cap at 1920px / quality 82.
    • GIFs are transcoded to muted H.264 MP4 via MediaCodec + MediaMuxer through an EGL input surface (typical 5–20× size reduction). No new Gradle deps.
    • Videos and other file types pass through unchanged.
  • imeta dim and Kind-15 size/dim tags are recomputed from post-pipeline bytes so clients see metadata that matches what was actually uploaded. Gallery-mode mixing check now treats GIFs as video since they become MP4s.

Test plan

  • Paste a note containing https://test-streams.mux.dev/x36xhzz/x36xhzz.m3u8 in the feed — renders as an inline video player and plays.
  • Paste a note with a .heic URL — renders as an inline image on API 28+.
  • Pick a 4000×3000 HEIC from the gallery as profile picture — upload succeeds, picture URL renders, bytes are ≤ ~40 KB, width ≤ 400, MIME image/jpeg.
  • Attach a large JPEG (≥ 4 MB) in compose — compressed bytes ≤ ~500 KB at 1920-wide; published note's imeta dim reflects compressed dimensions.
  • Attach a .heic in compose — uploads successfully as .jpg, renders in Damus/Amethyst.
  • Send an image in a DM — encrypted upload uses compressed bytes; Kind 15 dim/size reflect post-compression values.
  • Attach a GIF via the Android GIF keyboard in compose — uploaded URL ends in .mp4, dramatically smaller than the GIF, renders as an inline video that auto-plays muted.
  • Repeat GIF flow in DM.
  • Edge cases: 1-frame GIF, long GIF (>10s), tiny GIF (<100 px) — no crashes.
  • ./gradlew assembleDebug passes.

Extends the content parser's extension and MIME sets so HLS streams and
Apple HEIF images route to the existing video/image renderers instead of
falling through to plain links. HLS plays via the media3-exoplayer-hls
module that's already on the classpath; HEIC decodes via Coil3 ->
ImageDecoder on API 28+.
Every upload path (profile picture, compose attachment, DM attachment)
now runs through a shared preprocessing pipeline:

- image/gif -> transcoded to H.264 MP4 via MediaCodec + MediaMuxer
  through an EGL input surface (typical 5-20x size reduction)
- other image/* -> decoded with ImageDecoder (handles HEIC on API 28+),
  EXIF-rotated when needed, proportionally scaled, JPEG re-encoded
- videos and other types -> passed through unchanged

Profile pictures cap at 400px wide / JPEG quality 75. Compose and DM
uploads cap at 1920px / quality 82. imeta dim and Kind 15 size/dim
tags are recomputed from post-pipeline bytes so clients see accurate
metadata. Gallery-mode video/image mixing check treats GIFs as video
since they become MP4s.
@barrydeen barrydeen merged commit 4b927b3 into main Apr 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant