Letterbox art and remove BMP support - #168
Merged
Merged
Conversation
The scaling rule said the server "will scale images to fit within the specified dimensions while preserving aspect ratio", which reads as fit-inside: a square album cover requested at 800x480 would arrive 480x480. aiosendspin letterboxes instead, so the delivered image is 800x480 with black bars, and the spec never said which is correct. Letterboxing is the behavior to keep. It is what implementations already do, it keeps clients from having to lay out a variable-size image, and it makes the dimensions in stream/start match what the client asked for. - media_width and media_height are the dimensions of the delivered image, not upper bounds - The server MUST pad to those dimensions with black and MUST NOT crop - stream/start width/height MUST equal the declared media_width/media_height rather than merely not exceeding them The equality rule also settles the case that blocked scheduled artwork updates: with the delivered size pinned to the declaration, a later image cannot arrive at a different aspect ratio than the resolution stream/start announced. Gradient or artwork-derived padding was considered and left out; the padding is black.
BMP is in the format list because JPEG, PNG and BMP were what ESPHome decoded out of the box, not because a client needs it. It is the one format whose size is set by the pixel count rather than the image: an 800x480 24-bit BMP is about 1.1 MB, roughly 450 ms of wire time on a 20 Mbps link, and it has to be held in memory whole. The devices that would ask for BMP to avoid an image library are the constrained ones least able to absorb either cost. Clients keep 'jpeg' and 'png', both of which have decoders available for microcontrollers. This removes the largest artwork payloads but not the head-of-line blocking behind them; splitting large images across messages is still open in #134.
maximmaxim345
left a comment
maximmaxim345
left a comment
Member
There was a problem hiding this comment.
Can you also rename media_width and media_height to just width and height in this PR? See: #128 (comment)
A stream/request-format the server does not honor leaves the stream in its previous configuration (messaging.md), but the stream/start rule read as though every request moved the channel's current capability. That was harmless while width/height only had to not exceed the declaration; now that they must equal it, a declined request would put the continuing stream in violation. Scope the clause to the changes the server honored. Per-change rather than per-request, since the artwork request-format fields are each optional and a server can honor some and decline others.
media_width/media_height meant a maximum while stream/start reported the actual encoded width/height, so the two names described two quantities. Delivering at exactly the declared size collapsed that distinction: they are now the same number, and the prefix no longer disambiguates anything. Rename them to width and height, matching stream/start and the editorial rule of one canonical name per term. The stream/start requirement folds into a single clause now that both sides use the same names.
maximmaxim345
added a commit
that referenced
this pull request
Sep 2, 2026
An artwork image larger than one Noise frame (65518 bytes of payload) currently relies on transport fragmentation, which admits no interleaving: the sender must finish the fragmented message before any other frame, so the whole image is an indivisible burst that audio chunks cannot preempt. A player holding only `min_buffer_ms` of audio can underrun behind it; on a live stream that truncates audio. With BMP gone (#168) the worst case shrank, but photographic JPEG still exceeds one frame beyond roughly 500x500 (the spec's own `stream/start` example, 800x800 JPEG, is 3-5 frames), and PNG exceeds it at almost any artwork size. ## Changes: - An image is transferred as a fixed 14-byte announce message carrying the timestamp and the image's `total_size` (uint32), followed by parts carrying only image data, all on the channel's message type; a flags byte after the type byte distinguishes the message kinds, following the shape #172 gave fragmentation - The announce carries no image data, so a client can allocate its image buffer before any image bytes arrive and write each part directly into it; the transfer completes when the received data reaches `total_size` - Every artwork message is capped at 65518 bytes after the type byte so it never needs transport fragmentation; any other messages MAY be sent between the messages of a transfer, and transfers on different channels are independent - The pending image is now the most recently announced image, from its announce until it becomes current: an announce discards any held pending image, partly received or complete, and the pending image becomes current once its transfer completes and its timestamp is reached - A rejected announce (no active stream, or client not available) still starts its transfer, so its parts are rejected rather than treated as malformed - The clear message becomes an announce with `total_size` 0, which completes immediately with no parts - A two-byte cancel message (flags bit 1) discards the channel's pending image, partly received or complete, leaving the current image showing; it replaces resend-the-current-image as the cancel recipe in the scheduled-artwork rules, which under chunking would cost a full image retransfer - Malformed sequences (a message under 2 bytes or over the frame cap, an announce whose length is not 14 bytes, a cancel longer than 2 bytes, a part with no transfer in flight, data extending past `total_size`, a nonzero reserved flag bit, both flag bits set) close the connection, matching fragmentation The pending-image redefinition keeps the newest-wins rule from #135 applying at the announce, as it did at message arrival before chunking, and means a client needs at most two image buffers per channel, receiving the transfer directly into the pending one. Every image costs one extra small message, including images that fit in a single frame; the uniform format was judged worth ~31 bytes of ciphertext per image to give clients flexibility to transfer directly into a proper sized buffer. There is no explicit last-part flag; `total_size` already determines completion, and a redundant end marker would add mismatch states to define. Every artwork message changes shape: the flags byte shifts the timestamp, the header and image data travel in separate messages, and the empty clear message becomes a bare announce with `total_size` 0. Closes #134 ## Breaking changes: The wire format for artwork is completely changed. All servers and clients will have to re-implement to handle: - the announce binary message - the artwork specific split chunks versus using the protocol's generalized noise split - handling the pending slot; i.e., servers should now send a cancel message instead of re-sending the current image --------- Co-authored-by: Maxim Raznatovski <nda.mr43@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The previous artwork scaling rule said the server "will scale images to fit within the specified dimensions while preserving aspect ratio", which reads as fit-inside: a square cover requested at 800x480 would arrive 480x480. The spec will adopt what
aiosendspinalready does: letterbox.Changes:
width/heightare the dimensions of the delivered image, not upper bounds (renamed frommedia_widthandmedia_height)stream/startwidth/heightMUST equal the declaredwidth/heightrather than merely not exceeding them'bmp'from the artwork formats, leaving'jpeg'and'png'(see comments in Artwork transfers can starve audio: head-of-line blocking on the shared connection #134)The equality rule settles the case that blocked scheduled artwork updates: with the delivered size pinned to the declaration, a later image cannot arrive at a different aspect ratio than the resolution
stream/startannounced.I'll make a future PR splitting large images across messages, the remaining part of #134.
Closes #128
Breaking Changes
media_width/media_heightare renamed towidth/height