60fps output idea by 8x scaled Motion JPEG

[Ikadzuchi]

Hi.
I read that you tried Motion JPEG and it's too heavy.
Then, how about enlarging the image 8 times?
In JPEG, images in 8x8 px units are encoded only with DC components. Encoding can be done in pixel by pixel, with only lightweight operations like subtraction, table lookup and bit operations.
As of compression rate:
Setting quantization table to all 0xFF and the DC component will be in the range of -4 to +4 (enough for 4 shades),
then difference in range -8 to +8, and by limiting it to -7 to +7, only 0 to 3 bits are required for encoding each number.
By assigning Huffman codes as 3, 2 and 1 bit ones for the 0, 2, 3 bit number, 1 block (= original 1 pixel) can be represented in 4 or 3 bits.
With 1 bit for AC component (it is “all 0”), compression ratio of 5 bits/pixel will be possible.
I don’t know much about Motion JPEG, but I think this may be a feasible idea.
See the sample image file (made with GIMP). Note that however, the Huffman table is not assigned as described above. (Because it was difficult to create such a file.) It's optimal assignment for this image, not for minimizing worst case size.

[Staacks]

Sorry for the late reply - I have been on vacation.
I must say that I love the idea, but I have to admit that I only know the principle of how jpeg works. When I tried it, I used a library and hacked away at it to improve speed and take some shortcuts, but eventually gave up because the difference between required speed and what computation power I had to spare seemed insurmountable.

Your idea indeed sounds like there would be very little overhead and in context of USB Video there is not more to Motion JPEG than defining the correct USB descriptors and sending a series of JPEGs (if I am not forgetting something).

However, it will be a while before find the time to look into the details of an actual jpeg implementation and testing it. I am currently working on an entirely different project (deadline in about two weeks and I would like to make a Youtube video on the results) and I have a few other things on my list for the Interceptor that require less learning / research. But I will look into this for sure!

[Staacks]

So, I am finally looking into this and learning about how jpeg works in detail and I am very enthusiastic that we will indeed have a 60fps Interceptor soon.

One downside, however, would be that frame blending would not be possible as I could not encode the blended shades of gray that fall between the four colors of the Game Boy. This would be a problem for anyone streaming or recording 30fps without mixing the frames in software at their end.

So I thought about using a Huffman table with an additional code: 0x03, 0x02, 0x01 and 0x00 with 1, 2, 3 and 4 bits respectively. If I understand correctly, this would mean that I can encode every pixel with exactly 5 bit and encode any integer in a +/-7 range (i.e. -3 to +4). The fixed 5 bit length per pixel might even simplify the implementation. The size would be larger on average, but the worst-case scenario that I have to consider would be the same size.

Am I overlooking a downside here?

[Staacks]

It's working!

https://twitter.com/diconx/status/1670186665845874700

Still quite a few things to do before I can commit a proper test version:

• Status screens in JPEG
• OSD is currently not working because I start the JPEG conversion while drawing the lines. Only option might be to draw directly onto the finished JPEG, which should be possible because the pixels are now indeed perfectly aligned in 5bit blocks. Using the differential encoding, it might even be easier to just shift the brightness of a range to achieve a semi-transparent effect.
• There are a few glitches while restarting games and such

The great thing is that I managed to offload the encoding almost entirely from the CPU to the rp2040's PIOs and chained DMA channels.

Oh, and it also resolves the Mac incompatibility issue (#1). This is such a massive improvement to the GB Interceptor. Thank you so much for this ingenious idea!

[Ikadzuchi]

Great!
I'm happy that my idea was incorporated into your great work. (and sorry for not replying...)
I hadn’t thought about the blending. Encoding 0x03, 0x02, 0x01 and 0x00 with fixed 5 bits sounds good. I don't see any downside in it either.
It’s great that offloading to peripherals has become possible.

[Staacks]

Bad news: While MJPEG is generally a sequence of arbitrary JPEGs, the USB Video Class actually is a bit more specific about it and requires chroma channels with 2x subsampling. While some apps are happy with only the luma channel, others (like OBS) refuse to decode the MJPEG stream without chroma. So, I had to add an overhead of almost 3kB per frame and cannot quite reach 60fps any more (it's something around 56fps).

Not sure if I can squeeze out the last few frames. I will have to dive into TinyUSB to fix an issue with its UVC implementation anyway (the remainder of an older frame is transferred after restarting a video stream, breaking restarts in OBS), so maybe I find something to get those few percent that are still missing...

[Staacks]

Just realized that I might add a little update with good news here:
The current beta (and very likely the official release) of the firmware now allows switching between 30fps mode and 60fps mode. In 30fps mode the Interceptor encodes a JPEG with chroma, which should be compatible with almost any software out there. In 60fps mode the Interceptor ignores chroma and can achieve 60fps. I have made a pull-request that has already been merged into OBS that will support this 60fps mode in the next release of OBS.

[Staacks]

Just released version v1.2.0 with the feature.

[Staacks]

@Ikadzuchi Answering here as I am not sure if you got my message on Twitter or if it has been filtered:

I will be giving a talk about the GB Interceptor at the 37c3 (Chaos Communication Congress - not sure how well known it is outside Germany).
This will also include a section about how MJPEG encoding works on the rp2040 and my plan was to show a screenshot of your initial suggestion from the issue on GitHub. Is there any additional way you would like to be credited for the idea like a real name or handle on another platform?

[Ikadzuchi]

Oh, your Twitter message was received but apparently not notified. (How annoying Twitter is these days...)
I somewhat know c3. I think I watched some videos of it on YouTube or elsewhere.
I want to be credited as Ikadzuchi (name hear) or 雷更新世/pleist (name on Twitter).
Thank you.

[Staacks]

I have placed both on the slide :)
The talk will be today at 20:30 CET and can be seen live at https://streaming.media.ccc.de/37c3/granville. There will be a recording available later at https://media.ccc.de/c/37c3.

Edit: Recording can be found here:
https://media.ccc.de/v/37c3-11928-reconstructing_game_footage_from_a_game_boy_s_memory_bus