Quality Issues with uploaded GIFs since v4.2.0

[00-Evan]

Steps to reproduce the problem

1. upload a GIF as part of a Mastodon post (tested on web UI and Android app, Mastodon v4.2.1)
2. publish post, note the heavy compression artifacts in the converted MP4

Expected behaviour

mp4 video should have reasonable quality

Actual behaviour

mp4 video has heavy compression artifacting in the first ~1 second

Detailed description

The instance I use (mastodon.gamedev.place) has had quality issues with uploaded GIFs since its provider (Masto.host) upgraded to Mastodon v4.2.0. Uploaded GIFs have heavy compression artifacting in the first second or so.

Here's an example of the same GIF before v4.2.0: https://cdn.masto.host/mastodongamedevplace/media_attachments/files/111/075/446/608/181/342/original/c54d815b107ef206.mp4

And after: https://cdn.masto.host/mastodongamedevplace/media_attachments/files/111/365/649/893/183/634/original/8617945bc7d57d3b.mp4

Using a higher resolution mitigates the issue, but does not remove it entirely (compare the first frame to a frame at ~1 second to see the difference clearly): https://cdn.masto.host/mastodongamedevplace/media_attachments/files/111/398/651/304/019/902/original/fce7b6e49eb9c566.mp4

I emailed the owner of masto.host, and they asserted that they use vanilla Mastodon v4.2.1, and that they suspect that this change is the source of the issue: #23726

Thanks for your time.

Mastodon instance

mastodon.gamedev.place

Mastodon version

v4.2.0 & v4.2.1

Technical details

No response

[vmstan]

Do you have a link to the source GIF that has been unprocessed?

[00-Evan]

@vmstan sure

Here's the original GIF: https://github.com/mastodon/mastodon/assets/2347603/491b4ef0-a48a-4a63-8e20-708134153e9f

And the 3x upscaled version: https://github.com/mastodon/mastodon/assets/2347603/9b792733-1483-48b8-a300-b3b88af42af7

[vmstan]

I can confirm it's happening on my own server and mastodon.online which are using different ffmpeg versions right now, and I suspect it has to do with the source GIF bitrates. I think it's more likely a result of #26631 #26766 #27129 series of PRs as the one mentioned originally changed file size limits and removed client size resizing, but these actually changed video processing.

@Gargron

[vmstan]

One thing that is interesting is the source GIF has a pretty large range between the frame rate and the tbr when looking at ffprobe outputs, that I don't see from a GIF from a source like GIPHY or Tenor, and that I don't see this artifact appearing. I wonder if it's related to the particular screen capture app you're using, and how Mastodon 4.2 processes that.

The source GIF is Stream #0:0: Video: gif, bgra, 400x225, 33 fps, 100 tbr, 100 tbn
The pre-4.2 MP4 is Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 400x224, 183 kb/s, 42.60 fps, 60 tbr, 15360 tbn (default)
The post-4.2 MP4 is Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 400x224, 223 kb/s, 100 fps, 100 tbr, 12800 tbn (default)

[00-Evan]

I use ScreenToGIF to capture these, saved with its built-in encoder, octree quantizer. Here are a few more GIFs I've uploaded recently that also show this behavior, if that helps:

https://github.com/mastodon/mastodon/assets/2347603/937e398f-0918-45ed-843e-585efb80fe22

https://github.com/mastodon/mastodon/assets/2347603/963ca1e5-43ea-4a60-8e34-3b9e26e675d2

[vmstan]

Yeah these all have a similarly mismatched ratio of fps and tbr. I downloaded the app into a Windows machine and created a couple with the default settings, and with the octree quantizer, and poked around for a bit and couldn't find anywhere else in the app to adjust that setting.

I created a GIF using Cleanshot (on a Mac) and it came up with a 14.25 fps, 14.29 tbr ratio and don't see that artifact in the first frame being created after it's uploaded.

[00-Evan]

Okay then, sounds like this may warrant opening an issue on ScreenToGIF's repo as well. I'll look into that more later.

[vmstan]

Might be worth looking into, there's probably something we could do to deal with it better. Another workaround would be to upload the captures as native video files instead of letting Mastodon convert them. As long as they don't have audio associated with them will be treated the same as a GIF by clients.

[daprice]

I’ve also been seeing strange results with gifs I’ve uploaded on that same server. These were recorded by the Playdate SDK; the fps matches the tbr. These two both have similar content (1-bit dithered black-and-white animation), but one came out reasonably clear while the other one is completely unintelligible for its entire duration.

Here’s the one that’s entirely artifacts: https://cdn.masto.host/mastodongamedevplace/media_attachments/files/111/390/648/638/222/467/original/cd05a0a74a2e83f5.mp4

ffprobe output and original

Input #0, gif, from ‘~/Desktop/gradient.gif':
  Duration: 00:00:02.33, start: 0.000000, bitrate: 201 kb/s
  Stream #0:0: Video: gif, bgra, 400x240, 30 fps, 30 tbr, 100 tbn

---

versus the one I uploaded two days later that came out ok: https://cdn.masto.host/mastodongamedevplace/media_attachments/files/111/405/353/553/197/867/original/3f6491fe52529078.mp4

ffprobe output and original

Input #0, gif, from ‘~/Desktop/demo.gif':
  Duration: 00:00:17.17, start: 0.000000, bitrate: 197 kb/s
  Stream #0:0: Video: gif, bgra, 400x240, 30 fps, 30 tbr, 100 tbn

[TheEssem]

It seems like at least part of this issue applies to how it re-encodes small videos in general, not just GIFs. I attempted to upload both an 88x31 GIF and a manually converted MP4 version of the same GIF and got similar results.

Here's how it handles a GIF file.

Before:

After (pre-4.2.0):

stnicccgifuploadpre420.mp4

After (since 4.2.0):

stnicccgifupload.mp4

Here's how it handles an MP4 file.

Before (might not work on Firefox, but it does with Chromium and standalone players):

stnicccbadge.mp4

After (pre-4.2.0):

stnicccmp4uploadpre420.mp4

After (since 4.2.0):

stnicccmp4upload.mp4

All signs seem to point towards #26631 causing this behavior. Maybe 0.11 bits per pixel isn't exactly a silver bullet?