Certain video dimensions cause incomplete frame decoding or buffer overflow during YUV-to-RGB conversion

[shanrauf]

What

For certain video dimensions where the video is encoded in YUV format, sws_scale in AVProcessVideoFrame will sometimes skip the rightmost pixels in each row of the output image, causing a portion of the image to be cut off. Other times it'll write extra pixels per row, causing a buffer overflow.

Reproduction

(This may only happen on some hardware. I'm running on an x64-based Windows 11 PC with an AMD Ryzen processor)

#include "raymedia.h"

const char* ExampleTitle = "Example 01 - Basics";

const int   ScreenWidth = 1920 / 2;
const int   ScreenHeight = 1080 / 2;
const char* MovieFile = ASSETS_PATH"examples/broken1.mp4"; // You may need to change this line depending on how you load assets

int main(int argc, char** argv)
{
    InitWindow(ScreenWidth, ScreenHeight, TextFormat("raylib-media | %s", ExampleTitle));
    SetWindowState(FLAG_WINDOW_RESIZABLE);
    SetTargetFPS(60);
    InitAudioDevice();

    MediaStream videoMedia = LoadMedia(MovieFile);

    if (!IsMediaValid(videoMedia))
    {
        TraceLog(LOG_ERROR, "Failed to load media file: %s", MovieFile);
        CloseAudioDevice();
        CloseWindow();
        return -1;
    }

    SetMediaLooping(videoMedia, true);

    while (!WindowShouldClose())
    {
        UpdateMedia(&videoMedia);

        BeginDrawing();
        ClearBackground(DARKPURPLE);

        float scale = 1.0f;
        int displayWidth = videoMedia.videoTexture.width;
        int displayHeight = videoMedia.videoTexture.height;
        
        if (displayWidth > GetScreenWidth() || displayHeight > GetScreenHeight()) {
            float scaleX = (float)GetScreenWidth() / displayWidth;
            float scaleY = (float)GetScreenHeight() / displayHeight;
            scale = (scaleX < scaleY) ? scaleX : scaleY;
            
            displayWidth = (int)(displayWidth * scale);
            displayHeight = (int)(displayHeight * scale);
        }
        
        const int videoPosX = (GetScreenWidth() - displayWidth) / 2;
        const int videoPosY = (GetScreenHeight() - displayHeight) / 2;

        Rectangle sourceRect = {0, 0, (float)videoMedia.videoTexture.width, (float)videoMedia.videoTexture.height};
        Rectangle destRect = {(float)videoPosX, (float)videoPosY, (float)displayWidth, (float)displayHeight};
        Vector2 origin = {0, 0};

        DrawTexturePro(videoMedia.videoTexture, sourceRect, destRect, origin, 0, WHITE);

        EndDrawing();
    }

    UnloadMedia(&videoMedia);
    CloseAudioDevice();
    CloseWindow();

    return 0;
}

Sample files

The video where rightmost pixels are skipped (1078x2316)

1078.mp4

The video that causes buffer overflow (1068x2316)

broken1.mp4

Example of skipping pixels per row (1078x2316; vertical black bar on the right of the video)

Example of buffer overflow (1068x2316)

overflow_with_broken1.mp4

This seems to happen because sws_scale expects the output buffer's stride/linesize per row of the image to be 16-byte aligned at a minimum (that number may be larger though on different hardware I think. I'm running on an x64-based Windows PC with an AMD Ryzen processor). I don't think FFmpeg clearly documents this anywhere, but it's mentioned here and there online:

• "Some parts of FFmpeg, notably libavcodec, require aligned linesizes[]... For swscale/avfilter, I believe there is no absolute requirement, but you're recommended to still make it 32." https://stackoverflow.com/questions/35678041/what-is-linesize-alignment-meaning
• "The underlying issue is that the alignment passed to a bunch of libav functions is 1, but for sws_scale to work in general it is apparently supposed to be 32": Crash inside sws_scale with certain inputs OpenShot/libopenshot#576
  • https://stackoverflow.com/questions/31253485/how-do-you-resize-an-avframe/31270501#31270501
• OBS also had this bug with YUV-to-RGB conversion
  • yuv422p media sources cause out-of-bounds writes and heap corruption obsproject/obs-studio#3062
  • Fix FFmpeg YUV to RGB conversion by adding extra destination padding to line sizes obsproject/obs-studio#2836
    • Fixes https://obsproject.com/mantis/view.php?id=1032
    • Fixes https://obsproject.com/mantis/view.php?id=1177

• With the 4.3 release, swscale now crashes if provided with an output buffer that is not 16 bytes aligned for yuv2rgb conversions. It used to work in previous releases *

• https://trac.ffmpeg.org/ticket/10514

I also tested this experimentally: (Google Sheets link):

The data shows how most files crash when (W * 3) % 48 != 0 (and for the few cases where that's not true, those are the cases where not all pixels are processed, causing the black vertical bar on the right of the video). I believe all of this has something to do with the assembly (at least on x86) writing in 48-byte chunks, and something to do with this code, but I didn't dig much deeper since it didn't feel necessary.

I opened a PR to fix these issues here: #11