Memory leak with vaapi and h264 codec

[jnhmn]

mpv version and platform

mpv 0.25
kernel 4.10.11-1-ARCH
libva 1.8.1
libva-mesa-driver 17.0.4
mesa 17.0.4
xorg-server 1.19.3

Reproduction steps

mpv --hwdec=vaapi film.mkv

Expected behavior

Should not use my whole memory

Actual behavior

mpv slowly allocates all available memory until it crashes.
I can not provide any sample files since they are too large and I do not have the rights on them.
Video codec information: Video --vid=1 (*) (h264 1920x1080 23.976fps)

LOGs

$ vainfo
error: XDG_RUNTIME_DIR not set in the environment.
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva )
vainfo: Driver version: mesa gallium vaapi
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileNone : VAEntrypointVideoProc

$mpv film.mkv
Playing: film.mkv
(+) Video --vid=1 () (h264 1920x1080 23.976fps)
(+) Audio --aid=1 --alang=ger () (f) (ac3 6ch 48000Hz)
Audio --aid=2 --alang=eng (dts 6ch 48000Hz)
(+) Subs --sid=1 --slang=ger (*) (dvd_subtitle)
AO: [pulse] 48000Hz 5.1(side) 6ch float
Using hardware decoding (vaapi).
VO: [opengl] 1920x1080 vaapi[nv12]
AV: 00:00:01 / 01:44:31 (0%) A-V: 0.000

CPU: AMD A4-5000
GPU: Advanced Micro Devices, Inc. [AMD/ATI] Kabini [Radeon HD 8330]

[kolobos]

Same issue in both MPV 0.24.0 and 0.25.0 while watching Twitch streams. Downgrading to MPV 0.22.0-r2 (Gentoo Linux) fixes the memory leak.
GPU: AMD HD7790 (GCN 1.1 Bonaire)

[ghost]

Can you bisect it?

Does running ffmpeg with hw decoding also leak? (Use -hwaccel vaapi before -i.)

[AndyFurniss]

Seems to be an ffmpeg regression (avconv git also leaks)
Will find it later, don't know when, it's somewhere between head and n3.2.4

[AndyFurniss]

542a65d0b33abf81e5087ec9142bb11a54f23cde is the first bad commit
commit 542a65d0b33abf81e5087ec9142bb11a54f23cde
Author: Mark Thompson sw@jkqxz.net
Date: Sun Jul 31 22:51:45 2016 +0100

ffmpeg_vaapi: Convert to use hw_frames_ctx only

Most of the functionality here has moved into lavc.

(cherry picked from commit 3e8651a7ccd8e89cc2f162cf614a3c9f7f4d9fcf)

[ghost]

That really doesn't say much.

Maybe ping @fhvwy for further ideas how to debug this.

[AndyFurniss]

I ccd Mark with the issue on ffmpeg users.

[AndyFurniss]

Explanation of issue from Mark here

http://ffmpeg.org/pipermail/ffmpeg-user/2017-May/036232.html

[ghost]

Unbelievable. Only incompetent shitheads at Intel.

[ghost]

@xhaihao please explain this bullshit. How are we supposed to develop working software under these circumstances? Did you guys even think about what you're doing?

First the nonsense with the deinterlacing refs, now this...

[ghost]

This is an issue with vaapi with MESA 17.2.3 radeon vdpau backend using mpv 0.27 and ffmpeg 3.3.4.

The reason I got hit by it is that mpv will NOT do hardware decoding of 10-bit HEVC with vo=vdpau hwdec=vdpau for some reason but vo=vaapi and hwdec=vaapi works for some reason - except that it using vaapi leaks memory and it leaks it fast enough to fill 16 GB of RAM in 30 minutes.

I've read http://ffmpeg.org/pipermail/ffmpeg-user/2017-May/036232.html and I find that reasoning ... interesting. "lavc currently chooses (B) in both cases on the grounds that leaking memory is preferable to crashing randomly.". I guess crashing consistently after a period of time when memory is full and the OOM killer steps in is .. preferable to random crashes? Regardless, the result is that vaapi is unusable.

Are we just .. stuck with this situation now? Or will this be fixed? (I would prefer if paying 10-bit hevc with vdpau would be fixed, in all honestly, everything but 10-bit hevc works perfectly with that on RX 560)

[fhvwy]

It's fixed in libva 2. The specification there is clear, matches the behaviour of current drivers, and loading old drivers is not allowed.

Playback on AMD is still pretty dubious because the Intel-based interop doesn't work for it (you need to download the frames from the GPU and then upload them back again for output). This was implemented for libva 2 but delayed by Intel. It will hopefully be in libva 2.1, at which point playback will work properly.

[Traneptora]

This appears to be fixed, so I'm closing this issue. Open a new issue if there's new problems with libva 2.0.