Failed to decode h264 hwaccel in Mac OSX 10.13.4 with Intel GPU

[xswqaz]

mpv version and platform

[cplayer] mpv 0.28.2 (C) 2000-2017 mpv/MPlayer/mplayer2 projects
[cplayer]  built on Sat Feb 17 13:54:24 CET 2018
[cplayer] ffmpeg library versions:
[cplayer]    libavutil       56.7.101
[cplayer]    libavcodec      58.11.101
[cplayer]    libavformat     58.9.100
[cplayer]    libswscale      5.0.101
[cplayer]    libavfilter     7.12.100
[cplayer]    libswresample   3.0.101
[cplayer] ffmpeg version: git-2018-02-14-dc77e64

Reproduction steps

mpv sample.mp4

mpv.conf

ontop=yes
border=no
geometry=100%:0%

hwdec=auto

audio-spdif=ac3
af=scaletempo,lavcac3enc
audio-device=coreaudio/AppleHDAEngineOutput:1B,0,1,2:0

Expected behavior

hwaccel decode

Actual behavior

Failed to hwaccel decode

Log file

[cplayer] Command line options: 'sample.mp4' '-v'
[cplayer] mpv 0.28.2 (C) 2000-2017 mpv/MPlayer/mplayer2 projects
[cplayer]  built on Sat Feb 17 13:54:24 CET 2018
[cplayer] ffmpeg library versions:
[cplayer]    libavutil       56.7.101
[cplayer]    libavcodec      58.11.101
[cplayer]    libavformat     58.9.100
[cplayer]    libswscale      5.0.101
[cplayer]    libavfilter     7.12.100
[cplayer]    libswresample   3.0.101
[cplayer] ffmpeg version: git-2018-02-14-dc77e64
[cplayer] 
[cplayer] Configuration: waf configure --prefix=/Users/djinn/apps/homebrew/Cellar/mpv/0.28.2 --enable-zsh-comp --enable-libmpv-shared --enable-html-build --enable-lua --confdir=/Users/djinn/apps/homebrew/etc/mpv --datadir=/Users/djinn/apps/homebrew/Cellar/mpv/0.28.2/share/mpv --mandir=/Users/djinn/apps/homebrew/Cellar/mpv/0.28.2/share/man --docdir=/Users/djinn/apps/homebrew/Cellar/mpv/0.28.2/share/doc/mpv --zshdir=/Users/djinn/apps/homebrew/Cellar/mpv/0.28.2/share/zsh/site-functions --enable-javascript
[cplayer] List of enabled features: 51deb apple-remote asm atomics bsd-fstatfs build-date cocoa coreaudio cplayer cplugins cuda-hwaccel debug-build drmprime encoding fchmod ffmpeg gl gl-cocoa glob glob-posix gnuc gpl iconv javascript jpeg lcms2 libaf libass libass-osd libav-any libavcodec libavdevice libdl libm libmpv-shared lua optimize osx-thread-name plain-gl posix posix-or-mingw posix-spawn posix-spawn-native pthreads stdatomic videotoolbox-gl videotoolbox-hwaccel zlib zsh-comp
[cplayer] Reading config file /Users/.config/mpv/mpv.conf
[osc] lua-settings/osc.conf not found. 
[osd/libass] Shaper: FriBidi 1.0.1 (SIMPLE)
[osd/libass] Setting up fonts...
[ytdl_hook] lua-settings/ytdl_hook.conf not found. 
[osd/libass] Using font provider coretext
[osd/libass] Done.
[stats] lua-settings/stats.conf not found. 
[cplayer] Playing: sample.mp4
[cplayer] Running hook: ytdl_hook/on_load
[file] Opening sample.mp4
[cache] Cache size set to 20000 KiB (10000 KiB backbuffer)
[demux] Trying demuxers for level=normal.
[lavf] Found 'mov,mp4,m4a,3gp,3g2,mj2' at score=100 size=2048.
[demux] Detected file format: mov,mp4,m4a,3gp,3g2,mj2 (libavformat)
[cache] blocking for STREAM_CTRL 6
[cplayer] Opening done: sample.mp4
[find_files] Loading external files in .
[cplayer] Running hook: ytdl_hook/on_preloaded
[lavf] select track 0
[lavf] select track 1
[cplayer]  (+) Video --vid=1 (*) (h264 1920x1080 29.970fps)
[cplayer]  (+) Audio --aid=1 --alang=unk (*) (aac 2ch 48000Hz)
[vo/gpu] Probing for best GPU context.
[vo/gpu/opengl] Initializing GPU context 'cocoa'
[vo/gpu] can't find an ambient light sensor
[vo/gpu] GL_VERSION='4.1 INTEL-10.32.48'
[vo/gpu] Detected desktop OpenGL 4.1.
[vo/gpu] GL_VENDOR='Intel Inc.'
[vo/gpu] GL_RENDERER='Intel HD Graphics 5000 OpenGL Engine'
[vo/gpu] GL_SHADING_LANGUAGE_VERSION='4.10'
[vo/gpu] Loaded extension GL_APPLE_rgb_422.
[vo/gpu] Testing FBO format rgba16
[vo/gpu] Using FBO format rgba16.
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vo/gpu] Assuming 59.964023 FPS for display sync.
[vd] Container reported FPS: 29.969975
[vd] Codec list:
[vd]     h264 - H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd] Opening video decoder h264
[vd] Looking at hwdec h264-videotoolbox...
[vo/gpu] Loading hwdec driver 'videotoolbox'
[vo/gpu] Loading hwdec driver 'cuda-nvdec'
[vo/gpu/cuda-nvdec] Failed to load CUDA symbols
[vo/gpu] Loading failed.
[vd] Trying hardware decoding via h264-videotoolbox.
[vd] Selected video codec: h264 (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
[ad] Codec list:
[ad]     aac - AAC (Advanced Audio Coding)
[ad]     aac_fixed (aac) - AAC (Advanced Audio Coding)
[ad]     aac_at (aac) - aac (AudioToolbox)
[ad] Opening audio decoder aac
[ad] Requesting 1 threads for decoding.
[ad] Selected audio codec: aac (AAC (Advanced Audio Coding))
[cplayer] Starting playback...
[ffmpeg/audio] aac: channel element 1.2 is not allocated
[ad] Error decoding audio.
[ffmpeg/audio] aac: channel element 1.2 is not allocated
[ad] Error decoding audio.
[ffmpeg/audio] aac: channel element 1.2 is not allocated
[ad] Error decoding audio.
[ffmpeg/audio] aac: channel element 1.2 is not allocated
[ad] Error decoding audio.
[ffmpeg/audio] aac: channel element 1.2 is not allocated
[ad] Error decoding audio.
[af] Adding filter scaletempo 
[af] Adding filter lavcac3enc 
[af] Adding filter lavrresample 
[af] Adding filter lavrresample 
[af] Audio filter chain:
[af]   [in] 48000Hz 5.1 6ch floatp
[af]   [lavrresample] 48000Hz 5.1 6ch float [a]
[af]   [scaletempo] 48000Hz 5.1 6ch float
[af]   [lavrresample] 48000Hz 5.1 6ch floatp [a]
[af]   [lavcac3enc] 48000Hz stereo 2ch spdif-ac3
[af]   [out] 48000Hz stereo 2ch spdif-ac3
[af]   [ao] 48000Hz stereo 2ch spdif-ac3
[ao] Trying audio driver 'coreaudio'
[ao] Using preferred device 'AppleHDAEngineOutput:1B,0,1,2:0'
[ao/coreaudio] requested format: 48000 Hz, stereo channels, spdif-ac3
[ao/coreaudio] redirecting to coreaudio_exclusive
[ao/coreaudio_exclusive] requested format: 48000 Hz, stereo channels, spdif-ac3
[ao/coreaudio_exclusive] selected audio output device: Built-in Output (52)
[ao/coreaudio_exclusive] -  192000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  176400.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  96000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  88200.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  48000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  44100.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  32000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  192000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  176400.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  96000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  88200.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  48000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  44100.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  32000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  192000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  176400.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  96000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  88200.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  48000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  44100.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  32000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  96000.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] Using substream 0/1.
[ao/coreaudio_exclusive] our format: 48000.0Hz 16bit 3cac [12][4bpp][1fbp][4bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] -  192000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  176400.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  96000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  88200.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  48000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  44100.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  32000.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  192000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  176400.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  96000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  88200.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  48000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  44100.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  32000.0Hz 20bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] -  192000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  176400.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  96000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  88200.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  48000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  44100.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  32000.0Hz 16bit mcpl [12][4bpp][1fbp][4bpf][2ch] int LE S packed (s16)
[ao/coreaudio_exclusive] -  96000.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] -  48000.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] -  44100.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] setting stream physical format: 48000.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] format in use before switching: 44100.0Hz 24bit mcpl [4][8bpp][1fbp][8bpf][2ch] int LE S (-)
[ao/coreaudio_exclusive] actual format in use: 48000.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] input channel layout:
[ao/coreaudio_exclusive] layout: tag: <0>, bitmap: <0>, descriptions <2>
[ao/coreaudio_exclusive]  - description 0: label <4294967295, 64>,  flags: <0>, coords: <0.000000, 0.000000, 0.000000>
[ao/coreaudio_exclusive]  - description 1: label <4294967295, 64>,  flags: <0>, coords: <0.000000, 0.000000, 0.000000>
[ao/coreaudio_exclusive] mp chmap: unknown2
[ao/coreaudio_exclusive] input channel layout:
[ao/coreaudio_exclusive] layout: tag: <0>, bitmap: <0>, descriptions <2>
[ao/coreaudio_exclusive]  - description 0: label <1, 0>,  flags: <0>, coords: <0.000000, 0.000000, 0.000000>
[ao/coreaudio_exclusive]  - description 1: label <2, 1>,  flags: <0>, coords: <0.000000, 0.000000, 0.000000>
[ao/coreaudio_exclusive] mp chmap: stereo
[ao/coreaudio_exclusive] Channel layouts:
[ao/coreaudio_exclusive]  - stereo
[ao/coreaudio_exclusive]  - mono
[ao/coreaudio_exclusive] result: stereo
[ao/coreaudio_exclusive] virtual format 48000.0Hz 16bit 3cac [76][6144bpp][1536fbp][0bpf][2ch] int LE S packed (spdif-ac3)
[ao/coreaudio_exclusive] Latency property cntl: 5 frames
[ao/coreaudio_exclusive] Latency property zisf: 1536 frames
[ao/coreaudio_exclusive] Latency property tfas: 96 frames
[ao/coreaudio_exclusive] base latency: 34104 microseconds
[ao/coreaudio_exclusive] using soft-buffer of 10752 samples.
[cplayer] AO: [coreaudio_exclusive] 48000Hz stereo 2ch spdif-ac3
[cplayer] AO: Description: CoreAudio Exclusive Mode
[vd] Pixel formats supported by decoder: videotoolbox_vld yuv420p
[vd] Codec profile: High (0x64)
[vd] Requesting pixfmt 'videotoolbox_vld' from decoder.
[ffmpeg/video] h264: Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
[vd] Pixel formats supported by decoder: yuv420p yuv420p
[vd] Codec profile: High (0x64)
[vd] Requesting pixfmt 'yuv420p' from decoder.
[vd] Falling back to software decoding.
[vd] Detected 4 logical cores.
[vd] Requesting 5 threads for decoding.
[vd] Using software decoding.
[vd] Decoder format: 1920x1080 yuv420p auto/auto/auto/auto CL=mpeg2/4/h264 (auto 0.000000/0.000000/0.000000)
[vd] Using container aspect ratio.
[vf] Video filter chain:
[vf]   [in] 1920x1080 yuv420p bt.709/bt.709/bt.1886/limited SP=1.000000 CL=mpeg2/4/h264
[vf]   [out] 1920x1080 yuv420p bt.709/bt.709/bt.1886/limited SP=1.000000 CL=mpeg2/4/h264
[cplayer] VO: [gpu] 1920x1080 yuv420p
[cplayer] VO: Description: Shader-based GPU Renderer
[vo/gpu] Resize: 768x432
[vo/gpu] Window size: 768x432
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 768x432
[vo/gpu] Video scale: 0.400000/0.400000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Testing FBO format rgba16
[vo/gpu] Using FBO format rgba16.
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vo/gpu] Texture for plane 0: 1920x1080
[vo/gpu] Texture for plane 1: 960x540
[vo/gpu] Texture for plane 2: 960x540
[vo/gpu] Resize: 768x432
[vo/gpu] Window size: 768x432
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 768x432
[vo/gpu] Video scale: 0.400000/0.400000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[osd/libass] fontselect: (sans-serif, 400, 0) -> /System/Library/Fonts/Helvetica.ttc, -1, Helvetica
[cplayer] first video frame after restart shown
[cplayer] starting audio playback
[cplayer] playback restart complete

Sample files

[Akemi]

works as intended here with git master. make sure your file can actually be decoded by videotoolbox. other than this try with master and provide an example file.

[xswqaz]

How can I get binary for Mac OSX?

[Akemi]

you can try the bundle from my reply here. otherwise build it yourself, it's really not hard with homebrew. also try with this file. it works for me with videotoolbox. if it works for you too then most likely your file won't work with hwdec.

[xswqaz]

Your file is playing with hwaccel certainly
videotoolbox seems only activated on specific format
Could you check this file?

[Akemi]

your file works for me too with videotoolbox. though i have older hardware, macbook pro mid 2010 with an nvidia gt330m. same macOS version.

[xswqaz]

Here is mac mini 2014
Intel(R) Core(TM) i5-4260U CPU @ 1.40GHz HD 5000 and No additional gpu

Even though, No problem with QuickTime X ( Low cpu usage)
ffmpeg problem?

[Akemi]

i don't know. maybe try with mpv --no-config --hwdec=auto. maybe a log with --log-file=PATH might help.

[xswqaz]

It was success to decode hwaccel with 720p, But failed with 1080p

720p (High@L3.2)

[   0.650][v][vd] Pixel formats supported by decoder: videotoolbox_vld yuv420p
[   0.650][v][vd] Codec profile: High (0x64)
[   0.650][v][vd] Requesting pixfmt 'videotoolbox_vld' from decoder.
[   0.921][d][ffmpeg/video] h264: Reinit context to 1280x720, pix_fmt: videotoolbox_vld
[   0.945][i][vd] Using hardware decoding (videotoolbox).
[   0.945][v][vd] Decoder format: 1280x720 videotoolbox[nv12] auto/auto/auto/auto CL=mpeg2/4/h264 (auto 0.000000/0.000000/0.000000)
[   0.946][v][vf] Video filter chain:
[   0.946][v][vf]   [in] 1280x720 videotoolbox[nv12] bt.709/bt.709/bt.1886/limited SP=1.000000 CL=mpeg2/4/h264
[   0.946][v][vf]   [out] 1280x720 videotoolbox[nv12] bt.709/bt.709/bt.1886/limited SP=1.000000 CL=mpeg2/4/h264
[   0.946][i][cplayer] VO: [gpu] 1280x720 videotoolbox[nv12]
[   0.946][v][cplayer] VO: Description: Shader-based GPU Renderer

1080p(High@L4)

[   0.708][v][vd] Pixel formats supported by decoder: videotoolbox_vld yuv420p
[   0.708][v][vd] Codec profile: High (0x64)
[   0.708][v][vd] Requesting pixfmt 'videotoolbox_vld' from decoder.
[   0.790][d][ffmpeg/video] h264: VideoToolbox session not available.
[   0.790][e][ffmpeg/video] h264: Failed setup for format videotoolbox_vld: hwaccel initialisation returned er
ror.
[   0.790][v][vd] Pixel formats supported by decoder: yuv420p yuv420p
[   0.790][v][vd] Codec profile: High (0x64)
[   0.790][v][vd] Requesting pixfmt 'yuv420p' from decoder.
[   0.790][d][ffmpeg/video] h264: Reinit context to 1920x1088, pix_fmt: yuv420p
[   0.828][v][vd] Falling back to software decoding.
[   0.828][v][vd] Detected 4 logical cores.
[   0.828][v][vd] Requesting 5 threads for decoding.
[   0.833][d][ffmpeg/video] h264: Reinit context to 1920x1088, pix_fmt: yuv420p
[   0.877][v][vd] Using software decoding.
[   0.878][v][vd] Decoder format: 1920x1080 yuv420p auto/auto/auto/auto CL=mpeg2/4/h264 (auto 0.000000/0.000000/0.000000)
[   0.878][v][vf] Video filter chain:
[   0.878][v][vf]   [in] 1920x1080 yuv420p bt.709/bt.709/bt.1886/limited SP=1.000000 CL=mpeg2/4/h264
[   0.878][v][vf]   [out] 1920x1080 yuv420p bt.709/bt.709/bt.1886/limited SP=1.000000 CL=mpeg2/4/h264
[   0.879][i][cplayer] VO: [gpu] 1920x1080 yuv420p
[   0.879][v][cplayer] VO: Description: Shader-based GPU Renderer

What means "h264: VideoToolbox session not available."?
I don't know why 1080p file detected as 1920x1088

[Akemi]

i believe that means that videotoolbox was used by another process at the time you tried it. no your file is not detected to be 1920x1088, just the context is that size. also no idea what you are trying to prove with the 720p sample, since you tried my 1080p file and it worked.

[xswqaz]

There is no process use videotoolbox except mpv.
(No execution difference between 720 and 1080)

[Akemi]

that's what the error says though and that one is returned by the Framework via kVTVideoDecoderNotAvailableNowErr (Apple Doc).

videotoolbox can be a bit picky. try it several times and see if it always fails at [ffmpeg/video] h264: VideoToolbox session not available. for testing try to close every other App, especially browsers, since they could use it in the background. try VLC for testing, they use their own videotoolbox implementation and not the ffmpeg one i believe(?). if theirs work something isn't working correctly and i would say it's probably a problem with ffmpeg then.

[Argon-]

Does VLC have a log?
When you try to assess whether hwdec is running, please don't use CPU usage. This is not reliable as swdec for h264 is so good these days that it doesn't create noticeable load for most videos.

[xswqaz]

I tried VLC 3.0.1. and decoded with hwaccel no problem
CPU usage is very stable (less than 5%)
When I played same file with mpv, CPU usage over 10%

main debug: looking for video decoder module matching "any": 14 candidates
videotoolbox debug: new SPS parsed: 0
videotoolbox debug: new PPS parsed: 0
videotoolbox info: Using Video Toolbox to decode 'h264'
main debug: using video decoder module "videotoolbox"
main debug: looking for audio decoder module matching "any": 21 candidates
main debug: using audio decoder module "faad"
main debug: looking for meta reader module matching "any": 2 candidates
main debug: using meta reader module "taglib"
main debug: removing module "taglib"
main debug: `file:///Volumes/hdd/sample.mp4' successfully opened
mp4 debug: elst (0) gives 0ms (movie)-> 33ms (track)
mp4 debug: track[Id 0x1] using Sync Sample Box (stss)
mp4 debug: stss gives 1 --> 0 (sample number)
mp4 debug: elst (0) gives 0ms (movie)-> 0ms (track)
main debug: Buffering 0%
faad warning: Channel coupling not yet implemented
main debug: Buffering 25%
main debug: Buffering 50%
faad warning: decoded zero sample
main debug: reusing audio output
main debug: VLC is looking for: 'f32l' 48000 Hz Stereo frame=1 samples/8 bytes
auhal debug: attempting to use device 0
auhal debug: using default audio device 39
main debug: Buffering 75%
main debug: Buffering 100%
auhal debug: output layout of AUHAL is Stereo
auhal debug: selected 2 physical channels for device output
auhal debug: VLC will output: Stereo
auhal debug: Current AU format: [48000.000000][mcpl][9][8][1][8][2][32]
auhal debug: Current device has a latency of 0 us
auhal debug: analog output successfully opened
main debug: output 'f32l' 48000 Hz Stereo frame=1 samples/8 bytes
main debug: looking for audio volume module matching "any": 2 candidates
main debug: using audio volume module "float_mixer"
main debug: input 'f32l' 48000 Hz Stereo frame=1 samples/8 bytes
main debug: looking for audio filter module matching "scaletempo": 16 candidates
scaletempo debug: format: 48000 rate, 2 nch, 4 bps, fl32
scaletempo debug: params: 30 stride, 0.200 overlap, 14 search
scaletempo debug: 1.000 scale, 1440.000 stride_in, 1440 stride_out, 1152 standing, 288 overlap, 672 search, 2400 queue, fl32 mode
main debug: using audio filter module "scaletempo"
main debug: conversion: 'f32l'->'f32l' 48000 Hz->48000 Hz Stereo->Stereo
main debug: conversion pipeline complete
main debug: conversion: 'f32l'->'f32l' 48000 Hz->48000 Hz Stereo->Stereo
main debug: conversion pipeline complete
main debug: looking for audio resampler module matching "any": 3 candidates
main debug: using audio resampler module "samplerate"
main debug: Stream buffering done (1250 ms in 13 ms)
videotoolbox info: vt cvpx chroma: BGRA
main debug: looking for text renderer module matching "any": 3 candidates
main debug: using text renderer module "freetype"
main debug: looking for video converter module matching "any": 22 candidates
swscale debug: 32x32 (32x32) chroma: YUVA -> 16x16 (16x16) chroma: RGBA with scaling using Bicubic (good quality)
main debug: using video converter module "swscale"
main debug: looking for video converter module matching "any": 22 candidates
yuvp debug: YUVP to YUVA converter
main debug: using video converter module "yuvp"
main debug: Deinterlacing available
main debug: deinterlace -1, mode auto, is_needed 0
main debug: looking for vout window module matching "any": 1 candidates
macosx debug: Opening video window
macosx debug: Activated assertion NoIdleSleepAssertion through IOKit (38114)
macosx debug: Releasing old IOKit other assertion (38114)
macosx debug: Activated assertion NoDisplaySleepAssertion through IOKit (38115)
macosx debug: toggle playlist from state: removed splitview 0, minimized view 0. Event 2
macosx debug: toggle playlist to state: removed splitview 0, minimized view 0
macosx debug: returning videoview with proposed position x=0, y=0, width=1920, height=1080
main debug: using vout window module "macosx"
main debug: Opening vout display wrapper
main debug: looking for vout display module matching "any": 7 candidates
main debug: VoutDisplayEvent 'resize' 1920x1080
main debug: looking for glconv module matching "any": 1 candidates
main debug: using glconv module "glconv_cvpx"
main debug: VoutDisplayEvent 'resize' 1920x1080
main debug: using vout display module "macosx"
main debug: VoutDisplayEvent 'resize' 716x529
main debug: original format sz 1920x1088, of (0,0), vsz 1920x1080, 4cc CVPB, sar 1:1, msk r0x0 g0x0 b0x0
main debug: removing module "freetype"
main debug: looking for text renderer module matching "any": 3 candidates
main debug: using text renderer module "freetype"
main debug: Received first picture
videotoolbox info: Raising max DPB to 2
main debug: Decoder wait done in 96 ms
main debug: inserting 1414 zeroes

[Akemi]

then you should try again with mpv master and ffmpeg master. if the problem still persists it's most likely an ffmpeg problem.

[Akemi]

i reported this issue to ffmpeg https://trac.ffmpeg.org/ticket/7545.

[octoviaa]

I've similar issue (not able to use HW acceleration) in Mojave Mac Pro 5,1 with GTX 680.
Below are the error:
[vd] Pixel formats supported by decoder: videotoolbox_vld yuv420p
[vd] Codec profile: Main (0x4d)
[vd] Requesting pixfmt 'videotoolbox_vld' from decoder.
[ffmpeg/video] h264: Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.

However, I didn't have the following error
[ffmpeg/video] h264: VideoToolbox session not available.

Let me know if I should report it as different bug.

[Akemi]

if you can reproduce it with ffmpeg then reporting it to ffmpeg would be a good idea.

[Akemi]

reported to ffmpeg.