HEVC Crashing on Chromecast with Google TV via AVR

[sixones]

Environment

• ExoPlayer r2.13.1 and above (including r2.14.2).
• Chromecast with Google TV, connected to an AVR or soundbar (without the AVR or soundbar HEVC videos play correctly).
• Android 10.
• HEVC Main 10 video seems to be the common culprit.

Reproducing

• Play the attached video in the ExoPlayer demo application, or any HEVC Main 10 video.

Actual Behaviour

• Notice the video freezes randomly, frames are dropped or the device crashes and partially reboot (this takes a number of play throughs to replicate).
• Null pointer dereference error occurs from the surfaceflinger process: Unable to handle kernel NULL pointer dereference at virtual address 00000000, this then causes the device to partially reboot - it seems more like an OS reboot rather than a full device reboot.

Expected Behaviour

Playback is smooth without frame freezes or drops, and without rebooting the device.

Sample + Recorded Video

https://drive.google.com/drive/folders/1hpLZnBSBvF89MxOXtbOPMgfg_t-4k898

• HEVC 10 - EAC3 Atmos - Crash Sample.mkv is the sample video that can be played to replicate the behaviour.
• ExoPlayer v2.14.2 Demo on Chromecast with Google TV.mp4 provides a video recording showing the behaviour on the ExoPlayer r2.14.2 demo application (shows the video freezing and frames being dropped).
• adb bugreport is also zipped up in the folder.

Notes

Sounds like more of a device related issue rather than an issue with ExoPlayer, however the problem started occurring in ExoPlayer r.13.1. In r2.12.1 HEVC plays back smoothly on a Chromecast with Google TV device.

[christosts]

I'm afraid I could not repro. I'm using a chromecast with GoogleTV connected to a Sony TV and using a Samsung Soundbar. My chromecast device has the same OS build as yours. Can you also check the device firmware (see here how to check your device firmware version)?

The video was hard to play over the network (the device could not download fast enough - note the video is encoded at 40mbits) and I had to transfer it locally to the device. I also noticed a few network errors on your bugreport before the playback attempt, I wonder in the device was generally in a bad state when the playback happened.

Without a repro from our side, it's hard to assist further. Usually these issues tend to be unrelated to ExoPlayer and, as you said, device specific.

You mentioned the same video works fine with r2.12.1. Is this still occuring?

[sixones]

I will get hold of the device firmware information (another engineer has the device so I can't fetch this right now), the device ends up getting in a bad state after playing the file a few times with it getting noticeably worse over time.

The reports of HEVC working in r2.12.1 are from our users who have experienced the issue more frequently than we can replicate.

The size of the video should be fine, we've not seen any buffering issues and the video plays smoothly on lesser powered Fire TV devices and other Android devices so it seems to be specific to the Chromecast with Google TV device and as you can see from the recorded video its not a buffering issue but frame drops and freezes.

[christosts]

Note: we've had a similar report in #8557 with poor performance of HEVC on the same device but we were not able to reproduce internally either. We actually got some repros of the problem in #8557 on a specific device which, after a reboot, worked fine and never reproduced the issue.

[sixones]

After picking up the device myself I've been able to replicate the issue quite easily when connected to a full AVR that supports passthrough on the audio format I'm trying to play - originally when using a JBL soundbar I wasn't able to replicate but when switching to a Yamaha AVR the issue was happening straight away. Rebooting the device doesn't seem to make any difference.

In the logs, it looks like the audio is constantly underunning;

audioTrackUnderrun [eventTime=79.09, mediaPos=45.63, window=0, period=0, 40000, -9223372036854775807, 908]

After a minute or two, the HEVC decoder will fail:

2021-08-04 11:55:56.457 25900-26899 E: signalError(omxError 0x80001001, internalError -62)
2021-08-04 11:55:56.457 25900-26898 E: Codec reported err 0xffffffc2, actionCode 0, while in state 6

2021-08-04 11:55:56.692 25900-26861 E: Video codec error
      java.lang.IllegalStateException
        at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
        at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2790)
        at com.google.android.exoplayer2.mediacodec.SynchronousMediaCodecAdapter.dequeueOutputBufferIndex(SynchronousMediaCodecAdapter.java:100)

What's interesting is that the device then says the format is unsupported::

NoSupport [sizeAndRate.support, 3840x2160x23.976024627685547] [c2.android.hevc.decoder, video/hevc] [sabrina, Chromecast, Google, 29]

Disconnecting the AVR and running straight to a TV and the problem disappears, HEVC plays back smoothly.

There is definitely an issue here, if this could be forwarded on to the Chromecast device team that would be good as there is definitely a bug when using an AVR that's occurring on the device.

[sixones]

After more investigating, looks like this only happens on AVR's that only support 30, 25, 24Hz modes for RGB 4:4:4 8-bit or YCbCr 4:4:4 8-bit. On my Yamaha AVR, it's possible to switch between Mode 1: 4K at 60Hz / 50Hz and Mode 2: 4K at 30Hz / 25Hz / 24 Hz (details here: https://manual.yamaha.com/av/18/rxv685/en-US/366848907.html). Putting the AVR into Mode 1, and HEVC playback is perfect, switching to Mode 2 and HEVC playback is as described (stuttering with lots of frames dropped) and eventually crashes with the above errors.

[andrewlewis]

I tried to reproduce this using a Yamaha RX-V377 as the A/V receiver. The behavior was strangely intermittent! On the first few (3?) attempts, there were loads of dropped frames and audio kept underrunning then the decoder threw as reported here. Then I tried the exact same setup again (I didn't change modes on the A/V receiver) and this time audio played back fine and video was smooth. In all cases I think E-AC3 passthrough was being used.

[andrewlewis]

[Internal: b/197126165] has another bug report taken just after reproducing the issue.

[andrewlewis]

I've asked the relevant team to take a look and will update here if/when we learn more. Thanks.

[sixones]

Thanks for the update, odd that I saw it fix by changing the AVR mode, sounds like that was random and it might randomly work rather than to do the AVR mode.

[davidjmorin]

I have the same issue on all videos that are h.265
Plays fine when first watching, then on next episode it is zoomed in and wrong aspect ratio. Go back to home and chromecast restarts.

This is running through a Denon AVR

[dantebarba]

Same issue but not using AVR. CCwGtv connected to a Samsung TV and a Samsung soundbar via bluetooth. Playing HEVC 1080p 23 fps 1.8 Mpbs, audio MP3 Stereo.

It only happens sometimes using Plex though.

[WizardCornelius]

I can produce this 100% reliably on demand (every single next episode button press on plex app on the chromecast) if you'd like me to provide any data? It plays with the top left quarter of the image 'zoomed in' filling the screen, and sometimes crashes and reboots on exiting the episode, not often though.

Happy to carry out troubleshooting if it will assist. Just lmk what you would like me to do / provide and I'll get on it.

Versions:
App; com.plex.android 8.23.1.28053
Android TV OS 10
Kernel version 4.9.180
Android TV OS build: QTS1.210311.008.350836

File:
Video;1910x1080 -23fps 2.1 Mbps - HEVC 150 main 10
Audio; English - Opus - Stereo - 101kbps 48000khz

All available updates installed.

Also using a Yamaha YAS-107 soundbar interestingly.

[CakeOD36]

I can confirm this behavior with a direct TV connection (no AVR)

[ZoltrixGFC]

Also can confirm nothing to do with an AVR, was happening on two separate TVs for me. One with a soundbar, and one directly connected.

[TheSnyper]

I have four Chromecast with GoogleTV units. Three are connected to the TV only, one is connected to the TV with a bluetooth connection for audio. Every single unit has the same stretching/rebooting issue mentioned above when autoplaying the [next] HEVC video in a series on PLEX. The first video always plays fine, but continuing onto another one will produce this bug EVERY. SINGLE. TIME. This fault has only occurred since version 8.16.0 of ExoPlayer was incorporated into PLEX and every version since.

Please look at what changed in this version from the previous to try and narrow down the cause as these GoogleTV devices are pretty much useless for everybody that bought them specifically to run PLEX.

[ojw28]

It shouldn't be possible for the application layer (which is where ExoPlayer lives) to cause a device reboot, so the fact this is occurring indicates an issue in the underlying platform. I have requested an update and increased the priority of the issue that we previously filed on the relevant team.

[ZoltrixGFC]

@ojw28 lets us know if you need a hand testing.

[CakeOD36]

It shouldn't be possible for the application layer (which is where ExoPlayer lives) to cause a device reboot, so the fact this is occurring indicates an issue in the underlying platform. I have requested an update and increased the priority of the issue that we previously filed on the relevant team.

This is a critical observation. To date, this exceptional case has been provided no observable action by the Google development team even where they own the entire environment. I do understand this may be outside the scope of the ExoPlayer team but what actual action is being taken here?

[ojw28]

I do understand this may be outside the scope of the ExoPlayer team but what actual action is being taken here?

The internal issue was routed to the SOC manufacturer, who was unable to reproduce the issue and routed it back to the Google TV team. It's currently on the Google TV team pending next steps. We're continuing to escalate the issue with that team to try and get some more movement on it.

[ojw28]

I do have some updates on this issue:

1. The GoogleTV team and SOC vendor are now both able to reproduce the issue(s), including the stretched video issue and the kernel panic (reboot). It's unclear at this point whether they're related, but it seems likely.
2. The SOC vendor observed that the Plex app appears to be doing something unusual between the first and second playbacks. In particular, switching playback from non-tunnelled to tunnelled mode (background reading, if you're interested in what this means). It's thought that this is triggering an underlying bug in the SOC vendor provided part of the platform. The unusual nature of what the Plex app appears to be doing may explain why we've not received many reports of other applications being affected by the same issue.
3. I was able to reproduce the stretched video issue (but not the kernel panic) directly with the ExoPlayer demo app, by modifying it to toggle between non-tunnelled and tunnelled mode.
4. Plex can work around the issue in the short term by disabling tunnelled mode on GoogleTV. There may also be other workarounds that allow continued use of tunnelled mode, but this is the simplest short term fix. Plex have provided guidance here that the workaround will be included in their v8.26 beta release, which they're aiming to release next week. They've also provided an APK in the same post, if you're keen to try the fix before then.

We will keep this issue open to track the continued investigation into the underlying platform issue.

[ojw28]

Regarding the underlying platform issues:

1. A fix for the stretched video issue has been identified, and will be included in future updates to GoogleTV devices.
2. The kernel panic issue is still under investigation.

[ojw28]

Issues in the underlying platform should now be fixed.