CaptureFromTexture .MP4 files do not play in Chrome/Edge browers but do via VLC or Media Player

[Wayne-nulitics]

When recording using CaptureFromTexture the .mp4 file is playable using a Windows media player like VLC etc. However the video will not play in Chrome.

• Unity version: 2-21.3.19f1
• AVPro Movie Capture version: 5.1.3
• Operating system version: Android 10
• Capture component used: CaptureFromTexture
• Capture settings (resolution, frame-rate, codec): Output to Video File, Original Size, 30 FPS, H264, Audio Source Microphone, Streamable MP4.

To Reproduce
Download this to desktop and open with VLC or Media player:
https://storage.googleapis.com/notifiesu/test/8c62ad77-0bd4-4647-8ab3-0cfac68ad069v1.mp4
Open same link in Chrome Browser and try to play.

[Wayne-nulitics]

Also tried the HEVC codec and setting the codec tags in HTML5 video, just nothing in Chrome or Edge, Firefox works tho.

[Wayne-nulitics]

... found the bug, audio is mono in the .MP4 file and Metadata says stereo and missing ACC ID_END, so reopening.
See the output errors in Chrome, use the following to see the debug:
chrome://media-internals/
Then copy and paste the video url into a new tab: https://storage.googleapis.com/notifiesu/test/8c62ad77-0bd4-4647-8ab3-0cfac68ad069v1.mp4

In the media-internals tab you will get the breakdown of last played, and the audio warning.

info: Selected D3D11VideoDecoder for video decoding, config: codec: h264, profile: h264 high, level: not available, alpha_mode: is_opaque, coded size: [1280,720], visible rect: [0,0,1280,720], natural size: [1280,720], has extra data: true, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:SMPTE170M, matrix:BT709, range:LIMITED}
kAudioTracks: [object Object]
kVideoTracks: [object Object]
error: [object Object]

And then the error
{
"properties": {
"render_id": 1004,
"kAudioTracks": [
{
"bytes per channel": 4,
"bytes per frame": 4,
"channel layout": "MONO",
"channels": 1,
"codec": "aac",
"codec delay": 0,
"discard decoder delay": true,
"encryption scheme": "Unencrypted",
"has extra data": true,
"profile": "unknown",
"sample format": "Float 32-bit planar",
"samples per second": 44100,
"seek preroll": "0us"
}
],
"warning": "audio decoder fallback after initial decode error.",
"duration": 16.242472
{
"time": 16.10300000011921,
"key": "warning",
"value": "audio decoder fallback after initial decode error."
},
{
"time": 16.112999998033047,
"key": "error",
"value": "audio decoder reinitialization failed"
},
{
"time": 16.116999998688698,
"key": "error",
"value": "audio error during playing, status: PIPELINE_ERROR_DECODE"
},

[Wayne-nulitics]

If you download the .mp4 shared to a windows PC and you open it with VLC you will see the Metadata says Stereo (VLC bug).
If you look at what Chrome detects it is Mono.
If you open the .MP4 with another Windows tool like AudioCity you will only find one audio track.
If you use a tool like MediaInfo you will see the error: Missing ID_END

This is clearly a bug, either:
a) Correctly create the .MP4 file audio metadata
b) Add the ID_END

[Wayne-nulitics]

More information, if you use ffmpeg -i broken.mp4 fixed.mp4 you get a working file, here is the link:
https://storage.googleapis.com/notifiesu/test/fixed.mp4

During the ffmpeg process there is definatly errors in the source .mp4 created by CaptureFromTexture, so this is 100% a bug.

[aac @ 000002179eb3e340] Input buffer exhausted before END element found
Press [q] to stop, [?] for help
[aac @ 00000217a0ad00c0] Input buffer exhausted before END element found
Error while decoding stream #0:1: Invalid data found when processing input

[Wayne-nulitics]

I did some more testing, this time using HVEC and no pausing, here are the two files, one direct from device using CaptureFromTexture and one copied using ffmpeg, using this:
ffmpeg.exe' -i .\13e75efb-45c4-4b1f-a71d-03c6f4db2acd.mp4 -c:v copy 13e75efb-45c4-4b1f-a71d-03c6f4db2acd_copy.mp4

Stream copy is a mode selected by supplying the copy parameter to the -codec option. It makes ffmpeg omit the decoding and encoding step for the specified stream, so it does only demuxing and muxing. It is useful for changing the container format or modifying container-level metadata.

Thus all I did is correct the malformed .MP4 container.

You have this error in the source:
[aac @ 000002104dba9c40] Input buffer exhausted before END element found
Error while decoding stream #0:1: Invalid data found when processing input

And on the copied file where the structure is corrected:
[aac @ 000002104dd38000] Qavg: 47848.676

Files here:
https://storage.googleapis.com/notifiesu/test/13e75efb-45c4-4b1f-a71d-03c6f4db2acd.mp4
https://storage.googleapis.com/notifiesu/test/13e75efb-45c4-4b1f-a71d-03c6f4db2acd_copy.mp4

Using chrome://media-internals/ to debug

Original, fails on broken ACC audio metadata:

00:00:00.376	pipeline_state	"kStarting"
00:00:01.552	info	"FFmpegDemuxer: created video stream, config codec: hevc, profile: hevc main, level: not available, alpha_mode: is_opaque, coded size: [1280,720], visible rect: [0,0,1280,720], natural size: [1280,720], has extra data: true, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}"
00:00:01.552	info	"FFmpegDemuxer: created audio stream, config codec: aac, profile: unknown, bytes_per_channel: 4, channel_layout: MONO, channels: 1, samples_per_second: 44100, sample_format: Float 32-bit planar, bytes_per_frame: 4, seek_preroll: 0us, codec_delay: 0, has extra data: true, encryption scheme: Unencrypted, discard decoder delay: true, target_output_channel_layout: NONE, target_output_sample_format: Unknown sample format, has aac extra data: true"
00:00:01.552	kAudioTracks	[{"bytes per channel":4,"bytes per frame":4,"channel layout":"MONO","channels":1,"codec":"aac","codec delay":0,"discard decoder delay":true,"encryption scheme":"Unencrypted","has extra data":true,"profile":"unknown","sample format":"Float 32-bit planar","samples per second":44100,"seek preroll":"0us"}]
00:00:01.552	kVideoTracks	[{"alpha mode":"is_opaque","codec":"hevc","coded size":"1280x720","color space":{"matrix":"BT709","primaries":"BT709","range":"LIMITED","transfer":"BT709"},"encryption scheme":"Unencrypted","has extra data":true,"hdr metadata":"unset","natural size":"1280x720","orientation":"0°","profile":"hevc main","visible rect":"0,0 1280x720"}]
00:00:01.552	kMaxDuration	5.20127

00:00:01.566	pipeline_state	"kPlaying"
00:00:01.567	error	"Failed to send audio packet for decoding: {timestamp=0 duration=23220 size=2 is_key_frame=1 encrypted=0}"
00:00:01.567	warning	"audio decoder fallback after initial decode error."
00:00:01.567	error	"audio decoder reinitialization failed"
00:00:01.567	error	"audio error during playing, status: PIPELINE_ERROR_DECODE"
00:00:01.567	error	{"code":3,"data":{},"group":"PipelineStatus","message":"","stack":[{"file":"media\renderers\audio_renderer_impl.cc","line":884}]}

Copied version, working:

00:00:00.438	pipeline_state	"kStarting"
00:00:00.822	info	"FFmpegDemuxer: created video stream, config codec: hevc, profile: hevc main, level: not available, alpha_mode: is_opaque, coded size: [1280,720], visible rect: [0,0,1280,720], natural size: [1280,720], has extra data: true, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:SMPTE170M, matrix:BT709, range:LIMITED}"
00:00:00.822	info	"FFmpegDemuxer: created audio stream, config codec: aac, profile: unknown, bytes_per_channel: 4, channel_layout: MONO, channels: 1, samples_per_second: 44100, sample_format: Float 32-bit planar, bytes_per_frame: 4, seek_preroll: 0us, codec_delay: 0, has extra data: true, encryption scheme: Unencrypted, discard decoder delay: true, target_output_channel_layout: NONE, target_output_sample_format: Unknown sample format, has aac extra data: true"
00:00:00.822	kAudioTracks	[{"bytes per channel":4,"bytes per frame":4,"channel layout":"MONO","channels":1,"codec":"aac","codec delay":0,"discard decoder delay":true,"encryption scheme":"Unencrypted","has extra data":true,"profile":"unknown","sample format":"Float 32-bit planar","samples per second":44100,"seek preroll":"0us"}]
00:00:00.823	kVideoTracks	[{"alpha mode":"is_opaque","codec":"hevc","coded size":"1280x720","color space":{"matrix":"BT709","primaries":"BT709","range":"LIMITED","transfer":"SMPTE170M"},"encryption scheme":"Unencrypted","has extra data":true,"hdr metadata":"unset","natural size":"1280x720","orientation":"0°","profile":"hevc main","visible rect":"0,0 1280x720"}]
00:00:00.823	kMaxDuration	5.22449

00:00:00.854	pipeline_state	"kPlaying"
00:00:00.855	info	"D3D11VideoDecoder config change: profile: 16 chroma_sampling_format: 4:2:0 coded_size: (1280, 736)"
00:00:00.855	info	"D3D11VideoDecoder is using hevc main / 4:2:0"
00:00:00.855	info	"D3D11VideoDecoder producing NV12"
00:00:00.855	info	"D3D11VideoDecoder: Selected NV12"
00:00:00.855	info	"D3D11VideoDecoder output color space: (same as input)"
00:00:00.855	info	"D3D11VideoDecoder is binding textures"
00:00:00.855	info	"D3D11VideoDecoder is using single textures"
00:00:00.860	dimensions	"1280x720"
00:00:00.860	kResolution	"1280x720"
00:00:00.822	duration	5.22449
00:00:00.860	video_buffering_state	{"state":"BUFFERING_HAVE_ENOUGH"}
00:00:01.873	info	"Effective playback rate changed from 0 to 1"
00:00:01.873	event	"kPlay"
00:00:01.190	pipeline_buffering_state	{"for_suspended_start":false,"state":"BUFFERING_HAVE_ENOUGH"}

[MorrisRH]

Duplicate of #243

[Wayne-nulitics]

Hi, The issue is that metadata of the file created by the plugin, it is missing the END tag, this is not an Android device problem. If you rebuild the container it fixes the tag. Your .MP4 file is missing the end tag, I have another plugin that creates a correct tag but it has poor performance. This is a bug, not an Android OS problem. Sent from Android device On 26 Feb 2023 11:16, Morris Butler ***@***.***> wrote: Duplicate of #243<#243> — Reply to this email directly, view it on GitHub<#260 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR2MTXJYS7CBQGKVTENWL2TWZMNNHANCNFSM6AAAAAAVHY55WM>. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[Wayne-nulitics]

@MorrisRH I'm not sure how you wrote the plugin, I'm assuming you are calling some Java code for Android and using known MediaMuxer. ie. https://developer.android.com/reference/android/media/MediaMuxer

There is something called mEncoder.signalEndOfInputStream();
It should queue an empty input buffer with the BUFFER_FLAG_END_OF_STREAM flag.

Can you confirm that you guys are implementing this?
https://developer.android.com/reference/android/media/MediaCodec#signalEndOfInputStream()

[MorrisRH]

We're using the NDK Media module for the video encoding and can confirm that EOS is being handled correctly.

[Wayne-nulitics]

Hmmm, this is really odd, because I'm using both NatCorder and AVPro, NatCorder changed their model and no longer do a standalone recorder so we had to change, their .MP4 container is correct, the AVPro one is malformed, on same device.

Maybe the way you rewrite the moov causes it? I know its purely structural because demuxing and remuxing fixed the error.
We just can't manually process each recording using ffmpeg to fix a bug like this.
Command ffmpeg.exe' -v error -i source.mp4 shows there is no end element.
You can further debug the file and see the problem using this: ffmpeg -v trace -i videofile.mp4

Stumped, can be anything from presentationTimes being off to the way its written, do you have any way to validate or remux the file? This is really a big issue especially with the amount of AR/VR devices running Android not being able to write/save a correct MP4 container.

[Wayne-nulitics]

I'll see when I get some time, maybe test https://github.com/arthenica/ffmpeg-kit and write a demux/remux workaraound, until there's a fix. Based on this: https://github.com/arthenica/ffmpeg-kit/wiki/Using-FFmpegKit-in-Unity

Althought not for this version of ffmpeg, here is a guide on the previous version. https://sourceexample.com/en/671cf50fddfec13eb5e1/

[MorrisRH]

I've managed to track this down.

It was caused by differences between how the AMediaMuxer on different devices handle codec configuration packets. Samsung devices always discard the packet whereas the Google Pixel 4 writes it out.

The fix will make it into the next release.

[Chris-RH]

The latest version has been released. Please let us know if it has not fixed the issue for you.

[Wayne-nulitics]

The latest version has been released. Please let us know if it has not fixed the issue for you.

@Chris-RH Can confirm that the issue has been resolved and videos are now playable in Chrome browers.