MPEG-TS instead of MP4

[Anuskuss]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2020.07.28
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-o', u'video.mp4', u'-v', u'--merge-output-format', u'mp4', u'--recode-video', u'mp4', u'https://v.redd.it/16cqbkev2ci51']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 2.7.16 (CPython) - Linux-5.4.51-v7l+-armv7l-with-debian-10.4
[debug] exe versions: ffmpeg N-98802-g01a580f
[debug] Proxy map: {}
[RedditR] iduchl: Downloading JSON metadata
[Reddit] 16cqbkev2ci51: Downloading m3u8 information
[Reddit] 16cqbkev2ci51: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://v.redd.it/16cqbkev2ci51/HLS_540.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 3
[download] Destination: video.mp4
[download] 100% of 1.98MiB in 00:01
[debug] ffmpeg command line: ffmpeg -i 'file:video.mp4'
[ffmpeg] Not converting video file video.mp4 - already is in target format mp4

Description

Most videos from v.redd.it are DASH, which means audio and video get downloaded separately and then get combined into a (working) MP4. Some videos are MPEG-TS though and youtube-dl just downloades them and leaves them alone (even though I specified -o video.mp4). Those videos play fine but they don't work as HTML5 video.
I've tried adding --merge-output-format mp4 and --recode-video mp4 but youtube-dl keeps saying

[ffmpeg] Not converting video file video.mp4 - already is in target format mp4

As a workaround I've added my own exec (youtube-dl -o tmp.mp4 --exec "ffmpeg -i {} -c copy video.mp4 && rm {}" "https://v.redd.it/16cqbkev2ci51"), which runs ffmpeg for less than a second and actually gives me a web-browser playable MP4.
Also if it help you guys to debug this: I replaced my /bin/ffmpeg with echo $* > /tmp/ffmpeg.txt and as it turns out the system ffmpeg doesn't even get called.

[ghost]

FYI, another workaround: --hls-prefer-ffmpeg

[Anuskuss]

Maybe I didn't explain my issue all to well back then (was just getting started with youtube-dl).
The issue I have is that Reddit videos without audio (e.g. https://redd.it/iduchl) are being downloaded (HLS or DASH doesn't matter) and then put into "MP4" containers. But the output wasn't actually MPEG-4 though but rather MPEG-TS, hence the video doesn't play in web browsers (HTML5 video: <video src="video.mp4"></video>).
Downloading videos with audio or running them through ffmpeg afterwards (--exec "ffmpeg -i {} -c copy video.mp4 && rm {}", --hls-prefer-ffmpeg, etc.) works but the other flags (--merge-output-format mp4 or --recode-video mp4) don't:

[ffmpeg] Not converting video file video.mp4 - already is in target format mp4

I can't find a way to force it to proper MP4/MPEG4. Hopefully it's clearer now.

[Anuskuss]

Hm, I thought that I made it pretty clear now but alright, I'll give you an example:

Good: Video with Audio, bestvideo+bestaudio (DASH) -> MPEG-4 container

$ youtube-dl -o video.mp4 https://v.redd.it/8rj8b6c18ki51
[Reddit] 8rj8b6c18ki51: Downloading m3u8 information
[Reddit] 8rj8b6c18ki51: Downloading MPD manifest
[info] 8rj8b6c18ki51: Downloading 1 format(s): dash-video_2393788+dash-audio_0_131736
[download] Destination: video.fdash-video_2393788.mp4
[download] 100% of 22.14MiB in 00:03
[download] Destination: video.fdash-audio_0_131736.m4a
[download] 100% of 1.22MiB in 00:00
[Merger] Merging formats into "video.mp4"
Deleting original file video.fdash-video_2393788.mp4 (pass -k to keep)
Deleting original file video.fdash-audio_0_131736.m4a (pass -k to keep)

$ mediainfo video.mp4 | grep "Format  "
Format                                   : MPEG-4
Format                                   : AVC
Format                                   : AAC LC

Bad: Video without Audio, bestvideo (HLS) -> MPEG-TS container

$ youtube-dl -o video.mp4 https://v.redd.it/16cqbkev2ci51
[Reddit] 16cqbkev2ci51: Downloading m3u8 information
[Reddit] 16cqbkev2ci51: Downloading MPD manifest
[info] 16cqbkev2ci51: Downloading 1 format(s): hls-1352
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 3
[download] Destination: video.mp4
[download] 100% of 1.98MiB in 00:00

$ mediainfo video.mp4 | grep "Format  "
Format                                   : MPEG-TS
Format                                   : AVC

Good: Video without Audio, DASH -> MPEG-4 container

$ youtube-dl -o video.mp4 -f dash-video_1235784 https://v.redd.it/16cqbkev2ci51
[Reddit] 16cqbkev2ci51: Downloading m3u8 information
[Reddit] 16cqbkev2ci51: Downloading MPD manifest
[info] 16cqbkev2ci51: Downloading 1 format(s): dash-video_1235784
[download] Destination: video.mp4
[download] 100% of 1.81MiB in 00:00

$ mediainfo video.mp4 | grep "Format  "
Format                                   : MPEG-4
Format                                   : AVC

Bad: Video without Audio, bestvideo (HLS), Force MP4 -> MPEG-TS container

$ youtube-dl -o video.mp4 --merge-output-format mp4 --recode-video mp4 https://v.redd.it/16cqbkev2ci51
[Reddit] 16cqbkev2ci51: Downloading m3u8 information
[Reddit] 16cqbkev2ci51: Downloading MPD manifest
[info] 16cqbkev2ci51: Downloading 1 format(s): hls-1352
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 3
[download] Destination: video.mp4
[download] 100% of 1.98MiB in 00:00
[VideoConvertor] Not converting media file 'video.mp4'; already is in target format mp4

$ mediainfo video.mp4 | grep "Format  "
Format                                   : MPEG-TS
Format                                   : AVC

Good: Video without Audio, bestvideo (HLS), Custom ffmpeg command -> MPEG-4 container

$ youtube-dl -o tmp.mp4 --exec "ffmpeg -hide_banner -i {} -c copy video.mp4 && rm {}" https://v.redd.it/16cqbkev2ci51
[Reddit] 16cqbkev2ci51: Downloading m3u8 information
[Reddit] 16cqbkev2ci51: Downloading MPD manifest
[info] 16cqbkev2ci51: Downloading 1 format(s): hls-1352
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 3
[download] Destination: tmp.mp4
[download] 100% of 1.98MiB in 00:00
[Exec] Executing command: ffmpeg -hide_banner -i /tmp/tmp.mp4 -c copy video.mp4 && rm /tmp/tmp.mp4
Input #0, mpegts, from '/tmp/tmp.mp4':
  Duration: 00:00:12.30, start: 1.400000, bitrate: 1351 kb/s
  Program 1
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), none(progressive), 640x640 [SAR 1:1 DAR 1:1], 30 fps, 30 tbr, 90k tbn, 90k tbc
Output #0, mp4, to 'video.mp4':
  Metadata:
    encoder         : Lavf58.27.104
    Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), none(progressive), 640x640 [SAR 1:1 DAR 1:1], q=2-31, 30 fps, 30 tbr, 90k tbn, 90k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=  369 fps=0.0 q=-1.0 Lsize=    1981kB time=00:00:12.26 bitrate=1322.7kbits/s speed= 321x
video:1978kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.116154%

$ mediainfo video.mp4 | grep "Format  "
Format                                   : MPEG-4
Format                                   : AVC

Why does it matter?
I'm serving those videos over HTTP but MPEG-TS videos don't play. To see that for yourself, create a HTML file, write <video src="video.mp4" controls></video> into it, and try to play it. Simply enforcing DASH would work but gives you a worse video (480x480 instead of 640x640) in this case.

Yes, it does matter. No guarantee that HLS use the same codecs or containers.

Okay, you were right. I though that they just had different protocols serving the same files but this does matter here.

[Anuskuss]

Please remove any "good" examples. If something is "good" we dont need to see it.

No, the good cases provide a reference.

Please stop using those links, and use one of the proper links instead:

All links are "proper" links - the v.redd.it links are just redirects.

Using this command [1]:

reddit iduchl

resulting file plays fine in my browser.

Well yeah, of course. As per your output above

{BaseURL:DASH_480.mp4 Height:480 MimeType:video/mp4}

you will only get the 480x480 video (see case 3: dash-video_1235784), but with youtube-dl you get the superior 640x640 hls-1352. But as you hopefully can already tell, HLS results in a bad MP4 (MPEG-TS instead of MPEG-4). MPEG-TS usually comes with the .ts extension whereas MPEG-4 uses .mp4.

[Anuskuss]

If you are insisting on using the HLS result in a browser, you will need to do something like this:

Yes, I already did that (see case 5).

The fact that the unadulterated media result doesnt work in a browser, really doesnt have anything to do with YouTube-DL.

You don't seem to understand the issue here: It's not that it's not working in browser, that was just an example. Youtube-dl is creating a bad MP4 (AVC does not belong in a MPEG-TS container). It should either refuse to create a .mp4 or create a .ts instead. Or, you know, just make --recode-video mp4 work.

[Anuskuss]

Why are you so desperately trying to dismiss this issue as invalid? Read #26410 (comment)

or running them through ffmpeg afterwards

[Anuskuss]

Those videos play fine but they don't work as HTML5 video.

They play fine by pure chance. Giving a MPEG-TS container a .mp4 is not only pretty misleading, it's also against the MPEG specification to put MPEG-4 video into a MPEG-2 container. It only works because the player you're testing this on accounted for this (on your web browser didn't).

[Anuskuss]

Now that the issue is fixed downstream, I'm kinda debating if I should just delete all my comments here because it's quite embarrassing how I allowed myself to get trolled this hard by 89z (a random with 0 commits to this repo). But I guess it will serve as reference in case the project gets revived and a dev finds this? I'll leave it open just in case.

[Anuskuss]

To prevent further trolling, I have decided to block 89z now. Sorry for the noise.

[rautamiekka]

I don't think that was trolling in the slightest ...

[Anuskuss]

@rautamiekka Please keep your opinions to yourself. This is an issue tracker, not Twitter.