Problem with local m3u/m3u8 files

[atom-smasher]

Checklist

• I've verified that I'm running youtube-dl version 2019.09.12.1
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--ignore-config', u'-a', u'./640x360-test.m3u8']
[debug] Batch file urls: [u'https://walterebert.com/_/playground/videos/ts/640x3600.ts', u'https://walterebert.com/_/playground/videos/ts/640x3601.ts', u'https://walterebert.com/_/playground/videos/ts/640x3602.ts', u'https://walterebert.com/_/playground/videos/ts/640x3603.ts', u'https://walterebert.com/_/playground/videos/ts/640x3604.ts', u'https://walterebert.com/_/playground/videos/ts/640x3605.ts']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2019.09.12.1
[debug] Python version 2.7.16 (CPython) - Linux-5.0.0-13-generic-x86_64-with-Ubuntu-19.04-disco
[debug] exe versions: ffmpeg 4.1.1-1, ffprobe 4.1.1-1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 640x3600: Requesting header
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://walterebert.com/_/playground/videos/ts/640x3600.ts'
[download] Destination: 640x3600-640x3600.ts
[download] 100% of 993.61KiB in 00:02
[generic] 640x3601: Requesting header
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://walterebert.com/_/playground/videos/ts/640x3601.ts'
[download] Destination: 640x3601-640x3601.ts
[download] 100% of 803.59KiB in 00:02
[generic] 640x3602: Requesting header
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://walterebert.com/_/playground/videos/ts/640x3602.ts'
[download] Destination: 640x3602-640x3602.ts
[download] 100% of 1.14MiB in 00:02
[generic] 640x3603: Requesting header
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://walterebert.com/_/playground/videos/ts/640x3603.ts'
[download] Destination: 640x3603-640x3603.ts
[download] 100% of 900.53KiB in 00:02
[generic] 640x3604: Requesting header
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://walterebert.com/_/playground/videos/ts/640x3604.ts'
[download] Destination: 640x3604-640x3604.ts
[download] 100% of 475.51KiB in 00:02
[generic] 640x3605: Requesting header
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://walterebert.com/_/playground/videos/ts/640x3605.ts'
[download] Destination: 640x3605-640x3605.ts
[download] 100% of 116.95KiB in 00:01

After that finishes, I have these files:

ls -tr1
640x3600-640x3600.ts
640x3601-640x3601.ts
640x3602-640x3602.ts
640x3603-640x3603.ts
640x3604-640x3604.ts
640x3605-640x3605.ts
640x360-test.m3u8

Description

When using a remote m3u8 file, youtube-dl performs as expected. All individual files are downloaded and properly concatenated.

This works as expected:

youtube-dl https://walterebert.com/_/playground/videos/ts/640x360.m3u8

and outputs a single mp4 file, 640x360-640x360.mp4.

When using a local copy of an m3u8 file, in this case edited to include FQDNs, youtube-dl treats it as a normal batch-file and just downloads the transport-stream "pieces".

youtube-dl -v --ignore-config -a ./640x360-test.m3u8

Expected behaviour would be for youtube-dl to recognise that the batch-file is actually an m3u/m3u8 file and behave accordingly. Alternatively, there should be a command-line option to "force" a local file to be interpreted as an m3u/m3u8 file.

[atom-smasher]

Here's a copy of the m3u file used above, edited to include FQDNs.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:9.189889,
https://walterebert.com/_/playground/videos/ts/640x3600.ts
#EXTINF:8.916667,
https://walterebert.com/_/playground/videos/ts/640x3601.ts
#EXTINF:9.333333,
https://walterebert.com/_/playground/videos/ts/640x3602.ts
#EXTINF:10.666667,
https://walterebert.com/_/playground/videos/ts/640x3603.ts
#EXTINF:9.000000,
https://walterebert.com/_/playground/videos/ts/640x3604.ts
#EXTINF:5.083333,
https://walterebert.com/_/playground/videos/ts/640x3605.ts
#EXT-X-ENDLIST

[atom-smasher]

As a workaround, I tried handling the m3u file as a local URL.

youtube-dl --ignore-config file:///tmp/test-m3u8/640x360-test.m3u8

That failed, apparently as a result of a previous security fix, #8228

WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[generic] 640x360-test: Requesting header
WARNING: Could not send HEAD request to file:///tmp/test-m3u8/640x360-test.m3u8: <urlopen error file:// scheme is explicitly disabled in youtube-dl for security reasons>
[generic] 640x360-test: Downloading webpage
ERROR: Unable to download webpage: <urlopen error file:// scheme is explicitly disabled in youtube-dl for security reasons> (caused by URLError(u'file:// scheme is explicitly disabled in youtube-dl for security reasons',))

While that security fix makes sense, and a command-line option alone would be a poor way to over-ride it, I'm wondering if a config-file option and a command-line option together may be a reasonable way to over-ride that security fix.

Ultimately, the fix for this problem should be some combination of (a) a command-line option to force local files to be interpreted as m3u/m3u8 files, (b) automatic recognition of m3u/m3u8 files by parsing, (c) automatic recognition of m3u/m3u8 files by filename extension.

[atom-smasher]

Python is not a strong point of mine, but it looks like this is where remote URLs are parsed to see if they're M3U files -

youtube-dl/youtube_dl/extractor/generic.py

Line 2349 in 7cb51b5

# Is it an M3U playlist?

Should be fairly easy to apply the same test to local files...?

[dstftw]

Batch file has nothing to do with m3u8 and not supposed to be used with -a. What you can do is setup local webserver where to put m3u8 with all contents.

[atom-smasher]

Setting up a local web-server is not always a viable option.

Parsing a local file (automatically and/or with a command-line option) should be available.

Is there any reason why a local file could not be an M3U file???