sanitize_open problems

[pennsy-gg1]

sanitize_open is supposed to return a file object and a sanitized path name, but it currently returns the given filename always. Also, there is one call to sanitize_open that erroneously saves the sanitized name, and one call that discards the name instead of saving it. On FreeBSD, which uses US-ASCII file names, here is a URL that goes wrong: https://www.youtube.com/watch?v=DxEPR1BJEh8
It ends up with:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--proxy', u'socks5://mattapan:9050', u'-v', u'https://www.youtube.com/watch?v=DxEPR1BJEh8']
[debug] Encodings: locale US-ASCII, fs US-ASCII, out US-ASCII, pref US-ASCII
[debug] youtube-dl version 2017.11.26
[debug] Python version 2.7.14 - FreeBSD-10.3-RELEASE-p26-amd64-64bit-ELF
[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1, rtmpdump 2.4
[debug] Proxy map: {u'http': u'socks5://mattapan:9050', u'https': u'socks5://mattapan:9050'}
[youtube] DxEPR1BJEh8: Downloading webpage
[youtube] DxEPR1BJEh8: Downloading video info webpage
[youtube] DxEPR1BJEh8: Extracting video information
[youtube] DxEPR1BJEh8: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r1---sn-vgqskne6.googlevideo.com/videoplayback/id/0f110f475049121f/itag/136/source/youtube/requiressl/yes/mv/m/initcwndbps/665000/mn/sn-vgqskne6/mm/31/pl/24/ms/au/ei/kso2Wr38JMrUDZTph_AF/ratebypass/yes/mime/video%2Fmp4/otfp/1/gir/yes/clen/34813622/lmt/1512191792046732/dur/354.253/mt/1513540159/key/dg_yt0/signature/5B78F656BAA36D2DD34EB4AFFBE1158D81B1106B.62C000DF1B0CB08B2267098D5236FF2DF85E075A/ip/192.160.102.164/ipbits/0/expire/1513561842/sparams/ip,ipbits,expire,id,itag,source,requiressl,mv,initcwndbps,mn,mm,pl,ms,ei,ratebypass,mime,otfp,gir,clen,lmt,dur/'
[dashsegments] Total fragments: 68
[download] Destination: Bra (Turkey)-DxEPR1BJEh8.f136.mp4
[download] 1.5% of ~47.35KiB at 665.71KiB/s ETA 05:00Traceback (most recent call last):
File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/bin/youtube-dl/main.py", line 19, in
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 460, in main
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 450, in _real_main
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1986, in download
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 795, in extract_info
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 849, in process_ie_result
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1620, in process_video_result
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1887, in process_info
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1832, in dl
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 361, in download
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/dash.py", line 48, in real_download
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/fragment.py", line 114, in _append_fragment
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-2: ordinal not in range(128)

Here is the patch:

diff -ru youtube-dl/youtube_dl/downloader/fragment.py youtube-dl.new/youtube_dl/downloader/fragment.py
--- youtube-dl/youtube_dl/downloader/fragment.py	2017-12-10 11:17:26.000000000 -0500
+++ youtube-dl.new/youtube_dl/downloader/fragment.py	2017-12-12 14:02:33.867273000 -0500
@@ -168,7 +168,7 @@
                 self._write_ytdl_file(ctx)
                 assert ctx['fragment_index'] == 0
 
-        dest_stream, tmpfilename = sanitize_open(tmpfilename, open_mode)
+        dest_stream, _ = sanitize_open(tmpfilename, open_mode)
 
         ctx.update({
             'dl': dl,
diff -ru youtube-dl/youtube_dl/downloader/http.py youtube-dl.new/youtube_dl/downloader/http.py
--- youtube-dl/youtube_dl/downloader/http.py	2017-12-10 11:17:26.000000000 -0500
+++ youtube-dl.new/youtube_dl/downloader/http.py	2017-12-12 13:22:50.498057000 -0500
@@ -196,7 +196,7 @@
                 # Open destination file just in time
                 if ctx.stream is None:
                     try:
-                        ctx.stream, ctx.tmpfilename = sanitize_open(
+                        ctx.stream, _ = sanitize_open(
                             ctx.tmpfilename, ctx.open_mode)
                         assert ctx.stream is not None
                         ctx.filename = self.undo_temp_name(ctx.tmpfilename)
diff -ru youtube-dl/youtube_dl/utils.py youtube-dl.new/youtube_dl/utils.py
--- youtube-dl/youtube_dl/utils.py	2017-12-10 11:17:33.000000000 -0500
+++ youtube-dl.new/youtube_dl/utils.py	2017-12-12 13:23:38.133135000 -0500
@@ -452,7 +452,7 @@
                 msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
             return (sys.stdout.buffer if hasattr(sys.stdout, 'buffer') else sys.stdout, filename)
         stream = open(encodeFilename(filename), open_mode)
-        return (stream, filename)
+        return (stream, encodeFilename(filename))
     except (IOError, OSError) as err:
         if err.errno in (errno.EACCES,):
             raise

[dstftw]

Incorrect. sanitize_open returns sanitized name when necessary. encodeFilename has nothing to do with sanitization. The patch is incorrect.

[pennsy-gg1]

I confirm that 99081da fixes the problem.