Subprocess module unicode support

[mbnoimi]

It doesn't work under Windows in case I pointed to Unicode names (whether folder name or video name). The log doesn't show any useful info I just get this:

[04/13/15 11:53:36] WARNING: There's no description to write.

You can test it using this Arabic video under DailyMotion (this bug occurs for YouTube too)
In case I pointed to Arabic folder name it failes to download. If I pointed to Latin folder name it download the file as following:

 Royal Secrets _      -1_12-   .mp4

While it should be as following:

 سلسلة عمليات إحتيال دولية -1/12- وصية السيدة العجوز.mp4

[mbnoimi]

It seems this bug related to youtube-dl bug under Windows as mentioned in this issue

[MrS0m30n3]

Hi, @mbnoimi

First of all, if you try to download the same video using youtube-dl.exe from the Windows command line you still get the same result? If yes can you please add the -v (verbose) option and paste the output here?

[mbnoimi]

@MrS0m30n3 the log please:

C:\Python 3.5\Scripts>youtube-dl.exe -v --newline -o G:\tmp\%(title)s.%(ext)s --
proxy http://127.0.0.1:8118/ -i --write-description http://www.dailymotion.com/v
ideo/x2h0g6c_%D9%88%D8%AB%D8%A7%D8%A6%D9%82%D9%8A-royal-secrets-%D8%B3%D9%84%D8%
B3%D9%84%D8%A9-%D8%B9%D9%85%D9%84%D9%8A%D8%A7%D8%AA-%D8%A5%D8%AD%D8%AA%D9%8A%D8%
A7%D9%84-%D8%AF%D9%88%D9%84%D9%8A%D8%A9-1-12-%D9%88%D8%B5%D9%8A%D8%A9-%D8%A7%D9%
84%D8%B3%D9%8A%D8%AF%D8%A9-%D8%A7%D9%84%D8%B9%D8%AC%D9%88%D8%B2_shortfilms
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '--newline', '-o', 'G:\\tmp\\%(title)s.%(ext)s
', '--proxy', 'http://127.0.0.1:8118/', '-i', '--write-description', 'http://www
.dailymotion.com/video/x2h0g6c_%D9%88%D8%AB%D8%A7%D8%A6%D9%82%D9%8A-royal-secret
s-%D8%B3%D9%84%D8%B3%D9%84%D8%A9-%D8%B9%D9%85%D9%84%D9%8A%D8%A7%D8%AA-%D8%A5%D8%
AD%D8%AA%D9%8A%D8%A7%D9%84-%D8%AF%D9%88%D9%84%D9%8A%D8%A9-1-12-%D9%88%D8%B5%D9%8
A%D8%A9-%D8%A7%D9%84%D8%B3%D9%8A%D8%AF%D8%A9-%D8%A7%D9%84%D8%B9%D8%AC%D9%88%D8%B
2_shortfilms']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2015.04.09
[debug] Python version 3.5.0a3 - Windows-post2012Server-6.3.9600
[debug] exe versions: none
[debug] Proxy map: {'https': 'http://127.0.0.1:8118/', 'http': 'http://127.0.0.1
:8118/'}
[dailymotion] x2h0g6c: Downloading webpage
[dailymotion] x2h0g6c: Extracting information
[dailymotion] x2h0g6c: Downloading embed page
WARNING: There's no description to write.
[debug] Invoking downloader on 'http://www.dailymotion.com/cdn/H264-848x480/vide
o/x2h0g6c.mp4?auth=1429130807-2562-bwfc690w-362d90e339a2f7154915a1d6a566f314'
[download] Resuming download at byte 7284256
[download] Destination: G:\tmp\?????? Royal Secrets _ ?????  ?????? ?????? ?????
 -1_12- ???? ?????? ??????.mp4
[download]   2.5% of 276.51MiB at 63.99KiB/s ETA 01:11:53
[download]   2.5% of 276.51MiB at 191.96KiB/s ETA 23:57
[download]   2.5% of 276.51MiB at 447.90KiB/s ETA 10:16
[download]   2.5% of 276.51MiB at 959.79KiB/s ETA 04:47
[download]   2.5% of 276.51MiB at 165.31KiB/s ETA 27:49
[download]   2.5% of 276.51MiB at 149.32KiB/s ETA 30:48
[download]   2.6% of 276.51MiB at 162.55KiB/s ETA 28:17
[download]   2.6% of 276.51MiB at 263.22KiB/s ETA 17:27
[download]   2.7% of 276.51MiB at 247.76KiB/s ETA 18:32
[download]   2.8% of 276.51MiB at 320.03KiB/s ETA 14:20
[download]   2.9% of 276.51MiB at 406.51KiB/s ETA 11:16
[download]   3.2% of 276.51MiB at 440.33KiB/s ETA 10:22

ERROR: Interrupted by user

C:\Python 3.5\Scripts>

[mbnoimi]

BTW, I used same exactly used parameters by youtube-dlg

[MrS0m30n3]

Summary (work in progress)

Problem occurs in Python 2.* because the C implementation of the subprocess module can't handle unicode (http://bugs.python.org/issue1759845).

1. youtube-dlg uses subprocess to spawn a youtube-dl process
2. youtube-dl uses subprocess to spawn a ffmpeg process (Non-existent paths passed to ffmpeg/ffprobe/etc due to encodeFilename on Windows python2 ytdl-org/youtube-dl#4787)

---

(1.) youtube-dlg -> youtube-dl

• Youtube-dlg does not yet support Python 3.*
• On OS's that use UTF-8 (e.g. Linux) we can encode the unicode parameters at the youtube-dlg side and let the youtube-dl decode them at the other end without losing any characters
• The above solution does not work on Windows since they use encodings like cp1252, cp437, etc, that can't handle unicode characters (Windows suck!!!)
• On Windows we need a way to encode the unicode parameters at the youtube-dlg side using UTF-8 (we have that) and decode them at the youtube-dl side again using UTF-8 (we dont have that, see Way to specify additional encoding for options decoding ytdl-org/youtube-dl#6119)

(2.) youtube-dl -> ffmpeg

• Can be solved if we use a Python 3.* youtube-dl binary to do our job (Windows Unicode (utf16) not handled in filenames ? ytdl-org/youtube-dl#5094)
• Also check Alternative solution for issue #4787 ytdl-org/youtube-dl#5527

[mbnoimi]

After years of waiting it seems this bug still occurs

[elico]

@MrS0m30n3 How can someone resolve this bug?

[mbnoimi]

@MrS0m30n3 How can someone resolve this bug?

This project no longer maintained so I currently use https://github.com/axcore/tartube it's full of advanced options in addition to the classic ui