Build the Windows exe with Python 3.6

[yan12125]

• I've verified and I assure that I'm running youtube-dl 2016.07.05

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

Description of your issue, suggested solution and other information

Some Python bugs (#8132, #7951) is fixed upstream and included in new Python versions. However, the 3.4 branch of Python now receives only security fixes. As a result, Windows users have to install the latest Python 2.7 or 3.5 and optionally build their own youtube-dl.exe.

py2exe is dead and gets no new fixes since Oct 2014. Here I suggest to move to another bundling tool.

1. PyInstaller
   I got an error when installing PyInstaller - one of its dependencies is broken (erocarrera/pefile#105) Need to install the latest git-master of pefile. After installation everything looks fine (see test results later)
2. cx_Freeze
   There are still commits last month but no releases since Dec 2014. Sounds less promising so I didn't test it.

---

Testing results for PyInstaller

G:\Projects\youtube-dl>C:\Users\yen\AppData\Local\Programs\Python\Python35-32\Scripts\pyinstaller.exeScripts\pyinstaller.exe youtube_dl\__main__.py --onefile
G:\Projects\youtube-dl>C:

C:\Users\yen>G:\Projects\youtube-dl\dist\__main__.exe -v test:xuite
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'test:xuite']
[debug] Encodings: locale cp950, fs mbcs, out cp950, pref cp950
[debug] youtube-dl version 2016.07.05
[debug] Python version 3.5.2 - Windows-7-6.1.7601-SP1
[debug] exe versions: none
[debug] Proxy map: {}
[TestURL] Test URL: http://vlog.xuite.net/play/RGkzc1ZULTM4NjA5MTQuZmx2
[Xuite] RGkzc1ZULTM4NjA5MTQuZmx2: Downloading webpage
[Xuite] flv config: Downloading XML
[debug] Invoking downloader on 'http://5.mms.vlog.xuite.net/video/d90029/RGkzc1ZULTM4NjA5MTQuZmx2?k=83d2bffb6e41c13bef881851cf3433e6&q=mp3'
[download] Destination: 孤單南半球-歐德陽-3860914.mp3
[download] 100% of 3.77MiB in 00:00

C:\Users\yen>G:\Projects\youtube-dl\dist\__main__.exe -v test:youtube
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'test:youtube']
[debug] Encodings: locale cp950, fs mbcs, out cp950, pref cp950
[debug] youtube-dl version 2016.07.05
[debug] Python version 3.5.2 - Windows-7-6.1.7601-SP1
[debug] exe versions: none
[debug] Proxy map: {}
[TestURL] Test URL: http://www.youtube.com/watch?v=BaW_jenozKc&t=1s&end=9
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[youtube] BaW_jenozKc: Downloading MPD manifest
[debug] Invoking downloader on 'https://r5---sn-5njj-u2xe.googlevideo.com/videoplayback?mt=1467713206&sparams=cnr%2Cdur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&mv=m&ms=au&ip=140.112.230.216&requiressl=yes&mm=31&mn=sn-5njj-u2xe&id=o-AEkgVjFdJPzfmRjB0uKvDsKWLa2XSwYZ4jGFftQ_ja9N&pl=16&itag=22&initcwndbps=6036250&ratebypass=yes&expire=1467735204&fexp=9406174%2C9416126%2C9416891%2C9422596%2C9428398%2C9431012%2C9433096%2C9433223%2C9433946%2C9435526%2C9435876%2C9436965%2C9437066%2C9437553%2C9438327%2C9439652&ipbits=0&dur=9.891&sver=3&mime=video%2Fmp4&key=yt6&signature=2317B697B0D035F741776D0B4EE0E4CB6E773CBD.274C21B2BC22BC647D657CD9DD9FC5312266BF07&upn=3cWCeTQL_SY&source=youtube&cnr=14&lmt=1387961818043059'
[download] Destination: youtube-dl test video ''_ä↭𝕐-BaW_jenozKc.mp4
[download] 100% of 1.53MiB in 00:00

C:\Users\yen>

[dstftw]

Moving to python 3.5 means dropping support for Windows XP. I am aware it's been more than 2 years since the extended support for it ended but it's still used widely enough.

[yan12125]

I see the rationale. I guess XP users are much more than those who are affected by #8132 and #7951, and they can always build a customized youtube-dl.exe. Just leave this one as a future request, like #5697.

[Hrxn]

If a user on Windows XP has any issues, no one else is to blame but himself.
It's officially dead, I don't know why it should be taken into consideration at all.

[yan12125]

There are always good reasons for not upgrading, for example software or hardware compatibility. It's not a big problem to keep XP support, so just keep it.

[Hrxn]

No reason is good enough to use an OS that has officially reached end-of-life. You won't get security updates. You will run into problems, sooner or later. But being stupid is not illegal, of course.

Caveat: If you keep it isolated from the Internet, it will continue to work as usual. But who would do that, honestly? Apart from extreme corner cases, no end user I can think of.

[yan12125]

Using an obsoleted OS is definitely something bad. I guess most of them know the disadvantages, but have no choice but to stick with old systems. Some quick examples might be:

1. Their machines can not meet the hardware requirement of new systems, and they can't afford a new machine.
2. They need some old softwares working on XP only, and their machines can't support virtual machines.

[dstftw]

Alternatively two builds may be provided.

[yan12125]

Alternatively two builds may be provided.

Also a good idea. There are two options:

1. Use the 3.5 version as the default and provide a 3.4 version for XP
2. Stick with 3.4 and provide a 3.5 version for those who are affected by #8132 and #7951

Personally I prefer the first option. 3.5 is still maintained while 3.4 gets security fixes only. We can also know how many youtube-dl users are still using XP from Github's download statistics.

[wiiaboo]

@yan12125 have you considered pyinstaller as well?

[yan12125]

@wiiaboo Please search "pyinstaller" in this issue. And it would be even better if you read thorough this issue carefully.

[wiiaboo]

😱 For some reason I just read cx_freeze and glossed over PyInstaller.

[Invictaz]

I have it working with Pyinstaller 3.2.1. CXFreeze has some weird error with Python 3.5.0 so I didn't have that ready.

[Hrxn]

Why not Python 3.6.x, if we're at it? Since this is the official Python 3 version.

[yan12125]

@Invictaz: what's the error?

By the way, 3.6 supports Unicode console I/O. I guess the ctypes trick for write_string can be dropped since 3.6.

[Quaraxkad]

Original post was in 2016, it's now early 2018. Is there still a valid reason for not updating the windows binaries to Python 3.6+?

[yan12125]

Is there still a valid reason for not updating the windows binaries to Python 3.6+?

The original point is that youtube-dl.exe needs supporting Windows XP out-of-box. Some stats: there are still 3.36% users on the world [1], and 12.57% in China [2]. However, as Python 3.4 is going to reach its end-of-life this month [3], I'm considering switching again. It's OK to keep support for an old Python version on legacy systems, while it does not make sense to use it as the default.

[1] http://gs.statcounter.com/os-version-market-share/windows/desktop/worldwide
[2] http://gs.statcounter.com/windows-version-market-share/desktop/china
[3] https://www.python.org/dev/peps/pep-0429/

[wdlkmpx]

youtube-dl 2018.10.05

I'm using youtube-dl-gui through wine and i'm seeing this error:

[10/28/18 03:12:04] Traceback (most recent call last):
[10/28/18 03:12:04] File "main.py", line 19, in
[10/28/18 03:12:04] File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp5xdqinp2\build\youtube_dl_init_.py", line 472, in main
[10/28/18 03:12:04] File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp5xdqinp2\build\youtube_dl_init_.py", line 439, in _real_main
[10/28/18 03:12:04] File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp5xdqinp2\build\youtube_dl\YoutubeDL.py", line 414, in init
[10/28/18 03:12:04] File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp5xdqinp2\build\youtube_dl\YoutubeDL.py", line 2319, in _setup_opener
[10/28/18 03:12:04] File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp5xdqinp2\build\youtube_dl\utils.py", line 699, in make_HTTPS_handler
[10/28/18 03:12:04] File "C:\Python\Python34\lib\ssl.py", line 439, in create_default_context
[10/28/18 03:12:04] File "C:\Python\Python34\lib\ssl.py", line 390, in load_default_certs
[10/28/18 03:12:04] File "C:\Python\Python34\lib\ssl.py", line 382, in _load_windows_store_certs
[10/28/18 03:12:04] ValueError: Empty certificate data

I think the windows exe should be built with python 3.6...

[lvqcl]

However, as Python 3.4 is going to reach its end-of-life this month [3]

Currently PEP 429 includes this text: "There are no specific plans for the next release of Python 3.4. However, the expectation is that Python 3.4.10 will be released in March of 2019, and this will be the final release of Python 3.4."

I think the windows exe should be built with python 3.6...

Why 3.6.x when 3.7.1 is already available? (and why do you think that it will fix your issue?)

[wdlkmpx]

I don't understand.. if youtube-dl supports python 2.7.15, then why include python 3.4, 3.6, or 3.7 in the windows exe? python 2.7 is universally supported..

[wdlkmpx]

But if python 2.7 has the same bug as python 3.4... then it's time to use python 3.7...

[tylerszabo]

I tried using pyinstaller with python 3.7.2 on Windows 10 and it's working flawlessly.

As easy as installing pyinstaller with:

python.exe -m pip install pyinstaller

Then building the exe with:

pyinstaller.exe youtube_dl\__main__.py --onefile --name youtube-dl

So far I haven't had any unusual behaviors with the exe but I also don't know what paces to put it through. I'd encourage people to try it and see what breaks.

[albertosottile]

If I may, I have been working on resurrecting py2exe and make it compatible with the latest versions of python. If you want, you could give a try at the prebuilt wheels that I just released here: https://github.com/albertosottile/py2exe/releases/tag/v0.9.3.0

Please, let me know if you find any errors when packaging youtube-dl. This is a big project and I am sure it will provide a good benchmark for these new wheels.