Youtube-dl .EXE and Python version do not work the same

[ghost]

I was trying to download an encrypted YT Video ("Taylor Swift - Shake It Off").

I'm using WinXP and have Cygwin. I installed Livestreamer but it said to use Youtube-dl .

I obtained the Youtube-dl.exe and updated it. It does not work identically to the Python version.

I obtained the Python version of Youtube-dl, it can not update as easily (annoying) and so was not updated (I grabbed it a few hors ago).

The Python version works perfectly (well, I needed to play with the command lines because the plugin needs to be a bit smarter, but it works well).

These commands are entered from the Cygwin (WinXP) command line.

First we try the pure Python version of the Youtube-dl Program:

$ cd /cygdrive/c/Program Files/Livestreamer/youtube-dl

$ python -m youtube_dl --verbose -f 140/m4a/171/webm,137/mp4/248/webm http://www.youtube.com/watch?v=nfWlot6h_JM
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '-f', '140/m4a/171/webm,137/mp4/248/webm', 'http://www.youtube.com/watch?v=nfWlot6h_JM']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.29.2
[debug] Python version 2.7.3 - CYGWIN_NT-5.1-1.7.32-0.274-5-3-i686-32bit
[debug] Proxy map: {}
[youtube] Setting language
[youtube] Confirming age
[youtube] nfWlot6h_JM: Downloading webpage
[youtube] nfWlot6h_JM: Downloading video info webpage
[youtube] nfWlot6h_JM: Extracting video information
[youtube] nfWlot6h_JM: Encrypted signatures detected.
[youtube] {22} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {43} signature length 45.40, html5 player en_US-vfl15y_l6
... (WORKS)
[youtube] {140} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {171} signature length 45.40, html5 player en_US-vfl15y_l6
[info] nfWlot6h_JM: downloading video in 2 formats
[debug] Invoking downloader on u'https://r7---sn-ni5f-t8gl.googlevideo.com/videoplayback?initcwndbps=1908750&ipbits=0&expire=1412101 ...
[download] Destination: Taylor Swift - Shake It Off-nfWlot6h_JM.m4a
[download] 100% of 3.70MiB in 00:01
[debug] Invoking downloader on u'https://r7---sn-ni5f-t8gl.googlevideo.com/videoplayback?initcwndbps=1908750&ipbits=0&expire=1412101 ...
[download] Destination: Taylor Swift - Shake It Off-nfWlot6h_JM.mp4
[download] 100% of 79.95MiB in 00:29

$ ffmpeg.exe -i Taylor\ Swift\ -\ Shake\ It\ Off-nfWlot6h_JM.mp4 -i Taylor\ Swift\ -\ Shake\ It\ Off-nfWlot6h_JM.m4a -c copy Taylor\ Swift\ -\ Shake\ It\ Off.mp4

$ ls -l Taylor\ Swift\ -\ Shake\ It\ Off.mp4
-rwx------+ 1 HP_Administrator None 87734151 Sep 30 05:28 Taylor Swift - Shake It Off.mp4

$

Now lets try the .EXE version of the Youtube-dl Program.

$ cd ..

$ ./youtube-dl.exe -U
youtube-dl is up-to-date (2014.09.29.2)
$ ./youtube-dl.exe --verbose -f 140/m4a/171/webm,137/mp4/248/webm http://www.youtube.com/watch?v=nfWlot6h_JM
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '-f', '140/m4a/171/webm,137/mp4/248/webm', 'http://www.youtube.com/watch?v=nfWlot6h_JM']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2014.09.29.2
[debug] Python version 2.7.8 - Windows-XP-5.1.2600-SP3
[debug] Proxy map: {'http': 'http://127.0.0.1:9614', u'https': 'http://127.0.0.1:9614'}
[youtube] Setting language
[youtube] Confirming age
ERROR: Unable to confirm age: [Errno 10054] An existing connection was forcibly closed by the remote host; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type youtube-dl -U to update.
File "youtube_dl\extractor\common.pyo", line 224, in _request_webpage
File "youtube_dl\YoutubeDL.pyo", line 1266, in urlopen
File "urllib2.pyo", line 410, in open
File "urllib2.pyo", line 523, in http_response
File "urllib2.pyo", line 442, in error
File "urllib2.pyo", line 382, in _call_chain
File "urllib2.pyo", line 629, in http_error_302
File "urllib2.pyo", line 410, in open
File "youtube_dl\utils.pyo", line 808, in http_response
File "socket.pyo", line 351, in read
File "httplib.pyo", line 543, in read
File "httplib.pyo", line 612, in _read_chunked
File "httplib.pyo", line 658, in _safe_read
File "socket.pyo", line 380, in read
File "ssl.pyo", line 246, in recv
File "ssl.pyo", line 165, in read
(FAIL)

The .EXE version fails at the "[youtube] Confirming age" line but the pure Python version makes it past there.

Sometimes it fails at the "Setting Language" line, why it fails in different spots in different runs it uncertain.

[youtube] Setting language
WARNING: unable to set language: [Errno 10054] An existing connection was forcibly closed by the remote host

I've only had Youtube-dl for a few days and am not familiar with it but I think secure Sockets are broken in the .EXE .

If you keep re-running the Windows compiled Program over and over again (immediately) then one in five it might work, here is one time it did (though not perfectly, it missed the Language):

$ cd /cygdrive/c/Program Files/Livestreamer
$ ./youtube-dl.exe --verbose -f 140/m4a/171/webm,137/mp4/248/webm http://www.youtube.com/watch?v=nfWlot6h_JM
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '-f', '140/m4a/171/webm,137/mp4/248/webm', 'http://www.youtube.com/watch?v=nfWlot6h_JM']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2014.09.29.2
[debug] Python version 2.7.8 - Windows-XP-5.1.2600-SP3
[debug] Proxy map: {'http': 'http://127.0.0.1:9614', u'https': 'http://127.0.0.1:9614'}
[youtube] Setting language
WARNING: unable to set language: [Errno 10054] An existing connection was forcibly closed by the remote host
[youtube] nfWlot6h_JM: Downloading webpage
[youtube] nfWlot6h_JM: Downloading video info webpage
[youtube] nfWlot6h_JM: Extracting video information
[youtube] nfWlot6h_JM: Encrypted signatures detected.
[youtube] {22} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {43} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {18} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {5} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {36} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {17} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {137} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] nfWlot6h_JM: Downloading js player vfl15y_l6
[youtube] {248} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {136} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {247} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {135} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {244} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {134} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {243} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {133} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {242} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {160} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {140} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {171} signature length 44.40, html5 player en_US-vfl15y_l6
[youtube] {137} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {248} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {136} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {247} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {135} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {244} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {134} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {243} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {133} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {242} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {160} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {140} signature length 45.40, html5 player en_US-vfl15y_l6
[youtube] {171} signature length 45.40, html5 player en_US-vfl15y_l6
[info] nfWlot6h_JM: downloading video in 2 formats
[debug] Invoking downloader on u'https://r7---sn-ni5f-t8gl.googlevideo.com/videoplayback?itag=140&keepalive=yes&lmt=1408756070265690&key ...
[download] Destination: Taylor Swift - Shake It Off-nfWlot6h_JM.m4a
[download] 100% of 3.70MiB in 00:00
[debug] Invoking downloader on u'https://r7---sn-ni5f-t8gl.googlevideo.com/videoplayback?itag=137&keepalive=yes&lmt=1408758006531139&key ...
[download] Destination: Taylor Swift - Shake It Off-nfWlot6h_JM.mp4
[download] 100% of 79.95MiB in 00:22

There is something unreliable about the Windows XP version. It is not producing the same result each time it is run and it usually (but not always) fails.

Ideally we could just type the URL for the Page and the plugin would detect the TRUE best version of the Video and Audio, (the 'best' Videos on YT always have separate Video and Audio Streams) then and MUX them together, along with any Subtitles (which are rarely included on YT Videos).

Thanks for writing the Program and distributing it for free. It it probably the easiest "Video Downloader" there is (for difficult Videos).

YT,
Rob

[phihag]

Please do not engage in needless conspiracy theories. The .exe gets a proxy set, whereas the Python version does not. Check who's engaging in the proxy setting - most likely, it's the app that's listening on the proxy port and you'll have your culprit.

By the way, you can already use -f 'bestvideo+bestaudio/best' to let youtube-dl mux the best video quality. This is not the default yet since we do not have reliable detection whether muxing will work beforehand, and still may miss a couple of optimizations (such as not resuming downloads etc.).

Pull requests to that effect are very welcome. Also, you say you want the YouTube extractor to be a bit smarter. Can you elaborate about what the extractor needs to be smarter? If you make any changes to the code, please contribute them so that all youtube-dl users benefit.

[ghost]

My "Theory", as you refer to it, is a Bug Report; that was requested by an Error Message in the Program's output. The title of the Bug Report is the "Youtube-dl .EXE and Python version(s) do not work the same".

I go on to show an example where the same command, for the same Video, when used on the two versions of the Programs produces different results. It is no Theory, it is a fact that is demonstrated.

You say: "The .exe gets a proxy set, whereas the Python version does not.". Thus, you agree, there is a difference in the way the two Programs work. It you did not need to know about this that is OK but I do not accuse you of being in a conspiracy. This is my first Bug Report here, I was doing as requested and trying to help. It is unfortunate that this was SPAM, I won't spam you again.

Sorry for your trouble to read this, bye.

[phihag]

You misunderstand; I don't think this bug report was spam. You are very welcome to file bugs and thus contribute to youtube-dl, and we thank you for it! On second re-reading, my response was rather rough. I apologize.

The point I made is that the behavior you are seeing is extremely likely a result of these different proxy settings. These proxy settings are assigned by your operating system, so it's rather strange that one program does not get them and the other one does. While it is possible that there is a bug - this bug would be in the exe version then because it does not see the proxy - this is way more likely an effect of the operating system.

If you run the python and the exe version in the same shell, do you see that same behavior? What program is running on port 9614? Are you knowingly running a proxy?

Note that if the problem persists even after these hints (and you don't find out that by accident you started one shell before reconfiguring the proxy, have some random malware or anti-malware that assigns proxies or so, please do tell. We'll then reopen the port and try to help you to diagnose the problem.

Again, I apologize for answering in a rather rude fashion. There was no offense intended.