prefer http over https

[niltsh]

Hi

I make an app that take advantage of youtube-dl.

I found recently
when using ./youtube-dl -g http;//youtube.com/xxxxxxx

the URL returns https://....
However, my app could not process https in the near future.

Is there any way to return the http rather than https?

BRs,
Zongyao Qu

[jaimeMF]

I think you can replace https:// with http:// in the urls (at least the reverse is true) without any problem. I'm not sure about adding a new option, you can easily get the http url if you need it and I don't know if there's a good reason for downloading over http if you can use https.

By the way, which url are you using? in my case youtube-dl BaW_jenozKc -g gives an url that starts with http://

[niltsh]

let me try your idea.

I was using youtube.com, and I found it is random, even the same video, sometimes it is http, another day it is https

[jamesBrosnahan]

What app are you making that needs youtube-dl to output URLs in http? It does not seem to be a responsible feature to lower security just because a person wants to not implement security in their own code.

[qrtt1]

I also found it show https randomly. To replace https to http the server will response 403 to me.

[luluinZ]

"I also found it show https randomly. To replace https to http the server will response 403 to me."
This is the same for me.
It makes me sometimes fail to download the video using mplayer.
I can't see a reason not to add a new option when the video is actually downloaded over normal http. Please!

[luluinZ]

Or can you suggest another way to easily get the http url? thanks!

[jaimeMF]

The reason why we default to https is #739.
You can use mplayer by using a downloader that supports https, like: curl $(youtube-dl -g http://www.youtube.com/watch?v=BaW_jenozKc) | mplayer -cache 8192 -.
I think that adding an option for using http would require downloading all the pages with https, I think that will complicate the code.

[kokoko3k]

@jaimeMF:
The solution of using a downloader that supports https urls is fine until you don't need to seek.
About code complexity, i was very lucky in discovering (as i stated in #2562) that just issuing:

#cp /path/to/youtube.py /path/to/youtube.py.bak
#cat /path/to/youtube.py.bak | sed 's/https/http/' > /path/to/youtube.py

...Is enough to make youtube-dl to return prefectly playable and seekable non-secure http streams, that's why i was asking to provide a (non default) switch like -no-ssl.

What do you think about it? I could try to make a patch if you think is worth to consider the idea.

Thanks for your efforts and attention.

[jaimeMF]

@kokoko3k If the patch isn't too complex (and is maintainable) I think we won't oppose to it.

For all the people who want to directly play the urls with a player, I'm currently using mpv (a fork of mplayer2) and it supports https urls if the ffmpeg library is build with ssl support.

[kokoko3k]

Yes, mpv is a good alternative; i was speaking about a gui i develop
which is focused around mplayer.
In the next days, i'll take a deeper look into youtube-dl code and
will try to propose a patch.

Thanks

2014-03-15 14:04 GMT+01:00, Jaime Marquínez Ferrándiz
notifications@github.com:

@kokoko3k If the patch isn't too complex (and is maintainable) I think we
won't oppose to it.

For all the people who want to directly play the urls with a player, I'm
currently using mpv (a fork of
mplayer2) and it supports https urls if the ffmpeg library is build with ssl
support.

---

Reply to this email directly or view it on GitHub:
#2364 (comment)

[gitongit]

I try this in a script using curl, it does not work:

---

youtube-dl -F $1
read vidformat

playerurl=$(youtube-dl -f $vidformat -g "$1")
mplayer -fs $(curl $playerurl)

---

Only output is curl downloading the file:

---

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.0M 100 11.0M 0 0 577k 0 0:00:19 0:00:19 --:--:-- 607k

---

[kokoko3k]

@jaimeMF
Hi, i ended up in making the following patches; sorry if they are not in the form you'd expect them.
in init.py i just added a new option, --no-https.
Then in the youtube.py extractor i tested that option and set the string variable http to "https" or "http"
I use that variable to get the video_info_url.

Those patches seems quite simple and mantainable to me:

Gozer youtube_dl # diff -u init.py /tmp/new.init.py

--- __init__.py 2014-02-26 00:30:26.000000000 +0100
+++ /tmp/new.__init__.py        2014-03-17 13:20:14.874465206 +0100
@@ -253,7 +253,8 @@
         '--ignore-config',
         action='store_true',
         help='Do not read configuration files. When given in the global configuration file /etc/youtube-dl.conf: do not read the user configuration in ~/.config/youtube-dl.conf (%APPDATA%/youtube-dl/config.txt on Windows)')
-
+    general.add_option('--no-https',
+            action='store_true', dest='nohttps', help='Use insecure http youtube video urls')

     selection.add_option(
         '--playlist-start',
@@ -759,6 +760,7 @@
         'include_ads': opts.include_ads,
         'default_search': opts.default_search,
         'youtube_include_dash_manifest': opts.youtube_include_dash_manifest,
+        'nohttps': opts.nohttps,
     }

     with YoutubeDL(ydl_opts) as ydl:

Gozer youtube_dl # diff -u extractor/youtube.py /tmp/new.youtube.py

--- extractor/youtube.py        2014-02-25 10:41:20.000000000 +0100
+++ /tmp/new.youtube.py 2014-03-17 13:28:37.321129343 +0100
@@ -1148,6 +1148,11 @@
             player_url = None

         # Get video info
+        if self._downloader.params.get('nohttps'):
+            http="http"
+        else:
+            http="https"
+            
         self.report_video_info_webpage_download(video_id)
         if re.search(r'player-age-gate-content">', video_webpage) is not None:
             self.report_age_confirmation()
@@ -1162,7 +1167,7 @@
                                                   'asv': 3,
                                                   'sts':'1588',
                                                   })
-            video_info_url = 'https://www.youtube.com/get_video_info?' + data
+            video_info_url = http+'://www.youtube.com/get_video_info?' + data
             video_info_webpage = self._download_webpage(video_info_url, video_id,
                                     note=False,
                                     errnote='unable to download video info webpage')
@@ -1170,7 +1175,7 @@
         else:
             age_gate = False
             for el_type in ['&el=embedded', '&el=detailpage', '&el=vevo', '']:
-                video_info_url = ('https://www.youtube.com/get_video_info?&video_id=%s%s&ps=default&eurl=&gl=US&hl=en'
+                video_info_url = (http+'://www.youtube.com/get_video_info?&video_id=%s%s&ps=default&eurl=&gl=US&hl=en'
                         % (video_id, el_type))
                 video_info_webpage = self._download_webpage(video_info_url, video_id,
                                         note=False,

For your convenience, i uploaded the modified files here:
http://wpage.unina.it/aorefice/sharevari/ytdl/

Thanks.

[Crypto90]

@kokoko3k thanks for this patch, works fine!

[gitongit]

@kokoko3k You are clearly a genius --

I tried to re-install from scratch:

git clone https://github.com/rg3/youtube-dl.git

Returned error:

Initialized empty Git repository in /dirname/dir/youtube-dl/git_project/youtube-dl/.git/
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?

So I downloaded the zip file

https://github.com/rg3/youtube-dl/archive/master.zip

Unzipped it; then entered the "youtube-dl" directory and replaced the old "init.py" with your new version.

Entered the "youtube-dl/extractor" directory and replaced "youtube.py" with your new version.

Ran "make".

make

Received errors about program "pandoc" not being available. Commented related lines out in the makefile rather than waste time in dependency hell. Created empty "youtube-dl.1" file so that make could see and use it. Re-ran "make" again; it worked.

Ran "make install".

make install

It (seemed to) work.

Ran youtube-dl to watch heart-warming "sloth hugs cat" video and received friendly error message instead:

youtube-dl http://www.youtube.com/watch?v=VACbH_S5ZFo

Hi! We changed distribution method and now youtube-dl needs to update itself one more time.
This will only happen once. Simply press enter to go on. Sorry for the trouble!
From now on, get the binaries from http://rg3.github.io/youtube-dl/download.html, not from the git repository.

Pressed "Enter" and after pause, received message:

Done! Now you can run youtube-dl.

Re-ran youtube-dl to watch sloth-on-cat love:

youtube-dl --no-https http://www.youtube.com/watch?v=VACbH_S5ZFo

Returned error message:

Usage: youtube-dl [options] url [url...]

youtube-dl: error: no such option: --no-https

If I was a superintelligent five-year-old who wanted to watch her favorite sloth terrorizing her favorite cat on Youtube using your patched version of youtube-dl, how would I make this work?

P.S. Running "youtube-dl -g <url_here>" also still returns an "https://" URL.

[kokoko3k]

That patch is not official, so you won't get it from official "repos"; this mean that as you update youtube-dl you'll lose it.
What you need to do is to replace the files after you updated youtube-dl, but this trick won't work forever.

[gitongit]

@kokoko3k -- Okay, good point...

So I re-downloaded the sources from here:
https://github.com/rg3/youtube-dl/archive/master.zip

Then copied your files ("new.init.py" and "new.youtube.py") to the "youtube_dl/" and "youtube_dl/extractor" directories, respectively (and removed the "new." from their names, of course).

Ran youtube-dl and it updated.

Then I ran make and received this output:

python devscripts/bash-completion.py
Traceback (most recent call last):
File "devscripts/bash-completion.py", line 7, in
import youtube_dl
File "/dirname/dir/youtube-dl/source/youtube-dl-master/youtube_dl/init.py", line 86, in
from .extractor import gen_extractors
File "/dirname/dir/youtube-dl/source/youtube-dl-master/youtube_dl/extractor/init.py", line 293, in
from .youtube import (
ImportError: cannot import name YoutubeSearchURLIE
make: *** [youtube-dl.bash-completion] Error 1

The poor sloth and cat are waiting in Youtubeland, hoping that one day I'll be able to see their video...

[gitongit]

Since others are still asking about this, if anyone could give a step-by-step instruction set for making this patch work, it would be great... I'd really like to be able to use Youtube, and as far as I know (as of version 2014.03.20) the patch hasn't been integrated into the main version of youtube-dl yet.

Thanks for your help.

[phihag]

Since youtube-dl 2014.03.21 (update with youtube-dl -U), you can now pass in the --prefer-insecure option to let YouTube download the video information unencrypted. Note that this will only work for YouTube, and could break at any time. Instead, we may want to add a --play option that plays the video by handing the data to mplayer. Of course, this option would work great with HTTPS downloads.

[kokoko3k]

Thanks for your efforts,
about the discussed --play option, would it be just like a pipe?

2014-03-21 0:43 GMT+01:00, Philipp Hagemeister notifications@github.com:

Since youtube-dl 2014.03.21 (update with youtube-dl -U), you can now pass
in the --prefer-insecure option to let YouTube download the video
information unencrypted. Note that this will only work for YouTube, and
could break at any time. Instead, we may want to add a --play option that
plays the video by handing the data to mplayer. Of course, this option would
work great with HTTPS downloads.

---

Reply to this email directly or view it on GitHub:
#2364 (comment)

[phihag]

@kokoko3k I'm not sure, somebody mentioned that you can't seek in piped videos in mplayer. Note that you can already pipe the output with youtube-dl -o - | mplayer -.

[kokoko3k]

Yep, it was me to mention it :)

2014-03-21 11:30 GMT+01:00, Philipp Hagemeister notifications@github.com:

@kokoko3k I'm not sure, somebody mentioned that you can't seek in piped
videos in mplayer. Note that you can already pipe the output with
youtube-dl -o - | mplayer -.

---

Reply to this email directly or view it on GitHub:
#2364 (comment)