Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading videos from Lynda.com with Organization Login [SOLVED] #5926

Closed
jgozal opened this issue Jun 8, 2015 · 24 comments
Closed

Downloading videos from Lynda.com with Organization Login [SOLVED] #5926

jgozal opened this issue Jun 8, 2015 · 24 comments

Comments

@jgozal
Copy link

@jgozal jgozal commented Jun 8, 2015

Hello!

I am writing to present my solution to this problem which has appeared previously in issue #4888 and issue #5636 . It is quite easy and just requires a tiny bit of manual labor. I have searched the internet for other tools that could download videos from Lynda when having an organization account and youtube-dl seems to be the only one that's able to do the job.

A partial solution was presented in issue #4888 which talked about importing cookies in order to bypass authentication - This works. But you must be careful with the exact format of the cookie file you are importing from your browser. I tried a few extensions in Chrome and Firefox before I was able to succeed and the only one that worked well was a Firefox add-on named "Cookie Exporter 1.5.1-signed" by fliwatüt. Python does not only expect you to import a cookie.txt in a Netscape format but it also expects the cookie.txt to have a specific structure, so you will have to add a first line at the very top of the exported file being "# Netscape HTTP Cookie File" (without the quotes). If you don't add this line at the very top of the cookie.txt file, Python won't read your file, even if it actually is in a Netscape format. Note that before exporting the cookie.txt file from your browser, you must log into your Lynda organization account.

Good. Once that's done you just have to run "youtube-dl --cookies cookies.txt link-to-the-video" and it will download any video you want, but only ONE video - that is, youtube-dl won't collect all the Lynda videos in the course. Downloading say 150 videos from a course manually one by one can be a real pain, so I wrote a small script to automate this process:

EDIT: removed in order not to confuse, youtube-dl works fine with courses.

This script will do 3 things. First it will scrape all the links from a specific Lynda tutorial that you want to have, in this case the placeholder is "http://www.lynda.com/MySQL-tutorials/PHP-MySQL-Essential-Training/119003-2.html". Second, it will parse out only those links which have a key URL that all the videos share in common. In this case, the placeholder is 'http://www.lynda.com/MySQL-tutorials/'. This is extremely important; you need to put the key URL that all the videos share in common because there are many more links in the page and you don't want youtube-dl returning a bunch of invalid URL errors. Finally, after it parses all the links from the actual tutorial videos, it runs "youtube-dl --cookies cookies.txt" for every link that it found. There it is! Youtube-dl will collect all the videos and download them automatically. Remember you'll need to install BeautifulSoup library before you run the script!

A final couple of tips:

  • Remember to export the cookie.txt file every time you download another tutorial because if you don't it will tell you that your Lynda session has expired.
  • When downloading all the videos, everything will be disorganized and messy in your directory, but if you simply sort your files by date created, every video will be sorted in the correct order.

Thank you for this awesome library!

@jgozal jgozal changed the title Downloading videos from Lynda.com with Organization Login Downloading videos from Lynda.com with Organization Login [SOLVED] Jun 8, 2015
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jun 8, 2015

youtube-dl should download courses just fine. Could you run it with --verbose --write-pages on a course URL and post the output along with *.dump files?

@jgozal
Copy link
Author

@jgozal jgozal commented Jun 8, 2015

Sure thing. I'll do it tomorrow as soon as I get back to my laptop. When you say "youtube-dl should download courses just fine" you mean that it should be collecting all the videos and not just one video when importing cookies right?

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jun 8, 2015

Yes, since it's supported.

@pranavkaranjkar
Copy link

@pranavkaranjkar pranavkaranjkar commented Jun 24, 2015

@DevKiddo I am trying to use the script that you provided and I get following error. Terminal output with -v is included.

sudo python exposure.py
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--cookies', u'cookies.txt', u'http://www.lynda.com/Digital-Photography-tutorials/foundations-of-photography-exposure/71923-2.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.06.15
[debug] Python version 2.7.6 - Linux-3.16.0-41-generic-x86_64-with-elementary_OS-0.3-freya
[debug] exe versions: ffmpeg 2.7, ffprobe 2.7
[debug] Proxy map: {}
[lynda:course] 71923: Downloading course JSON
ERROR: Unable to download webpage: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 312, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1730, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Can you help figuring out whats going wrong?

Thanks

@pranavkaranjkar
Copy link

@pranavkaranjkar pranavkaranjkar commented Jun 24, 2015

@dstftw I did try youtube-dl directly with a coursepage but it only downloads a few videos (throughout all the chapters) from a course.

UPDATE: I did try this on a windows machine. And everything works fine after using a --cookies option for an institutional login. So, the log in my post above is specific to OS X.

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Jul 11, 2015

I'm getting the following error on doing everything that you said (before your python script) (I'm trying this for an individual lynda link)

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-u', '', '--cookies', 'Desktop/cookies.txt', '--verbose', 'http://www.lynda.com/Ubuntu-tutorials/Introducing-Ubuntu/159637/179555-4.html']
[debug] Encodings: locale 'UTF-8', fs 'UTF-8', out 'UTF-8', pref: 'UTF-8'
Type account password and press return:
[debug] youtube-dl version 2014.02.17
[debug] Python version 2.7.6 - Linux-3.13.0-57-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] Proxy map: {'ftp': 'ftp://172.16.15.32:2193', 'http': 'http://172.16.15.32:2193', 'https': 'https://172.16.15.32:2193'}
[lynda] Logging in as aver39@sapient.com
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out>
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 191, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1157, in urlopen
return self._opener.open(req)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/dist-packages/youtube_dl/utils.py", line 572, in https_open
return self.do_open(HTTPSConnectionV3, req)
File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
raise URLError(err)

@jgozal
Copy link
Author

@jgozal jgozal commented Jul 11, 2015

Hi @pranavkaranjkar and @AV12111994,

I've been super busy lately and I'm not sure if I'll have the time to debug your error logs. If this helps, very recently I realized that the lynda iPhone app lets you download all videos of a course to your smartphone, regardless of your account type. It may be just a lot easier to do that and then jailbreak into your phone with software like PhoneBrowse in order to import the downloaded videos into your computer.

Hope this helps!

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Jul 11, 2015

@DevKiddo I had done that on android meanwhile, is solving the purpose, but I want to work it out on my computer too. I can wait.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jul 11, 2015

@AV12111994 does your proxy alive at all? You are using an outdated version of youtube-dl. Update and post the *.dump files when running youtube-dl with --write-pages.

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Jul 11, 2015

@dstftw http, https proxies were set and I wasn't using any proxy network, so I removed them from .bashrc file. After that I updated the youtube-dl version. By doing this I was able to download the videos from youtube, but on doing

youtube-dl --write-pages --cookies Desktop/cookies.txt --verbose http://www.lynda.com/WordPress-tutorials/Downloading-correct-version-BitNami/114908/122081-4.html

I'm getting this error log:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--write-pages', u'--cookies', u'Desktop/cookies.txt', u'--verbose', u'http://www.lynda.com/WordPress-tutorials/Downloading-correct-version-BitNami/114908/122081-4.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.07.07
[debug] Python version 2.7.6 - Linux-3.13.0-57-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {}
[lynda] 122081: Downloading video JSON
ERROR: Unable to download webpage: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 312, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1729, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

@manrodriguezf
Copy link

@manrodriguezf manrodriguezf commented Jul 24, 2015

Actually I made an httpclient to do that automatically (in Java), just provide a valid cookie and output format. PM me for info guys.

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Aug 1, 2015

@manrodriguezf Could you tell what's wrong with my error that I put up above?

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Aug 1, 2015

@DevKiddo I've got mad trying to work it out, if you could help...

@jgozal
Copy link
Author

@jgozal jgozal commented Aug 2, 2015

@AV12111994 what's your OS? For some reason people using Linux have had issues with this solution. I only tested this on my windows

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Aug 2, 2015

@DevKiddo Ubuntu 14.04. I also put --verbose --print-traffic --dump-pages output and cookie structure here :
#6429 (comment)

@IgorGee
Copy link

@IgorGee IgorGee commented Sep 16, 2015

Thank you so much for this solution. This seems to be a common problem with people who use institutional log ins.

@travman34
Copy link

@travman34 travman34 commented Oct 30, 2015

Thanks for the info, saved tons of time versus downloading 1 video at a time. On my windows machine running "youtube-dl --cookies cookies.txt link-to-the-video" would download the entire list after importing cookies from firefox, but wasn't working on my debian machine. So thanks again!

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Oct 30, 2015

After the help from this issue, I wrote the following tutorials for someone who'd want to work it out on linux(and hence hopefully mac) or windows.

Hope this helps new people trying it out :

EDIT: removed, youtube-dl works fine with courses out of the box without any kind of wrappers.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Oct 30, 2015

@gdad-s-river you don't need this wrapper, youtube-dl works fine with courses out of the box.
Closing since cookies usage is now documented in FAQ.

@dstftw dstftw closed this Oct 30, 2015
@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Oct 30, 2015

@dstftw , I couldn't understand what wrapper is, that you are talking about... could you explain?

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Oct 30, 2015

I'm about python code in your tutorial. youtube-dl works fine with course URLs directly.

@gdad-s-river
Copy link

@gdad-s-river gdad-s-river commented Oct 30, 2015

Oh, didn't know that! Good to know.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Jun 27, 2016

@henryjackylucky allavsoft is completely unrelated to youtube-dl. Please refrain from proposing other softwares here, unless it provides some clues to improve youtube-dl.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jun 27, 2016

@yan12125 it's just a spammer.

@ytdl-org ytdl-org locked and limited conversation to collaborators Jun 27, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
11 participants
You can’t perform that action at this time.