'Did not find necessary cookies' output when trying to download courses #547

rolfmorel · 2016-07-01T19:13:22Z

Subject of the issue

coursera-dl does not seem to authenticate through cookies correctly and always exits with the message: Could not authenticate: Did not find necessary cookies.

I tried it for a number of course names: sdn1, sdn1-001, modelthinking, modelthinking-004, and the one I am currently interested in: crypto, for which I also tried learn/crypto.

Your environment

Operating System (name/version): ArchLinux, Linux 4.6.2
Python version: 3.5.1
coursera-dl version: latest checkout 0.6.1 with commit 81b9d40 (also tried release version 0.6.1)

Steps to reproduce

Tell us how to reproduce this issue, please don't forget to include
enough information so that you can help us help you:

Is the problem happening with the latest version of the script?
- yes, checkout of the current repo at commit 81b9d40
Do you have all the recommended versions of the modules? See them in the
file requirements.txt.
- Performed sudo pip install -r requirements.txt --upgrade on the repo checkout.
What is the course that you are trying to access?
- The cryptography course with this link.
What is the precise command line that you are using.
- coursera-dl -u 'username@gmail.com' -p 'XXXXXXX' crypto
- coursera-dl -u 'username@gmail.com' -p 'XXXXXXX' learn/crypto
- Note: I tried the --on-demand command switch, but this only outputs a depreciated option message extra.
What are the precise messages that you get? Please, use the --debug
option before posting the messages as a bug report. Please, copy and paste
them. Don't reword/paraphrase the messages.

$ coursera-dl -u 'username@gmail.com' -p 'XXXXXXX' crypto --debug
root[main] coursera_dl version 0.6.1
root[main] Downloading class: crypto
root[download_class] Downloading old style class crypto
root[get_cookies_from_cache] Trying to get cookies from the cache.
root[load_cookies_file] Loading cookie file /tmp/rolf_coursera_dl_cache/cookies/rolfmorel@gmail.com.txt into memory.
root[get_cookies_from_cache] Could not load cookies from the cache.
root[login] Initiating login.
root[login] There were no .coursera.org cookies to be cleared.
requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): class.coursera.org
requests.packages.urllib3.connectionpool[_make_request] "GET /crypto HTTP/1.1" 200 None
root[login] Forging cookie header: csrftoken=5jeRB9VL13vGBq5QZ; csrf2_token_oiH1R=____D0VB0Q2I3POZciw6Woio.
requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): www.coursera.org
requests.packages.urllib3.connectionpool[_make_request] "POST /api/login/v3 HTTP/1.1" 200 None
root[login] Logged in on coursera.org.
requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): class.coursera.org
requests.packages.urllib3.connectionpool[_make_request] "GET /crypto/auth/auth_redirector?type=login&subtype=normal HTTP/1.1" 200 None
root[down_the_wabbit_hole] Following https://class.coursera.org/crypto/auth/auth_redirector?type=login&subtype=normal to authenticate on class.coursera.org.
root[down_the_wabbit_hole] Exiting "deep" authentication.
root[main] Could not authenticate: Did not find necessary cookies.

Expected behaviour

Having used coursera-dl some time ago, I expected it to download the course.

Actual behaviour

If I run the command coursera-dl -u 'username@gmail.com' -p 'XXXXXXX' crypto , or with any other course name, the output that I get back is as follows:

coursera_dl version 0.6.1
Downloading class: crypto
Starting new HTTPS connection (1): class.coursera.org
Starting new HTTPS connection (1): www.coursera.org
Logged in on coursera.org.
Starting new HTTPS connection (1): class.coursera.org
Could not authenticate: Did not find necessary cookies.

The text was updated successfully, but these errors were encountered:

balta2ar · 2016-07-01T19:35:14Z

@Polyphemus This is a known issue. Please see this: #508 (comment)

rolfmorel · 2016-07-01T19:40:44Z

Thank you, forcing the call to be to download_on_demand_class(...) instead of download_old_style_class(...) did indeed work.

tourist-py · 2016-07-02T04:02:59Z

I think the same error(#154) got the different reason.

jaidevd · 2016-07-02T05:27:16Z

Thanks, @Polyphemus for raising this issue, I too started seeing this only today. Thanks, @balta2ar , your suggestion works.

oliyiyi · 2016-07-02T05:41:45Z

Thanks, I got it.

jiangtiandao · 2016-07-02T07:11:12Z

https://github.com/coursera-dl/coursera-dl/blob/master/coursera/coursera_dl.py#L1320

befreshdumbie · 2016-07-02T13:08:48Z

Thank @jiangtiandao , your code line worked for me

shkp · 2016-07-02T19:59:06Z

I am having the same issue. I was able to download courses just fine, till yesterday, today I'm seeing this error. I have seen all the older threads and successfully updated both the script and the requirements. However, I could not quite figure out how to use --clear-cache and also the solutions posted on this thread (how can I change and access sections of the script).

I am relatively, a noob, when it comes to programming. In fact, this is exactly what I'm trying to get better at through Coursera. I hope you guys can help me out. Thanks in advance. And thanks to the makers of this project for helping all of us learn new things.

balta2ar · 2016-07-02T20:34:11Z

@shk3 What courses are you downloading? Old courses were shut down yesterday, only on-demand courses are available now.

shkp · 2016-07-03T07:51:49Z

@balta2ar I believe the course i was trying to download was an on-demand course. It was Front-end Javascript Frameworks: Angular JS. The error, as i mentioned above is "Could not authenticate: Did not find necessary cookies."

C:\Users\shkp\venv1\Scripts>coursera-dl -u user@gmail.com -p XXXXXXX angular-js --debug root[main] coursera_dl version 0.6.1 root[main] Downloading class: angular-js root[download_class] Downloading old style class angular-js root[get_cookies_from_cache] Trying to get cookies from the cache. root[load_cookies_file] Loading cookie file C:\Windows\TEMP\shkp_coursera_dl_cache\cookies\user@gmail.com.txt into memory. root[get_cookies_from_cache] Could not load cookies from the cache. root[login] Initiating login. root[login] There were no .coursera.org cookies to be cleared. requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): class.coursera.org requests.packages.urllib3.connectionpool[_make_request] "GET /angular-js HTTP/1.1" 200 None root[login] Forging cookie header: csrftoken=QmCbpWk31D6qbQi6SsI5; csrf2_token_Q9Aedfcj=A4OttJdduuvDVjn3N2g3SY8I. requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): www.coursera.org requests.packages.urllib3.connectionpool[_make_request] "POST /api/login/v3 HTTP/1.1" 200 None root[login] Logged in on coursera.org. requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): class.coursera.org requests.packages.urllib3.connectionpool[_make_request] "GET /angular-js/auth/auth_redirector?type=login&subtype=normal HTTP/1.1" 200 None root[down_the_wabbit_hole] Following https://class.coursera.org/angular-js/auth/auth_redirector?type=login&subtype=norma l to authenticate on class.coursera.org. root[down_the_wabbit_hole] Exiting "deep" authentication. root[main] Could not authenticate: Did not find necessary cookies.

I do see that when i use the --debug attribute that it is trying to download the course as an 'old style course'. But I'm not sure what the solution is, to that. I also, still cannot figure out how to use the --clear-cache attribute.coursera-dl -u <user> -p <pass> angular-js --clear-cache gives me an error saying NameError: name 'download_class' is not defined. I installed this using pip in a virtualenv. Any help is appreciated. Thanks in advance.

balta2ar · 2016-07-03T08:18:13Z

@shkp Confirming. Old classes do not work anymore and unfortunately because of that downloader fails to switch to on-demand class download. You need to hack the code. See my comment above in this issue. We will fix it in the next version.

shk3 · 2016-07-03T23:47:39Z

@balta2ar I guess you meant to mention @shkp right? I am not actively using the downloader for a while.

balta2ar · 2016-07-04T05:29:33Z

Yes, sorry! Github completion confused me.

l1th1um · 2016-07-05T07:45:34Z

In the newest version, I'm switching the exception in download_class function. So it will try the new version first, then the old version

try:
    logging.debug('Downloading new style (on demand) class %s', class_name)
        return download_on_demand_class(args, class_name)           
except ClassNotFound:
    logging.debug('Downloading old style class %s', class_name)
        return download_old_style_class(args, class_name)

simratsingh96 · 2016-07-05T14:12:54Z

@balta2ar I am getting the same error inspite of changing all occurrences of 'download_class' to 'download_on_demand_class'.

balta2ar · 2016-07-05T14:17:37Z

@simratsingh96 Which course?

simratsingh96 · 2016-07-05T15:45:06Z

https://www.coursera.org/learn/probability-intro/home : Introduction to Probability and Data

balta2ar · 2016-07-05T17:46:37Z

@simratsingh96 Could you use the latest version from github? Support for old courses is already removed there, I think it should work for you. This change will be in the next release.

simratsingh96 · 2016-07-05T18:04:49Z

@balta2ar I am using the most recent version 0.6.1

balta2ar · 2016-07-05T18:32:35Z

@simratsingh96 The changes that I'm talking about are not yet tagged, they are not in pip yet. Follow this: https://github.com/coursera-dl/coursera-dl#alternative-installation-method-for-unix-systems

simratsingh96 · 2016-07-05T18:47:25Z

@balta2ar I use a mac, the instructions seem to be for a windows environment. Are there terminal specefic instructions availible?

balta2ar · 2016-07-05T19:21:31Z

@simratsingh96 they work for mac

simratsingh96 · 2016-07-06T05:15:08Z

Thank you for the suggestions @balta2ar. Worked perfectly for me

edoardovivo · 2016-07-09T14:22:49Z

Thank you @balta2ar ! Works fine now!

samrus98 · 2016-07-14T05:49:25Z

@balta2ar, I am not able to make it work.
I have Python 3.5 and I have already reinstalled and upgraded coursera-dl. After that I located the coursera-dl.py file at path C:\Users\AppData\Local\Programs\Python\Python35\Lib\site-packages\coursera and replaced download_class to download_on_demand_class but I am still getting the error as follows:

I tried downloading the following courses:

Alternatively when I tried to use the coursera-dl.py file you pointed to in post https://github.com/coursera-dl/coursera-dl/blob/master/coursera/coursera_dl.py#L1293 after making the same replacement, I am getting the error as shown below:

Please let me know what I should do differently?

-Sam

balta2ar · 2016-07-14T06:51:59Z

@samrus98 Have you tried this instruction? https://github.com/coursera-dl/coursera-dl#alternative-installation-method-for-unix-systems Pay attention that this instruction does not use pip to install coursera-dl (only for dependencies).

samrus98 · 2016-07-14T18:49:54Z

@balta2ar I am currently trying to do this in Windows command prompt...are you indicating that I should use unix environment? I am assuming that you are talking about the following piece of code:

cd /directory/where/I/want/my/courses
virtualenv my-coursera
cd my-coursera
source bin/activate
git clone https://github.com/coursera-dl/coursera-dl
cd coursera
pip install -r requirements.txt
./coursera-dl ...

balta2ar · 2016-07-14T21:17:55Z

Well, in Windows you can do pretty much the same. I think there is git for windows, so it's possible. In fact, you don't necessarily need to clone, you can simply download master.zip and unpack it. I think virtualenv should work for Windows too.

neokito · 2016-07-15T22:17:19Z

I have the same problem :/

course: https://www.coursera.org/learn/climate-change-mitigation/home/week/1

zjfjyc · 2016-07-17T06:25:34Z

@balta2ar Sorry to disturb you, I have met the same problem as others. I installed the latest version 0.6.1, and tried to fix this problem as what you said in this issue, but it still doesn‘t work on my win 10 laptop.

kinshuk4 · 2016-07-17T06:51:03Z

Thanks @balta2ar your suggestion for replacing download_class to download_on_demand_class in main() method worked 👍

ee7klt · 2016-07-19T03:46:30Z

in which directory do you find coursera-dl.py after installation?

balta2ar · 2016-07-19T04:21:21Z

That depends on a lot of things: your OS, installation method, Python version.

samrus98 · 2016-07-19T05:44:04Z

Unfortunately i could not make the suggested solution work (yet). When are
you planning to release an update?

-Sam

On Jul 14, 2016 14:18, "Yuri Bochkarev" notifications@github.com wrote:

Well, in Windows you can do pretty much the same. I think there is git for
windows, so it's possible. In fact, you don't necessarily need to clone,
you can simply download master.zip and unpack it. I think virtualenv should
work for Windows too.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#547 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AKYCS3ijC-0WKAKqCrQnqLLn-7ci3vslks5qVqeNgaJpZM4JDaSL
.

balta2ar · 2016-07-19T07:35:46Z

Only one thing is left in my plans for the next release: quiz download. I've started work on it but it appeared slightly more complicated than I anticipated. I don't want to give any concrete deadlines but I hope to finish it soon.

whyfuture · 2016-07-20T21:24:37Z

@balta2ar QUIZ-download would make this insanely cool.

ghost · 2016-07-21T21:30:33Z

@balta2ar Thank you! Alternative install working properly.
Debian 8.5.0 and Python 2.7.9.

tlatforlz · 2016-07-22T17:34:10Z

@kinshuk4 how you can fix that? you fix in main() function or down_load function?

caselmer · 2016-07-22T23:44:36Z

As line numbers have changed:
In which line am I supposed to replace "download_class" with "download_on_demand_class"?

i) def download_class(args, class_name):
ii) if download_class(args, class_name):
iii) both

thanks

caselmer · 2016-07-23T02:00:47Z

OK, I can answer my own question.
Only change line with "if download_class(args, class_name):"
:-)

abhigenie92 · 2016-07-28T02:31:32Z

I am getting the same issue. I am using the latest version on pip.

abhishek Algos1 $ coursera-dl -u abhigenie@live.com -p XXXXXX algorithm-design-analysis
coursera_dl version 0.6.1
Downloading class: algorithm-design-analysis
Starting new HTTPS connection (1): class.coursera.org
Starting new HTTPS connection (1): www.coursera.org
Logged in on coursera.org.
Starting new HTTPS connection (1): class.coursera.org
Could not authenticate: Did not find necessary cookies.

Debug output:

abhishek Algos1 $ coursera-dl -u abhigenie@live.com -p XXXXXXXXX algorithm-design-analysis-2 --debug
root[main] coursera_dl version 0.6.1
root[main] Downloading class: algorithm-design-analysis-2
root[download_class] Downloading old style class algorithm-design-analysis-2
root[get_cookies_from_cache] Trying to get cookies from the cache.
root[load_cookies_file] Loading cookie file /tmp/abhishek_coursera_dl_cache/cookies/abhigenie@live.com.txt into memory.
root[get_cookies_from_cache] Could not load cookies from the cache.
root[login] Initiating login.
root[login] There were no .coursera.org cookies to be cleared.
requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): class.coursera.org
requests.packages.urllib3.connectionpool[_make_request] "GET /algorithm-design-analysis-2 HTTP/1.1" 200 None
root[login] Forging cookie header: csrftoken=PXVZ5ebBNOiYWf2j29XX; csrf2_token_pEoMCy19=Z4YjKl9Lm3H1tQMpblYzz6eo.
requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): www.coursera.org
requests.packages.urllib3.connectionpool[_make_request] "POST /api/login/v3 HTTP/1.1" 200 None
root[login] Logged in on coursera.org.
requests.packages.urllib3.connectionpool[_new_conn] Starting new HTTPS connection (1): class.coursera.org
requests.packages.urllib3.connectionpool[_make_request] "GET /algorithm-design-analysis-2/auth/auth_redirector?type=login&subtype=normal HTTP/1.1" 200 None
root[down_the_wabbit_hole] Following https://class.coursera.org/algorithm-design-analysis-2/auth/auth_redirector?type=login&subtype=normal to authenticate on class.coursera.org.
root[down_the_wabbit_hole] Exiting "deep" authentication.
root[main] Could not authenticate: Did not find necessary cookies.

Environment:
Kubuntu 16.04 and python3.
Version: 0.6.1. The latest one on coursera-dl github repo.

caselmer · 2016-07-28T02:56:42Z

Environment:
Kubuntu 16.04 and python3.
Version: 0.6.1. The latest one on coursera-dl github repo.

I am no expert, but this seems not enough. Inside of my virtual environment I had to edit
./lib/python3.5/site-packages/coursera/coursera_dl.py

Look my previous posts

balta2ar · 2016-07-28T14:21:20Z

This should be fixed in 0.7.0.

neokito mentioned this issue Jul 18, 2016

TreeBuilderForHtml5lib 'module' object has no attribute '_base' #554

Closed

tlatforlz mentioned this issue Jul 20, 2016

Error when "pip install coursera-dl --upgrade" #558

Closed

whyfuture mentioned this issue Jul 20, 2016

Authentication issue #557

Closed

balta2ar closed this as completed Jul 28, 2016

'Did not find necessary cookies' output when trying to download courses #547

'Did not find necessary cookies' output when trying to download courses #547

Comments

rolfmorel commented Jul 1, 2016 • edited

Subject of the issue

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

balta2ar commented Jul 1, 2016

rolfmorel commented Jul 1, 2016

tourist-py commented Jul 2, 2016

jaidevd commented Jul 2, 2016

oliyiyi commented Jul 2, 2016

jiangtiandao commented Jul 2, 2016 • edited

befreshdumbie commented Jul 2, 2016

shkp commented Jul 2, 2016

balta2ar commented Jul 2, 2016

shkp commented Jul 3, 2016 • edited

balta2ar commented Jul 3, 2016

shk3 commented Jul 3, 2016

balta2ar commented Jul 4, 2016

l1th1um commented Jul 5, 2016

simratsingh96 commented Jul 5, 2016 • edited

balta2ar commented Jul 5, 2016

simratsingh96 commented Jul 5, 2016

balta2ar commented Jul 5, 2016

simratsingh96 commented Jul 5, 2016

balta2ar commented Jul 5, 2016

simratsingh96 commented Jul 5, 2016

balta2ar commented Jul 5, 2016

simratsingh96 commented Jul 6, 2016

edoardovivo commented Jul 9, 2016

samrus98 commented Jul 14, 2016

balta2ar commented Jul 14, 2016 • edited

samrus98 commented Jul 14, 2016

balta2ar commented Jul 14, 2016

neokito commented Jul 15, 2016 • edited

zjfjyc commented Jul 17, 2016

kinshuk4 commented Jul 17, 2016

ee7klt commented Jul 19, 2016

balta2ar commented Jul 19, 2016

samrus98 commented Jul 19, 2016

balta2ar commented Jul 19, 2016

whyfuture commented Jul 20, 2016

ghost commented Jul 21, 2016 • edited by ghost

tlatforlz commented Jul 22, 2016

caselmer commented Jul 22, 2016

caselmer commented Jul 23, 2016

abhigenie92 commented Jul 28, 2016 • edited

caselmer commented Jul 28, 2016

balta2ar commented Jul 28, 2016

rolfmorel commented Jul 1, 2016 •

edited

jiangtiandao commented Jul 2, 2016 •

edited

shkp commented Jul 3, 2016 •

edited

simratsingh96 commented Jul 5, 2016 •

edited

balta2ar commented Jul 14, 2016 •

edited

neokito commented Jul 15, 2016 •

edited

ghost commented Jul 21, 2016 •

edited by ghost

abhigenie92 commented Jul 28, 2016 •

edited