NameError: global name 'className' is not defined

[bit-man]

When trying to download any course the error in the subject is shown :

./coursera-dl -u user@email --path /home/user/Downloads/coursera eqp-001
Coursera password for user@email: 
Downloading class: eqp-001
Starting new HTTP connection (1): proxy.xxxxxxxx.com
Traceback (most recent call last):
  File "./coursera-dl", line 1154, in <module>
    main()
  File "./coursera-dl", line 1134, in main
    if download_class(args, class_name):
  File "./coursera-dl", line 1090, in download_class
    login(session, class_name, args.username, args.password)
  File "./coursera-dl", line 189, in login
    raise ClassNotFound(className)
NameError: global name 'className' is not defined

[bit-man]

eqp-001 course HTML : http://pastebin.com/PguhVJ38

[jonasdt]

What do you get if you replace className with class_name on line 189?

[bit-man]

Could not find class: eqp-001

[bit-man]

I guess it's a proxy issue. Will try when not behind a proxy and let you
know.

[jonasdt]

yup, I just saw the
Starting new HTTP connection (1): proxy.xxxxxxxx.com

[rbrito]

@bit-man, so, the script doesn't bomb anymore with the change? Unfortunately, I myself can't get eqp-001 to be redownloaded also. I kept getting 5xx errors from coursera's side.

OTOH, if I login personally, I can get stuff. I will only be able to investigate such stuff this weekend.

And I use no proxies (not that my ISP can't be using a transparent one that I don't know about)...

[jonasdt]

@bit-man Maybe it works with proxies if you change allow_redirects=False to allow_redirects=True on line 184. Can you try that please?
Or, you can try http://docs.python-requests.org/en/latest/user/advanced/#proxies

[bit-man]

Unfortunately it doesn't work either by changing to allow_redirects=True

[jonasdt]

@bitman and with http://docs.python-requests.org/en/latest/user/advanced/#proxies

[jonasdt]

@bitman can you paste the output that you get with the proxies disabled?

[bitman]

I think you may have the wrong man, are you peer-reviewing?

On 19 June 2013 15:14, Jonas De Taeye notifications@github.com wrote:

@bitman https://github.com/bitman can you paste the output that you get
with the proxies disabled?

—
Reply to this email directly or view it on GitHubhttps://github.com/jplehmann/coursera/issues/124#issuecomment-19686506
.

[rbrito]

Oh, just for the record, I couldn't access eqp-001 with the older
version, without requests. So, this may be indicative of something in
lower layers of the stack. But, again, I am sleep deprived right now
and I am certainly talking gibberish. :)

Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://cynic.cc/blog/ : github.com/rbrito : profiles.google.com/rbrito
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br

[jonasdt]

the only thing that we do on this line is getting the csrf_token by hitting the class url, so it must be a problem with your internet configuration

[jonasdt]

@bitman sorry ;) bitman != bit-man

[bitman]

I'm confused enough :)

No problem.

Scott

Regards,

Scott Pritchett

See my support page http://support.webguys.co

On 19 June 2013 15:20, Jonas De Taeye notifications@github.com wrote:

@bitman https://github.com/bitman sorry ;) bitman != bit-man

—
Reply to this email directly or view it on GitHubhttps://github.com/jplehmann/coursera/issues/124#issuecomment-19686906
.

[jonasdt]

the specified class doesn't even matter, so maybe we should return a better error message

[bit-man]

Thanks @jonasdt but the proxy env. variables were already set.

[rbrito]

@jonasdt, you meant that "the specified class doesn't even matter" for me? The context isn't clear.

[rbrito]

@bit-man, can you unset the proxies? Can you use a "regular" browser without the proxy? If you can, then that would help with the bug hunting...

[jonasdt]

@rbrito check https://github.com/jplehmann/coursera/blob/master/coursera/coursera_dl.py#L182
it is this request that fails for @bit-man, but in fact the class_name doesn't matter in this request, coursera always redirects to www.coursera.org and sets a csrf_token
try to download the class 'test', you'll see that the script fails when you try to authenticate on class.coursera.org, but not on line 189. So that is why we should have a better error message

[bit-man]

If the proxy is unset can't access the site and "no route to host is shown" :

./coursera-dl -u user@email --path /home/Downloads/coursera eqp-001

Coursera password for user@email:
Downloading class: eqp-001
Starting new HTTPS connection (1): class.coursera.org
Traceback (most recent call last):
  File "./coursera-dl", line 1154, in <module>
    main()
  File "./coursera-dl", line 1134, in main
    if download_class(args, class_name):
  File "./coursera-dl", line 1090, in download_class
    login(session, class_name, args.username, args.password)
  File "./coursera-dl", line 184, in login
    r = requests.get(class_url, allow_redirects=False)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 206, in send
    raise ConnectionError(sockerr)
requests.exceptions.ConnectionError: [Errno 113] No route to host

[jonasdt]

I'm sorry, I think it is a problem with your internet configuration. From the requests documentation

In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a ConnectionError exception.

There is not much that we can do about that ...
Have you used the script before?

[bit-man]

I used it through a proxy or direct connection, no problem about it.
Problems started yesterday, but may be the problem is some code change
before that

BTW I'm having problems to download from ANY Coursera course but can access
it from any browser without problems.

[jonasdt]

Can you try git bisect?

[bit-man]

Cool stuff !! I'll try later.

[webhat]

I just got the following mail 14 minutes ago:

tl;dr: the archive was disabled for eqp-001, it has been reenabled

The latest information from Exploring Quantum Physics on Coursera.
There was a glitch that caused the class archives to be closed. We don't know why - it was not our intention - anyway they are open again now.

Our thanks to those who brought this to our attention - and to all who participated in this class!

Charles and Victor

[rbrito]

@webhat, thanks for that hint. At least the eqp-001 would be fixed, but the original poster still seems to have problems.

@bit-man, @jonasdt, can we copy the relevant parts of the bug so that we can close this one (the undefined variable as already fixed) and continue with another one, to keep things tidy?

[rbrito]

Oh, just for the record, I tested re-downloading things from eqp-001 and it works fine now.

Now, @bit-man, to get you started with the git bisecting, here is a hint:

git bisect start
git bisect bad
git bisect good e68d405c

And try to download stuff. At each step, issue git bisect good or git bisect bad according to if it works or not.

Hope this helps.

[bit-man]

I've setup tinyproxy in my box and was able to sucessfully downloaded a web page using wget.
On coursera-dl things wen't weird :

$ echo $http_proxy
http://localhost:8888

$ ./coursera-dl -u user@mail --path /home/Downloads/coursera eqp-001
Coursera password for user@mail: 
Downloading class: eqp-001
Starting new HTTP connection (1): localhost
Starting new HTTP connection (1): localhost
Logged in on www.coursera.org.
Starting new HTTP connection (1): localhost
... same message about 30 times ...
Starting new HTTP connection (1): localhost
Traceback (most recent call last):
  File "./coursera-dl", line 1086, in <module>
    main()
  File "./coursera-dl", line 1072, in main
    if download_class(args, class_name):
  File "./coursera-dl", line 1030, in download_class
    get_authentication_cookies(session, class_name)
  File "./coursera-dl", line 250, in get_authentication_cookies
    down_the_wabbit_hole(session, class_name)
  File "./coursera-dl", line 227, in down_the_wabbit_hole
    r = session.get(auth_redirector_url)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 310, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 289, in request
    history = [r for r in gen] if allow_redirects else []
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 88, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

[bit-man]

@rbrito After using git bisect a few times I've found that commit 8bba26e is the one that introduced the issue

[jonasdt]

Do you mean that this commit introduced the issue (the issue with the proxy), or that it breaks the program (but with other errors). The master branch is a stable branch, but that does not mean that every commit is stable. The commit above breaks the program as it removes a function which is still used (at that time) in the rest of the program.

Can you verify that, with your new setup, coursera-dl works with commit f764480, and fails with commit 9236558. If that is the case, than there might be a problem with requests. Maybe your proxy makes a lot of redirects, so try something like adding requests.settings.max_redirects = 50 (in coursera_dl.py, under the import statements.) Though, I think that 30 redirects are not normal ...

[bit-man]

I tested with a different proxy and the commit 8bba26e introduces the proxy issue.
This time the 30 redirects issue didn't happen, blame it to tinyproxy, but the commit that broke proxy functionality is the same for both proxies.

[jonasdt]

ok, I think you lost me here ...
When I do git checkout 8bba26e, and then try to run the program, it crashes and gives the following error
NameError: global name 'extract_session_and_csrftoken_from_cookiejar' is not defined
This is, as explained above, the expected behaviour. I cannot see how this commit could possible introduce the proxy issue.

Can you please do the following:

• make sure the proxy works
• git checkout f764480 and try to run the program
• git checkout 9236558, and again try to tun the program

If you got an error, please paste it in this thread.

[bit-man]

You're right that's the error but if you checkout to the previous commit the proxy stuff works fine (download using a proxy can be performed)

[jonasdt]

ok good, but what if you do
git checkout 9236558, and try to run the program?

[bit-man]

Sorry forgot to do the second checkout :-P
Nope it doesn't work either.

On Sat, Jun 22, 2013 at 10:27 AM, Jonas De Taeye
notifications@github.comwrote:

ok good, but what if you do
git checkout 9236558, and try to run the program?

—
Reply to this email directly or view it on GitHubhttps://github.com/jplehmann/coursera/issues/124#issuecomment-19856768
.

Víctor A. Rodríguez (http://www.bit-man.com.ar)
El bit Fantasma (Bit-Man)
Programming: love it or leave it

[jonasdt]

and what is the error :)

[bit-man]

https://github.com/jplehmann/coursera/issues/124#issuecomment-19820596 for
tinyproxy and the next for proxy at work :

# ./coursera-dl -u user@mail --path /home/Downloads/coursera eqp-001
Coursera password for user@mail:
Downloading class: eqp-001
Starting new HTTP connection (1): proxy.xxxxxxxxx.com
Could not find class: eqp-001

[jonasdt]

What version of requests do you use?

python -c "import requests; print requests.__version__"

[bit-man]

$ python -c "import requests; print requests.version"
1.1.0

[jonasdt]

Please install the latest version:
from the top level directory of the repo

pip install -r requirements.txt

Now checkout the master brach and try again. Paste the output in your comment.

[bitman]

Now, I love following y'all, but I think I was added in error, could someone remove me from the thread :) I (bitman) was mistaken for bit-man.
—
Sent from Mailbox for iPhone

On Tue, Jun 25, 2013 at 2:00 PM, Jonas De Taeye notifications@github.com
wrote:

Please install the latest version:
from the top level directory of the repo
pip install -r requirements.txt

Now checkout the master brach and try again. Paste the output in your comment.

Reply to this email directly or view it on GitHub:
https://github.com/jplehmann/coursera/issues/124#issuecomment-19974046

[rbrito]

Scott, if you go to your github notifications, you can click on a
speaker for "muting" this conversation for you.

I don't know otherwise how one can un-add another person from a github
converstation.

Regards,

Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://cynic.cc/blog/ : github.com/rbrito : profiles.google.com/rbrito
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br

[bitman]

Many thanks,

​Done.

Scott 

—
Sent from Mailbox for iPad

On Tue, Jun 25, 2013 at 3:19 PM, Rogério Brito notifications@github.com
wrote:

Scott, if you go to your github notifications, you can click on a
speaker for "muting" this conversation for you.
I don't know otherwise how one can un-add another person from a github
converstation.

Regards,

Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://cynic.cc/blog/ : github.com/rbrito : profiles.google.com/rbrito

DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br

Reply to this email directly or view it on GitHub:
https://github.com/jplehmann/coursera/issues/124#issuecomment-19979081

[rbrito]

@bit-man, did installing a newer version of requests help here?

[bit-man]

Sadly it didn't help :

$ sudo pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade):

argparse==1.2.1 in /usr/lib/python2.7 (from -r requirements.txt (line
1))
Downloading/unpacking beautifulsoup4==4.1.3 (from -r
requirements.txt (line 2))
Downloading beautifulsoup4-4.1.3.tar.gz (131kB): 131kB downloaded
Running setup.py egg_info for package beautifulsoup4

Downloading/unpacking nose==1.3.0 (from -r requirements.txt (line 3))
  Downloading nose-1.3.0.tar.gz (404kB): 404kB downloaded
  Running setup.py egg_info for package nose

    no previously-included directories found matching 'doc/.build'
Downloading/unpacking requests==1.2.3 (from -r requirements.txt (line 4))
  Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded
  Running setup.py egg_info for package requests

Installing collected packages: beautifulsoup4, nose, requests
  Found existing installation: beautifulsoup4 4.1.2
    Uninstalling beautifulsoup4:
      Successfully uninstalled beautifulsoup4
  Running setup.py install for beautifulsoup4

  Running setup.py install for nose

   no previously-included directories found matching 'doc/.build'
    Installing nosetests script to /usr/local/bin
    Installing nosetests-2.7 script to /usr/local/bin
  Found existing installation: requests 1.1.0
    Uninstalling requests:
      Successfully uninstalled requests
  Running setup.py install for requests

Successfully installed beautifulsoup4 nose requests

$ ./coursera-dl -u user@email --path /home/Downloads/coursera eqp-001
Downloading class: eqp-001
Starting new HTTP connection (1): proxy.xxxxxx.com
Could not find class: eqp-001

[jonasdt]

I'm afraid I'm out of ideas for the moment ... I tried installing tinyproxy, but it started to take too much time :) That said, I did find this thread in the requests issue list, perhaps it has some useful information.

[rbrito]

@bit-man, just for the record, do you need the proxy (say, due to network policies) to access the web?

Have you installed a newer version of requests? Any word on this? Can we close this?

[rbrito]

Ping.