ST2 won't go through my Proxy #93

Closed
dargoan opened this Issue Feb 29, 2012 · 40 comments

Comments

Projects
None yet
6 participants

dargoan commented Feb 29, 2012

Hello,

I have some issue with ST2, and my proxy.
My proxy is a php script, and I don't what it's doing exactly. The fact is that I can surf on the net, but I had trouble with my Eclipse IDE to update components inside Eclipse. In Eclipse, I solved the problem by setting, in the Proxy settings:
HTTP
-> Host : Dynamic,
-> Port : Dynamic,
-> Provider : Native
-> Auth : False

In ST2, I don't know how to do that. When I try to install a package using Package Control, I have a popup that says to me :
"Package control : There are non packages available for installation".

Have you got any idea that can help me?
Many thanks by advance.

Best regards,
Johan

Owner

wbond commented Feb 29, 2012

I don't have a proxy like yours to test against, so there is little I can do other than tell you how it works and give you some debugging tips.

As of right now, the default channel file (http://sublime.wbond.net/repositories.json) is served over HTTP. On all platforms this is downloaded via urllib2 in Package Control.py. If you open the file and search for the UrlLib2Downloader class, you can see if the http_proxy setting is set, it creates a new urllib2 ProxyHandler object with your proxy settings. http://docs.python.org/library/urllib2.html

I would try writing some python using the examples at the bottom of the urllib2 page to see if you can get any requests to go through your proxy. If you find a way to get it to work, then we can figure out why it isn't working with Package Control.

dargoan commented Mar 1, 2012

Thank you Will.

I'll try and see what I can do, and I'll study the PackageControl.py file to see more about http_proxy setting.
I'm looking forward to see that urllib2 page you talked about.

Best regards,
Johan

Contributor

schlamar commented Mar 6, 2012

The default repository for me was https://sublime.wbond.net/repositories.json. Switching to http worked for me without setting proxy parameters.

Contributor

schlamar commented Mar 6, 2012

Well, but the rest of the downloads are broken. Have a look at this: http://stackoverflow.com/questions/1030113/does-urllib2-in-python-2-6-1-support-proxy-via-https

This bad workaround (fallback to HTTP) works for now in UrlLib2Downloader.download (line 594).

#handlers.append(VerifiedHTTPSHandler(ca_certs=bundle_path))
url = 'http' + url[5:]
Owner

wbond commented Mar 6, 2012

@ms4py I highly recommend against that solution since:

  1. It opens you back up to man-in-the-middle automatic upgrade code injection
  2. The next version of Package Control will overwrite your version, breaking it again

According to the post you linked (which I have read before) the bug was fixed in Python 2.6.3. Sublime ships with 2.6.6 or newer depending on your platform.

Contributor

schlamar commented Mar 6, 2012

Yes, I know. Any idea to further debug this?

Owner

wbond commented Mar 6, 2012

@ms4py I don't have any experience debugging proxy issues - I've never even been on a network with a proxy.

The only way I know forward would be for someone with the same kind of proxy as you (such as you, or someone you know who knows python) to write some Python that can successfully download from https://sublime.wbond.net/repostiories.json. Once we figure out how urllib2 needs to be contorted, then I can fix Package Control.

dargoan commented Mar 6, 2012

Thanks for trying to solve this issue.

I've been testing what you said Will... but for now, it's not successfull.
I've got a timeout error message, for now.

Contributor

schlamar commented Mar 8, 2012

I did some research and google is working with my urllib code, but I cannot fetch the repositories.json

import urllib2
proxy_handler = urllib2.ProxyHandler({'https': 'http://proxy:1234/'})
opener = urllib2.build_opener(urllib2.HTTPSHandler(), proxy_handler)
urllib2.install_opener(opener)
print urllib2.urlopen('https://www.google.com').read()
print urllib2.urlopen('https://sublime.wbond.net/repostiories.json').read()

Fetching the second URL returns a httplib.BadStatusLine. Is this related to the certificate stuff you are doing? Can you give me more details on this to dig further?

Owner

wbond commented Mar 12, 2012

@ms4py I am wondering if there might be some issues with the old version of Apache and OpenSSL on my server. I am moving to a new server in the next week or two, which should hopefully fix this, and some other SSL issues people are reporting.

dsaputo commented Mar 23, 2012

I ran ms4py's code against my setup. Google worked, but the second URL returned urllib2.URLError: <urlopen error [Errno 11004] getaddrinfo failed> for me.

Owner

wbond commented Mar 24, 2012

@dsaputo I believe that means your network is not resolving the IP address for sublime.wbond.net.

dsaputo commented Mar 26, 2012

@wbond Yes, I could not ping it at work.I hard-coded 207.58.165.193 in my hosts file and then received urllib2.URLError: <urlopen error Tunnel connection failed: 407 Proxy Authentication Required> when fetching the second URL.

Contributor

schlamar commented Mar 27, 2012

After fixing some bugs (#115) I tried the CurlDownloader, it seems definitely an error in your webserver:

curl: (35) Unknown SSL protocol error in connection to sublime.wbond.net:443 downloading https://sublime.wbond.net/repositories.json.

When will you upgrade?

Owner

wbond commented Mar 29, 2012

sublime.wbond.net has moved to a new server, can everyone check and see if they get any different results now?

dsaputo commented Mar 29, 2012

@wbond What is the ip address? I still can't ping it at work, as it doesn't resolve.

dsaputo commented Mar 29, 2012

@wbond I hard-coded to 50.116.34.243 and got the same 407 error as above.

Owner

wbond commented Mar 29, 2012

@dsaputo Yeah, 407 is a proxy authentication error, which probably means there is some python that needs to be written to fix it

dsaputo commented Mar 29, 2012

@wbond The code I'm using to test with is ms4py's above with my configuration, and Google has always worked fine. I've been fighting this proxy for four years, and what works for something doesn't work for something else...

Contributor

schlamar commented Mar 30, 2012

Well, the redirect could be the case of the error:

$ curl --user-agent "Sublime Package Control" --cacert "/cygdrive/c/Users/schla
ich/Sublime Text 2/Data/Packages/Package Control/certs/ca-bundle.crt" https://s
ublime.wbond.net/repostiories.json
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://wbond.net/sublime_packages/">here</a>.
</p>
</body></html>
Contributor

schlamar commented Mar 30, 2012

Here is the verbose output: https://gist.github.com/2249233
The SSL stuff seems correct. Could it be the problem that a proxy is rewriting the user agent or something like this?

Contributor

schlamar commented Mar 30, 2012

Another example:

>>> import httplib
>>> conn = httplib.HTTPConnection('...', 4128)
>>> conn.set_debuglevel(1)
>>>
>>> conn.request('GET', 'https://sublime.wbond.net/repostiories.json',
...              headers={'User-Agent': 'Sublime Package Control'})
send: 'GET https://sublime.wbond.net/repostiories.json HTTP/1.1\r\nHost: sublime
.wbond.net\r\nAccept-Encoding: identity\r\nUser-Agent: Sublime Package Control\r
\n\r\n'
>>> res = conn.getresponse()
reply: 'HTTP/1.1 301 Moved Permanently\r\n'
header: Date: Fri, 30 Mar 2012 08:23:20 GMT
header: Server: Apache
header: Location: http://wbond.net/sublime_packages/
header: Content-Length: 242
Contributor

schlamar commented Mar 30, 2012

Oh, you have disabled any proxy handlers by the direct socket connection in the CertValidatingHTTPSConnection.connect method. Fix is in the referenced pull request.

schlamar referenced this issue Mar 30, 2012

Merged

fixes #93 #116

Owner

wbond commented Mar 30, 2012

@ms4py The redirect you are seeing is due to a typo in the URL.

Contributor

schlamar commented Mar 30, 2012

Oh, in this case the curl part is working and the first test, too ;-)

But this has nothing to do with the fix. It adresses the right cause of the problem.

tousdan commented Apr 5, 2012

If this can help you in anyway, I experienced similar problems with a HTTP Proxy installed locally (Fiddler, in this case). The progress indicator would loop for a while and then abandon loading the repository.

If I disable my HTTP proxy, it seems to work othewise, it doesnt.

Contributor

schlamar commented Apr 11, 2012

Why don't you accept or comment the pull request (#116), it fixes this issue.

Owner

wbond commented Apr 12, 2012

@ms4py I haven't had time to work on Package Control recently. This and some other issues need to be merged and then I need to get a small groups of users to test the new version. After that I can push it out as an official release.

pomeh commented Apr 13, 2012

@wbond I have the same problem about proxy. Switching to http url fixes it, but as you said, this is not a good solution.
I'm available if you need help to test the new version, just let me know :)

Contributor

schlamar commented Apr 14, 2012

@pomeh the correct solution should already be in #116 if you want to test it

pomeh commented Apr 16, 2012

I tried the solution proposed in #116, I have the following results behind a proxy:

  • without the modification: it only works with HTTP, either with proxy settings on or off,
  • with the modification: it's the same behavior as above about HTTP config, BUT it's also work with Proxy on over HTTPS; HTTPS without proxy settings fails (this sounds correct to me)

So to me the patch is working, but I didn't test it without proxy at all.

wbond closed this in d9c19a9 Jun 11, 2012

dargoan commented Jul 24, 2012

Hello,

I installed ST2 v2.0.1, build 2217. My question is : how do i get your last modifications of #93?
Because, for now, it doesn't work.
Maybe I have to do something, but I don't know what?

Thanks

dargoan commented Jul 24, 2012

@wbond :
I changed the 2 lines of code according to #93.
I restarted ST2.
When I do : "Package Control" > Install Packages...
I get this :
"Exception in thread Thread-1:
Traceback (most recent call last):
File ".\threading.py", line 532, in __bootstrap_inner
File ".\Package Control.py", line 2484, in run
File ".\Package Control.py", line 1699, in record_usage
File ".\Package Control.py", line 1016, in download_url
File ".\Package Control.py", line 630, in download
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 106: ordinal not in range(128)

Exception in thread Thread-2:
Traceback (most recent call last):
File ".\threading.py", line 532, in __bootstrap_inner
File ".\Package Control.py", line 1990, in run
File ".\Package Control.py", line 1825, in make_package_list
File ".\Package Control.py", line 1130, in list_available_packages
File ".\Package Control.py", line 1075, in list_repositories
File ".\Package Control.py", line 190, in get_repositories
File ".\Package Control.py", line 163, in fetch_channel
File ".\Package Control.py", line 1016, in download_url
File ".\Package Control.py", line 630, in download
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 97: ordinal not in range(128)"

I also tried your testing package (http://sublime.wbond.net/testing/Package%20Control.sublime-package) and installed it in "Installed Packages" directory. But it did not change the fact that I don't have anything downloading.

Is there something that I don't get (hopefully, so I can try make things work).

Thanks by advance.
Regards.

Contributor

schlamar commented Jul 25, 2012

Have you set your proxy information in settings? What's the error in the console with the testing package?

dargoan commented Jul 25, 2012

@ms4py : Thanks for your help.
In fact, I did position the automatic script I have in Chrome settings (script for automatic configuration) in proxy information in settings.
When I try to do "Install Package", ST2 tries to load repository "https://sublime.wbond.net/repositories.json" from "repository_channels". It waits for 10 seconds about, and then shows me :

"Exception in thread Thread-6:
Traceback (most recent call last):
File ".\threading.py", line 532, in __bootstrap_inner
File ".\Package Control.py", line 1990, in run
File ".\Package Control.py", line 1825, in make_package_list
File ".\Package Control.py", line 1130, in list_available_packages
File ".\Package Control.py", line 1075, in list_repositories
File ".\Package Control.py", line 190, in get_repositories
File ".\Package Control.py", line 163, in fetch_channel
File ".\Package Control.py", line 1016, in download_url
File ".\Package Control.py", line 630, in download
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 97: ordinal not in range(128)"

dargoan commented Jul 25, 2012

@wbond & @ms4py :
At line 630 of Package Control.py : I was having the previous errors (UnicodeDecodeError) for "str(e.reason)".
Instead, I used :

"print '%s : %s URL error | downloading %s ' % (name, error_message, url)"
wich gave me :
"Package Control | Error submitting usage information. | https://sublime.wbond.net/submit?package_control_version=1.5.0.12&sublime_platform=windows&package=Package+Control&sublime_version=2217&version=1.5.0.12&operation=install
Package Control: Skipping automatic upgrade, last run at 2012-07-25 10:54:13, next run at 2012-07-25 11:54:13 or after
Package Control | Error downloading channel. | https://sublime.wbond.net/repositories.json
error: Package Control: There are no packages available for installation."

Owner

wbond commented Jul 25, 2012

@dargoan Please see the directions at wbond#130 (comment) to ensure you are running a known version of the code. There are quite a number of different bug fixes in 1.5.0.11+.

dargoan commented Aug 1, 2012

@wbond :
I totally reinstalled ST2 and downloaded Package Control. It is not working through my proxy.

So, I've done what you said :
I still have the issue "Traceback (most recent call last):
File ".\threading.py", line 532, in __bootstrap_inner
File ".\Package Control.py", line 1990, in run
File ".\Package Control.py", line 1825, in make_package_list
File ".\Package Control.py", line 1130, in list_available_packages
File ".\Package Control.py", line 1075, in list_repositories
File ".\Package Control.py", line 190, in get_repositories
File ".\Package Control.py", line 163, in fetch_channel
File ".\Package Control.py", line 1016, in download_url
File ".\Package Control.py", line 630, in download
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 97: ordinal not in range(128)"
when I try to install a new package.

In Package Controle.py, I've changed the lines 629-630 that way : "print '%s: %s URL error | downloading %s.' % (name, error_message, url)"
So that str(e.reason) do no throw a UnicodeDecodeError.

When I try to install a package, I have this :
"Package Control: Error submitting usage information. URL error | downloading https://sublime.wbond.net/submit?package_control_version=1.5.0.12&sublime_platform=windows&package=Package+Control&sublime_version=2217&version=1.5.0.12&operation=install.
Package Control: Error downloading channel. URL error | downloading https://sublime.wbond.net/repositories.json.
error: Package Control: There are no packages available for installation.
Package Control: Error downloading channel. URL error | downloading https://sublime.wbond.net/repositories.json.
Package Control: Error downloading channel. URL error | downloading https://sublime.wbond.net/repositories.json.
Package Control: No updated packages"

About my proxy problem, I am in a domain.
So, as a proxy_user, I used : domain_name\proxy_user, as I've read somewhere.
But it's not working for me.

Also, I've tested some codes with proxy handlers, as you suggested in the beginning of the track, but for now, nothing worked...

dargoan commented Nov 9, 2012

Hello,

With version 1.5.0.18, it's finally working, though I still needed some code to fix UnicodeDecodeError (unicode(e.reason) didn't seem to work for me),
So, I managed to get a list of plugins to install, I was glad !! In fact, my settings were wrong : repository => with "http" and "submit_url" with "http".

With theses settings, ST2 does not seem to be able to download.
I have:
"Package Control: Urllib2 Debug Proxy
http_proxy: myProxy:port
https_proxy:
proxy_username: username
proxy_password: password
Package Control: Urllib2 HTTPS Debug General
Connecting to nodeload.github.com on port 443
Package Control: Error downloading package. URL error downloading https://nodeload.github.com/joshnh/CSS-Snippets/zip/master.
ignored packages updated to: []"

So, anybody here could tell me if there is a setting that I don't use correctly ?

Once again, thanks a lot for your help, and amazing work...!

Owner

wbond commented Nov 10, 2012

Please update to version 1.5.0.19 (http://wbond.net/sublime_packages/package_control/installation#Testing_Version), which should fix the UnicodeDecodeError, and which also reverts a change in proxy handling so that your proxy will be used for HTTPS URLs also unless you set https_proxy to false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment