Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Could not get file by 'skydrive-cli get' #1

Closed
jas0nliu opened this Issue October 16, 2012 · 11 comments

2 participants

jas0nliu Mike Kazantsev
jas0nliu

skydrive-cli tree;
skydrive-cli ls;
skydrive-cli quota;
All those commands succeed;

When using skydrive-cli get Pictures/xxx.jpg > test.jpg, get HTTP 104 error:

raceback (most recent call last):
File "./skydrive-cli", line 9, in
load_entry_point('python-skydrive==12.10.14', 'console_scripts', 'skydrive-cli')()
File "/volume1/@appstore/Python/usr/lib/python2.7/site-packages/skydrive/cli_tool.py", line 212, in main
resolve_path(optz.file), byte_range=optz.byte_range ))
File "/volume1/@appstore/Python/usr/lib/python2.7/site-packages/skydrive/api_v5.py", line 232, in get
return self(join(obj_id, 'content'), dict(download='true'), raw=True, **kwz)
File "/volume1/@appstore/Python/usr/lib/python2.7/site-packages/skydrive/api_v5.py", line 200, in call
try: return self.request(api_url(), **kwz)
File "/volume1/@appstore/Python/usr/lib/python2.7/site-packages/skydrive/api_v5.py", line 65, in request
raise raise_for.get(code, ProtocolError)(code, err.message)

skydrive.api_v5.ProtocolError: (None, error(104, 'Connection reset by peer'))

Mike Kazantsev
Owner

It's doesn't seem to be an http error (which are handled a bit differently and return http status code in place of "None" in exception args tuple), but rather a tcp socket error, which shouldn't actually happen in normal http interactions (errors should be reported via http).

Unfortunately, it's nearly impossible to say why server drops your connection from that data, so the main question is whether it's reproducible error, so one can e.g. monitor traffic and code paths when it happens.

Does it fail consistently (i.e. every time you try it) or only occasionally?

Especially if it's consistent, I can suspect the issues with TLS implementations on either side.
I can recommend trying to update "requests" library if you have less-than-latest version, since it actually helped me with these particular Microsoft servers in the past (TLS 1.2 advertisement issue).
Or maybe check which version you have and write it here, so I can try to reproduce the issue with it.

Another interesting thing is that I've seen persistent connections occasionally hang on ms servers (though with a bit different code involved), so maybe look into /volume1/@appstore/Python/usr/lib/python2.7/site-packages/skydrive/api_v5.py file and try changing default value for "session" argument in request() method to "False" (on line 35 or so).
That should disable re-using of tcp/tls connections, and if problem goes away with that, it gives more info as to where it might be as well as a reasonable workaround.

jas0nliu

Thanks for quick response.

  1. It's reproducible, consistently (every time);
  2. requests-0.14.1-py2.7.egg-info, requests is 0.14.1, the latest;
  3. set the session default to 'False', same error and call stack.

Question:
If this is socket error, why it just happens for the 'get' command?

jas0nliu

I think I found the root cause.
Request lib will use ssl module with default ssl_version=ssl.PROTOCOL_SSLv23, this doesn't work in my environment. Need to add the ssl_version=sl.PROTOCOL_SSLv3 to /site-packages/requests/packages/urllib3/connectionpool.py:102:

self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file,
cert_reqs=self.cert_reqs,

ca_certs=self.ca_certs,

ssl_version=ssl.PROTOCOL_SSLv3)

Now it's working.
Cheers ~

Mike Kazantsev
Owner

Hm, that's interesting, thanks for tracking down the issue.

Note that ssl_version=ssl.PROTOCOL_SSLv3 forces connection to use SSLv3 only, while SSLv23 allows less-broken protocols like TLSv1 (and even there, TLSv1.0 is susceptible to practical BEAST attacks), so I'd try to use such thing only as a temporary workaround.

If this is socket error, why it just happens for the 'get' command?

I think the reason is that "get" will actually use a different server (depending on your ISP and physical location), presumably with different ssl/tls implementation.

Try running skydrive-cli --debug quota 2>&1 | grep 'Starting new HTTPS connection, then the same for "put" and finally "get". Note that while non-get commands are served from "api.live.com", "get" requests get redirected to something like "3bfzag.bay.livefilestore.com".
While I think both non-get and get requests are certainly to be served by some CDN (api.live.com gets resolved through Akamai DNS as well, they are still directed to a different hosts, which are probably different ones for you than they're for me, unless you connect through the same area or networks.

Still, I wouldn't blame it on server right off the bat, since it might also be your openssl version or some very restrictive (or outrightly malicious) ISP on the way to the livefilestore.com host in question.

I suggest you try following:

  • openssl s_client -state -connect public.bay.livefilestore.com:443 (should give "connection reset by peer" or something to that effect)
  • openssl s_client -ssl3 -state -connect public.bay.livefilestore.com:443 (should work, i.e. go past "New, TLSv1/SSLv3, Cipher is ..." message and hang there)
  • openssl s_client -tls1 -state -connect public.bay.livefilestore.com:443 (I wonder if it might work)
  • openssl s_client -tls1_2 -state -connect public.bay.livefilestore.com:443
  • same things, but with the last server which shows up during aforementioned "get + grep" command (in my case, it's "3bfzag.bay.livefilestore.com").
  • same things with "65.54.191.43" (IP I get for "3bfzag.bay.livefilestore.com").

In my case, TLSv1.0 is the most recent thing that works, but it'd be interesting if you'll get different results with the host you get and the last IP.
If that's the case (different results, only SSLv3 works for your server), I suggest you try filing a support ticket at Microsoft (iirc SkyDrive had the link somewhere at the bottom of the interface), maybe they care enough to fix it if one more user bugs them about it ;)

In the meantime, guess I'll add some CLI switch to enable workarounds for SSL/TLS and similar option to the api (though maybe requests/urllib3 already have the feature, need to look).

jas0nliu

I was actually using curl to test the REST GET api, which ended with those errors:

  • About to connect() to qy2iiw.bay.livefilestore.com port 443 (#1)
  • Trying 65.54.191.43...
  • connected
  • Connected to qy2iiw.bay.livefilestore.com (65.54.191.43) port 443 (#1)
  • SSLv3, TLS handshake, Client hello (1): } [data not shown]
  • Unknown SSL protocol error in connection to qy2iiw.bay.livefilestore.com:443
  • Closing connection #1 curl: (35) Unknown SSL protocol error in connection to qy2iiw.bay.livefilestore.com:443
  • Closing connection #0

* SSLv3, TLS alert, Client hello (1):

This made me notice that, the SSL protocol might be the root cause. Then I went through the Requests library and finally found it uses the default value when wrapping up the SSL socket. The Requests lib does not have the option to configure this, so I just updated the source code to make it trick.

Mike Kazantsev
Owner

I don't get the same error from curl -v https://65.54.191.43 or curl -v https://qy2iiw.bay.livefilestore.com - you seem to have "curl: (35) Unknown SSL protocol error in connection to qy2iiw.bay.livefilestore.com:443", while for me it just hangs (which is expected of these buggy servers on TLSv1.2 request).

I believe requests (at least 0.14.0+) actually can work around the issue I'm having with the server you use (which I've mentioned in README and first comment here), but you seem to have a different issue there, and it's not because of the server (it seems, unless it chooses different protocols per-client or forwards tcp connection to different backends), since same IP works fine for me.

Also, curl can use (i.e. be compiled with) different SSL/TLS implementation (gnutls) than python/requests (openssl), so it's behavior in this case might not mean anything at all.

jas0nliu

Initially, I tried the command line below to download the file, got the above SSL error. Then I tried to add '-3' to force it use SSLv3, it works.
curl -k -G -v -L "https://apis.live.net/v5.0/file.xxxxx/content" -d "download=true" –d "access_token=xxxx" -o abc.zip

Mike Kazantsev
Owner

Just looked into making SSL version configurable and it seem to be in pull-request queue for both requests and urllib3 already.

Also interesting that I've narrowly managed to avoid the bug myself by using requests-0.14.0 all this time, where the workaround simlar to yours was accidentally merged.
With 0.14.1 I can totally reproduce the problem, guess one shouldn't underestimate the difference between pre-latest and the latest patchset ;)

Anyways, since proposed "clean" fixes are just a few days old and discussion around them still rages on, guess I'll try taping over the issue by monkey-patching known-bad urllib3/requests versions.

Mike Kazantsev mk-fg referenced this issue from a commit October 23, 2012
Mike Kazantsev Workaround for issue #1 dc1f9a6
Mike Kazantsev
Owner

While current solution (dc1f9a6) isn't particulary great, I think it should work until aforementioned patches will be merged.

I'd also like to note that I've used ssl_version = ssl.PROTOCOL_TLSv1, which should be preferrable over SSLv3 you've used above, though still broken badly.

Feel free to reopen the issue if the ahem fix doesn't work for you.
Thanks again for reporting it and taking your time to look for the source of the problem.

Mike Kazantsev mk-fg closed this October 22, 2012
jas0nliu
Mike Kazantsev
Owner

Good catch, guess it'd have caused some "inheritance overflow" at some point, after a bunch of requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.