Add HttpFileWrapper as a separate module

[sigmavirus24]

This just provides a basic way to wrap a file object so you can force a
specific amount of data to be returned at a minimum.

So far, basic tests at the interactive console have shown it to work as
expected. This still needs real tests, however.

Closes #75

[Lukasa]

I prefer the if size in (-1, 0) or size > force_size: idiom here.

[Lukasa]

Proposed alternative names:

[18:38:57] lukasa_home: May as well rename it the 'ScrewYouIWontDoWhatYouTellMe' wrapper
[18:39:25] sigmavirus24:    ScrewYouIAmAnAdult
[18:39:29] sigmavirus24:    IThrewItOnTheGround
[18:39:57] lukasa_home: HttplibSucksWrapper
[18:40:04] sigmavirus24:    BreakAllExpectationsWrapper
[18:40:09] lukasa_home: YoloWrapper
[18:40:12] sigmavirus24:    WhyYouDoThisWrapper
[18:40:23] lukasa_home: ItsMoreASetOfGuidelinesWrapper
[18:40:57] sigmavirus24:    NeedForSpeedWrapper
[18:41:26] sigmavirus24:    FastererererWrapper
[18:41:54] lukasa_home: ThisIsWhatHappensWhenYouDontConfirmYourConstraintsWrapper
[18:42:46] sigmavirus24:    GiantFileUploadFastererWrapper
[18:43:05] sigmavirus24:    TakeLessThanAnHourToUploadSomethingRidiculouslyLargeWrapper
[18:43:17] lukasa_home: BoyIHopeYou'reUsingTheSocketAPIProperlyWrapper

[sigmavirus24]

Or we could just make this into a documentation chapter instead of bothering to write it for someone and distribute it.

[sigmavirus24]

More names

12:40.57  sigmavirus>NeedForSpeedWrapper
12:41.25  sigmavirus>FastererererWrapper
12:41.55  lukasa_hom ThisIsWhatHappensWhenYouDontConfirmYourConstraintsWrapper
12:42.46  sigmavirus>GiantFileUploadFastererWrapper
12:43.05  sigmavirus>TakeLessThanAnHourToUploadSomethingRidiculouslyLargeWrapper
12:43.17  lukasa_hom BoyIHopeYou'reUsingTheSocketAPIProperlyWrapper

[lifeofdave]

Hey sigmavirus, thanks for pointing this out. Had a quick test with this but it seemed to make performance a bit to a lot worse, depending on the chunk size. Would I need any changes on the django backend to take advantage of this?

[Lukasa]

@lifeofdave What chunk sizes were you using?

[lifeofdave]

@Lukasa After retesting today it turns out I was incorrect, the File Wrapper doesn't make transfer speeds worse (don't know why that seemed to be the case yesterday), but also it doesn't improve them.

I've been testing with a 5MB file, and code looks like this:

chunk_size = 16384
m = MultipartEncoder(fields={'myfile': (fname, HttpFileWrapper(open(fname, 'rb'),chunk_size))})
fup = self.session.post(self.SITE_URL + 'api/uploads/', data=m, headers={'Content-Type':m.content_type},)

Results:
Not using HttpFileWrapper:
http: 3.9 seconds
https: 10.8 seconds

Using HttpFileWrapper:
chunk_size = 8192
http: 4 seconds
https: 10.4 seconds

chunk_size = 16384
http: 3.8 seconds
https: 10.7 seconds

chunk_size = 32768
http: 3.8 seconds
https: 10.9 seconds

chunk_size = 65536
http: 4 seconds
https: 10.4

Using curl
http: ~4 seconds
https: ~4 seconds

[Lukasa]

Given that our speeds are about the same as curl's with HTTP, the problem is unlikely to be in the read() logic in httplib. Instead, the problem seems to be in the interface with OpenSSL. What platform are you on?

[lifeofdave]

Windows 7, python 2.7.10 64 bit

[Lukasa]

So the first thing to note is that your curl probably isn't using OpenSSL at all: what's the output of curl --version?

[lifeofdave]

$ curl --version
curl 7.47.1 (x86_64-unknown-cygwin) libcurl/7.47.1 OpenSSL/1.0.2f zlib/1.2.8 libidn/1.29 libpsl/0.12.0 (+libidn/1.29) libssh2/1.5.0 nghttp2/1.6.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: Debug IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets Metalink PSL

Thanks

[Lukasa]

Ohh, but it's a cygwin thing, so it is linked against OpenSSL. That's tough.
Can you also show me the output of pip freeze please?

[lifeofdave]

pip freeze outputs nothing

[Lukasa]

Uh..where are you getting your copy of requests from?

[lifeofdave]

I'm only using cygwin for curl, I've installed requests using the pycharm package manager, which I assume is the same as pip install requests. Requests version is 2.9.1, requests-toolbelt is 0.6.0. Sorry if that wasn't clear!

[Lukasa]

Hmmm. Well, in that case you're using the standard library's SSL module, which is presumably the source of the majority of your slowdown. If you'd like to use something like cprofile to get a feel for where you're spending most of your time, that's probably a good idea.

[lifeofdave]

Here's the openssl version (if it helps):

import ssl
print ssl.OPENSSL_VERSION
OpenSSL 1.0.2a 19 Mar 2015

Is it simple to use a different SSL module?
I'll check out cprofile

[Lukasa]

You can try using PyOpenSSL instead: if you install pyopenssl, ndg-httpsclient and pyasn1 requests will automatically switch to using that.

[lifeofdave]

Same results with PyOpenSSL, I'll have a look into cprofile. Thanks so much for your help..