Re-use the open filehandle rather than open a new one. #112
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch adds support for custom S3 servers in the connection string. It also adds explicit support for setting the server port, and whether or not to use SSL, both as paramaters to the smart_open function as well as within the connection string. These changes are neccessary to be able to connect to s3proxy and other custom s3 servers which don't run on the default port, or neccessarily use SSL.
Adds support for opening vanilla HTTP and HTTPS addresses. Supports efficient streaming, gzip and bz2 compression, as well as Kerberos and username/password (basic) http authentication.
We still want to maintain Python 2.6 compatibility, so don't rely on contextlib.closing.
Now they don't require internet access, and will test for Basic authentication in the HTTP header.
http => https, and remove old versions of the tests.
Conflicts: CHANGELOG.rst smart_open/smart_open_lib.py
This allows one to use any filehandle-like object instead of just local posix. It also avoids unneccessary filesystem syscalls.
|
New unit tests for compressed files via http are needed. And the re-use of file handle test too. |
This breaks out the http tests into their own test class. Also fixed a few behaviors in the HttpReader uncovered by the new tests (yay).
Works now on Python 2 as well as Python 3.
|
OK -- the lastest version add unit tests for .gz and .bz2 over HTTP. It also fixes a couple of issues discovered by those tests. |
|
Thanks for the fix. Will merge as is. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a permanent fix to the issue in #109 and #110.