Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
67 changed files
with
7,246 additions
and
1,442 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #### Problem description | ||
|
|
||
| Be sure your description clearly answers the following questions: | ||
|
|
||
| - What are you trying to achieve? | ||
| - What is the expected result? | ||
| - What are you seeing instead? | ||
|
|
||
| #### Steps/code to reproduce the problem | ||
|
|
||
| In order for us to be able to solve your problem, we have to be able to reproduce it on our end. | ||
| Without reproducing the problem, it is unlikely that we'll be able to help you. | ||
|
|
||
| Include full tracebacks, logs and datasets if necessary. | ||
| Please keep the examples minimal ([minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example)). | ||
|
|
||
| #### Versions | ||
|
|
||
| Please provide the output of: | ||
|
|
||
| ```python | ||
| import platform, sys, smart_open | ||
| print(platform.platform()) | ||
| print("Python", sys.version) | ||
| print("smart_open", smart_open.__version__) | ||
| ``` | ||
|
|
||
| #### Checklist | ||
|
|
||
| Before you create the issue, please make sure you have: | ||
|
|
||
| - [ ] Described the problem clearly | ||
| - [ ] Provided a minimal reproducible example, including any required data | ||
| - [ ] Provided the version numbers of the relevant software |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| #### Title | ||
|
|
||
| Please **pick a concise, informative and complete title** for your PR. | ||
| The title is important because it will appear in [our change log](https://github.com/RaRe-Technologies/smart_open/blob/master/CHANGELOG.md). | ||
|
|
||
| #### Motivation | ||
|
|
||
| Please explain the motivation behind this PR in the description. | ||
|
|
||
| If you're fixing a bug, link to the issue number like so: | ||
|
|
||
| ``` | ||
| - Fixes #{issue_number} | ||
| ``` | ||
|
|
||
| If you're adding a new feature, then consider opening a ticket and discussing it with the maintainers before you actually do the hard work. | ||
|
|
||
| #### Tests | ||
|
|
||
| If you're fixing a bug, consider [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development): | ||
|
|
||
| 1. Create a unit test that demonstrates the bug. The test should **fail**. | ||
| 2. Implement your bug fix. | ||
| 3. The test you created should now **pass**. | ||
|
|
||
| If you're implementing a new feature, include unit tests for it. | ||
|
|
||
| Make sure all existing unit tests pass. | ||
| You can run them locally using: | ||
|
|
||
| pytest smart_open | ||
|
|
||
| If there are any failures, please fix them before creating the PR (or mark it as WIP, see below). | ||
|
|
||
| #### Work in progress | ||
|
|
||
| If you're still working on your PR, include "WIP" in the title. | ||
| We'll skip reviewing it for the time being. | ||
| Once you're ready to review, remove the "WIP" from the title, and ping one of the maintainers (e.g. mpenkov). | ||
|
|
||
| #### Checklist | ||
|
|
||
| Before you create the PR, please make sure you have: | ||
|
|
||
| - [ ] Picked a concise, informative and complete title | ||
| - [ ] Clearly explained the motivation behind the PR | ||
| - [ ] Linked to any existing issues that your PR will be solving | ||
| - [ ] Included tests for any new functionality | ||
| - [ ] Checked that all unit tests pass | ||
|
|
||
| #### Workflow | ||
|
|
||
| Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress. | ||
| Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -56,3 +56,12 @@ target/ | |
| # vim | ||
| *.swp | ||
| *.swo | ||
|
|
||
| # PyCharm | ||
| .idea/ | ||
|
|
||
| # VSCode | ||
| .vscode/ | ||
|
|
||
| # env files | ||
| .env | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,8 @@ | ||
| include LICENSE | ||
| include README.rst | ||
| include CHANGELOG.rst | ||
| include setup.py | ||
| include MIGRATING_FROM_OLDER_VERSIONS.rst | ||
| include CHANGELOG.md | ||
| recursive-include sampledata *.zip | ||
| recursive-include smart_open/tests/test_data *.txt *.xz *.gz | ||
| include integration-tests/README.md integration-tests/*.txt | ||
| recursive-include integration-tests *.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| Migrating to the new ``open`` function | ||
| ====================================== | ||
|
|
||
| Since 1.8.1, there is a ``smart_open.open`` function that replaces ``smart_open.smart_open``. | ||
| The new function offers several advantages over the old one: | ||
|
|
||
| - 100% compatible with the built-in ``open`` function (aka ``io.open``): it accepts all | ||
| the parameters that the built-in ``open`` accepts. | ||
| - The default open mode is now "r", the same as for the built-in ``open``. | ||
| The default for the old ``smart_open.smart_open`` function used to be "rb". | ||
| - Fully documented keyword parameters (try ``help("smart_open.open")``) | ||
|
|
||
| The instructions below will help you migrate to the new function painlessly. | ||
|
|
||
| First, update your imports: | ||
|
|
||
| .. code-block:: python | ||
| >>> from smart_open import smart_open # before | ||
| >>> from smart_open import open # after | ||
| In general, ``smart_open`` uses ``io.open`` directly, where possible, so if your | ||
| code already uses ``open`` for local file I/O, then it will continue to work. | ||
| If you want to continue using the built-in ``open`` function for e.g. debugging, | ||
| then you can ``import smart_open`` and use ``smart_open.open``. | ||
|
|
||
| **The default read mode is now "r" (read text).** | ||
| If your code was implicitly relying on the default mode being "rb" (read | ||
| binary), you'll need to update it and pass "rb" explicitly. | ||
|
|
||
| Before: | ||
|
|
||
| .. code-block:: python | ||
| >>> import smart_open | ||
| >>> smart_open.smart_open('s3://commoncrawl/robots.txt').read(32) # 'rb' used to be the default | ||
| b'User-Agent: *\nDisallow: /' | ||
| After: | ||
|
|
||
| .. code-block:: python | ||
| >>> import smart_open | ||
| >>> smart_open.open('s3://commoncrawl/robots.txt', 'rb').read(32) | ||
| b'User-Agent: *\nDisallow: /' | ||
| The ``ignore_extension`` keyword parameter is now called ``ignore_ext``. | ||
| It behaves identically otherwise. | ||
|
|
||
| The most significant change is in the handling on keyword parameters for the | ||
| transport layer, e.g. HTTP, S3, etc. The old function accepted these directly: | ||
|
|
||
| .. code-block:: python | ||
| >>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
| >>> session = boto3.Session(profile_name='smart_open') | ||
| >>> smart_open.smart_open(url, 'r', session=session).read(32) | ||
| 'first line\nsecond line\nthird lin' | ||
| The new function accepts a ``transport_params`` keyword argument. It's a dict. | ||
| Put your transport parameters in that dictionary. | ||
|
|
||
| .. code-block:: python | ||
| >>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
| >>> params = {'session': boto3.Session(profile_name='smart_open')} | ||
| >>> open(url, 'r', transport_params=params).read(32) | ||
| 'first line\nsecond line\nthird lin' | ||
| Renamed parameters: | ||
|
|
||
| - ``s3_upload`` -> ``multipart_upload_kwargs`` | ||
| - ``s3_session`` -> ``session`` | ||
|
|
||
| Removed parameters: | ||
|
|
||
| - ``profile_name`` | ||
|
|
||
| **The profile_name parameter has been removed.** | ||
| Pass an entire ``boto3.Session`` object instead. | ||
|
|
||
| Before: | ||
|
|
||
| .. code-block:: python | ||
| >>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
| >>> smart_open.smart_open(url, 'r', profile_name='smart_open').read(32) | ||
| 'first line\nsecond line\nthird lin' | ||
| After: | ||
|
|
||
| .. code-block:: python | ||
| >>> url = 's3://smart-open-py37-benchmark-results/test.txt' | ||
| >>> params = {'session': boto3.Session(profile_name='smart_open')} | ||
| >>> open(url, 'r', transport_params=params).read(32) | ||
| 'first line\nsecond line\nthird lin' | ||
| See ``help("smart_open.open")`` for the full list of acceptable parameter names, | ||
| or view the help online `here <https://github.com/RaRe-Technologies/smart_open/blob/master/help.txt>`__. | ||
|
|
||
| If you pass an invalid parameter name, the ``smart_open.open`` function will warn you about it. | ||
| Keep an eye on your logs for WARNING messages from ``smart_open``. | ||
|
|
Oops, something went wrong.