Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add version_id transport parameter for fetching a specific S3 object version #325

Merged
merged 47 commits into from Jul 15, 2019
Merged

Add version_id transport parameter for fetching a specific S3 object version #325

merged 47 commits into from Jul 15, 2019

Conversation

interpolatio
Copy link
Contributor

@interpolatio interpolatio commented Jun 16, 2019

Added file check by id.
Added check for the existence of the passed id.

Fix #306

Added check for the existence of the passed id.
Copy link
Collaborator

@mpenkov mpenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good start, please see my comments and add some thorough unit tests.

smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Outdated
@@ -112,6 +114,7 @@ def open(
fileobj = SeekableBufferedInputBase(
bucket_id,
key_id,
version_id,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass keyword parameters by name.

Suggested change
version_id,
version_id=version_id,

smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
1. Updated help revision_id;
2. Added version validation when getting the length of an object s3 in BufferedInputBase;
3. Style adjustment.
smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
….ClientError - creates an exception IOerror
Copy link
Collaborator

@mpenkov mpenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still needs tests

smart_open/s3.py Outdated Show resolved Hide resolved
interpolatio added 5 commits June 22, 2019 13:50
Copy link
Collaborator

@mpenkov mpenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice progress. Left you some more comments, please take a look.

smart_open/tests/test_s3_version.py Outdated Show resolved Hide resolved
smart_open/tests/test_s3_version.py Outdated Show resolved Hide resolved
smart_open/tests/test_s3_version.py Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Show resolved Hide resolved
smart_open/tests/test_s3_version.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@mpenkov mpenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let you some more comments.

Also, you're missing a top-level test. You should make sure that working with your version_id parameter via the top-level smart_open.open function works as intended.

Finally, it looks like the documentation isn't picking up your new parameter. It should appear here but does not.

README.rst Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
smart_open/s3.py Show resolved Hide resolved
@mpenkov
Copy link
Collaborator

mpenkov commented Jun 26, 2019

Also, looks like unit tests are failing, please fix:

======================================================================
ERROR: test_readline_tiny_buffer (smart_open.tests.test_s3.SeekableBufferedInputBaseTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/moto/core/models.py", line 71, in wrapper
    result = func(*args, **kwargs)
  File "/home/travis/build/RaRe-Technologies/smart_open/smart_open/tests/test_s3.py", line 249, in test_readline_tiny_buffer
    with smart_open.s3.BufferedInputBase(BUCKET_NAME, KEY_NAME, buffer_size=8) as fin:
  File "/home/travis/build/RaRe-Technologies/smart_open/smart_open/s3.py", line 222, in __init__
    self._content_length = _get(self._object, self._version_id)['ContentLength']
AttributeError: 'BufferedInputBase' object has no attribute '_version_id'

… the version is not None.

Added test for this.
smart_open/tests/test_s3_version.py Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
smart_open/s3.py Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
@mpenkov mpenkov changed the title Added version_id for s3_object. Add version_id transport parameter for fetching a specific S3 object version Jul 15, 2019
Copy link
Collaborator

@mpenkov mpenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, this is ready to merge

@mpenkov mpenkov merged commit 0995cfc into piskvorky:master Jul 15, 2019
@mpenkov
Copy link
Collaborator

mpenkov commented Jul 15, 2019

@interpolatio Congrats on your very first contribution to smart_open! 🥇 Keep them coming!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failed file access should raise IOError not ValueError (S3)
3 participants