/pandas

ENH Recognize 's3n' and 's3a' as an S3 address #11071

Merged
merged 1 commit into from Sep 14, 2015
+19 −2

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
Data IO
Projects
None yet
Milestone
  0.17.0
2 participants
@stephen-hoover @jreback
@stephen-hoover

stephen-hoover commented Sep 12, 2015

This PR allows read_csv to recognize that "s3n://" designates a valid AWS S3 address. Partially addresses issue #11070 .

@jreback jreback commented on an outdated diff Sep 12, 2015

pandas/io/tests/test_parsers.py
@@ -4246,6 +4246,15 @@ def test_parse_public_s3_bucket(self):
tm.assert_frame_equal(pd.read_csv(tm.get_data_path('tips.csv')), df)
@tm.network
+ def test_parse_public_s3n_bucket(self):
+ # Read from AWS s3 as "s3n" URL
+ import nose.tools as nt
+ df = pd.read_csv('s3n://nyqpug/tips.csv', nrows=10)
+ nt.assert_true(isinstance(df, pd.DataFrame))
@jreback

jreback Sep 12, 2015

Contributor

self.assertTrue

and like

stephen-hoover changed the title from ENH Recognize 's3n' as an S3 address to ENH Recognize 's3n' and 's3a' as an S3 address Sep 12, 2015

@stephen-hoover

stephen-hoover commented Sep 12, 2015

Fixed. I also realized (reading the links I added for issue #11070 ) that "s3a" is a valid S3 designator as well, so I added that too.
@stephen-hoover

stephen-hoover commented Sep 12, 2015

The one test failure appears unrelated to the code change in this PR:

FAIL: test_numeric_column_names (pandas.io.tests.test_stata.TestStata)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/io/tests/test_stata.py", line 480, in test_numeric_column_names
    original.to_stata(path)
  File "/home/travis/miniconda/envs/pandas/lib/python3.3/contextlib.py", line 55, in __exit__
    next(self.gen)
  File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 2021, in assert_produces_warning
    % extra_warnings)
AssertionError: Caused unexpected warning(s): ['ResourceWarning'].
@jreback
Contributor

jreback commented Sep 12, 2015

no it's just hitting the actual error there
you have something failing

jreback added the Data IO label Sep 12, 2015

stephen-hoover referenced this pull request Sep 12, 2015

Closed

Improvements for read_csv from AWS S3 #11070

4 of 4 tasks complete
@jreback
Contributor

jreback commented Sep 13, 2015

add a whatsnew note in 0.17.0 (enhancements section)

jreback added this to the 0.17.0 milestone Sep 13, 2015

@stephen-hoover

stephen-hoover commented Sep 14, 2015

@jreback , green here too!

@jreback jreback added a commit that referenced this pull request Sep 14, 2015

@jreback jreback Merge pull request #11071 from stephen-hoover/recognize-s3n 
ENH Recognize 's3n' and 's3a' as an S3 address
 95110ac

@jreback jreback merged commit 95110ac into pandas-dev:master Sep 14, 2015
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jreback
Contributor

jreback commented Sep 14, 2015

thanks!

stephen-hoover deleted the stephen-hoover:recognize-s3n branch Sep 14, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment