Skip to content
Permalink
Browse files
Adding anonymous flag to s3 (#70)
* Adding anonymous flag to s3

* Fixing missing comma

* Info about s3 credentials
  • Loading branch information
drabastomek committed Sep 10, 2021
1 parent 5d43a86 commit 9750a6402436f0379a9a7bde4184076c615f5a93
Showing 1 changed file with 26 additions and 2 deletions.
28 python/source/io.rst 100644 → 100755
@@ -394,7 +394,10 @@ partitioned data coming from remote sources like S3 or HDFS.
from pyarrow import fs

# List content of s3://ursa-labs-taxi-data/2011
s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem(region="us-east-2"))
s3 = fs.SubTreeFileSystem(
"ursa-labs-taxi-data",
fs.S3FileSystem(region="us-east-2", anonymous=True)
)
for entry in s3.get_file_info(fs.FileSelector("2011", recursive=True)):
if entry.type == fs.FileType.File:
print(entry.path)
@@ -419,7 +422,7 @@ by ``month`` using

.. testcode::

dataset = ds.dataset("s3://ursa-labs-taxi-data/2011",
dataset = ds.dataset("s3://ursa-labs-taxi-data/2011",
partitioning=["month"])
for f in dataset.files[:10]:
print(f)
@@ -447,6 +450,27 @@ or :meth:`pyarrow.dataset.Dataset.to_batches` like you would for a local one.
It is possible to load partitioned data also in the ipc arrow
format or in feather format.

.. warning::

If the above code throws an error most likely the reason is your
AWS credentials are not set. Follow these instructions to get
``AWS Access Key Id`` and ``AWS Secret Access Key``:
`AWS Credentials <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html>`_.

The credentials are normally stored in ``~/.aws/credentials`` (on Mac or Linux)
or in ``C:\Users\<USERNAME>\.aws\credentials`` (on Windows) file.
You will need to either create or update this file in the appropriate location.

The contents of the file should look like this:

.. code-block:: bash
[default]
aws_access_key_id=<YOUR_AWS_ACCESS_KEY_ID>
aws_secret_access_key=<YOUR_AWS_SECRET_ACCESS_KEY>
Write a Feather file
====================

0 comments on commit 9750a64

Please sign in to comment.