Skip to content
Browse files
Adding anonymous flag to s3 (#70)
* Adding anonymous flag to s3

* Fixing missing comma

* Info about s3 credentials
  • Loading branch information
drabastomek committed Sep 10, 2021
1 parent 5d43a86 commit 9750a6402436f0379a9a7bde4184076c615f5a93
Showing 1 changed file with 26 additions and 2 deletions.
28 python/source/io.rst 100644 → 100755
@@ -394,7 +394,10 @@ partitioned data coming from remote sources like S3 or HDFS.
from pyarrow import fs

# List content of s3://ursa-labs-taxi-data/2011
s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem(region="us-east-2"))
s3 = fs.SubTreeFileSystem(
fs.S3FileSystem(region="us-east-2", anonymous=True)
for entry in s3.get_file_info(fs.FileSelector("2011", recursive=True)):
if entry.type == fs.FileType.File:
@@ -419,7 +422,7 @@ by ``month`` using

.. testcode::

dataset = ds.dataset("s3://ursa-labs-taxi-data/2011",
dataset = ds.dataset("s3://ursa-labs-taxi-data/2011",
for f in dataset.files[:10]:
@@ -447,6 +450,27 @@ or :meth:`pyarrow.dataset.Dataset.to_batches` like you would for a local one.
It is possible to load partitioned data also in the ipc arrow
format or in feather format.

.. warning::

If the above code throws an error most likely the reason is your
AWS credentials are not set. Follow these instructions to get
``AWS Access Key Id`` and ``AWS Secret Access Key``:
`AWS Credentials <>`_.

The credentials are normally stored in ``~/.aws/credentials`` (on Mac or Linux)
or in ``C:\Users\<USERNAME>\.aws\credentials`` (on Windows) file.
You will need to either create or update this file in the appropriate location.

The contents of the file should look like this:

.. code-block:: bash
Write a Feather file

0 comments on commit 9750a64

Please sign in to comment.