Error when attempting to use filters

[jeremyakers]

I'm attempting to filter data from a Parquet file to reduce memory usage:

import awswrangler as wr

df = wr.s3.read_parquet(
         path="s3://bucket/path/part-00000.snappy.parquet",
         filters=[('iso_country_code', '==', 'US')]
)

This results in:

Traceback (most recent call last):
  File "filter_parquet_from_aws.py", line 5, in <module>
    filters=[('iso_country_code', '==', 'US')]
  File "/home/jeremyakers/.local/lib/python3.6/site-packages/awswrangler/s3.py", line 1726, in read_parquet
    s3_additional_kwargs=s3_additional_kwargs,
  File "/home/jeremyakers/.local/lib/python3.6/site-packages/awswrangler/s3.py", line 1603, in _read_parquet_init
    split_row_groups=False,
  File "/home/jeremyakers/.local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1200, in __init__
    self._filter(filters)
  File "/home/jeremyakers/.local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1314, in _filter
    accepts_filter = self.partitions.filter_accepts_partition
AttributeError: 'NoneType' object has no attribute 'filter_accepts_partition'

There don't seem to be many examples or clear docs on how to use this filters option, so I'm not sure what I might be doing wrong.

[jeremyakers]

I'm wondering if this might be a bug. Pandas supports this if you add the option use_legacy_dataset=False but that option doesn't seem to be supported in awswrangler:

Traceback (most recent call last):
  File "filter_parquet_from_aws.py", line 10, in <module>
    filters=[('iso_country_code', '==', 'US')]
TypeError: read_parquet() got an unexpected keyword argument 'use_legacy_dataset'

[igorborgest]

Hi @jeremyakers thanks for reaching out!

1 - Unfortunately AWS Data Wrangler does not support filters on physical columns, only on partitions. (Docs updated in the commit above). It's not only a matter of pass use_legacy_dataset=False, seems that the new dataset approach does not have support to receive boto3 sessions.

2 - Fortunately there are others strategies to mitigate your lack of memory, did you take a look on others argumentes like columns, chunked and categories? Maybe someone could help you.

[igorborgest]

Closing due to inactivity.