pd.read_sas with chunksize option raises IndexError

[jicky94]

Good morning,

Using Python 3.6.

My problem seems to be close to issue #14734 but with a different error type though. However, please forgive my lack of competence, but I am not able to understand 1. if my issue is really similar and 2. if the issue that seemed to surround some sas files has been solved or if there might still be some probleùs with some sas files (that i cannot provide for reasons detailed just below).

I have read the rules about posting but i cannot attach a sample of my data or reproduce the entire error message as the data i am working on is located on a server without access to internet. I apologize for this inconvenience. I’ll try to reproduce most of what is requested however below.

I am working with very big sas files (data on each job, hence millions of lines) and got memory error when i was trying to simple read them (they open fine in R or stata strangely). Therefore i searched and find the pandas.read_sas option to work with chunks of the data. My code is now the following:

import pandas as pd
df_chunk = pd.read_sas(r'file.sas7bdat', chunksize=500)

for chunk in df_chunk:  
    chunk_list.append(chunk)

At this point i get the following error (I am reproducing it here manually as i cannot copy paste):

line 660, in _chunk_to_dataframe
if self.column_formats[j] in const.sas_date_formats:
IndexError: list index out of range

I am aware the exposition of my issue is truncated and probably incomplete but many thanks for any help you could provide,
Axelle

[MarcoGorelli]

Thanks for the report, Axelle

i cannot attach a sample of my data or reproduce the entire error

There's some sas files in pandas/tests/io/sas/data - in order to expedite resolution of this issue, could you try reproducing this with one of them?

[jicky94]

Thanks, i tried with several of the datasets provided and it worked perfectly well.. Unfortunately i cannot provide the datasets i am using to check further. Do you think there still might be some issues with some sas files? Many thanks again.

[MarcoGorelli]

Do you think there still might be some issues with some sas files?

Don't know :) Perhaps you could try making your own sas file which mimics the format of the data you're using on your server, and then see if you can reproduce the issue there?

[MarcoGorelli]

I'll close this for now as it looks like we don't have a reproducible example (https://pandas.pydata.org/pandas-docs/stable/development/maintaining.html#cleaning-up-old-issues)