Chunking MDF to Parquet

[DonDaps]

Hi All,

Firstly thanks to the authors for a great MDF package, has saved a lot of headaches and supported building lots of useful tools.

I am trying to export MDF channels to Parquet in their native logging frequencies. Whilst creating the df in the background, I am regularly running out of memory (16GB) for MDF files <500mb. The important interpolation arguments are set to false to avoid bloating.

I am exploring chunking the df, with the intention of building Parquet files in chunks also (like this)

However, i am getting the following error if i set the 'chunk_ram_size' to any value smaller than the file itself.
ValueError: could not broadcast input array from shape (372292,) into shape (13,)

Full disclosure, chunking is new to me. I'm hoping I haven't missed something obvious here...Can you help point me in the right direction please?

x = mdf.iter_to_dataframe(
chunk_ram_size=1000000,
reduce_memory_usage = True,
use_interpolation=False,
only_basenames=True,
interpolate_outwards_with_nan=False)

try:
val = next(x)
print(val.head(10))
except StopIteration:
print('the end')

[danielhrisca]

Hello, looks like you found a bug. Is there any way you could send the file for analysis?

[danielhrisca]

@DonDaps I believe you've send a dropbox invitation, but I cannot access it because my dropbox account does not contain the . (dot) in the e-mail address. Could you send it again?

[DonDaps]

Has that worked?

[danielhrisca]

I've downloaded the file and I can reproduce the problem

[danielhrisca]

please try the development branch code

[DonDaps]

Thanks very much for the super quick turnaround. The generator works, so the bug is fixed - thanks!

The problem i'm now having is creating and appending a parquet file with the df chunks. The source of the headache is creating a nullable schema, for instances where there are chunks of nulls that don't abide by the schema metadata: data type for example.

Chunking an MDF (via a df) to Parquet would be an extremely useful function for your package to achieve. Export to Parquet regularly saturates memory unfortunately.

Here is an example that reproduces the issue:

 from asammdf import MDF
import pyarrow as pa
import pyarrow.parquet as pq

fname_source = r"\ExampleMDF.mdf"
mdf = MDF(fname_source, remove_source_from_channel_names=True)

# set chunk size
chunksize=100*1000*1000 

# create generator to chunk MDF file
x = mdf.iter_to_dataframe(
    chunk_ram_size=chunksize,
    reduce_memory_usage = True,
    use_interpolation=False,
    only_basenames=True,
    interpolate_outwards_with_nan=False)


pqwriter = None
i=0
while(True):
    
    try:
        df = next(x)
        table = pa.Table.from_pandas(df)
        
        # an attempt to remove metadata to see if it fixes issue
        # schema = table.schema
        # schema = schema.remove_metadata()
        if i == 0:
            # create a parquet write object giving it an output file
            pqwriter = pq.ParquetWriter('sample.parquet', table.schema)
            i+=1 
        # this is where the code errors
        pqwriter.write_table(table)
    
    except StopIteration:
        print('the end')