Store pandas dataframe as single object in data frame

[jan-janssen]

Converting to dictionary is terribly slow:

from pyiron_base import DataContainer
from pyiron_base import Project
import pandas
with open("test.csv", "w") as f:
    f.write(",a,b\n")
    for i in range(100000): 
        f.write(str(i) + ",1,2\n")
df = pandas.read_csv("test.csv", index_col=0)
pr = Project(".")
hdf = pr.create_hdf(".", "test")
dc = DataContainer(df.to_dict())
dc.to_hdf(hdf)

This takes forever, while with a small change it is in the order of ms:

dc = DataContainer(df)

[jan-janssen]

This fixes my issues with #463

[pmrv]

Have you checked that this actually stores the data? In my testing conversion from pandas is broken.

df = DataFrame({'a': range(4), 'b': range(4,8), 'c': range(8,12)})
dc = DataContainer(df)
print(dc)
> DataContainer([])
dc = DataContainer(df.to_dict())
print(dc)
> DataContainer({'a': DataContainer([0, 1, 2, 3]), 'b': DataContainer([4, 5, 6, 7]), 'c': DataContainer([8, 9, 10, 11])})

The reason I didn't save it via pandas is that I expect people to put data into a pyiron_table that is HDF-serializable to us, but not to pandas (e.g. structures). I'll have a look today at #463 and #559.

[pmrv]

This will break without pytables or not?

[jan-janssen]

Yes, I rely on pytables - but as long as that is the recommended solution by pandas I guess that is the only option. Alternatively we could also pickle the object and store the pickled string, but I do not think that is a better alternative.

[pmrv]

Then there's no need to put the DataFrame into the data container at all and we can just DataFrame.to_hdf/read_hdf, I suppose. Then we also need to put tables back into our environment.yml.

[jan-janssen]

That I exactly what I do in the latest commit.

[pmrv]

Ah, I missed that. Looks good to me then, we can always come back to this if someone really wants to store "complex" data in their pyiron tables.

EDIT: The dependency still should be added, though.

[jan-janssen]

The reason I didn't save it via pandas is that I expect people to put data into a pyiron_table that is HDF-serializable to us, but not to pandas (e.g. structures). I'll have a look today at #463 and #559.

That is definitely a limitation of the current solution, currently I sure the structures as JSON objects before inserting them into the pyiron table, but that requires extra work on the user side. The advantage is the size difference. Already if I only load the first 10000 entries of the file in the example above the old code resulted in a 124MB HDF5 file. So the conversion to dictionary and storing in the Datacontainer is definitely not scalable for large pyiron tables.

[pmrv]

I'm looking into the file size right now and it seems like an issue with how we write to HDF5 in general. When I do a h5repack on the file in the example, I directly get a factor ~20 decrease in file size.

[jan-janssen]

The dependency is still included https://github.com/pyiron/pyiron_base/blob/pyiron_table_store_using_pandas/setup.py#L45