Create a structure for the GEMDAT tool

[v1kko]

It might look something like this:

option 1

• GEMDAT
  • plot_all
  • plot_1
  • plot_2
  • plot_3

from GEMDAT import plot_1, plot_all
plot_1(<data_and_config>)
plot_all(<data_and_config>)

---

Or something like this:

option 2

• GEMDAT
  • plot(<data_and_config>, )
  • plots
    • plot_1(<data_and_config>)
    • plot_2(<data_and_config>)

from GEMDAT import plot
plot(<data_and_config>, ['diffusivity', 'MSD'])  

from GEMDAT.plots import plot_diffusivity
plot_diffusivity(<data_and_config>)

---

data_and_config:
There will probably be the need to adjust plots with some configuration (splitting into x number of multiple smaller simulations, or cutting away the first few timesteps). I think it would be okay to just pass those through **kwargs and let all plot functions accept **kwargs, so they can extract those keywords that they will use.
All the possible keywords should be listed in the plot (or plot_all for option 1) function, but how they are implemented can be explained in the specific plot_xxx function.

---

I prefer option two, what are your thoughts about this, @stefsmeets ?

[stefsmeets]

I think having some sort of plot_all or plot(..., [list of plots]) makes sense to have something that resembles the analyse_md function.

Other than that, I would keep all data manipulation (generating displacements, calculating jumps, etc) out of the plots.

[v1kko]

Hmm yes I agree,

how about some structure where the data can be retrieved or calculated (transparently to the user):

from GEMDAT import Data
data = Data()
data.structure = structure #  etc...

from GEMDAT import plot
plot(data=data, cutoff=xxx, all=True)

A data object would then be able to provide Data::jumps if it was already calculated, or calculate it if it was not yet available.

---

The only thing I am then still thinking about is these intermediate files, clearly they are useful, but can we nicely integrate them without the user having to do something like:

from GEMDAT import Data
from pymatgen import VaspRun
data = Data().from_vasprun(VaspRun("xml file")
data.save_intermediate("something")

If we have to do something like that, we should think about the correct data-format for such intermediate file.
I would prefer to only have one file containing everything, to not clutter folders, but there is something to be said for saving specific arrays in independent files as well.

[stefsmeets]

I'm also wondering to which end we need to cache these intermediate data. So far, none of the computations are particularly demanding. Reading the data is by far the most time consuming.

All the computations I have encountered so far can be vectorized very nicely and run in under a second.

[v1kko]

I think its good to define "intermediate" as the data format that can be fed directly to the computations, so after parsing with pymatgen, but before computing extra properties. Because the parsing of pymatgen still takes more than a minute (for me).
We could try to use the to_dict() functions of pymatgen for this?

[stefsmeets]

The to_dict() functionality seems horribly inefficient to me. On my computer it takes longer to serialize the data than to read it again from the vasprun.xml. It also eats all my memory (10GB+ for the python process) so it's continually swapping to disk.

For small objects (Lattice, Species, one or two Structures) it seems OK. From what I've seen so far, the data structures in pymatgen are not optimized for the number of time steps we are working with.

I think for this project we will mostly be working with the Trajectory class. This is essentially just an array of atomic coordinates. Way more efficient. And even that is easier to store as a numpy file than to go through the as_dict() / from_dict() machinery.

If we store the data as a numpy file, we can construct the Trajectory class ourselves if it is useful (so far I've been working with the coordinates directly though).

[stefsmeets]

Does it make sense to have some sort of persistant data storage that we can just shove data into when we generate something?

Could something simple like shelve work?

Or maybe something outside the standard library like klepto, which seems like a more flexible big brother of shelve.

[v1kko]

I like the interface offered by the numpy.savez function.
Or otherwise a hdf5 dump might work?

I dislike shelve a bit I think as it seems to be based on pickle, which is known to break at times

[stefsmeets]

I was thinking about hdf5 as well, don't you find it a bit overkill here?

I was thinking of shelve mainly because the interface is simple (like a python dict), I don't mind that it uses pickle. Pickle usually works well with simple data structures.

I just tried np.savez and it also uses pickle on the backend when you pass non-numpy objects (like a python dict). It seems like a nice option too.

from tempfile import TemporaryFile
outfile = TemporaryFile()
np.savez(outfile, data=data, traj_coords=traj_coords)
outfile.seek(0)
npzfile = np.load(outfile, allow_pickle=True)
data2 = npzfile['data'].tolist()  # get the dict back
traj_coords2 = npzfile['traj_coords']

[v1kko]

Yes, figuring out the same thing here (just not pushed yet 😉 ). so I will add a numpy.savez version to the pull request to have only a single cache file.

[stefsmeets]

If we store data using np.savez I want to have a small tool that extracts the parameters / metadata to plain text (yaml) so that they can be inspected.

[v1kko]

that makes sense, we should store the version/commit_hash of GEMDAT, creation time, the input file location, maybe the pymatgen version as well., then make a small script which can print that information (and with a simple breakpoint() inspect the data)

[v1kko]

okay big downside of numpy.savez is that it throws away information when saving object, so trying simple pickle next, we can take it from there