Multiple Datasets in YAML

[tgmaxson]

I have two datasets where one is a small cell and another is a larger cell. While I can use something like ASE to combine the two datasets into a single one, I would like to simply supply both to the yaml. Is this something that can easily be done?

Further, I would like to manually specify the training, validation, and test sets if possible. I think I can do this by putting it in order and turning off shuffling, but it would be great to be able to just give a separate dataset for each.

[Linux-cpp-lisp]

Hi @tgmaxson ,

This is supported for separate train, validation, and test sets. Train and validation can be specified separately using options prefixed by dataset_ and validation_dataset_ respectively (see https://github.com/mir-group/nequip/blob/main/configs/full.yaml#L146-L157). Test sets are not handled by nequip-train; arbitrary test sets can be using by passing a new YAML file with dataset_ options to nequip-evaluate --dataset-config whatever.yaml.

We do not, however, support combining multiple datasets into a single training set in the YAML; generally we try to avoid badly integrating things like dataset construction that can be done more easily and in a more customizable way externally. Note that extxyz files can be combined by simply concatenating them, i.e. cat 1.xyz 2.xyz > full.xyz.