Software metadata in LH5 files

[gipert]

We should consider storing software metadata relevant for data preservation consistently in all LH5 files. One thing could be certainly pygama.__version__, but I'm not sure whether also versions of some dependencies like NumPy or Numba would be relevant.

We should also retain the possibility to do checksum-based file comparisons. Storing things like file creation time would make this more difficult and would at least require custom checksumming utilities.

We should also consider versioning the LH5 specification and store the version a file was created with.

[jasondet]

@iguinn and I suggest adding to the attributes a key "provenance" that is a dict with this info. E.g. for a typical DSP output column it would contain:

provenance:
    creation_date: YYMMDDZhhmmss
    processor: module.processor_name
    module_versions:
        numpy: v...
        dspeed: v...
        etc.

[MoritzNeuberger]

Do I understand correctly that this "provenance" information would be added to the appropriate table in the build_dsp, build_hit,... script level and no change to LH5Store would be necessary?

[gipert]

Yes, the issue mentioned by @jasondet should be moved to dspeed.