Possible data provenance functionality

[Datseris]

With @JonasIsensee , @tamasgal and @sebastianpech we discussed that smaller groups of scientists may not find it sensible to opt for large data management software such as CaosDB. But it would still be great to have basic data provenance for forms out data outside .bson.

.bson and similar formats are covered satisfactorily by DrWatson due to the automatic adding of git info, and the automatic adding of source file that generated them. This is not possible for e.g. figures or CSV files.

What could be possible is to have a central file, next to Project.toml, that is also .toml or .yml based, and works as a dictionary. It maps unique identifiers to a set of properties, the first of which is file, and it just contains the file path relative to the project main folder. The advantage of using toml is that it is human readable and can be searched with Ctrl+F. Notice that specialized parameter searches are more suited for the result of a function like collect_data and thus do not need to be considered for this functionality.

Other properties could be added, like source file used, date produced, savename of parameters used, author, git commit, etc.

All in all this is a great compromise between the complexity of a full data manager and having data provenance for figures, CSV, etc.

[Datseris]

Actually, I don't immediatelly see why the mapping should map unique identifiers. Seems to me that the format could directly map a file name (with its relative path) to the dictionary. The file name is also unique after all.

[sebastianpech]

The advantage of using toml is that it is human readable and can be searched with Ctrl+F

I think one advantage of using a binary file format is that we can attach julia types as metadata. So I could theoretically directly attach the parameter config dict that led to this specific file, instead of converting it into a string before.

The search functionality must the be of course implemented in DrWatson.

Actually, I don't immediatelly see why the mapping should map unique identifiers. Seems to me that the format could directly map a file name (with its relative path) to the dictionary. The file name is also unique after all.

I think using the filenames as identifiers is fine. It also suggests that the database file is only used for storing metadata for files and not for storing arbitrary data entries.

[Datseris]

I think one advantage of using a binary file format is that we can attach julia types as metadata. So I could theoretically directly attach the parameter config dict that led to this specific file, instead of converting it into a string before.

I thought about this as well, but it has a significant downside: filesize will explode quickly...? We should compare. In fact, if we use a central .bson file as the provenance, we can do this provenance thingy pretty much immediately. Writing a Julia function that does this isn't a big deal...

The search functionality must the be of course implemented in DrWatson.

This is really hard to do though, and probably not worth the effort. Searching withing dictionaries of arbitrary type is also dubious; if a user gives "p" you have to search all keys, and all values that could potentially include "p" or :p, and fields of custom types as well. Too complicated I feel, and would be a pain in the butt to debug for all possible use cases.

[Datseris]

See #152 for a quick and dirty sketch of the idea.

[sebastianpech]

Writing a Julia function that does this isn't a big deal...

Definitely not too difficult to do. Fitting it into the DrWatson workflow is a little harder.
So one fundamental question: Would this functionality replace savename?

[Datseris]

Would this functionality replace savename?

What never! I use savename for figure titles :D

[sebastianpech]

What never! I use savename for figure titles :D

Clever :)

So we would promote two approaches that have a similar purpose

1. The current one. With savename, tagsave, ... that only fully works if you can store the metadata alongside you results. (Though, savename is universal, that's what's so nice about it. So storing the parameter set can always be done, no matter the file type)
2. Storing all info about the simulation in the central database file

[Datseris]

So we would promote two approaches that have a similar purpose

yeap, precisely.

[JonasIsensee]

Before we go off talking about implementation details, I believe we should think clearly about what we actually want from this software and what we need it to truly add value to the workflow. (or the reproducibility.)
For example: The filename may be unique enough to identify a file but it can't tell if the file has been modified / overwritten. In that case hashvalues would be good.

Also - how can we do this to keep it as extensible as possible?

2. Storing all info about the simulation in the central database file

Saving into a central file is dangerous when generating data on multiple workers in parallel.
I think collecting the metadata into a database after the fact would be safer.

[Datseris]

I think collecting the metadata into a database after the fact would be safer.

Yeah, but that is why the external-server-complicated-detached database software like CaosDB exist. Like you said, we really have to consider what we want to do. I think we all agree that we don't want this to be leading into any heavy dependencies....

We should also discuss whether we want to match existing functionalities. Personally, I don't see a reason to try and match the complexity (and capabilities) of those data management software, as there are several software that provide such options. The same thing holds for being able to tell if a file is modified / overwritten: it again can be managed by these advanced software.

It is also a matter of effort: I definitely can't spend a lot on time on this.