Workflow for managing multiple output files, or files with too many parameters, and metadata

[stillyslalom]

I've been using DrWatson.jl to organize my preprocessing code/analysis for an ensemble of experiments performed in a large, complex facility. I've struggled to find an ergonomic workflow that captures the entire pipeline. Issues include:

• Too many experiment-relevant settings to capture in savename format without exceeding filesystem length limits
• Experimental ensembles are collected using the same nominal conditions for each data series, but contain idiosyncrasies (inoperable instruments, flubs, sensor/timing tweaks) that may need to be handled on a case-by-case basis
• External tools (ImageJ, Matlab) and user input fit awkwardly in the middle of a produce_or_load data-processing pipeline that assumes hands-off, end-to-end Julia code
• Data in instrument-specific formats is best kept in the filesystem where it can be handled with specialized external software rather than packed into JLD

This issue is partly a reminder to myself to write suitable documentation once I arrive at a good workflow.

[sebastianpech]

That sound very familiar to me. You might want to take a look at https://github.com/sebastianpech/DrWatsonSim.jl. It s kind of a spin-off of DrWatson and supports storing metadata, which I use instead of savename. Besides that I guess it's almost impossible to find a project independent approach for capturing your whole workflow.

What I do is exporting all parameter configurations for all my simulations to a note taking app and add additional annotations and documentation there. This works quite well and can mostly be automated. As DrWatsonSim works with simulation IDs instead of unique savenames, I can always find my notes and simulation results by this id.

[Datseris]

@stillyslalom can you try this out and let us know how it works for you? @sebastianpech I suggest that we start with writing a "real world example" in the DrWatson docs that showcases (briefly of course) your workflow. Based on that perhaps we can think of integrating DrWatsonSim directly into DrWatson if possible? Seems like many other people have asked for something similar and it always goes back to something like your approach...

[sebastianpech]

@Datseris Yes, seems so. I'll open a PR and polish the code a bit. I think I kept it general enough to integrate it in DrWatson. And it's all opt-in features, so no existing workflows will break and the metadata directory is only created when a metadata-related function is called the first time, so the folder structure also remains unchanged.

[sebastianpech]

Oh and @Datseris which file format do we prefer. The version on the master branch uses BSON for storing metadata, but I also have a separate branch using JLD2. Haven't merged it yet as I didn't want to update all my old project folders, but I guess as we have JLD2 as a dependency in DrWatson, I will do the switch.

[Datseris]

Yeah there is a clear statement in DrWatson now for JLD2 preference since 2.0. Do note that JLD2 cannot save functions though. Do you need this?

[sebastianpech]

Aha! I remember why I didn't do the switch yet. Yes I store all kinds of stuff in the bson files. However, it's not a requirement for the metadata functions to work. I think it's easy to workaround this. I will add a note to the docs.

[Datseris]

We have a similar difficulty in Agents.jl where we can't store functions-part of the model, only parameters that are not functions. However, JLD2 can store function-like objects with e.g., singleton dispatch.

struct Object end
(o::Object)(x, y) = x+y

instances of Object can be stored and when loaded will behave like functions.

[sebastianpech]

Interesting syntax. Never seen it. Cool. Good to know.

[Datseris]

Isn't it possible to make the system backend independent? Just choose via a keyword or environment variable which save backend to use?

[sebastianpech]

Sure, that's a good option.

[JonasIsensee]

We have a similar difficulty in Agents.jl where we can't store functions-part of the model, only parameters that are not functions. However, JLD2 can store function-like objects with e.g., singleton dispatch.

struct Object end
(o::Object)(x, y) = x+y

instances of Object can be stored and when loaded will behave like functions.

Hi, quick note:
This only works, when the Object and its method is defined in the new session already.

[sebastianpech]

I just tested this

foo(x) = x^2
save("test.jld2", Dict("fun"=>foo))
load("test.jld2")["fun"](2) # Gives 4

and it works. @Datseris what did you mean by storing functions? Do you mean actually storing them without the need to define them in the file you are loading them again?

I just ran through 1000+ metadata files I stored and I can convert all of them from BSON to JLD2