Tooling for a make.jl file in the project root

[goerz]

As explained in https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368, it might be nice to have some tooling that would allow me to write a script make.jl in the root of my research project that would automatically run the scripts to generate any missing output files or plots.

I realize this might a stretch, but it wouldn't seem entirely out-of-scope for a project like DrWatson to include something that takes a Dict defining a map of target files to dependencies and runs some code to produce those targets automatically.

[Datseris]

This seems like it boils down to having a bunch of .jl files that use produce_or_load, and then a make.jl file what just does incluide( all files ). My question is, what do you do if for the same .jl file you'd want to run this with different parameters?

[goerz]

I'm not sure we're exactly on the same page :-)

What I'm looking for is a way to automatically run the .jl files. Like a make.jl file that contains

RULES = [
    ScriptRule(
        [datadir("oct", savename("baseline_oct_result", Dict(:iter_stop=>100, :ν_max=>0.0), "jld2"))],
        scriptsdir("2022-01-09_baseline.jl")
    )
]

where the 2022-01-09_baseline.jl script would be run automatically run if the jld2 doesn't exist. The script itself in this case still has produce_or_load, but that only takes care of not unnecessarily re-creating the jld2 file if it already exists. It's somewhat redundant if someone is using the make.jl, but still important if they call the script directly.

The above snippet is currently in https://github.com/goerz-research/2022-01_Rydberg_Krotov_Spectral_Constraints, see also https://discourse.julialang.org/t/julia-based-makefile-replacement-for-research-workflows/74368/4?u=goerz

That system could actually specifically handle the situation of running the same .jl file with different arguments that you mention: You could have, e.g.,

RULES = [
    ScriptRule(
        [datadir("oct", savename("baseline_oct_result", Dict(:iter_stop=>100, :ν_max=>0.1), "jld2"))],
        scriptsdir("2022-01-09_baseline.jl"),
        args=[0.1]
    ), 
    ScriptRule(
        [datadir("oct", savename("baseline_oct_result", Dict(:iter_stop=>100, :ν_max=>0.5), "jld2"))],
        scriptsdir("2022-01-09_baseline.jl"),
        args=[0.5]
    )
]

under the assumption that the script takes the value for ν_max as an argument (I haven't actually implemented a ScriptRule with support for args, but it would be easy to do)

[Datseris]

Okay, but that ScriptRule you describe seems an okay solution? What's the problem with implementing that? Yes, you would have to manually write down every single ScriptRule for your project, but I don't see another way for the program to deduce this information.

If the ScriptRule works we can add here via a PR, it fits to the package and doesn't bring any dependencies nor disturbs the workflow.

What is a bit annoying is that you have to manually write down all files like datadir("oct", savename("baseline_oct_result", Dict(:iter_stop=>100, :ν_max=>0.5), "jld2")). Would be nice if we could automate this.

[goerz]

Yeah, there’s no problem :-)

The Discourse thread was just inquiring whether someone had already implemented a system like that (it seems not).

This issue is to find out if there’s any interest in having make-like features built in to DrWatson (it seems like “maybe”). So what I’ll do is let my makerules.jl gestate for a while through a few different projects, and then think of making a pull request to get that functionality into DrWatson

[goerz]

As for having to declare the output/target files: I agree, especially because it requires duplicating the call to savename and thus declaring all the parameters again in the "makefile". It would be much nicer if the output files could be automatically extracted from the script, so that they're only declared in one place. I just haven't come up with a good way to achieve that.

[goerz]

The problem is that the output files generally depend on the arguments, so figuring out the output filenames requires running the code at some level. The only reasonable idea I've come up with would require produce_or_load/@produce_or_load to have a dry_run option, so that

produce_or_load([path="",] config, f; dry_run=true, kwargs...) -> exists, s

where exists is a boolean indicating whether the file already exists or not, and s is the full path to the output file, as before. With that, scripts could self-report their output file(s): They'd have to be run once in dry-run-mode when instantiating the rule, and then once more for the actual run.

The first output, exists, is actually arbitrary for this purpose: I'd only need s. An existence-indicator seemed like the most straightforward thing (apart from nothing/an empty dict), but maybe there's something else that is cheap to calculate and carries more information for other potential use cases.

P.S.: A small nitpick about the documentation. Maybe instead of

produce_or_load([path="",] config, f; kwargs...) -> file, s

it would be clearer to have

produce_or_load([path="",] config, f; kwargs...) -> data, filename

I've gotten confused by file and s every time I've looked at that :-)

[goerz]

I'd note that dry_run would be very similar to the existing loadfile, but unlike loadfile, it doesn't produce anything on disk

[Datseris]

The problem is that the output files generally depend on the arguments, so figuring out the output filenames requires running the code at some level.

I was thinking of something much, much simpler. Julia has a folder watching functionality (that's how Revise.jl works actually). You can simply use the folder watching package, and tell it "watch these folders. Whenever a file is added in these folders, add its path and name to the X container". X will be a container that can be read by our make file business.

Difficulties:

1. How to ensure this folder watching works in the backfground all the time
2. how to ensure than when you save an output file to X, it also knows what's the script it produced it?

Perhaps we can kill two birds with one stone by adding this functionality to the macros @tagsave and @produce_or_load. Because they are macros, they have perfect knowledge of where they are being used (in fact, that's how the script entry of @tagsave! works). So we can add an option to the macros "save both the file name and the script used to run it as a pair script => filename to the X thingy"

Actually, this should work, and seems to be generic enough for your make idea? It also doesn't need folder watching functionality, the macros @produce_or_load do everything by themselves.

[Datseris]

P.S.: A small nitpick about the documentation. Maybe instead of

Please consider doing a pull request that fixes this, seems a rather straightforward documentation improvement!

[Datseris]

The X thingy can be a jld2 data file which can be read by a function generate_makefile that would read the pairs "script" => "filename" and generate the make file you wish for.

[goerz]

Please consider doing a pull request that fixes this, seems a rather straightforward documentation improvement!

Done in #318

I was thinking of something much, much simpler. Julia has a folder watching functionality
[...]
So we can add an option to the macros "save both the file name and the script used to run it as a pair script => filename to the X thingy

Yeah, I don't think folder watching would even be required if you hook something into the existing macros

Actually, this should work, and seems to be generic enough for your make idea?

I'm not sure... it would have to record the command line arguments (ARGS) as well.

For clarity, let me just re-iterate what I'm trying to achieve:

So, let's say I have a DrWatson-based project on Github, which I want people to be able to clone and reproduce. It contains scripts in the scripts subfolder, which produce jld2 files in the data subfolder. Generally, these jld2 files would not be committed to the repository (except maybe if they take more than a couple of hours to produce). Thus, the person trying to reproduce the project will have to run the appropriate script files, possibly with different arguments (!), and possibly in a specific order (if one script uses the output jld2 of another script as input).

That's the part I'm trying to automate: Ideally, the person reproducing the project can simply run make (or julia make.jl). Right now, I'm working with a make.jl that manually specifies which output jld2 files should be generated from which scripts/arguments. The next step I was thinking of would be to have a dry_run option for each script that returns the list of filenames it will generate for a particular set of ARGS, so that the make.jl only needs the list of (script, arguments) and can figure out the names of the jld2 files automatically.

What you're proposing, I think, is to hook into tagsave to store a mapping jld2_file => (script, arguments) in some central location (another jld2 file) that would be committed to the repo and used by make.jl. Actually, I think you may have considered only jld2_file => script which would be easy but not generic enough for a make. The general concept would work, though, at least in principle. I don't think it would require file watching: just every time tagsave produces an output file that's not already in the database, it has to append a record.

I'd see two difficulties, though.

The first is with recording the arguments (ARGS). Specifically, the ARGS are not necessarily the same as the config dict (oct_args) passed to produce_or_run/tagsave. I'm not sure to what extent the macro has access to the arguments. I guess ARGS is a global variable, but if I run the scripts by include instead of from the command line, the way my main functions are currently written, I'm actually not using ARGS but a passed variable args. 🤷 So I guess @tagsave would need an extra args argument to override ARGS as well, which would seem like a bit of a pain.

The second problem is that I might want to play around with parameters and run scripts in the "exploratory" stage of the project, without wanting to record the result for the make. So there'd have to be a way to skip recording, or to clean up the database manually (maybe a human-readable format would be better than jld2 in that case). Another possibility would be to generally record ARGS/args in all output jld2 files generated by @tagsave. There could then be a manual command that looks at all current jld2 files in the project and records their name => (script, args) mapping in the database (under the assumption that I've removed any "exploratory" `jld2 files). Keeping the ordering might be tricky with that approach: I guess for any new files, they should be ordered by modification time, but even that isn't guaranteed to work.

So it seems like it's possible, but not easier than having a dry_run option. Unless I misunderstood what you had in mind :-)

I'd also say that dry_run doesn't seem like a terrible feature on its own, independent of its use in a possible make-like system. Do you think I should make a PR just for that?