[ideas] Reading in previous versions of targets

[noamross]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I've created a couple of functions to read previous versions of targets in cloud storage (e.g., AWS S3 versioning), using the git reference to look at git history and then download the appropriate cloud versions referenced in the meta file: https://gist.github.com/noamross/e05d3884b8ec8b4dce8380e24163851d .

The purpose is to do comparison of objects across history without fully checking out to those project states. For instance, one could create a report of how model performance changes across commits or branches:

current_model <- tar_read(model_object)
old_model <- tar_read_version(model_object, "HEAD~1")
alternate_model <- tar_read_version(model_object, "feature/penalization")
AIC(current_model, old_model, alternate_model)

The approach is to:

• Extract the _targets (or other store directory) from git history into a temporary directory named for the SHA of that subtree
• (Skip extraction if the directory already exists, as the SHA is universally unique)
• Run tar_read() or tar_load() using the temporary directory as the store.

One problem with this is, due to current limitations of git2r, the meta file can be extracted from the git repo history, but not the binary (rds/qs/...) target object files. So this only works with targets that are stored on AWS or "file" type targets. I'd like it to work for cases where one commits target objects to git history directly. See ropensci/git2r#460 for what is needed to do this.

Another limitation is if file targets need to be read in and compared, they lie outside the _targets/store directory. But I already have what I need to make a tar_file_version("filename.txt", "HEAD~3") that extracts the file and returns the temporary path to it. The binary issue still applies, though.

Both of these could be fixed if, instead of the current approach, I use a processx::run() call to the CLI git to check out the whole repo or a subtree at a given commit to a temporary directory. I think this would be heavier, more fragile and complex across platforms or git versions, though, so I hope the git2r limitation can be fixed.

I'm interested in some feedback:

• Is this a good general approach? Where else might it fail?
• Where might these functions live? Would @wlandau be interested in a PR to tarchetypes?

Give it a go, and let me know what you think!

@emmamendelsohn @n8layman

[noamross]

Wow, that was fast! The git2r maintainer already implemented my suggestion. Installing ropensci/git2r@raw-blob-content (requires compiling from source and the libgit2 system library for now) allows for extracting binary files and I've updated the functions to do this. Next considering:

• Having tar_read/load_versioned() check if the target loaded is a local file, and if so automatically extracting that file from the same git commit to a temporary location are returning the path to it
• Using a system-level temporary directory rather than tempdir() to store the extracted files, so they persist across R sessions (and thus tar_make() calls).

[noamross]

I wonder if this actually belongs in a git helper package. Plucking a file or subtree out of git history is an obscure bit of git-fu, and there are plenty of non-targets related workflows for which it would be helpful to have something like git_version_copy("plots/fig-1.png", "HEAD~3", to="old_fig1.png"). From a quick search (https://r-universe.dev/search/?q=git) I don't see anything that provides this kind functionality - it needs to build built up from the low-level API in git2r/libgit2

Then tar_read/load_version() would be bonuses, and targets would be a suggested dependency.

[wlandau]

This all sounds like a nice reimagining of what maybe gittargets should have been.

As for external files: recent versions of targets create a _targets/user/ folder for user data (e.g. that gittargets users might want to use). Maybe that could help with format = "file" targets that you want to put under version control?

[noamross]

Hmm. This will work as-is with _targets/user/ (which is also a nice solution for things like git-lfs, as well). But I think there are lots of cases where things are some kind of code-data hybrid (e.g., Rmd files) where people are likely to want to store things in folders like data or outputs. (In my own case the _targets.R file is itself usually a target, so that the result of tar_mermaid() is updated into the README when the pipeline is changed!).

[wlandau]

On reflection: how important is it to expand the scope beyond cloud-based targets? Local files can be large and unpredictable, and even Git LFS is not powerful enough to really be a data tool in many situations. A cloud-only approach could have nice clear boundaries, met expectations, and low/predictable overhead.

[noamross]

Cloud-based targets are definitely my most common use case and necessary for a certain scale, but I've had small projects where I save intermediate RDS files and outputs like images and rendered reports and they have worked fine and would work in targets as well. I do think many things fall in the code/data hybrid space (e.g., CSV files in the repo that are edited by hand in Excel - I've had plenty of those!).

Probably more tooling to make "local" cloud stuff easier with a tool like MinIO is a practical way to make versioning work out of the box for individual users. For that it would be useful to have some "run before/after the pipeline" hooks. (They may also be useful for things like #1177) . I recently found that https://github.com/nektos/act, which I use for running GH actions locally, does something similar, launching a server in the background of the build process to emulate GH's cache and artifact storage.

The git helper package just starting to come together here: https://github.com/ecohealthalliance/relic/ . So far it is is mostly functions for plucking files from git history. I'm just starting the targets-specific stuff.