Allow relative paths in Sipnet run script #3418
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Modifies the Sipnet job.sh template to make it compute the absolute paths of OUTDIR, RUNDIR, SITE_MET, and BINARY at runtime, thus allowing paths in pecan.xml to be relative to the workflow's run directory and thus potentially host-agnostic.
Paths specified as absolute will continue to work as before.
I also went ahead and implemented path expansion for $BINARY while I was at it -- even if most people keep specifying this as an absolute path, it may be useful for side-by-side testing of different Sipnet builds on the same system before deployment.
Motivation and Context
The existing PEcAn standard practice is to populate machine-specific absolute paths in the XML file, either directly in text or by specifying database identifiers that are then looked up in BETYdb and expanded to absolute paths during the settings check process. This is all well and good when all files can be assumed tracked in BETYdb, but becomes annoying in db-free runs, and it means one XML file cannot be reliably reused across hosts.
A popular* alternate design is to use portable project directories: If all files needed for a run live in the same directory and all paths are expressed relative to the root of that directory, then paths stay valid when that directory is copied as a unit, either within or between machines. This PR attempts to support portable directories without breaking workflows that specify absolute paths -- if you see a case I missed that will change absolute paths, please flag it!
A particular motivating use case for me: I often create a project directory on my machine and then test-run it on multiple systems by mounting the same folder into different Docker images, meaning the same file might live in quick succession at
/Users/chrisb/projects/foo/met/bar.clim,/work/met/bar.clim, and/projectnb/dietzelab/chrisb/foo/met/bar.clim. Sure, I could write another script to edit the XML each time... or I could write it as./met/bar.climand always run the workflow from the directory that contains the XML.*(Yes, by "a popular" I mean "my preferred." Guilty as charged.)
Review Time Estimate
Types of changes
Checklist: