Repository navigation
Best practices for reusing expensive upstream results #1085
Replies: 1 comment 5 replies
|
I recommend using a single signac workspace. You can use a dictionary key to distinguish whether a given directory is of type "p1" or "p2" (row has examples for this using "subproject"). When executing an action on "p2", you can read the corresponding "p1" job from the same workspace. row only allows preconditions as a function of a single job. Once you create a "p2" directory, there is no way to require that a corresponding "p1" exist before running the first action on it in When you run action A1, do you have a way to compute all the "p2" values that stem from it? If so, your A1 action could create all of the corresponding "p2" directories when it completes. That way, the first |
Uh oh!
There was an error while loading. Please reload this page.
Hello I’m evaluating signac + row for a research workflow and I’d appreciate guidance on whether this tool is appropriate for my needs and on best practices to avoid unnecessary recomputation of expensive steps. I really like signac’s data model and row’s action dependency model, and I’m hoping to use them to manage large parameter explorations. However, I have not found in the documentation a clear explanation on how to address my workflow.
I need to run a multi-stage pipeline composed of several expensive calculations. Upstream calculations are independent of downstream calculations, thus from the result of one upstream calculation I can run multiple downstream ones with different parameters.
In my current understanding the normal way to store data with signac is through a job which encodes all the actions row will execute. This leads me to believe that the expected way of working with signac would be to encode in the state point the parameters of upstream and downstream actions. This would then lead to recompute deterministic upstream actions even when I'm just changing downstream parameters.
If I have A1 and A2 in which A2 depends on output from A1 and encode my sp as
(p1, p2)for every value of p2 I will have to recompute A1(p1). Moreover, I will have to store the output of A1 in duplicates.As far as I understand it using multiple projects could be used to address this by setting up two separate projects one for A1 and one for A2 and in A2 to somehow reference jobs from A1 and retrieve results from there.
I read the signac and row documentation but did not find a clear recommended methodology for this kind of pattern. Could you please let me know if this kind of setup is possible with signac + row and what would be the best practices to set it up?
Thank you
All reactions