Improve DataSpaces usability and documentation #6
Conversation
FLEXPATH is the default if this option is omitted. This requires a different workflow to launch the server, and a utility script to check if the server is running. The user still needs to change heat_transfer.xml file.
Added instructions to edit xml file for DATASPACES
This should make documentation easier
…er debug statement.
kshitij-v-mehta
requested review from
j-woz,
ianfoster,
jychoi-hpc,
bd4 and
kshitij-v-mehta
|
This is a new experience for me. I'd like to try this, but I think that I need it integrated it into savanna@develop so I can pull it. Is that feasible? |
|
This looks fine to me. In the near future we will want to be able to use the same workflow.swift for both modes FLEXPATH and DATASPACES so that the science logic is reused. But for now this is great. |
dataspaces.conf
Outdated
| @@ -2,10 +2,10 @@ | |||
| ## Config file for DataSpaces | |||
|
|
|||
| ndim = 2 | |||
| dims = 600,600 | |||
| dims = 1600,1500 | |||
There was a problem hiding this comment.
@philip-davis - Could you elaborate a bit on what these mean and why you are modifying them? Thanks.
There was a problem hiding this comment.
@kshitij-v-mehta Thanks for catching these. They were artifacts from some earlier testing that I had left in. The dims values are the default global dimensions of the space. These must be larger than the dimensions being written by heat_transfer, but if they are too large the space will not be partitioned efficiently in the case of multiple servers. Optimal values powers of two (and the same for all dimensions), so I set them to 256,256 for the release.
|
It looks like one step is missing. After running DataSpaces server and before running heat transfer app and stager, "conf" should be read and export two env variables: P2TNID and P2TPID. In bash, we usually do as follows: --Then, run heat_transfer_adios2 and stage_write @philip-davis Do we need this step? Am I right? |
|
Thanks for looking at this @jychoi-hpc. Sourcing the conf isn't necessary in recent versions of DataSpaces. The clients read from the conf file directly now. |
|
I see. In my test run, it hangs without proceeding. I will check my run again. |
|
My fault. I forgot to change xml file. It works now. |
| stc -p -u -I $LAUNCH -r $LAUNCH workflow.swift | ||
| turbine -n $PROCS $MACHINE workflow.tic | ||
| stc -p -u -I $LAUNCH -r $LAUNCH workflow.swift | ||
| turbine -n $PROCS $MACHINE workflow.tic -s=${STAGING} |
There was a problem hiding this comment.
I know this is already merged, but what does -s do here? Tubine help page says we should be providing a settings.sh file here
| int dsproc = 1; | ||
|
|
||
| availproc = turbine_workers(); | ||
| rmethod = argv("s", "FLEXPATH"); |
There was a problem hiding this comment.
@kshitij-v-mehta The -s flag is being passed as an argument to workflow.tic, rather than as a turbine option. That allows run-workflow.sh to pass on the transport type. I had thought I had picked something that didn't stomp on any turbine flags, but apparantly not 😬. Perhaps a different letter would be less confusing? -t for transport maybe?
There was a problem hiding this comment.
Sorry, my stupidity, its obviously not a flag to turbine.
Modified run-workflow.sh and workflow.swift to launch correctly-configured dataspaces_server based on command line arguments. This is done by adding a second argument to the run-workflow.sh script, which is used to
Also, paramaterized process counts in workflow.swift using a set of int values at the top of the script to make it easier to change process count and arrangement.
It is now possible to switch between DataSpaces and FlexPath by just changing the second argument to run-workflow.sh and appropriately modifying heat_transfer.xml.
These steps have been documented as well.