File Moving Utility #58
Replies: 2 comments
-
|
Beta Was this translation helpful? Give feedback.
-
An example use case for a "local copy" would be for the utility to take in a The utility in this particular use case is responsible for copying the files (the filename contains the source), to the destination (again destination contains the path). The paths can be absolute or relative. The utility is also creating "empty" directories, if desired. There is parsing going on The example here is from an EWOK based unified workflow using SOLO from JCSDA. |
Beta Was this translation helpful? Give feedback.
-
Scope
https://jira-epic.woc.noaa.gov/browse/UW-116
The File Handler tool is meant to be a tool that will provide a generic interface to any other workflow tool that may need to move files from one location to another.
It will need to be provided with parameterized information like the protocol, the source, and the destination. Depending on the protocol, it may need additional arguments (e.g. HPSS files have a path on tape, and path inside an archive file, and those archive files may be zip or tar files).
The File Handler Tool should not have to deal with any formatted strings (no need for it to be aware of datetime or ensemble member, for example), but should (probably?) incorporate the use of wildcards for moving multiple similar files simultaneously, as this will improve performance as opposed to calling the Python tool once per file. We may want to consider being able to pass a list of files (data files, not archive files) for the HPSS option because it would be a huge waste of resources to pull an entire archive every time we want to grab a data file from it.
Appropriate logging and error handling should be incorporated.
Design
Beta Was this translation helpful? Give feedback.
All reactions