File Moving Utility

[ryanlong1004]

Scope

https://jira-epic.woc.noaa.gov/browse/UW-116

The File Handler tool is meant to be a tool that will provide a generic interface to any other workflow tool that may need to move files from one location to another.

It will need to be provided with parameterized information like the protocol, the source, and the destination. Depending on the protocol, it may need additional arguments (e.g. HPSS files have a path on tape, and path inside an archive file, and those archive files may be zip or tar files).

The File Handler Tool should not have to deal with any formatted strings (no need for it to be aware of datetime or ensemble member, for example), but should (probably?) incorporate the use of wildcards for moving multiple similar files simultaneously, as this will improve performance as opposed to calling the Python tool once per file. We may want to consider being able to pass a list of files (data files, not archive files) for the HPSS option because it would be a huge waste of resources to pull an entire archive every time we want to grab a data file from it.

Appropriate logging and error handling should be incorporated.

Design

1. What design pattern makes sense for this tool?
2. Will it automatically use a hard link by default when it can?
3. Where does it live? Probably src/workflow_tools/file_mover?
4. Do we want checksums?
5. Do we want a lightweight option to ping remote data sources to make sure they exist before we try to transfer them? This could be useful for enabling workflow dependencies.

[ryanlong1004]

• Since we'll already be in the interpreter, dropping to OS offers negligible (if any) speed increases while adding another layer of complexity.
• This will likely end up as async (or better threads if it makes sense), so more the reason to stay in the interpreter.

It sounds like he has a script that's locating particular files and moving them. In that case (since he's already in python) shutil.move(stuff) is cleaner & safer to write than os.system('mv stuff'); Once you're already running python interpreter, the difference is moot since shutil.move just calls the system's move.

Source

[aerorahul]

An example use case for a "local copy" would be for the utility to take in a yaml such as:
https://github.com/NOAA-EMC/ufs-workflow/blob/develop/ewok/stage_files.yaml.

The utility in this particular use case is responsible for copying the files (the filename contains the source), to the destination (again destination contains the path). The paths can be absolute or relative.

The utility is also creating "empty" directories, if desired.

There is parsing going on ${} and $() when the yaml file is being loaded and configured.
The syntax for ${} follows substituting variables from the environment and $() follows substituting key values at runtime.
The parsing of yaml is part of another ticket, but its relevance here is to point out the use case.

The example here is from an EWOK based unified workflow using SOLO from JCSDA.