Filesystem optimization #31

villarrealas · 2019-03-11T14:14:48Z

This PR sets up scripts and adds them to the Parsl workflow in order to:

Check a list of files to make sure they were transferred and, if not, transfer them and unpack them.
At the end of a run, give an automatic script that will transfer completed files to longterm storage - can also be run manually.

…nsfer/untar/clean as necessary

…ng script

villarrealas · 2019-03-12T20:00:47Z

Fixed this so it now operates on Cori as desired.

benclifford

talked through a bunch of this with @villarrealas - seems ok

Antonio Villarreal and others added 10 commits February 19, 2019 09:27

added script to validate worklist

75e5db3

adjusted and prototyped in rsync

8e6623c

prototype code to check an existing worklist of tarball paths and tra…

60cce44

…nsfer/untar/clean as necessary

added in decompressing and removal functionality

7d264f7

prototype code in order to move files to archival

c8d1b96

added new bash apps

5609823

changed name and inputs to be more reflective of transfer needs

4d728be

fixed parsl bash app for validate-transfer appropriately

a56e0e8

added in some code for the validate transfer as part of the overarchi…

e89d9e3

…ng script

cleaned up the archive and added it to the workflow

2611561

villarrealas requested a review from benclifford March 11, 2019 14:14

removed validate transfer to be part of a pre-run operation script

c3acfec

benclifford approved these changes Mar 14, 2019

View reviewed changes

benclifford merged commit 60ee82c into master Mar 14, 2019

Provide feedback