You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add tools for validating the backup specifications in etc/desi.json, in particular:
do any regexes not match any files?
are any files in the directory tree not covered by a regex and not in the exclude list?
are any files covered by more than one regex?
would any tar specification result in a tarball > 1TB?
a --manifest option (or some equivalent name) that would generate a manifest of what files would go into what tarballs. e.g. perhaps this would point to a directory and it would create fake *.tar files in the same hierarchy and naming that would end up on HPSS, but the contents of those files would be the listing of file paths rather than the contents of the files themselves (like tar -tf blat.tar would report). This would allow someone to do a dry run and check if the results are intended before actually trying to send TB of data to HPSS.
Note that this validation would need to be run at NERSC since it needs access to the actual files; this isn't just the json syntax check of the travis test.
The text was updated successfully, but these errors were encountered:
Add tools for validating the backup specifications in
etc/desi.json
, in particular:--manifest
option (or some equivalent name) that would generate a manifest of what files would go into what tarballs. e.g. perhaps this would point to a directory and it would create fake *.tar files in the same hierarchy and naming that would end up on HPSS, but the contents of those files would be the listing of file paths rather than the contents of the files themselves (liketar -tf blat.tar
would report). This would allow someone to do a dry run and check if the results are intended before actually trying to send TB of data to HPSS.Note that this validation would need to be run at NERSC since it needs access to the actual files; this isn't just the json syntax check of the travis test.
The text was updated successfully, but these errors were encountered: