Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add etc/desi.json validation tools #4

Closed
sbailey opened this issue Jul 26, 2017 · 1 comment · Fixed by weaverba137/hpsspy#2
Closed

add etc/desi.json validation tools #4

sbailey opened this issue Jul 26, 2017 · 1 comment · Fixed by weaverba137/hpsspy#2
Assignees
Labels
enhancement New feature or request

Comments

@sbailey
Copy link
Contributor

sbailey commented Jul 26, 2017

Add tools for validating the backup specifications in etc/desi.json, in particular:

  1. do any regexes not match any files?
  2. are any files in the directory tree not covered by a regex and not in the exclude list?
  3. are any files covered by more than one regex?
  4. would any tar specification result in a tarball > 1TB?
  5. a --manifest option (or some equivalent name) that would generate a manifest of what files would go into what tarballs. e.g. perhaps this would point to a directory and it would create fake *.tar files in the same hierarchy and naming that would end up on HPSS, but the contents of those files would be the listing of file paths rather than the contents of the files themselves (like tar -tf blat.tar would report). This would allow someone to do a dry run and check if the results are intended before actually trying to send TB of data to HPSS.

Note that this validation would need to be run at NERSC since it needs access to the actual files; this isn't just the json syntax check of the travis test.

@weaverba137 weaverba137 self-assigned this Jul 27, 2017
@weaverba137 weaverba137 added the enhancement New feature or request label Jul 27, 2017
@weaverba137
Copy link
Member

For my own notes, I think item 5 already exists, in the form of an intermediate data file used to actually create the hsi/htar commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants