Daily nightwatch transfer #21
Conversation
There was a problem hiding this comment.
Looks good. I especially like the automated alternative if $CSCRATCH isn't available. Confirming: does any big data flow through that, or only logs and/or small temporary files? i.e. do we need to worry about the small $HOME quota? If anything big (like actual data files) flows through that, consider using a CFS dir as the scratch alternative instead.
|
Yes, the scratch directory is only used for setting up temporary files used by the morning "catch-up" transfers and by the HPSS backups. No actual data lands there. |
|
@sbailey, while trying to do a full re-sync of spectro/nightwatch/kpno, I noticed a lot of files are now missing from KPNO. Is that because at some point you went from individual spectrograph files, to files containing data from all spectrographs? Or did entire classes of files become obsolete? For example, 20200308/00054444/qfiberflat-b0-00054444.fits is gone. |
|
Ah, yes, I removed additional semi-optional files from KPNO to clean up disk space for new data. This will be an ongoing "feature" of nightwatch: occasionally removing a subset of files while leaving the most important ones. Eventually Klaus is going to do that to the raw data too, removing directories that previously were there. Could you configure the NERSC rsync to retain whatever files it had previously successfully synced, without generating N>>1 false alerts about files that disappear on the KPNO side? If you let me know when you do the daily sync, I can coordinate that any cleanup won't be happening at that time so that we don't remove files out from under you during a sync. |
|
The current default for the daily transfer is to not delete anything on the NERSC side (i.e. The daily transfer is initiated at 12:00 MST. |
|
Daily nightwatch transfers are working fine, & I want to get some other minor changes into the tag, so I'll merge this as soon as tests pass. |
This PR fixes #20.
In addition, adds an
ensure_scratch()function so that if${CSCRATCH}is unavailable, an alternate temporary directory can be chosen.I'm still testing the new rsync commands and exclude files, but it looks good so far.