Desi pipe sync spectra

[julienguy]

Adding option desi_pipe sync --force-spec-done to change the state of spectra and redshift tasks if spectra files exist even in case there is no cframe files on disk. This is useful to redo a redshift run starting from a pre-existing prod.

[tskisner]

Just trying to understand the motivation a little more... So some cframes exist, the spectra file is generated, then the cframes are deleted. Normally this would mark the spectra file as not done, since one of its inputs is missing.

In your case, what happened to the cframes? Or are you starting from an empty production and manually copying in the spectra files from elsewhere?

If we are supporting the general case of "starting in the middle" of a production (for example, copying just the cframe files into place), then maybe we should have a general option to desi_pipe sync which says something like, "Only look at the files and not their dependencies". So if any file exists it is marked as done, otherwise as waiting, with no dependency checking.

Thoughts?

[julienguy]

• "Or are you starting from an empty production and manually copying in the spectra files from elsewhere?"
  Yes. This is to test/debug redrock.
• It's a good point that we may want to have this feature more generic.

[tskisner]

Ok, I don’t want to hold up your current work, but maybe in the future we could change this to a “—no-deps “ or other generic option that applies to all objects and simply uses file existence to choose between done or waiting. No urgency though.

…

On Aug 29, 2018, at 3:31 PM, julienguy ***@***.***> wrote: "Or are you starting from an empty production and manually copying in the spectra files from elsewhere?" Yes. This is to test/debug redrock. It's a good point that we may want to have this feature more generic. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.