restore pipeline operations on KNL

[sbailey]

This PR has a potpourri of updates that I made while trying to restore the ability of the pipeline to run efficiently on KNL. Most of them turned out to be unrelated, but I'm opening this PR to re-establish a working baseline for further updates. Changes include:

• dashboard outputs to $DESI_SPECTRO_REDUX/$SPECPROD/dashboard by default instead of $HOME
• more user friendly time logging in desi_proc output:
  • instead of dumping full json into the log, log only the duration.max per step (details are still in the *.json files)
  • include the total duration as the last line in the log
• parallelize over cameras while merging the PSFs in desi_proc_joint_fit
  • The previous code was asking for 200 cores but apparently only using 1; now it asks for 30 ranks and uses all of them.
  • There has been some churn of how many ranks this step asks for, so I highlight this for attention to make sure I'm not misunderstanding the reason for earlier changes
• add srun --cpu-bind=cores option for steps that are pure MPI (extraction, redrock), and --cpu-bind=none for steps that use multiprocessing (fluxcalib, spectra regrouping). See freeze_iers problematically slow on KNL desiutil#180 for context
• when generating the desi_proc batch script, derive from the requested command line whether --nightlybias applies instead of passing a separate parameter through the function calls that redundantly tracks that info
• More specex logging, e.g. log the command that will be run, not just log the commands that fail
• Keep OMP_NUM_THREADS=(available threads per rank) for PSF fits (where we have profiled that helps), but go back to OMP_NUM_THREADS=1 for other steps.
  • This was motivated by OMP_NUM_THREADS impact on extractions #1515, though in the end my concern about OMP_NUM_THEADS>1 may have been a red herring. Regardless, I've kept here was emperically works in test runs, and then we can separately test whether re-increasing OMP_NUM_THREADS helps extractions or not.

I've tested this branch for dark+nightlybias, arc, flat, science, and redshifts in /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f2k (still running, and includes some NERSC-outage inflicted failures, but basically working).

@akremin

For the record: my biggest problem with testing turned out to be missing KMP_AFFINITY=false which is needed for using MPI+numpy on KNL. Other pieces that may have helped are being more explicit about --cpu-bind and adding the PSF merging parallelism.

[coveralls]

Coverage decreased (-0.01%) to 25.676% when pulling 74cbc93 on f2knl into 2de7942 on master.

[akremin]

A useful set of potpourri. I left two comments, one giving full context for the dashboard change to verifying our mutual thoughts and another checking if all the srun's are accounted for.

Note: I didn't do any trial runs of the code or verify that the appropriate cpu-bind is used in each circumstance.

[akremin]

Line 567 above is another srun where you haven't added --cpu-bind. It pertains to arcs and flats. Was that intentional?

[sbailey]

Full disclosure: I wasn't sure what the right answer was in that case and not specifying it was working so I left it alone. We can revisit that one later.