emlinefit hangs on 82406/20211118/coadd-5-82406-thru20211118.fits

[sbailey]

@araichoor the following emlinefit command hangs (no errors, no crash, it just gets stuck doing something):

desi_emlinefit_afterburner \
  --coadd /global/cfs/cdirs/desi/spectro/redux/iron/tiles/cumulative/82406/20211118/coadd-5-82406-thru20211118.fits \
  --redrock /global/cfs/cdirs/desi/spectro/redux/iron/tiles/cumulative/82406/20211118/redrock-5-82406-thru20211118.fits \
  --output $SCRATCH/emline-5-82406-thru20211118.fits

The same occurs for spectrograph 6, but the other petals from that tile work fine.

This causes this ztile job to timeout with those two missing emline files. Please check and see if you can implement a fix that wouldn't change the answer for anything that has already been run.

[araichoor]

investigation report:

I guess the reason is because it s a tile with insane ebv.. it s a specialbackup test tile at galactic b=-2...
in short:

• in emlinefit, the flux is corrected for Galactic extinction, here:
  

  desispec/py/desispec/io/emlinefit.py

  Lines 239 to 246 in e8c541d

  	if mwext_corr:
  	# AR TBD: use a smarter way with no loop on tids...
  	# AR TBD: but as 500 rows at most, ~ok
  	tmpexts = ext_odonnell(tmpw, Rv=rv)
  	for i in range(nspec):
  	tmp_mw_trans = 10 ** (-0.4 * ebvs[i] * rv * tmpexts)
  	tmpfl[i, :] /= tmp_mw_trans
  	tmpiv[i, :] *= tmp_mw_trans ** 2

• some fibers have flux > 1e4, so it shoots up to 1e6-1e7 after correction, likely being the reason for the code hanging.

a possibility is to simply remove that tile :)
if we want a code fix for that, nulling the ivar for insane fluxes works.
i.e. add at the end of this loop the following lines

                # AR nulling ivar for insane flux
                sel = tmpfl[i, :] > 1e6
                if sel.sum() > 0:                                                                                                                                                                   
                    tmpiv[i, sel] = 0

we could add a condition to perform this test only if TILEID=82406, in which case we would be 100% certain that we do not change any computation for other tiles.

few further infos/details:

here is the position of this tile in Galactic coordinates (displaying all iron tiles):

(and here the tile design: https://desi.lbl.gov/trac/wiki/SurveyOps/TileDesigns#a82399-82411-BACKUPprogramtests)

I looked at the first row where it hangs, i.e. TARGETID=2305843012414221480 (index 103; it s a Gaia/G2 star of mag ~15).
note that it also hangs for other rows.

the code hangs here when trying to fit the OII line in the 3687.2 A < lambda_obs < 3769.6 A region:

desispec/py/desispec/emlinefit.py

Lines 218 to 227 in e8c541d

	popt, pcov = curve_fit(
	myfunc,
	waves[keep_line],
	fluxes[keep_line],
	p0=p0,
	sigma=1. / np.sqrt(ivars[keep_line]),
	maxfev=10000000,
	gtol=1.49012e-8,
	bounds=bounds,
	)

here is the flux for that spectrum (with highlighting in light grey the region used for the fit):

the flux there can be of ~1e4 1e-17 ergs/s/cm2/A; and, when corrected with the super-high extinction factor, it shoots up to ~1e6-1e7 ergs/s/cm2/A.

[moustakas]

I agree it's a pretty extreme position in the sky, but---FWIW---FastSpecFit doesn't crash on this tile, so I'm not sure throwing it away completely is the right choice.

[sbailey]

Accepting 2 missing afterburner files is indeed better than discarding the entire tile, even if it was a backup program test tile :)

Although the flux numbers are big, they don't seem big enough to run into overflow issues that would actually cause a fit failure. I think there is something else making this a pathological case.

[sbailey]

fixed by PR #2195