Error while running redrock in mock spectra

[LuzGarciaP]

When trying to run redrock on a small set of mocks located here: /global/cscratch1/sd/alxogm/BAL_mocks

I'm getting this error:
Traceback (most recent call last):
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/code/redrock/master/bin/rrdesi", line 9, in
desi.rrdesi(comm=None)
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/code/redrock/master/py/redrock/external/desi.py", line 804, in rrdesi
raise err
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/code/redrock/master/py/redrock/external/desi.py", line 695, in rrdesi
targets = DistTargetsDESI(args.infiles, coadd=(not args.allspec),
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/code/redrock/master/py/redrock/external/desi.py", line 219, in init
scores = encode_table(Table(hdus["SCORES"].data,
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/conda/lib/python3.9/site-packages/astropy/io/fits/hdu/hdulist.py", line 323, in getitem
self._positive_index_of(key))
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/conda/lib/python3.9/site-packages/astropy/io/fits/hdu/hdulist.py", line 780, in _positive_index_of
index = self.index_of(key)
File "/global/common/software/desi/cori/desiconda/20211217-2.0.0/conda/lib/python3.9/site-packages/astropy/io/fits/hdu/hdulist.py", line 761, in index_of
raise KeyError(f'Extension {key!r} not found.')
KeyError: "Extension 'SCORES' not found."

Would it be possible to make SCORES an optional HDU again, please?

Best regards,
Luz García

[moustakas]

Thanks for the crash report @LuzGarciaP.

I'm looking at this issue in the scores-optional branch but I'm running into a related issue in that your mock spectra don't contain resolution information either, which is much more "baked" into redrock than the SCORES HDU.

I made a small change to make it further along into redrock without the resolution information but I am now crashing during the redshift scanning. E.g., with my branch

rrdesi -i /global/cscratch1/sd/alxogm/BAL_mocks/exp-1000/0/0/spectra-16-0.fits -n 3 -o zbest.fits --mp 1
Running with 1 processes
WARNING:  using multiprocessing, but the OMP_NUM_THREADS
WARNING:  environment variable is not set- your system may
WARNING:  be oversubscribed.
Loading targets...
Read and distribution of 3 targets: 5.8 seconds
DEBUG: Read templates from /global/common/software/desi/cori/desiconda/20211217-2.0.0/code/redrock-templates/master
DEBUG: Using default redshift range -0.0050-1.6997 for rrtemplate-galaxy.fits
DEBUG: Using default redshift range 0.0500-5.9934 for rrtemplate-qso.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-A.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-B.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-CV.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-F.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-G.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-K.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-M.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-WD.fits
Read and broadcast of 10 templates: 0.1 seconds
Rebinning templates: 33.5 seconds
Computing redshifts
  Scanning redshifts for template GALAXY
--- Process 0 raised an exception ---
Proc 0: Traceback (most recent call last):
Proc 0:   File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/external/desi.py", line 735, in rrdesi
    scandata, zfit = zfind(targets, dtemplates, mpprocs,
Proc 0:   File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/zfind.py", line 148, in zfind
    results = calc_zchi2_targets(targets, templates, mp_procs=mp_procs)
Proc 0:   File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/zscan.py", line 292, in calc_zchi2_targets
    tg.sharedmem_pack()
Proc 0:   File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/targets.py", line 237, in sharedmem_pack
    s.sharedmem_pack()
Proc 0:   File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/targets.py", line 59, in sharedmem_pack
    self._ndiag = self.R.data.shape[0]
Proc 0: AttributeError: 'NoneType' object has no attribute 'data'

Traceback (most recent call last):
  File "/global/homes/i/ioannis/code/desihub/redrock/bin/rrdesi", line 9, in <module>
    desi.rrdesi(comm=None)
  File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/external/desi.py", line 816, in rrdesi
    raise err
  File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/external/desi.py", line 735, in rrdesi
    scandata, zfit = zfind(targets, dtemplates, mpprocs,
  File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/zfind.py", line 148, in zfind
    results = calc_zchi2_targets(targets, templates, mp_procs=mp_procs)
  File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/zscan.py", line 292, in calc_zchi2_targets
    tg.sharedmem_pack()
  File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/targets.py", line 237, in sharedmem_pack
    s.sharedmem_pack()
  File "/global/homes/i/ioannis/code/desihub/redrock/py/redrock/targets.py", line 59, in sharedmem_pack
    self._ndiag = self.R.data.shape[0]
AttributeError: 'NoneType' object has no attribute 'data'

I suspect that for DESI-style input spectra (via the use of rrdesi) we should always expect the resolution matrix in the input spectra, but I'd prefer getting @sbailey's assessment before continuing further.

[alxogm]

If redrock expects to have the resolution matrix always present we can re-run the mock Luz is using to include it.
Update: I have actually re-ran the pixel that is being used for testing the fix for the original issue. Unfortunately rrdesi failed with the following issue, which again is due to a mismatch in the data model, I guess this can also be fixed for backward compatibility?

rrdesi -i /global/cscratch1/sd/alxogm/BAL_mocks/exp-1000/0/0/spectra-16-0.fits -n 3 -o zbest.fits --mp 1
Running with 1 processes
Loading targets...
Read and distribution of 3 targets: 8.4 seconds
DEBUG: Read templates from /global/common/software/desi/cori/desiconda/20211217-2.0.0/code/redrock-templates/master
DEBUG: Using default redshift range -0.0050-1.6997 for rrtemplate-galaxy.fits
DEBUG: Using default redshift range 0.0500-5.9934 for rrtemplate-qso.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-A.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-B.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-CV.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-F.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-G.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-K.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-M.fits
DEBUG: Using default redshift range -0.0020-0.0020 for rrtemplate-star-WD.fits
Read and broadcast of 10 templates: 0.2 seconds
Rebinning templates: 43.0 seconds
Computing redshifts
  Scanning redshifts for template GALAXY
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 8.1 seconds
  Scanning redshifts for template QSO
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 5.4 seconds
  Scanning redshifts for template STAR:::A
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.7 seconds
  Scanning redshifts for template STAR:::B
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.5 seconds
  Scanning redshifts for template STAR:::CV
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.5 seconds
  Scanning redshifts for template STAR:::F
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.6 seconds
  Scanning redshifts for template STAR:::G
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.6 seconds
  Scanning redshifts for template STAR:::K
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.6 seconds
  Scanning redshifts for template STAR:::M
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.6 seconds
  Scanning redshifts for template STAR:::WD
    Progress:   0 %
    Progress:  10 %
    Progress:  20 %
    Progress:  30 %
    Finished in: 0.6 seconds
  Finding best fits for template GALAXY
    Finished in: 3.1 seconds
  Finding best fits for template QSO
    Finished in: 1.4 seconds
  Finding best fits for template STAR:::A
    Finished in: 1.5 seconds
  Finding best fits for template STAR:::B
    Finished in: 1.4 seconds
  Finding best fits for template STAR:::CV
    Finished in: 1.2 seconds
  Finding best fits for template STAR:::F
    Finished in: 1.3 seconds
  Finding best fits for template STAR:::G
    Finished in: 1.4 seconds
  Finding best fits for template STAR:::K
    Finished in: 1.4 seconds
  Finding best fits for template STAR:::M
    Finished in: 1.5 seconds
  Finding best fits for template STAR:::WD
    Finished in: 1.5 seconds
Computing redshifts took: 34.2 seconds
--- Process 0 raised an exception ---
Proc 0: Traceback (most recent call last):
Proc 0:   File "/global/u1/a/alxogm/desi/code/redrock/py/redrock/external/desi.py", line 802, in rrdesi
    targets.tsnr2,
Proc 0: AttributeError: 'DistTargetsDESI' object has no attribute 'tsnr2'

Traceback (most recent call last):
  File "/global/u1/a/alxogm/desi/code/redrock/bin/rrdesi", line 9, in <module>
    desi.rrdesi(comm=None)
  File "/global/u1/a/alxogm/desi/code/redrock/py/redrock/external/desi.py", line 816, in rrdesi
    raise err
  File "/global/u1/a/alxogm/desi/code/redrock/py/redrock/external/desi.py", line 802, in rrdesi
    targets.tsnr2,
AttributeError: 'DistTargetsDESI' object has no attribute 'tsnr2'

[sbailey]

FWIW, I think rrdesi should not require SCORES or TNSR2 values, but it should require resolution matrix data and the sims should provide that. Since the full resolution matrix format is large and probably overkill for fast sims, a compromise could be to define a Gaussian sigma (or FWHM) vs. wavelength HDU, and update redrock to support converting that into a resolution matrix to use.

[moustakas]

@LuzGarciaP @alxogm, I'm going to close this ticket as resolved, but please feel free to reopen (or open a new one) if this is still an issue.