generate templates at given/input redshifts

[moustakas]

This isn't ready to merge, but I'd appreciate eyes on some of the core code before I go and break/change everything else. The main point of this PR is to address #104, i.e., enabling templates to be generated at a given set of input redshifts, which we need for our full-survey data challenge. This turned out to be a harder problem than I had first anticipated, and required a complete rewrite of the Class for each spectral class; so far I've only rewritten the desisim.templates.ELG() Class. For example, there's some chunking and vectorized calls to the speclite filter functions as kindly suggested by @dkirkby to speed things up.

If the user passes a redshift for which none of the templates pass the nominal color-cuts then a warning is issued and the output spectra and metadata table rows for those objects are simply returned empty; maybe we want this to be a more dramatic exception. I'm not sure, but @sbailey probably has an opinion.

However, in this situation it takes quite a long time (many 10s of seconds) to consider all ~8000 templates at the desired redshift, which is too long. For example, if you ask for 1000 templates at z=(0.1-0.4) which also pass the DESI ELG color-cuts then you better go for a long, long run instead of waiting around to be told that none of the templates were generated (i.e., not possible). I note that there's no clean, fast way of knowing ahead of time which templates will work out; you have to do the math (or pre-assign templates to redshifts, e.g., in the mocks, which is something we should pursue).

I plan on addressing this slowness issue by creating a more streamlined set of ELG basis templates. In the current set there's considerable overlap of the templates in color-color and physical parameter space, so we probably don't need more than a few hundred templates.

I still need to add a handful of unit tests, so stand by. And this PR also fixes #129 (for ELGs at least).

[moustakas]

After many struggles with get_targets, tests are finally passing.

[sbailey]

Odd. The last test run failed with:

======================================================================
FAIL: test_newexp (desisim.test.test_obs.TestObs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/desihub/desisim/py/desisim/test/test_obs.py", line 82, in test_newexp
    self.assertTrue(maxflux > 0.1, 'suspiciously low {} flux ({}); wrong units?'.format(objtype, maxflux))
AssertionError: suspiciously low LRG flux (0.0); wrong units?

even though the last commit was just for documentation and adding two additional test assertions in a different test, and the previous test run had passed. I've run test_obs 7 times in a row on my laptop without failure, but restarting it on Travis failed again in the same way. The test is intended to catch cases where the input templates changed units (which has happened) but perhaps this is clumsy. I'll submit some extra debugging printouts to see why this is failing on Travis but not my laptop...

[moustakas]

I spent about an hour debugging this last night but it ended up being a problem in targets.get_targets (which obs.new_exposure calls) whereby template spectra (in my case ELGs) were being assigned to (empty) sky spectra.

I don't know why the failure isn't reproducible but it's probably related to the fact that we're not fixing the random seed in test_obs (and we're only generating a few test spectra), so it's only failing only some of the time. This is bad, of course.

I'm making 500 LRG spectra (using bin/simulate-templates) to see if I can find any that are empty/zeros.

[sbailey]

Comparing typical maxflux values, this test was too close to threshold. I loosened it. Merge conflict is due to doc/changes.rst . I will resolve that my hand and merge.

[sbailey]

Ah, I didn't see @moustakas comment prior to merge. If you can reproduce this, please open new PR fix based on a branch off of master.

[moustakas]

How's the timing with the new code? I haven't profiled it but it'd be great to know/check whether the bottleneck is still in the velocity dispersion convolution. I think @dkirkby had checked this previously.

[weaverba137]

Note that the branch hasn't been deleted from GitHub.

[sbailey]

It takes about a second per spectrum, which is a bit problematic but not a deal breaker (14k CPU hours to simulate 50M DESI spectra...). As you noted separately, getting this correct was a good first step; making it faster would be helpful but can be treated as a separate issue.

It would also be helpful to make the tests faster, which could be done just by checking multiple things per template set generation instead of re-generating templates multiple times just to check one thing each time.

I'll delete the branch that Ben noticed was still there.