Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

discrepancy between specsim.simulator and CMX spectra #110

Open
changhoonhahn opened this issue Apr 17, 2020 · 12 comments
Open

discrepancy between specsim.simulator and CMX spectra #110

changhoonhahn opened this issue Apr 17, 2020 · 12 comments
Assignees

Comments

@changhoonhahn
Copy link

I noticed that the CMX spectra from mini-SV2 and SV0 were noisier than the simulated spectra I was getting from a BGS version of specsim. As a test, in this jupyter notebook I compare a bright BGS galaxy spectrum from CMX to a simulated spectrum constructed using specsim.simulator.Simulator.

image

To reduce the number of moving parts, I smooth the CMX spectrum and use it as the source flux (black dashed). I also use the actual sky surface brightness from CMX for the surface brightness [^1]. The simulated spectrum is substantially less noisy and has much lower inverse variances.

tl;dr: The spectra I get from specsim.simulator.Simulator is substantially less noisy than observed spectra from CMX.

[^1] @dkirkby pointed out that the way I correct for flux calibration in the raw sky data overestimates the sky flux because it corrects for fiberloss. I didn't correct this since a fainter sky would make the problem worse.

@julienguy
Copy link
Contributor

Interesting. A possibility is that the particular fiber you are looking at is on average not well positioned on its target. Can you compare the value of FIBERFLUX_R in the fibermap with the integral of the calibrated spectrum?

@dkirkby dkirkby self-assigned this Apr 20, 2020
@dkirkby
Copy link
Member

dkirkby commented Apr 20, 2020

Thanks for documenting this nice test @changhoonhahn. I am looking into this now...

@changhoonhahn
Copy link
Author

@julienguy @dkirkby I found the issue for all the galaxies spectra that I compared to so I don't think it's caused by a particular fiber.

I've updated the notebook to print out the specific fiber number and also compared a second galaxy spectrum.

@dkirkby
Copy link
Member

dkirkby commented May 12, 2020

I think I have finally gotten to the bottom of this. The discrepancy you found is because specsim assumes a nominal transparency and fiber acceptance fraction that is significantly higher than actually achieved in this exposure. As a result, the relative contribution of the signal shot noise is much larger than specsim predicts.

Here is a summary plot for the TARGETID 35185736613364265 on EXPID 52113 of TILEID 70502 from 20200225 that you studied:
ffrac_20200225_70502_52113_35185736613364265

The cyan curve shows the predicted signal (in flatfielded electrons / bin) for each camera, assuming 100% transparency and fiber fraction. The actual (smoothed) signal + sky is shown as the blue / red / black curves. Comparing these, the actual r-band TRANSP x FFRAC is only 4 %. The PSF-like sources in this exposure have r-band TRANSP x FFRAC ~ 13%, so the large size of this galaxy is playing a role here too.

@changhoonhahn
Copy link
Author

changhoonhahn commented May 13, 2020

Hmm, I didn't realize transparency would be such a large effect!

Regarding the fiber acceptance fraction, in my example, I used the smoothed spectra as the source. In that case, FFRAC should not play a role since the source signal is already scaled down --- right?

To account for transparency, should I be scaling my exposure times by a factor of (transp/1)^2 as you did for your depth calculations?

@dkirkby
Copy link
Member

dkirkby commented May 13, 2020

FFRAC still plays a role since the CFRAME spectrum is corrected for the actual fiber acceptance measured with stars in this exposure. Specsim then applies the nominal fiber acceptance for 1.1" seeing, which could be more or less than the actual value. The bigger effect here is probably the large source size, which decreases the FFRAC for his target relative to a PSF-like source.

@changhoonhahn
Copy link
Author

changhoonhahn commented May 15, 2020

I updated the comparison to include TRANSP and FFRAC, estimated from the GFA data, in the fiber_acceptance_fraction of specsim.Simulator.simulate. Also, instead of the incorrectly flux calibrated sky, I included sky brightness derived from converting the throughput corrected sky data.

Now specsim produces spectra much closer to the actual CMX observations:
image

Still, the inverse variance of the specsim spectrum doesn't agree very well for r and z spectrograph:
image

I suspect the throughput corrected sky brightness (blue), which by eye looks like it is overestimated in r and z:
image
Nominal dark sky included for comparison.

@dkirkby I derived the sky brightnesses as (see notebook for details):

(sky surface brightness) = (sky data)/(throughput)/(photons per bin)/(exp. time)/(fiber area)

Is this consistent with how you got the sky brightness in the comparison you had in your survey planning talk during the sci-com telecon?

@dkirkby
Copy link
Member

dkirkby commented May 16, 2020

I don't see anything with units of energy in your formula. Here is the code I use:

def convert_sky_rate(inc):
    # The incident rate is in phot/sec per 0.8A wavelength bin.
    # Convert to surface brightness units (sbunit) suitable as input to specsim.
    wave = inc.wave * u.Angstrom
    dwave = np.gradient(wave)
    ephot = (h * c / wave / u.ph).to(u.erg / u.ph)
    etendue = (DESI.instrument.effective_area * DESI.fiber_area.mean()).to(u.cm ** 2 * u.arcsec ** 2)
    # Calculate surface brightness on the 0.8A reduction grid.
    surfbrightness = ((inc.flux * u.ph / u.s) / dwave * ephot / etendue).to(sbunit)
    # Upsample to 0.1A specsim simulation grid. This means that resolution will be reapplied by specsim, but
    # should at least give the correct normalization and smoothed shape. Because we are upsampling a density,
    # no rescaling for the new bin size is required. The result will be padded to the wider simulation grid
    # by repeating the first and last values.
    return np.interp(simwave, inc.wave, surfbrightness.data) * sbunit

The input inc has attributes wave and flux with throughput corrections already applied (and coadded over the cameras). This gives the following sky surface brightness suitable for input to specsim:
surfb

@changhoonhahn
Copy link
Author

@dkirkby Thanks! Turns out it was a bug in my code with the wavelength binning.

I reran the test with the updated sky surface brightness and now there's better agreement between the specsim output and the CMX data overall:
image
image

There's still some discrepancy, specsim slightly underestimates the noise for the b spectrograph and overestimates the noise for the z spectrograph. I'm not sure if discrepancies on these levels are worth the scrutiny.

@dkirkby
Copy link
Member

dkirkby commented May 19, 2020

Thanks for the update @changhoonhahn. We should still be able to achieve better agreement than this, at least for the FRAME ivar (which should dominate and is what specsim actually calculates), but is this good enough for your studies?

@changhoonhahn
Copy link
Author

Here's a comparison with the FRAME ivar:
download
There are still some significant discrepancies.

I'm currently quantifying how much this will impact redshift success rates, which is what I'm ultimately interested in estimating. Do you have any suggestions on what other factors may be causing this discrepancy?

@dkirkby
Copy link
Member

dkirkby commented May 21, 2020

Is this for the same bright TARGETID 35185736613364265? What are you assuming for the throughput (transparency x fiberfrac) for this target? This doesn't look right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants