Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create version/option in quickquasars to generate Lyman alpha mocks for eBOSS #437

Open
andreufont opened this issue Oct 18, 2018 · 71 comments

Comments

@andreufont
Copy link
Contributor

We would like to use quickquasars to generate Lyman alpha mocks for eBOSS. That would be useful for eBOSS, but it would also increase the number of eyes and hands on quickquasars and the London mocks.

@belaa already has a version of specsim that works on eBOSS specs, we just need to work on the rest of quickquasars (the easy part). Today I only started to think about it, but I plan to work on this tomorrow (wifi permitting).

The first thing we need to decided is: do we want --eBOSS to be an option of quickquasars, or do we want to have a separate script quickquasars-eBOSS.py ?

The advantage of the first would be that we don't have duplicated code, but it could potentially make our beloved DESI script a bit messier...

I believe the mess would be quite localized though:

  • Instead of using the DESI footprint it uses the eBOSS footprint function (that probably calls BOSS-eBOSS footprint and eBOSS footprint. We probably want the density in BOSS-only footprint to be around 15, and eBOSS footprint to be 25, or something like that.
  • We should have a somewhat different distribution of luminosities, based on the distribution of r-band magnitudes in eBOSS DR14Q. @alxogm is now adding code that can randomly choose magnitudes from a QLF file, so that should be work (we just need a different file).
  • quickquasars does not call specsim, it calls quickspectra, that calls specsim. So we should modify quickquasars to call specsim with the eBOSS configuration...

Ok, it might be more work than I thought. We probably want to have a different script.

@julienguy @moustakas @sbailey @dkirkby @ngbusca

@ngbusca
Copy link

ngbusca commented Oct 18, 2018

I vote for having a --eboss (all lowercase) option, rather than duplicating code. There's not reason for it to become messier just because of the extra option. What would require more thought is how to get the eboss+boss sky density, but I don't think we absolutely need this from the get-go as long as we have the right mean density.

For the geometric part of the footprint we will need external data (e.g. the list of boss+eboss tiles), (would we add that to a desi repository?) @kdawson1000 might know where to get those input files.

@kdawson1000
Copy link

Nicolas,
There are several ways that we can track the eBOSS footprint. I will try to find you in person so we can settle on the easiest way.

@julienguy
Copy link
Contributor

I would vote for

  • opening a new github repo under desihub or igmhub, something like desilya
  • moving there quickquasars
  • try and isolate the desi simulation part
  • add the --eboss option.

The reason is it would be problematic to force new version of desisim (which is the main DESI simulation repo) each time we make an improvement in the eboss sim.
But I agree a single quickquasars code used for both projects would be better.

@ngbusca
Copy link

ngbusca commented Oct 20, 2018

We worked quite a bit yesterday with @belaa it seems that the changes are quite minimal:

  • When instantiating the _simulator object in quickspectra (not quickquasars) we can add the choice of config, currently defaulting to desi
  • The rest of the work is totally on specsim, where we plan to add a dummy z camera with zero throughput and minimal wavelength coverage so desi doesn't complain about the lack of z (the r, b, z cameras are hard-coded in several places, which defies the desired properties of the general framework, but that's another discussion)

@andreufont
Copy link
Contributor Author

I agree it would be better to have a minimal change in desisim.

However, I agree with Julien that we should also have a Lyman alpha repo where we can play with freely (for instance, for mini-test that Julian is working on).

@andreufont
Copy link
Contributor Author

I started a branch to do that, but if someone else is already working on a different branch I can move the couple of lines there.

https://github.com/desihub/desisim/tree/eboss-quickquasars

@ngbusca
Copy link

ngbusca commented Oct 21, 2018

After taking a closer look at the desisim code, I've encountered a few more blocking points to the point that I think we should actually decouple from desisim altogether and interface directly with specsim. @belaa could you point us to an example on how given a noiseless, high resolution eboss spectrum we can obtain a noisy version of it using specsim? @dkirkby mentioned that this could be even done as a function of plate/mjd, is that the case?

@belaa
Copy link
Contributor

belaa commented Oct 22, 2018

@ngbusca Here is a notebook on my eboss_config branch that goes through how to simulate eBOSS spectra. As far as I know, I think you have to provide the fluxes for each source you're using either in the config file or as an argument to Simulator.simulate(), so I don't think it's as easy as just giving a plate/mjd - but maybe @dkirkby can confirm this.

@dkirkby
Copy link
Member

dkirkby commented Oct 22, 2018

@belaa is correct that the true spectrum needs to be specified for each simulated fiber.

@ngbusca: what properties for a given plate/mjd do you want to reproduce? (sky, read noise, wlen dispersion, targets, ...)

@ngbusca
Copy link

ngbusca commented Oct 23, 2018

@dkirkby @belaa to zeroth order we would like to get the right s/n, sky, etc. on coadds (spPlates) on average (I think we already have that, right?). But a more realistic way to do it could be:
a. for each simulated quasar get its RA,DEC and calculate the plate (or plates) were it would have been observed
b. assign to it random fibre (such that the fiber isn't already assigned to another quasar)
c. use the info available for this plate/mjd/fiberid (exptime, focal XY, moon, atmospheric depth, ...) to add read noise, sky, etc.

I'm still not sure how a. would interplay with simulating quasar reobservations (ideas?).

Would c. add a lot of overhead? would c. add correlated sky residuals? The zeroth order might be enough by the way.

@andreufont
Copy link
Contributor Author

Do we really need this, Nico? It sounds quite complicated, specially when taking into account reobservations (and how these targest were selected).

For now, I would work under the assumption that all spectra in DR14Q are random draws from the same SNR distribution, and that the only difference between eBOSS and BOSS_minus_eBOSS footprint is the density of high-z quasars. But it would be great to see these distributions of SNR in BOSS, in eBOSS, for the reobserved quasars...

@ngbusca
Copy link

ngbusca commented Oct 23, 2018

@andreufont I totally agree, that's my order zeroth above and we should definitely try it out and check the SNR, etc. before complicating things. I was just asking looking ahead, certainly not requesting anything new from specsim!

@dkirkby
Copy link
Member

dkirkby commented Oct 24, 2018

Here are some notes as I try to understand the layers of quickquasars. Corrections welcome.

The nested calling sequence is:

desisim.scripts.quickquasars.simulate_one_healpix
  -> desisim.scripts.quickspectra.sim_spectra
    -> desisim.simexp.simulate_spectra
      -> specsim.simulator.Simulator.simulate

At the lowest level, in desisim.simexp.simulate_spectra, specsim is used as follows:

    desi = desisim.specsim.get_simulator(config, num_fibers=nspec,
        camera_output=psfconvolve)
    ...
    desi.atmosphere.seeing_fwhm_ref = obsconditions['SEEING'] * u.arcsec
    desi.observation.exposure_time = obsconditions['EXPTIME'] * u.s
    desi.atmosphere.airmass = obsconditions['AIRMASS']
    desi.atmosphere.moon.moon_phase = np.arccos(2*obsconditions['MOONFRAC']-1)/np.pi
    desi.atmosphere.moon.moon_zenith = (90 - obsconditions['MOONALT']) * u.deg
    desi.atmosphere.moon.separation_angle = obsconditions['MOONSEP'] * u.deg
    ...
    desi.observation.exposure_start = astropy.time.Time(obsconditions['MJD'], format='mjd')
    ...
    desi.instrument.fiberloss_method = 'fastsim'
    ...
    desi.simulate(source_fluxes=flux, focal_positions=xy, source_types=source_types,
                  source_fraction=source_fraction,
                  source_half_light_radius=source_half_light_radius,
                  source_minor_major_axis_ratio=source_minor_major_axis_ratio,
                  source_position_angle=source_position_angle)

However, the following top-level call, from desisim.scripts.quickquasars.simulate_one_healpix, already contains most of the info needed by desi.simulate:

    sim_spectra(qso_wave,qso_flux, args.program, obsconditions=obsconditions,spectra_filename=ofilename,
                sourcetype="qso", skyerr=args.skyerr,ra=metadata["RA"],dec=metadata["DEC"],targetid=targetid,
                meta=specmeta,seed=seed,fibermap_columns=fibermap_columns,use_poisson=False)

What output files do you need for the eboss mode? Are you only simulating quasars, or also contaminants?

@ngbusca
Copy link

ngbusca commented Oct 24, 2018

I tried pushing as far as I could, and eventually I got an error claiming that the requested X,Y position of the fiber was outside the FOV of the telescope. I think this is related to positioning the fibers in the focal plane according to Mayall but trying to simulate them with SDSS, which is why I suggested to give up using desisim altogether...

@dkirkby
Copy link
Member

dkirkby commented Oct 24, 2018

Many of the steps being performed now are not relevant for quasar sims (e.g. setting moon conditions and galaxy size & shape). That's not necessarily a problem, unless it brings in a lot of hardcoded desi assumptions (like the field of view).

@andreufont
Copy link
Contributor Author

I think it's fair to say that we (Julien?) Copy pasted code to get things running, but if you can identify code that is not relevant feel free to remove it :-)

@andreufont
Copy link
Contributor Author

Nicolas,
There are several ways that we can track the eBOSS footprint. I will try to find you in person so we can settle on the easiest way.

@kdawson1000 - Any progress on this? A part from the issues with the noise simulator, we also need to address the footprint discussion. Could you provide a function / file that defines the footprint for BOSS and for eBOSS?

@kdawson1000
Copy link

I thought Nicolas had a solution to this by simply using the DR16Q catalog.

@andreufont
Copy link
Contributor Author

Yes, I forgot about this. @ngbusca , let me know if you would like to discuss this (on skype?) or whether you already have a good plan for how to proceed with the eBOSS mocks.

@londumas
Copy link
Contributor

londumas commented Nov 7, 2018

@andreufont, Could I be of any help to have this moving?

@belaa
Copy link
Contributor

belaa commented Nov 7, 2018

An update on my end: I got quickquasars to run on the eboss specsim config, although I'm not sure how to interpret the output. I get three FITS files:

spectra-16-333.fits truth-16-333.fits zbest-16-333.fits

I had to tweak some things in the specsim config file to get it to run which I'm trying to understand, but I was able to run it without any errors.

@londumas
Copy link
Contributor

londumas commented Nov 7, 2018

@belaa, I be happy to have a look at these files. Could you tell me where you ran quickquasar and where are the output files? Could you give me reading permissions?

@belaa
Copy link
Contributor

belaa commented Nov 7, 2018

@londumas I've been running it on my computer but have moved the output files here:

/global/cscratch1/sd/belaa/desi/quickquasar_output

I think you should have reading permissions but let me know if you can't access it for some reason.

@londumas
Copy link
Contributor

londumas commented Nov 7, 2018

@belaa, I don't have permissions. could you run fix_permissions.sh?

@belaa
Copy link
Contributor

belaa commented Nov 7, 2018

@londumas Ok I ran it and it should work now

@londumas
Copy link
Contributor

londumas commented Nov 7, 2018

@belaa, Could you make sure you ran on the first level of the directories? i.e on /global/cscratch1/sd/belaa/? I still don't have permissions.

@belaa
Copy link
Contributor

belaa commented Nov 7, 2018

@londumas Hi Helion, I'm sorry, does it work now?

@londumas
Copy link
Contributor

londumas commented Nov 7, 2018

@belaa, Yes it does. I will have a look at it.

@londumas
Copy link
Contributor

londumas commented Nov 8, 2018

@belaa, everything looks good for the moment.
I have plotted a spectrum, it looks good with only two spectrographs instead of three in DESI.
However, the flux doesn't go down to 3600 A. Do you know why?
The same as for the upper value, it doesn't go above 10,000 A

Could you run on way more (~200,000) so I can look at the footprint,
at the noise level, at the magnitude distribution...

@andreufont
Copy link
Contributor Author

Hi @londumas - there are two things that need to be done:

  • modify the code so that it can add eBOSS-like noise, that's what @belaa is working on.
  • modify the code so that it generates spectra only in the BOSS+eBOSS footprint, and with the right density of high-z quasars (z>2.1) per square degree (around 16 in the BOSS only area, 24 in the BOSS+eBOSS area). I started a branch to do the latter, but I never finished it. So you could take over this task if you had time :-)

@londumas
Copy link
Contributor

@andreufont, Perfect, thank you very much. Let me try this first today when I find the time.
Let's call if I fail to do it.

@londumas
Copy link
Contributor

@andreufont, I managed to run quickquasar.
Some transmission files did not work.
I wonder if it is not because of the error message at the end: BASS-MzLS-BASS-MzLS not a valid photo system. @julienguy do you know anything about that?
The test script at then end gives the same error message.

Data:
eboss_data

Mocks:
eboss_mocks

INFO:quickquasars.py:256:simulate_one_healpix: Select QSOs in BOSS+eBOSS footprint 2438 -> 0
WARNING:quickquasars.py:258:simulate_one_healpix: No intersection with BOSS+eBOSS footprint
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "<me>/Programs/imcgreer/simqso/simqso/sqphoto.py", line 316, in load_photo_map
    photSys = supported_photo_systems[photSysName][survey]
KeyError: 'BASS-MzLS'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/common/software/desi/edison/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/global/common/software/desi/edison/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "<me>/Programs/desi/code/desisim/py/desisim/scripts/quickquasars.py", line 629, in _func
    return simulate_one_healpix(**arg)
  File "<me>/Programs/desi/code/desisim/py/desisim/scripts/quickquasars.py", line 421, in simulate_one_healpix
    noresample=True, seed=seed, south=issouth)
  File "<me>/Programs/desi/code/desisim/py/desisim/templates.py", line 2725, in make_templates
    nocolorcuts=nocolorcuts, noresample=noresample, south=south)
  File "<me>/Programs/desi/code/desisim/py/desisim/templates.py", line 2493, in _make_simqso_templates
    qsometa.loadPhotoMap([('BASS-MzLS', 'BASS-MzLS'), ('WISE', 'AllWISE')])
  File "<me>/Programs/imcgreer/simqso/simqso/sqgrids.py", line 1000, in loadPhotoMap
    self.photoMap = sqphoto.load_photo_map(photoSys)
  File "<me>/Programs/imcgreer/simqso/simqso/sqphoto.py", line 319, in load_photo_map
    (photSysName,survey))
ValueError: BASS-MzLS-BASS-MzLS not a valid photo system
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<me>/Programs/desi//code//desisim/bin/quickquasars", line 9, in <module>
    sys.exit(main())
  File "<me>/Programs/desi/code/desisim/py/desisim/scripts/quickquasars.py", line 726, in main
    pool.map(_func, func_args)
  File "/global/common/software/desi/edison/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/global/common/software/desi/edison/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
ValueError: BASS-MzLS-BASS-MzLS not a valid photo system
srun: error: nid00033: task 0: Exited with exit code 1
srun: Terminating job step 11349230.20
#!/bin/bash

downsampling=0.40
idir=/project/projectdirs/desi/mocks/lya_forest/london/v4.0/
outdir=/project/projectdirs/desi/mocks/lya_forest/london/v4.0/eboss-0.0/test-0-0/

if [ ! -d $outdir ] ; then
    mkdir -p $outdir
fi
if [ ! -d $outdir/logs ] ; then
    mkdir -p $outdir/logs
fi
if [ ! -d $outdir/spectra-16 ] ; then
    mkdir -p $outdir/spectra-16
fi

#file=/project/projectdirs/desi/mocks/lya_forest/london/v4.0/0/0/transmission-16-0.fits
file=/project/projectdirs/desi/mocks/lya_forest/london/v4.0//1/137/transmission-16-137.fits

quickquasars -i $file --outdir $outdir/spectra-16 --zbest --mags --eboss --seed 123

@londumas
Copy link
Contributor

@julienguy and @andreufont, sory for the noise. I simply had to update SIMQSO to master. And it works now.

@moustakas
Copy link
Member

Tag v1.2.3 of simqso would also have worked:
https://github.com/imcgreer/simqso/releases/tag/v1.2.3

@londumas
Copy link
Contributor

@andreufont, I have rerun quickquasar on the branch eboss-quickquasars: /project/projectdirs/desi/mocks/lya_forest/london/v4.0/eboss-0.0. For the moment, everything looks good: footprint, SNR, magnitude, ...
I will run do_deltas.py tomorrow and the correlation functions.

Here are things we could work on:

  • downsampling in redshift according to the DR14Q n(z)
  • Work better on the footprint, the high density of SEQUELS doesn't appear in the mocks
  • The maximum lambda_obs is lower in mocks than in data.
  • In eBOSS the pixels are binned in log lambda, in these mocks they are binned in lambda.

Redshift distribution
redshift_distribution

Footprint data
footprint_data

Footprint mocks
footprint_mocks

@belaa
Copy link
Contributor

belaa commented Nov 13, 2018

@londumas Sorry for the lag - my computer was out of commission for a couple days. I have looked into the code and think I may know why it is showing a limited spectral range, but need to investigate this further. As for running this on more realizations, I'm afraid I am still unfamiliar with the details of how quickquasars works - is each quasar specified by a transmission file? And if so, can you point me to where these are located?

@andreufont
Copy link
Contributor Author

Hi @londumas - I'm glad it worked, and very curious to see how the correlation/BAO looks like!

A couple of comments on your comments:

H) downsampling in redshift according to the DR14Q n(z)
A) there are two options. We could run SDSS-only CoLoRe boxes, but it would be better if we could recycle the DESI boxes. I agree with you that for this the easiest would be to find some sort of function p(z) that we could use to downsample the quasars in a redshift-dependent way to match the distribution in SDSS.

H) Work better on the footprint, the high density of SEQUELS doesn't appear in the mocks
A) Well, here it gets tricky. The code assumes that the selection function can be split into a function of redshift, n(z), and an angular selection (density as a function of angular position). However, in SDSS this is not true, since the n(z) changes a lot from BOSS to eBOSS, with the latter having a lot more low-z quasars. This means that the plots of the footprint are highly dependent on the minimum redshift cut used. I used zmin=2.15 to define what I consider the footprint, since this cut is probably the sensible decision for Lyman alpha auto-correlation. With this cut, the SEQUELS area is not that different, I believe... What zmin did you use? In any case, feel free to add an alternative footprint file!

H) The maximum lambda_obs is lower in mocks than in data. In eBOSS the pixels are binned in log lambda, in these mocks they are binned in lambda.
A) I would decouple these two tasks. My branch (and the PR I'll submit soon) only tackles the density of quasars and the footprint, while instrument-specific settings should be addressed in a different branch. This is what @belaa is working on, to simulate the eBOSS instrument response. Once she has that part of the code running it should be easy to combine.

@andreufont
Copy link
Contributor Author

andreufont commented Nov 13, 2018

About the footprint in DR14Q and in the mocks, I add here the footprint I get when using the cut in zmin=2.15.

sdss_footprint
sdss_mock_footprint

I also attach the python script in case you want to play with it.

import numpy as np
import os
import matplotlib.pyplot as plt
import fitsio
import healpy

def pix2ang_deg(nside,pix,nest=True):
    pix_rad_dec,pix_rad_ra = healpy.pix2ang(nside,pix,nest=nest)
    pix_ra = pix_rad_ra * 180 / np.pi
    pix_dec = (np.pi/2.-pix_rad_dec)*180./np.pi
    return pix_ra,pix_dec

def pixfile(basedir,nside,pix):
    pix_str=str(pix)
    pix_100_str=str(int(pix/100))
    fname='zbest-'+str(nside)+'-'+pix_str+'.fits'
    return basedir+'spectra-'+str(nside)+'/'+pix_100_str+'/'+pix_str+'/'+fname

def plot_pixel(basedir,nside,pix,pix_area):
    fname=pixfile(basedir,nside,pix)
    if os.path.isfile(fname):
        f = fitsio.FITS(fname)
        z=f[1]['Z'][:]
        f.close()
        #print(pix,'found',len(z),'quasars')
        highz=(z>2.15)
        z=z[highz]
        nz=len(z)
        #print(nz,'of which are high-z')
        dens=nz/pix_area
        #print('density',dens)
    else:
        nz=0
        dens=0.0
    ra,dec=pix2ang_deg(nside,pix)
    if dens>25.0: 
        plt.errorbar(ra,dec,fmt='o',markersize=3,color='green')
    elif dens>15.0: 
        plt.errorbar(ra,dec,fmt='o',markersize=3,color='orange')
    elif dens>0.0: 
        plt.errorbar(ra,dec,fmt='o',markersize=3,color='blue')
    else:
        return 0
        #plt.errorbar(ra,dec,fmt='o',markersize=1,color='gray')
    return nz

def add_legend(nside):
    # use one of the pixels that will then be painted on top
    pix=0
    ra,dec=pix2ang_deg(nside,pix)
    plt.errorbar(ra,dec,fmt='o',markersize=3,color='blue',label='all')
    plt.errorbar(ra,dec,fmt='o',markersize=3,color='orange',label='>15')
    plt.errorbar(ra,dec,fmt='o',markersize=3,color='green',label='>25')
    plt.legend()

nside=16
npix=healpy.nside2npix(nside)
print('npix',npix)
pix_area = healpy.nside2pixarea(nside,degrees=True)

# create legend
add_legend(nside)

basedir='/data/london-skewers/v4.0/eboss-0.0/'
total_nz=0
total_pix=0
for pix in range(npix):
    nz = plot_pixel(basedir,nside,pix,pix_area)
    if nz > 0:
        total_pix += 1
        total_nz += nz

print('pixels with high-z quasars =',total_pix)
print('total high-z quasars =',total_nz)
plt.xlim(-10,370)
plt.ylim(-25,90)
plt.xlabel('RA [deg]')
plt.ylabel('Dec [deg]')
plt.title('Density of z>2.15 quasars in SDSS mock')
plt.savefig('sdss_mock_footprint.png')
plt.show()

@londumas
Copy link
Contributor

@belaa, I did not realize the version I was testing only @andreufont version and not yours.
I can run easily on more transmission files. What branch should I checkout and what option should I use? Are all the changes in your current branch?

@londumas
Copy link
Contributor

@belaa, your mocks have a pretty good range of wavelength.
We could work on it still. Curently we have:

  • Version lambda_obs_min lambda_obs_max
  • Mocks 3650.39990234375 9912.0
  • Data 3554.6758199906994 10401.596438661036

It would be nice to have the mocks o down to at least 3600.
Would that be difficult to fix?

distribution_wave_bela

@belaa
Copy link
Contributor

belaa commented Nov 13, 2018

@londumas I don't think it should be difficult to fix. I will take a look later today.

@belaa
Copy link
Contributor

belaa commented Nov 13, 2018

@londumas I just pushed my eboss_quickquasars branch on desihub/desisim, which should be run while on my desihub/specsim eboss_config branch. I am currently calling quickquasars with the following options:

quickquasars --exptime 4000. -i path_to_transmission_files/transmission-16-333.fits --outdir . --downsampling 0.1 --zbest --mags --overwrite --nmax 10 --no-simqso

Please let me know if you have any problems running this so I can make sure to fix them.

@londumas
Copy link
Contributor

@belaa, could you merge your branch with master? It is very far behind.

@londumas
Copy link
Contributor

@belaa, I merged the local version of your branch with master and got the following error.
Could you have a look at what is happening?

DEBUG:simexp.py:410:simulate_spectra: creating specsim desi simulator
Traceback (most recent call last):
  File "<me>/Programs/desi//code//desisim/bin/quickquasars", line 9, in <module>
    sys.exit(main())
  File "<me>/Programs/desi/code/desisim/py/desisim/scripts/quickquasars.py", line 692, in main
    footprint_healpix_weight,footprint_healpix_nside,bal=bal)
  File "<me>/Programs/desi/code/desisim/py/desisim/scripts/quickquasars.py", line 531, in simulate_one_healpix
    meta=specmeta,seed=seed,fibermap_columns=fibermap_columns,use_poisson=False, specsim_config_file="eboss") # use Poisson = False to get reproducible results.
  File "<me>/Programs/desi/code/desisim/py/desisim/scripts/quickspectra.py", line 180, in sim_spectra
    psfconvolve=True, specsim_config_file=specsim_config_file)
  File "<me>/Programs/desi/code/desisim/py/desisim/simexp.py", line 413, in simulate_spectra
    camera_output=psfconvolve)
  File "<me>/Programs/desi/code/desisim/py/desisim/specsim.py", line 53, in get_simulator
    camera_output=camera_output)
  File "<me>/Programs/desi/code/specsim/specsim/simulator.py", line 75, in __init__
    self.instrument = specsim.instrument.initialize(config, camera_output)
  File "<me>/Programs/desi/code/specsim/specsim/instrument.py", line 618, in initialize
    allow_convolution=camera_output))
  File "<me>/Programs/desi/code/specsim/specsim/camera.py", line 147, in __init__
    .format(self.name))
RuntimeError: Wavelength grid min does not cover b-camera response.

@belaa
Copy link
Contributor

belaa commented Nov 13, 2018

@londumas Are you also pointing to the eboss_config branch in specsim?

@londumas
Copy link
Contributor

@belaa, yes.

@londumas
Copy link
Contributor

@belaa, I have tried to also merge desihub/specsim eboss_config branch with master and it gives me the same error: RuntimeError: Wavelength grid min does not cover b-camera response. Could you have a look?

@londumas
Copy link
Contributor

@andreufont, I have updated the footprint file to remove DR7 and start at z=1.8. Can I commit the changes into your branch?
Here is the result. We have to have a lower downsampling. We have twice in the mocks, what we have in the DR14Q data. Do you know how to fix this?
The mean redshift is not the same because of the redshift distribution.
I am working on this.

footprint

@belaa
Copy link
Contributor

belaa commented Nov 14, 2018

@londumas Yes I will look into this and get back to you.

@londumas
Copy link
Contributor

@andreufont, you can forget my remarks. I found the solutions.
I have been pushing to the branch and I have nearly finished implementing the addition of the SDSS redshift distribution.

@londumas
Copy link
Contributor

@andreufont, the latest version of desisim:eboss-quickquasars branch is ready to be reviewed and merged.
It gives:

  • the correct redshift distribution
  • the correction footprint density

Is there other things you want to have a look to before we do a PR?

cat nb_tot nb_zmin_1.8_zmax_3.5
zcat 262,071 256,529
Data(DR14) 446,495 248,318

hist_redshift

footprint_data

footprint_mocks

@belaa
Copy link
Contributor

belaa commented Nov 16, 2018

@londumas Can you please look at the output in /global/cscratch1/sd/belaa/desi/quickquasar_output/run2 and confirm that the wavelength goes below 3600/above 10000A?

@londumas
Copy link
Contributor

@belaa, Thank you very much for the work. You have fixed the small wavelength which is very important, but not the large wavelength. If it is too much work to fix them, it is not important.
If it is easy, let's do it.

Mocks 3533.39990234375 9912.0
Data 3554.6758199906994 10401.596438661036

@londumas
Copy link
Contributor

@belaa, do you think you would have time to work on this?
If not, it is ok, let's have you do a PR.

@belaa
Copy link
Contributor

belaa commented Nov 20, 2018

@londumas yes, I can do this tomorrow.

@londumas
Copy link
Contributor

@belaa, perfect. Thank you.

@londumas
Copy link
Contributor

@belaa, I did a merge master for eboss_quickquasars to put it up to date.

@belaa
Copy link
Contributor

belaa commented May 29, 2020

Now that the eboss_quickquasars branch has been merged, can we close this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants