GalSim benchmarks

GalSim benchmarks

The code in the galsim/benchmarks directory uses GalSim to generate synthetic star and galaxy imagery and uses these images to benchmark Celeste's accuracy.

The code in this directory is in the process of being partially replaced by Accuracy benchmarks. However, this directory will still hold code for setting up the GalSim distribution and generating synthetic imagery using GalSim.

Fetching pre-built test images

Run make fetch in benchmark/galsim.

Generating test images

If you wish the generate test images yourself, the script galsim_benchmarks.py generates the images using GalSim. If you have GalSim installed on your local Python you can run it directly (see bootstrap.sh, or follow the GalSim installation instructions Installing GalSim is a bit cumbersome, though, so you may wish to use the included Vagrantfile:

1. Install Vagrant.
2. From this directory, run vagrant up. This will take a while the first time (downloading a machine image and installing GalSim, a process handled by bootstrap.sh) but should be less than a minute thereafter.
3. Run vagrant ssh. You will now be logged into the virtual machine with GalSim installed.
4. From within the VM, run cd /vagrant && python galsim_benchmarks.py.

You will then have a file output/galsim_benchmarks_*.fits (under benchmark/galsim) containing the test images in a multi-extension FITS file. The "*" holds an MD5 hash of the file contents, which serves to uniquely identify different versions of the file. The script also writes the generated FITS filename to latest_filenames/latest_galsim_benchmarks.txt, which gets checked into git.

Running Celeste on test images

Once you have output/galsim_benchmarks_*.fits, use julia run_galsim_benchmark.jl to run Celeste on each test case and report inferred and ground truth values.

Using the test images yourself

The generated FITS file contains five images ("extensions") per test case, one for each of the five color bands. The first five extensions correspond to the first test case, the next five to the second case, etc. The FITS header on each extension includes all ground truth data along with metadata describing the test case. See Glossary-and-guide-to-Celeste-parameters for explanations of these parameters; that page, in combination with the comments present in the FITS header, should be sufficient to understand all the header data. Here's an example of the custom header fields present, in addition to standard ones describing the WCS:

CLCASEI =                   23 / Celeste: test case index
CLDESCR = 'three_sources_two_overlap' / Celeste: comment
CLIOTA  =               1000.0 / Celeste: counts per nMgy
CLSKY   =                 0.01 / Celeste: "epsilon" sky level (nMgy each px)
CLNOISE =                    F / Celeste: was Poisson noise added?
CLSIGMA =                    4 / Celeste: Gaussian PSF sigma (px)
CLBAND  =                    5 / Celeste: color band
CLNSRC  =                    3 / Celeste: number of sources
CLRES   =              0.00011 / Celeste: resolution (degrees/px)
CLX001  = 0.003946111111111111 / Celeste: X center in world coordinates (deg)
CLY001  = 0.006723888888888889 / Celeste: Y center in world coordinates (deg)
CLFLX001=                   40 / Celeste: reference (=3) band brightness (nMgy)
CLC12001=    3.990977443609023 / Celeste: ratio of flux in band 2 to band 1
CLC23001=     1.88394875659382 / Celeste: ratio of flux in band 3 to band 2
CLC34001=               1.3179 / Celeste: ratio of flux in band 4 to band 3
CLC45001=    1.169815615752333 / Celeste: ratio of flux in band 5 to band 4
CLTYP001= 'star    '           / Celeste: "star" or "galaxy"?
CLX002  = 0.005890555555555556 / Celeste: X center in world coordinates (deg)
CLY002  = 0.006723888888888889 / Celeste: Y center in world coordinates (deg)
CLFLX002=                   10 / Celeste: reference (=3) band brightness (nMgy)
CLC12002=    1.243458759033142 / Celeste: ratio of flux in band 2 to band 1
CLC23002=    2.004008016032064 / Celeste: ratio of flux in band 3 to band 2
CLC34002=               1.4031 / Celeste: ratio of flux in band 4 to band 3
CLC45002=    1.265055947544722 / Celeste: ratio of flux in band 5 to band 4
CLTYP002= 'galaxy  '           / Celeste: "star" or "galaxy"?
CLANG002=                   35 / Celeste: major axis angle (degrees from x-axis)
CLRTO002=                  0.2 / Celeste: minor/major axis ratio
CLRDA002=                  1.5 / Celeste: half-light radius (arcsec)
CLRDP002=    3.787878787878788 / Celeste: half-light radius (pixels)

The per-source headers are suffixed with an integer 1-CLNSRC indexing the relevant light source (e.g., CLFLX001, CL_FLX002, etc).

Defining new test images

Take a look at the bottom of galsim_benchmarks.py. There are many simple examples and you can add a new (decorated) function to define a new test case.

Generating a synthetic field

In addition to the above benchmarks, there is a separate script, benchmarks/galsim/galsim_field.py, which generates synthetic images the size of an SDSS frame based on a ground truth catalog. You can learn more about generating ground truth catalogs at Accuracy benchmarks.

In the Vagrant VM, run galsim_field.py /celeste/benchmark/accuracy/output/<catalog CSV> to generate the FITS file from the catalog (with five images, one per color band).