Add basic centroid file capability. #112

cwwalter · 2018-05-05T15:42:49Z

I am working on adding more pixel level truth to the simulation but thought it would be useful to first add basic 'centroid file' capability (requested in #32).

My work on adding information based on what each object looks like on the sensor is currently adding a lot of CPU run time and I need more work to optimize it. So, this initial PR uses only truth information about the objects in the centroid file. This makes it more difficult to use this to do imSim/PhoSim validation comparisons (because the PhoSim centroid values are average X and Y values on the sensor) but it is better for comparing catalog truth with the sensor since the true values are not affected by being near the edge of the sensor etc. Eventually we may want both.

This code will make centroid files corresponding to each fits file in the output directory. For example:

wasabi:fits % ls
centroid_197356_R22_S01_r.txt	lsst_e_197356_R22_S01_r.fits
centroid_197356_R22_S10_r.txt	lsst_e_197356_R22_S10_r.fits
centroid_197356_R22_S11_r.txt	lsst_e_197356_R22_S11_r.fits
centroid_197356_R22_S12_r.txt	lsst_e_197356_R22_S12_r.fits
centroid_197356_R22_S21_r.txt	lsst_e_197356_R22_S21_r.fits

With a small piece of code this can be converted into a ds9 region file for overlay. This pandas code example puts a circle at each true object position with the radius scaled by the log of the number of photons for one of the centroid files:

import math
import numpy as np
import pandas as pd

centroids = pd.read_csv('fits/centroid_197356_R22_S11_r.txt', delim_whitespace=True, comment="#")

with open('region_file.reg','w') as outfile:
    outfile.write('physical\n')
    centroids.query('Photons>0').to_string(outfile, columns=['xpos','ypos','Photons'], 
                                           header=False, index=False,
                                           formatters=['circle {:10.3f}'.format,
                                                       '{:10.3f}'.format,
                                                       lambda x: '{:10.3f}'.format(math.log(x))])

Loading this regions file after loading the fits file results in:

To my eye, there is a small shift between the circle centers and the objects. I'm not sure if this is a plotting issue, a misunderstanding on my part of the variables I used, or some small bug in the code.

The imSim code only sets the options to turn this on and sets the file prefix in the config file. The code to open, close and write to the files is in sims_GalSimInterface. I will open a separate PR there.

Currently, this code only is in the non-sensor version of the GalSimInterpreter class. I would like some feedback from @jchiang87, @rmjarvis and @danielsf before I add much more to make sure this approach seems reasonable to people.

I started by making this look like a PhoSim centroid file since I thought this might make things easier on the people running the pipeline but in principle we could also add more information into the file.

danielsf · 2018-05-07T18:55:03Z

Regarding the offset between the green circles and the objects in the image above:

Was this with or without the sensor model, @cwwalter ? The pixel positions you are printing to the centroid files are calculated before the sensor model is applied. That might have something to do with it.

cwwalter · 2018-05-08T13:56:58Z

Was this with or without the sensor model, @cwwalter ? The pixel positions you are printing to the centroid files are calculated before the sensor model is applied. That might have something to do with it.

This was without the sensor model (Just calling the drawObject version in GalSimInterpreter). I also ran with the sensor model turned on which uses GalSimSiliconInterpeter. There I saw what looked to my eye like a smaller shift if it was still there in another direction.

cwwalter · 2018-07-04T23:24:28Z

This last change (along with a PR in the sims_GalSimInterface) moves the centroid information into a data structure in memory. This way it can be stored with checkpoints. The centroid files are now written out at the end of the simulation.

in the fits directory with filenames corresponding to the fits files. The uiniqueID, the true flux and the true object position in detector coordinates are written to each file. The code to write obejct information into the filesis in sims_GalSimInterface.

files at the end of the process. This is so we can use this with the checkpointing system.

coveralls · 2018-07-12T21:48:20Z

Coverage decreased (-0.2%) to 70.534% when pulling 25b9aeb on centroid into e4b4bfc on master.

jchiang87

Looks good...just a few minor change requests.

jchiang87 · 2018-07-13T04:15:27Z

python/desc/imsim/ImageSimulator.py

@@ -34,7 +34,7 @@ class ImageSimulator:
    Class to manage the parallel simulation of sensors using the
    multiprocessing module.
    """
-    def __init__(self, instcat, psf, numRows=None, config=None, seed=267,
+    def __init__(self, instcat, create_centroid_file, psf, numRows=None, config=None, seed=267,


I would prefer to make create_centroid_file a keyword argument with a default value of False rather than make it a required argument, so that existing scripts won't break. I would also add it towards the end of the argument list, say after apply_sensor_model.

jchiang87 · 2018-07-13T04:17:40Z

python/desc/imsim/ImageSimulator.py

@@ -125,6 +126,10 @@ def _make_gs_interpreters(self, seed, sensor_list, file_id):
                                        self.phot_params,
                                        self.obs_md)

+            if self.create_centroid_file is True:
+                self.gs_interpreters[det_name].centroid_base_name = self.outdir \


It mostly doesn't matter, but it's probably better to use os.path.join for concatenating path components.

jchiang87 · 2018-07-13T04:30:00Z

python/desc/imsim/ImageSimulator.py

@@ -125,6 +126,10 @@ def _make_gs_interpreters(self, seed, sensor_list, file_id):
                                        self.phot_params,
                                        self.obs_md)

+            if self.create_centroid_file is True:


if self.create_centroid_file is True: -> if self.create_centroid_file:

… False instead of a required argument. Also fix up a couple of trivial PEP008 issues.

cwwalter · 2018-07-15T03:08:32Z

Requested changes are done in this repo.

The code in close_centroid_files in sims_GalSimInterface was moved into write_centroid_files.

cwwalter mentioned this pull request May 5, 2018

Add centroid file capability. lsst-sims/legacy_sims_GalSimInterface#51

Merged

cwwalter requested a review from jchiang87 May 5, 2018 15:47

cwwalter added 3 commits July 12, 2018 16:46

Wrote the centroid objects to memory and then write them all out to

a0c6d84

files at the end of the process. This is so we can use this with the checkpointing system.

Missed changing one gs_interpreter to the multiprocessing list.

d38c96e

cwwalter force-pushed the centroid branch from 8588510 to d38c96e Compare July 12, 2018 21:32

jchiang87 requested changes Jul 13, 2018

View reviewed changes

cwwalter added 3 commits July 14, 2018 22:46

Make create_centroid_file be a keyword argument with default value of…

75433f4

… False instead of a required argument. Also fix up a couple of trivial PEP008 issues.

Use os.path.join to concatenate the path to the base name.

93351b6

Minor syntax change for review.

4b674d3

Remove call to close_centroid_files

25b9aeb

The code in close_centroid_files in sims_GalSimInterface was moved into write_centroid_files.

jchiang87 approved these changes Jul 15, 2018

View reviewed changes

cwwalter merged commit e311a5f into master Jul 15, 2018

cwwalter deleted the centroid branch July 15, 2018 15:39

cwwalter mentioned this pull request Jul 15, 2018

Add a centroid file capability #32

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic centroid file capability. #112

Add basic centroid file capability. #112

cwwalter commented May 5, 2018

danielsf commented May 7, 2018

cwwalter commented May 8, 2018

cwwalter commented Jul 4, 2018 •

edited

Loading

coveralls commented Jul 12, 2018 •

edited

Loading

jchiang87 left a comment

jchiang87 Jul 13, 2018

jchiang87 Jul 13, 2018

jchiang87 Jul 13, 2018

cwwalter commented Jul 15, 2018

Add basic centroid file capability. #112

Add basic centroid file capability. #112

Conversation

cwwalter commented May 5, 2018

danielsf commented May 7, 2018

cwwalter commented May 8, 2018

cwwalter commented Jul 4, 2018 • edited Loading

coveralls commented Jul 12, 2018 • edited Loading

jchiang87 left a comment

Choose a reason for hiding this comment

jchiang87 Jul 13, 2018

Choose a reason for hiding this comment

jchiang87 Jul 13, 2018

Choose a reason for hiding this comment

jchiang87 Jul 13, 2018

Choose a reason for hiding this comment

cwwalter commented Jul 15, 2018

cwwalter commented Jul 4, 2018 •

edited

Loading

coveralls commented Jul 12, 2018 •

edited

Loading