Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-9926: reduce memory footprint of meas_mosaic #7

Merged
merged 3 commits into from
Mar 30, 2017
Merged

DM-9926: reduce memory footprint of meas_mosaic #7

merged 3 commits into from
Mar 30, 2017

Commits on Mar 30, 2017

  1. SourceReader: don't read calexp 

    Reading the calexp is slow, uses excessive memory and unnecessary:
* the detector is available from the camera (no need to worry about cameras
  that don't index camera by dataId["ccd"]: we're only supporting HSC)
* the dimensions of the image are available from the calexp_md. This is
  a little dirty, but it's used elsewhere in this file and we only expect
  to have to support meas_mosaic a little longer.
    @PaulPrice
    PaulPrice committed Mar 30, 2017
    Configuration menu
    Copy the full SHA
    68e7b5b View commit details
    Browse the repository at this point in the history

  2. use std_vector.reserve before doing a bunch of push_back 

    Probably not important here, but good practice in general because it
guards against memory fragmentation.
    @PaulPrice
    PaulPrice committed Mar 30, 2017
    Configuration menu
    Copy the full SHA
    819eb0c View commit details
    Browse the repository at this point in the history

  3. MosaicTask.readCatalog: reduce memory footprint in multiprocessing 

    We do this in two ways:
1. Use Pool with maxtasksperchild=1. This stops memory fragmentation
   from compounding, because although memory may be fragmented in each
   process, after each read the process exits and is replaced with a
   fresh slate.
2. Join the pool when done. This stops a bunch of python processes from
   hanging around when iterating over tracts in a single job.
    @PaulPrice
    PaulPrice committed Mar 30, 2017
    Configuration menu
    Copy the full SHA
    2ecc8fd View commit details
    Browse the repository at this point in the history