Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup jointcal and separate astrometry/photometry tickets/DM-8552 #27

Merged
merged 13 commits into from
Feb 16, 2017

Commits on Jan 24, 2017

  1. Big speed improvement and cleanup dead ccdImage code

    Only read the calexps when building the ccdImages (and not when
    building the output to write) and save the Calib to its respective
    ccdImage to use it when preparing output. Use that calib in ccdImage
    to convert flux to magnitude, instead of custom code.
    
    No need for the calexp_md now: just extract the values from the calexp
    (eventually this should come from some metadata datasetType, e.g.
    `get('info')`).
    
    Add SOURCE_IO_NO_FOOTPRINTS flag to get('src') for a big speedup,
    since I don't need the heavy footprints (thanks @TallJimbo for the
    suggestion).
    
    Add individual profiling for each slow step, activated via
    profile_jointcal option.
    
    Remove a bunch of variables and getters from ccdImage that are totally
    unused.
    parejkoj committed Jan 24, 2017
    Configuration menu
    Copy the full SHA
    dab02ba View commit details
    Browse the repository at this point in the history
  2. Remove skipped tests for fewer visits.

    With plot_jointcal_results, I can now extract statistics from any
    jointcal run, so I don't need those tests sitting around here.
    parejkoj committed Jan 24, 2017
    Configuration menu
    Copy the full SHA
    cf60c48 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2017

  1. photometry/astrometry can be run independently, with passing tests.

    Split photo/astro code with Config parameters, and a test for each option in the
    lsstSim test class. The tests check for empty Wcs/Calib, if they shouldn't have
    been written.
    
    Pulled apart _testJointcalTask so I could use the run/plot bits in the "only
    one" tests, since those have quite different assert conditions.
    
    Also support the split in JointcalStatistics: the nested ifs are a bit
    excessive, but they get the job done.
    
    Fixed inconsistent capitalization of jointcal in tests.
    parejkoj committed Feb 16, 2017
    Configuration menu
    Copy the full SHA
    40b323e View commit details
    Browse the repository at this point in the history
  2. Make refObjLoaders configurable

    Add astrometry/photometryRefObjLoaders as configurables and make
    jointcal.__init__() take a butler argument to configure them. JointcalRunner
    has to be derived from ButlerInitializedTaskRunner to get said butler.
    
    Clean up "default" filter handling somewhat (see DM-9093 for what else needs to
    be done), to use the most common filter in the dataRefs. CollectLSSTRefStars
    now takes the actual fluxField, instead of building it on the fly.
    parejkoj committed Feb 16, 2017
    Configuration menu
    Copy the full SHA
    82456de View commit details
    Browse the repository at this point in the history
  3. astro/photo split now compiles and runs; new test metrics.

    Add jointcal._do_load_refcat_and_fit() to do both photometry and astrometry
    via kwargs.
    
    Add metrics dictionary in jointcal.py, for basic values to test, that may
    turn into validation/verification SQUASH tests later.
    
    associateCatalogs now properly cleans up when run a second time. This would
    be best handled via better memory management, but that's DM-4043.
    
    Fix some range-based for loops and cleaned up pointers/references.
    
    Remove bit-rotted Jointcal class and other unused code.
    
    More renaming of badly-named methods and variables. Filed DM-9135 to deal
    with this in bulk.
    
    Print some interesting values in when running JointcalStatistics verbosely.
    
    Fixed a bug I had introduced in RefStar re:setFittedStar behavior with nullptr.
    parejkoj committed Feb 16, 2017
    Configuration menu
    Copy the full SHA
    ac9d1b1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f5ea857 View commit details
    Browse the repository at this point in the history
  5. Add GAIA refcat and no-photo/no-astro HSC tests

    Add hsc config for filterMap and tweak test to work with it.
    
    Add every star number and chi2 metric for every test dataset.
    
    Metrics tests always occur in JointcalTestBase._runJointcalTask(). Default
    to checking every metric that gets returned, so they'll fail if that metric
    isn't defined by the test method. A metric can be skipped by setting it to None.
    parejkoj committed Feb 16, 2017
    Configuration menu
    Copy the full SHA
    ff8115e View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    cf00394 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    55c7fb3 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    5f70e89 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    2473da4 View commit details
    Browse the repository at this point in the history
  10. Compute common tangent plane after all ccdImages are loaded

    Tweak metrics after solving common tangent plane problem.
    parejkoj committed Feb 16, 2017
    Configuration menu
    Copy the full SHA
    16c7c0d View commit details
    Browse the repository at this point in the history
  11. sort dataRefs for consistent py2/3 ordering

    tweak metrics for sorted dataRefs
    parejkoj committed Feb 16, 2017
    Configuration menu
    Copy the full SHA
    6ec66b3 View commit details
    Browse the repository at this point in the history