Faster contrast curves #136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When using csv files for logs, save version with darks to a new file (marked by "_with_darks.csv") so that darks are not appended repeatedly when the pipeline is rerun.
Set default reduction method using bracket notation instead of dot notation because the column isn't actually added to the data frame when using dot notation.
Check if specified reduction directory exists and create it if needed.
Corrected description of reddir.
Share original filenames with plot_array so that the shift1_cube.png files can be annotated with filenames.
If ShARCS dataset contains BrG-2.16 data and no corresponding flat, use the Ks flat instead.
Label each subpanel with filename or star name. Add new functionality for plotting reduced images of all stars observed on a single night.
For ShARCS, if no BrG-2.16 flat exists, use Ks flat for reducing BrG-2.16 data. Also removed leftover stray comments.
Removed previously commented out text changing head.update to head.set.
Corrected typo (missing = in equality check).
Corrected typo (missing : before if/then statement)
Changed literal eval to avoid issues when filenames have leading zeros.
Added ability to save time by using existing darks, flats, and skies.
Added if statement to tell the code when to use the "all_stars" version of the plot.
Added snames=None to ensure that all plotting routines are compatible with the addition of plot_all_stars
Corrected typos in plot_all_stars.
Truncate error array to the inner 10" so that lim_arc_lengths, delta_mags, and lim_stds are all the same length.
correct typo in comments (afrer --> after)
Automatically run contrast curve code for each star and save plots and csv files showing contrast as a function of angular separation.
fixed typo in generation of lim_stds
Flipped y-axis to show brighter objects at top.
Produce grid of all images and plot showing all contrast curves.
added a space to improve readability
Produce grid showing reduced images for all stars observed and plot showing all contrast curves.
When plotted in a grid, flats are now shown with a narrower, adaptive color scaling range and labeled by file name.
When run with verbose=False, warning messages from concur are suppressed to avoid messing up the progress bars.
Cast header['CRPIX1'] and header['CRPIX2'] as floats to prevent errors that occur when they are read in as integers.
Don't run darks, flats, and sky drivers again after registering images.
Don't print to screen unless VERBOSE is set. Allow all types of flats to match (e.g., sky flats, dome flats, flats) and improve matching of images taken in J+Ch4-1.2 by allowing different filter capitalization and whitespace in the logsheet.
Return configuration file when user requests to skip data reduction.
corrected a minor typo in comments
Made larger intermediate images (800x800 pixels versus 600x600) to avoid edge effects in final 600x600 pixel images. Changed contrast curve spacing to <= 0.5" beyond FWHM. Made larger fake images to avoid cases where sky annulus falls outside the fake image for high FWHM.
Move contrast curves into a central directory and format as needed for TRICERATOPS runs.
Now use 1000x1000 pixel images for majority of process instead of 800x800 pixel images. Removed hardcoding of initial image size when trimming down to the final image.
TRICERATOPS can't handle missing contrast curve values, so now rows with missing values are removed from the contrast curve dataframe before saving to file.
Restricted possible image centers to central region when using shift_bruteforce to avoid selecting bright image edges.
Use astropy.convolution to interpolate over NaNs before registration. Fixes issue with shift (from scipy.ndimage.interpolation) generating images that are completely NaN if the input image contains NaNs. Also fixed typo in comment in drivers.
Courtney updates
Properly handle observations taken in J+Ch4-1.2 by reading both filters in the FITS header and saving files accordingly.
Make separate folders for J+Ch4-1.2 files
Increased default maximum allowed image shift from 200 pixels to 350 pixels in registration.py. Improved handling of NaNs in skies and image reduction. Corrected typos in comments.
Added option to run image drivers and registration for only a subset of stars. Need to improve tracking of reduction methods for selected star registration. Currently assuming "saturated" by default.
Sped up contrast curve code 20-30x by using the FWHM to set the size of the fake image.
Updated size used for fake images to 5 pixels wider than the size used for the outer radius of the sky annulus.
Courtney updates
Deal with gaps in list of images. Without this fix, only the final set of images is used for each star/filter combination.
Also deal with breaks in the lists of flats, darks, and skies. Previously, the code just used a single set rather than combining sets of images with a given exposure time and filter.
corrected typo (skis --> skies)
Streamline handling of flats and skies.
Corrected bug in splitting skies by filter. Switched to adaptive color scaling for grids of skies.
Courtney updates
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #136 +/- ##
==========================================
- Coverage 76.31% 76.04% -0.26%
==========================================
Files 25 25
Lines 2296 2346 +50
==========================================
+ Hits 1752 1784 +32
- Misses 544 562 +18
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Sped up contrast curve generation by using the measured FWHM to set the sizes of the fake images.
Motivation and Context
Contrast curve generation was previously much slower (roughly 160 s per image). Depending on the FWHM, contrast curve generation can now be completed in under 10 seconds.
How Has This Been Tested?
Compared to contrast curves previously generated by David Ciardi using his IDL code.
Screenshots (if appropriate):
Types of changes
Checklist: