Skip to content

Commit

Permalink
Merge pull request #34 from mackjenn/master
Browse files Browse the repository at this point in the history 
Mask IR persistence and Recombine with AstroDrizzle
  • Loading branch information
@stscicrawford
stscicrawford committed Oct 4, 2018
2 parents f545759 + 3f2b9d4 commit 9d14f3e
Showing 1 changed file with 319 additions and 0 deletions.
319 changes: 319 additions & 0 deletions docs/notebooks/wf3persist.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,319 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"Image persistence in the IR array occurs whenever a pixel is exposed to light that exceeds more than about half of the full well of a pixel in the array. Persistence can occur within a single visit, as the different exposures in a visit are dithered. Persistence also occurs from observations in a previous visit of completely different fields.\n",
"\n",
"Image persistence is seen in a small, but non-negligible fraction of WFC3/IR exposures. Its properties are discussed in the WFC3 Instrument Handbook in [Chapter 8](http://www.stsci.edu/hst/wfc3/documents/handbooks/currentDHB/Chapter8_persistence1.html). Persistence is primarily a function of the degree to which a pixel is filled (in electrons) and the time since this occurred. Additional information is available from the [WFC3 Persistence Webpage](http://www.stsci.edu/hst/wfc3/ins_performance/persistence/)\n",
"\n",
"This notebook shows how to use persistence products to update the data quality (DQ) array of the IR calibrated FLT dataset for use with AstroDrizzle. When the data is sufficiently dithered, affected regions of the detector may be replaced with 'good' pixels from other exposures in the visit.\n",
"\n",
"The notebook is organized as follows:\n",
"\n",
" Download the data from MAST \n",
" Download the persistence data products\n",
" Define a lower threshold in the persistence model for defining a mask (eg. >0.005 e/s)\n",
" For affected pixels, add flag 16384 to the DQ array values\n",
" Redrizzle the FLT data to produce a cleaned DRZ product"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Setup\n",
"\n",
"This notebook assumes you have installed a recent version of AstroConda .\n",
"\n",
"Two additional astropy packages must be installed in your conda environment before downloading the data. To do this, type the following command in the terminal before starting the notebook:\n",
"\n",
" conda install -c astropy astroquery ccdproc\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from ccdproc import ImageFileCollection\n",
"from astroquery.mast import Observations\n",
"from astropy.io import fits\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import os \n",
"import glob \n",
"import shutil"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Data\n",
"\n",
"Here we obtain WFC3/IR observations from GLASS program 13459, Visit 29 in the F140W filter. These exposures were impacted by persistence from grism exposures obtained just prior to these exposures. \n",
"\n",
"The following commands query MAST and then downloads the FLT and DRZ data products to the current directory."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_list = Observations.query_criteria(obs_id='ica529*',filters='F140W')\n",
"\n",
"Observations.download_products(data_list['obsid'],mrp_only=False,download_dir='./data',\n",
" productSubGroupDescription=['FLT','DRZ'])\n",
"\n",
"science_files = glob.glob('data/mastDownload/HST/*/*fits')\n",
"\n",
"for im in science_files:\n",
" root = im.split('/')[-1]\n",
" os.rename(im,'./'+root)\n",
"shutil.rmtree('data/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Obtain the Persistence Model Products \n",
"\n",
"To find the URL of the tar file containing the persistence fits files, visit the [PERSIST Search Form](https://archive.stsci.edu/prepds/persist/search.php) and search for dataset='ica529*'.\n",
"\n",
"Hover over the link in the Visit colum to get the URL to the gzipped tar file for Visit 29. \n",
"The persistence model data products we will use to create a mask are named 'rootname_persist.fits' and contain contributions from both external and internal persistence. \n",
"\n",
"We define external persistence as residual signal that is generated by an earlier visit, and internal persistence as that generated within the same visit as the image in question. External persistence typically comes from a prior scheduled WFC3/IR program and is not within the control of the observer. Internal persistence can be mitigated by the observer by dithering the exposures within a given visit. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import urllib\n",
"request=urllib.request.urlopen('https://archive.stsci.edu/pub/wfc3_persist/13459/Visit29/13459.Visit29.tar.gz')\n",
"\n",
"tar=request.read()\n",
"with open('13459.Visit29.tar.gz', \"wb\") as gzfile:\n",
" gzfile.write(tar)\n",
"\n",
"!tar -zxvf 13459.Visit29.tar.gz"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Observation Summary\n",
"\n",
"Visit 29 contains 4 consecutive dithered exposures in the F140W filter which are obtained in a single orbit. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"collec = ImageFileCollection('./',\n",
" keywords=[\"asn_id\",\"targname\",\"filter\",\"samp_seq\",\"nsamp\",\"exptime\",\n",
" \"postarg1\",\"postarg2\",\"date-obs\",\"time-obs\",], glob_include=\"ica529*flt.fits\", ext=0)\n",
"out_table = collec.summary\n",
"out_table"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Display the Images\n",
"\n",
"The DRZ product combines the 4 individual FLT exposures and shows faint residual persistence from grism observations obtained just prior to these data. An example of the 'Persistence Removal Evaluation' for the first dataset ica529rmq may be found [here](https://archive.stsci.edu/pub/wfc3_persist/13459/Visit29/ica529rmq_persist.html).\n",
"\n",
"The DRZ pipeline product is shown on the left, and we can see that the persistence was partiallym, but not completely, removed by the cosmic-ray rejection functionality in AstroDrizzle. The persistence model for the first FLT dataset is shown on the right. A red box is overplotted on each image to highlight a portion of the detector which is impacted by the grism signal. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"flt1=fits.getdata('ica529rsq_flt.fits',ext=1)\n",
"per1=fits.getdata('13459.Visit29/ica529rsq_persist.fits',ext=1)\n",
"drz =fits.getdata('ica529030_drz.fits',ext=1)\n",
"\n",
"fig=plt.figure(figsize=(20,8))\n",
"ax1=fig.add_subplot(1,2,1)\n",
"ax2=fig.add_subplot(1,2,2)\n",
"\n",
"ax1.imshow(drz, vmin=-0.1,vmax=0.35,cmap='Greys_r',origin='lower')\n",
"ax2.imshow(per1, vmin=0.0,vmax=0.005,cmap='Greys_r',origin='lower')\n",
"\n",
"ax1.set_title('ica529030_drz.fits (Drizzled Product = 4 FLTs)',fontsize=20)\n",
"ax2.set_title('ica529rsq_persist.fits (Model for a single FLT)',fontsize=20)\n",
"\n",
"ax1.plot([0,300,300,0,0],[250,250,350,350,250],c='red')\n",
"ax2.plot([0,300,300,0,0],[250,250,350,350,250],c='red')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Use the persistence model to add DQ flags to each FLT\n",
"\n",
"For any pixels in the model which have a signal greater than 0.005 electrons/sec, we add a flag of 16384 to the current DQ array values in each FLT frame. This threshold is flexible should be determined empirically by the user based on their specific data.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"flt1_dq=fits.open('ica529rmq_flt.fits',mode='update')\n",
"per1 =fits.getdata('13459.Visit29/ica529rmq_persist.fits',ext=1)\n",
"\n",
"flt2_dq=fits.open('ica529rsq_flt.fits',mode='update')\n",
"per2 =fits.getdata('13459.Visit29/ica529rsq_persist.fits',ext=1)\n",
"\n",
"flt3_dq=fits.open('ica529s0q_flt.fits',mode='update')\n",
"per3 =fits.getdata('13459.Visit29/ica529s0q_persist.fits',ext=1)\n",
"\n",
"flt4_dq=fits.open('ica529s6q_flt.fits',mode='update')\n",
"per4 =fits.getdata('13459.Visit29/ica529s6q_persist.fits',ext=1)\n",
"\n",
"flt1_dq[3].data[per1 > 0.005] += 16384\n",
"flt2_dq[3].data[per2 > 0.005] += 16384\n",
"flt3_dq[3].data[per3 > 0.005] += 16384\n",
"flt4_dq[3].data[per4 > 0.005] += 16384\n",
"\n",
"flt1_dq.close()\n",
"flt2_dq.close()\n",
"flt3_dq.close()\n",
"flt4_dq.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Redrizzle the FLT data with the new DQ flags\n",
"\n",
"Now, we recombine the FLT data with AstroDrizzle using the updated DQ arrays and compare with the pipeline data products."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from drizzlepac import astrodrizzle\n",
"astrodrizzle.AstroDrizzle('ica529*flt.fits',output='ica529030_pcorr',preserve=False,final_bits=576,build=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Display the Original and the Corrected DRZ files"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"drz =fits.getdata('ica529030_drz.fits',ext=1)\n",
"drz_corr =fits.getdata('ica529030_pcorr_drz.fits',ext=1)\n",
"\n",
"fig=plt.figure(figsize=(20,8))\n",
"ax1=fig.add_subplot(1,2,1)\n",
"ax2=fig.add_subplot(1,2,2)\n",
"\n",
"ax1.imshow(drz, vmin=-0.1,vmax=0.35,cmap='Greys_r',origin='lower')\n",
"ax2.imshow(drz_corr,vmin=-0.1,vmax=0.35,cmap='Greys_r',origin='lower')\n",
"\n",
"ax1.set_title('ica529030_drz.fits (Pipeline SCI)',fontsize=20)\n",
"ax2.set_title('ica529030_pcorr_drz.fits (Corrected SCI)',fontsize=20)\n",
"\n",
"ax1.plot([0,300,300,0,0],[250,250,350,350,250],c='red')\n",
"ax2.plot([0,300,300,0,0],[250,250,350,350,250],c='red')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Display the Original and the Corrected WHT files\n",
"\n",
"When final_wht_type='EXP', the drizzled WHT images provide an effective exposure time map of the combined array. \n",
"\n",
"In the plots below, we can see that the pipeline products have lower weight in the region impacted by the grism, but that only a single frame was flagged and rejected by the cosmic ray algorithm. The grey horizontal bars have a weight ~500 seconds compared to the total exposure ~700 seconds.\n",
"\n",
"In the redrizzled 'Corrected' WHT image, only a single exposure contributed to the darkest horizontal bars which have a value of ~200 seconds. The adjacent grey bars have a weight ~500 seconds and the rest of the WHT image is ~700 seconds. Users will need to experiment with the persistence masking thresholds and consider the size of the dithers in their individual datasets to best combine t"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"wht =fits.getdata('ica529030_drz.fits',ext=2)\n",
"wht_corr =fits.getdata('ica529030_pcorr_drz.fits',ext=2)\n",
"\n",
"fig=plt.figure(figsize=(20,8))\n",
"ax1=fig.add_subplot(1,2,1)\n",
"ax2=fig.add_subplot(1,2,2)\n",
"\n",
"ax1.imshow(wht, vmin=0,vmax=800,cmap='Greys_r',origin='lower')\n",
"ax2.imshow(wht_corr,vmin=0,vmax=800,cmap='Greys_r',origin='lower')\n",
"\n",
"ax1.set_title('ica529030_drz.fits (Pipeline WHT)',fontsize=20)\n",
"ax2.set_title('ica529030_pcorr_drz.fits (Corrected WHT)',fontsize=20)\n",
"\n",
"ax1.plot([0,300,300,0,0],[250,250,350,350,250],c='red')\n",
"ax2.plot([0,300,300,0,0],[250,250,350,350,250],c='red')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

0 comments on commit 9d14f3e

Please sign in to comment.