Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

For commenting on 01-05-Calibration-overview.ipynb (part of review-865a5c2) #218

Closed
wants to merge 1 commit into from
Closed

For commenting on 01-05-Calibration-overview.ipynb (part of review-865a5c2) #218

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
341 changes: 341 additions & 0 deletions notebooks/01-05-Calibration-overview.ipynb
View file
Open in desktop
@@ -0,0 +1,341 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Calibration overview\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An image of the sky contains counts from several sources. The task of data\n",
"reduction (another name for image calibration) is to remove all non-celestial\n",
"counts from the image and to correct for non-uniform sensitivity.\n",
"\n",
"At the end of the previous notebook we arrived at an expression for the counts\n",
"in a science image in terms of the sources of counts:\n",
"\n",
"$$\n",
"\\text{raw image} = \\text{bias} + \\text{noise} + \\text{dark current} + \\text{flat} \\times (\\text{sky} + \\text{stars}).\n",
"$$\n",
"\n",
"Solving for the counts just from the stars is as follows:\n",
"\n",
"$$\n",
"\\text{stars} + \\text{noise} = \\frac{\\text{raw image} - \\text{bias} - \\text{dark current}}{\\text{flat}} - \\text{sky}\n",
"$$\n",
"\n",
"**It is *impossible* to remove the noise from the raw image because the noise is\n",
"random.**\n",
"\n",
"The dark current is typically calculated from a *dark frame* (aka dark image).\n",
"Such an image has bias and read noise in it as well, so:\n",
"\n",
"$$\n",
"\\text{dark current} + \\text{noise} = (\\text{dark frame} - \\text{bias})/(\\text{dark exposure time})\n",
"$$\n",
"\n",
"Once again, note that the noise cannot be removed."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## This noise cannot be removed from CCD images\n",
"\n",
"To demonstrate that you cannot remove the noise from an image, let's construct\n",
"an image with just stars and noise and try to subtract a noise image created\n",
"with the same parameters. The amount of noise here is exaggerated to make it\n",
"clear in the images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"%matplotlib inline\n",
"from matplotlib import pyplot as plt\n",
"\n",
"from astropy.visualization import hist\n",
"from astropy.stats import histogram\n",
"\n",
"import image_sim as imsim\n",
"from convenience_functions import show_image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use custom style for larger fonts and figures\n",
"plt.style.use('guide.mplstyle')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### First, some stars with noise\n",
"\n",
"The image below shows stars (the larger \"blobs\" in the image) but shows quite a\n",
"bit of noise as well (the much smaller \"dots\")."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image = np.zeros([2000, 2000])\n",
"gain = 1.0\n",
"noise_amount = 1500 \n",
"\n",
"stars_with_noise = imsim.stars(image, 50, max_counts=2000, fwhm=10) + imsim.read_noise(image, noise_amount, gain=gain)\n",
"\n",
"show_image(stars_with_noise, cmap='gray', percu=99.9)\n",
"plt.title('Stars with noise')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now an *incorrect* attempt at reducing noise\n",
"\n",
"Notice that the call to the noise function has exactly the same arguments as\n",
"above, in much the same way your camera's electronics will have the same noise\n",
"properties every time you read out an image.\n",
"\n",
"However, the amount of noise has **increased**, not decreased. It's much harder\n",
"to pick out the stars in this image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"incorrect_attempt_to_remove_noise = stars_with_noise - imsim.read_noise(image, noise_amount, gain=gain)\n",
"\n",
"show_image(incorrect_attempt_to_remove_noise, cmap='gray', percu=99.9)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Every image has noise\n",
"\n",
"Every image, including calibration images like bias and dark frames, has noise.\n",
"If we tried to calibrate images by taking a single bias image and a single dark\n",
"image, the final result might well look worse than before the image is reduced.\n",
"\n",
"For demonstration, we'll see what happens below.\n",
"\n",
"Note that here we construct *realistic* bias and dark, but leave read noise out\n",
"of the flat; we'll return to that point later."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### First, set parameters for the CCD\n",
"\n",
"These are the same as in the previous notebook, except for the read noise, which\n",
"is 700$e-$, 100 times larger than in the previous notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gain = 1.0\n",
"star_exposure = 30.0\n",
"dark_exposure = 60.0\n",
"dark = 0.1\n",
"sky_counts = 20\n",
"bias_level = 1100\n",
"read_noise_electrons = 700\n",
"max_star_counts = 2000"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generate the images, with noise"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bias_with_noise = (imsim.bias(image, bias_level, realistic=True) + \n",
" imsim.read_noise(image, read_noise_electrons, gain=gain))\n",
"\n",
"dark_frame_with_noise = (imsim.bias(image, bias_level, realistic=True) + \n",
" imsim.dark_current(image, dark, dark_exposure, gain=gain, hot_pixels=True) +\n",
" imsim.read_noise(image, read_noise_electrons, gain=gain))\n",
"\n",
"flat = imsim.sensitivity_variations(image)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"realistic_stars = (imsim.stars(image, 50, max_counts=max_star_counts) +\n",
" imsim.dark_current(image, dark, star_exposure, gain=gain, hot_pixels=True) +\n",
" imsim.bias(image, bias_level, realistic=True) +\n",
" imsim.read_noise(image, read_noise_electrons, gain=gain)\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Uncalibrated image\n",
"\n",
"Below we display the uncalibrated image; in a moment we'll compare it to the\n",
"calibrated version. Even though they don't stand out there really are stars in\n",
"it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(12, 12))\n",
"show_image(realistic_stars, cmap='gray', percu=99.9, figsize=(9, 9))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reduce (calibrate) the star image\n",
"\n",
"First we calculate the dark current, scaled to the exposure time of our light\n",
"image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"scaled_dark_current = star_exposure * (dark_frame_with_noise - bias_with_noise) / dark_exposure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we subtract the bias and dark current from the star image and then apply\n",
"the flat correction."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"calibrated_stars = (realistic_stars - bias_with_noise - scaled_dark_current) / flat"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_image(calibrated_stars, cmap='gray', percu=99.9)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reducing the image cleans up the image a bit\n",
"\n",
"The stars stand more clearly than in the unreduced image.\n",
"\n",
"This image does not look *much* better than the uncalibrated image, but remember\n",
"that the read noise used in this simulated image, 700 $e^-$ per pixel, is\n",
"unrealistically high."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reducing the image increases the noise in the image\n",
"\n",
"The histogram below shows pixel values before and after calibration. The width\n",
"of the distribution is a measure of the read noise. As expected, reducing the\n",
"image increases the read noise. One reason one takes several calibration images\n",
"of each type is to reduce the amount of noise in the calibration image. That\n",
"will, in turn, keep the noise in the final image as small as possible."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(9, 9))\n",
"hist(calibrated_stars.flatten(), bins='freedman', label='calibrated star image', alpha=0.5)\n",
"hist(stars_with_noise.flatten(), bins='freedman', label='raw star image', alpha=0.5)\n",
"plt.legend()\n",
"plt.grid()\n",
"plt.xlabel('Count level in image')\n",
"plt.ylabel('Number of pixels with that count');"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
}
},
"nbformat": 4,
"nbformat_minor": 2
}