imaging_mode/imaging_mode_stage_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='top'></a>\n",
    "# Imaging Mode Data Calibration: Part 2 - Calibrated Slope Images\n",
    "---\n",
    "**Author**: Bryan Hilbert (hilbert@stsci.edu)| **Latest Update**: June 10, 2022\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <h3><u><b>Notebook Goals</b></u></h3>\n",
    "    <ul>Working with the Stage 2 Calibration Pipeline, we will:</ul>    \n",
    "<ul>\n",
    "    <li>Look at the different ways to call the pipeline</li>\n",
    "    <li>Examine exactly what each pipeline step does to the science data</li>    \n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "* [Introduction](#intro)\n",
    "* [Pipeline Resources and Documentation](#resources)\n",
    "   * [Installation](#installation)\n",
    "   * [Reference Files](#reference_files)\n",
    "* [Imports](#Imports_ID)\n",
    "* [Convenience Functions](#convenience_functions)\n",
    "* [Download Data](#download_data)\n",
    "* [Association Files](#associations)\n",
    "* [Methods for calling steps/pipelines](#calling_methods)\n",
    "* [calwebb_image2 - Calibrated slope images](#image2) \n",
    "   * [Run the entire pipeline](#image2_at_once)\n",
    "       * [Using the run() method](#run_method)\n",
    "       * [Using the call() method](#call_method)\n",
    "       * [Using the command line](#command_line)\n",
    "       * [Exercise: run Image2 pipeline on MIRI data](#exercise)\n",
    "   * [Run the individual pipeline steps](#image2_step_by_step)\n",
    "       * [The `WCS Creation` step](#assign_wcs)\n",
    "       * [The `Flat Fielding` step](#flatfield)\n",
    "       * [The `Photometric calibration` step](#photom)\n",
    "       * [The `Resample` step](#resample)\n",
    "* [Exercise Solution](#exercise_solution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='intro'></a>\n",
    "## Introduction\n",
    "\n",
    "This notebook covers part 2 of the imaging mode data calibration module. In this notebook we'll review Stage 2 of the JWST calibration pipeline for imaging data, also known as *calwebb\\_image2*. \n",
    "\n",
    "The [Stage 2 pipeline](https://jwst-pipeline.readthedocs.io/en/stable/jwst/pipeline/calwebb_image2.html) applies instrumental corrections and calibrations to the slope images output from Stage 1. This includes background subtraction, the creation of a full World Coordinate System (WCS) for the data, application of the flat field, and flux calibration. In most cases the final output is an image in units of surface brightness. Whereas the input files had suffixes of `*_rate.fits*`, the output files have suffixes of `*_cal.fits*`.\n",
    "\n",
    "In addition to the steps above, the Stage 2 pipeline will also run the [resample](https://jwst-pipeline.readthedocs.io/en/stable/jwst/resample/main.html) step on the calibrated images, in order to remove the effects of instrument distortion. This step outputs files with the suffix `*_i2d.fits*` that contain \"rectified\" images. However, these files are meant only for user examination of the data. It is the `*_cal.fits*` files that are passed on to Stage 3 of the pipeline.\n",
    "\n",
    "To illustrate how the steps of the pipeline change the input data, we will download several sample files and run them through the pipeline, examining the results at several places along the way.\n",
    "\n",
    "All JWST imaging mode data, regardless of instrument, are processed through the *calwebb\\_image2* pipeline. The steps and the order in which they are performed is the same for all data. For the purposes of this notebook, we will continue with the processing of the NIRCam data used in the Stage 1 notebook. We will also provide example MIRI files that can be used in a separate exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='resources'></a>\n",
    "## Pipeline Resources and Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several different places to find information on installing and running the pipeline. This notebook will give a shortened description of the steps pulled from the detailed pipeline information pages, but to find more in-depth instructions use the links below.\n",
    "\n",
    "* [JWST Documentation (JDox) for the Stage 2 pipeline](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_image2) including short a short summary of what each step does.\n",
    "\n",
    "* [High-level description of all pipeline stages and steps](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/main.html)\n",
    "\n",
    "* [`jwst` package documentation](https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html) including how to run the pipeline, input/output files, etc.\n",
    "\n",
    "* [`jwst` package GitHub repository, with installation instructions](https://github.com/spacetelescope/jwst/blob/master/README.md)\n",
    "\n",
    "* [**Help Desk**](https://stsci.service-now.com/jwst?id=sc_cat_item&sys_id=27a8af2fdbf2220033b55dd5ce9619cd&sysparm_category=e15706fc0a0a0aa7007fc21e1ab70c2f): **If you have any questions or problems regarding the pipeline, submit a ticket to the Help Desk**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='installation'></a>\n",
    "### Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    During the JWebbinar, we will be working in a pre-existing environment where the <b>jwst</b> package has already been installed, so you won't need to install it yourself.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "If you wish to run this notebook outside of this JWebbinar, you will have to first install the <b>jwst</b> package.<br>\n",
    "\n",
    "For more detailed instructions on the various ways to install the package, see the [installation instructions](https://github.com/spacetelescope/jwst/blob/master/README.md) on GitHub.\n",
    "\n",
    "The easiest way to install the pipeline is via `pip`. Below we show how to create a new conda environment, activate that environment, and then install the latest released version of the pipeline. You can name your environment anything you like. In the lines below, replace `<env_name>` with your chosen environment name.\n",
    "\n",
    ">`conda create -n <env_name> python`<br>\n",
    ">`conda activate <env_name>`<br>\n",
    ">`pip install jwst`\n",
    "\n",
    "If you wish to install the development version of the pipeline, which is more recent than (but not as well tested compared to) the latest released version:\n",
    "\n",
    ">`conda create -n <env_name> python`<br>\n",
    ">`conda activate <env_name>`<br>\n",
    ">`pip install git+https://github.com/spacetelescope/jwst`\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='reference_files'></a>\n",
    "### Reference Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Calibration reference files](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/jwst-data-calibration-reference-files) are a collection of FITS and ASDF files that are used to remove instrumental signatures and calibrate JWST data. For example, the dark current reference file contains a multiaccum ramp of dark current signal to be subtracted from the data during the dark current subtraction step. \n",
    "\n",
    "When running a pipeline or pipeline step, the pipeline will automatically look for any required reference files in a pre-defined local directory. If the required reference files are not present, they will automatically be downloaded from the Calibration Reference Data System (CRDS) at STScI.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    During the JWebbinar, our pre-existing existing environment is set up to correctly use and store calibration reference files, and you do not need to set the environment variables below.\n",
    "</div>\n",
    "    \n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "If you wish to run this notebook outside of this JWebbinar, you will have to specify a local directory in which to store reference files, along with the server to use to download the reference files from CRDS. To accomplish this, there are two environment variables that should be set prior to calling the pipeline. These are the CRDS_PATH and CRDS_SERVER_URL variables. In the example below, reference files will be downloaded to the \"crds_cache\" directory under the home directory.\n",
    "\n",
    ">`$ export CRDS_PATH=$HOME/crds_cache`<br>\n",
    ">`$ export CRDS_SERVER_URL=https://jwst-crds-pub.stsci.edu`<br>\n",
    "OR:<br>\n",
    "`os.environ[\"CRDS_PATH\"] = \"/user/myself/crds_cache\"`<br>\n",
    "`os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds-pub.stsci.edu\"`<br>\n",
    "\n",
    "The first time you run the pipeline, the [CRDS server](https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html#crds) should download all of the context and reference files that are needed for that pipeline run, and dump them into the CRDS_PATH directory. Subsequent executions of the pipeline will first look to see if it has what it needs in CRDS_PATH and anything it doesn't have will be downloaded from the STScI cache. \n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">NOTE: Users at STScI should automatically have access to the Calibration Reference Data System (CRDS) cache for running the pipeline, and can skip setting these environment variables.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of Notebook](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=#Imports_ID></a>\n",
    "## Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import packages necessary for this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Packages that allow us to get information about objects:\n",
    "from glob import glob\n",
    "#Modify the path to a directory on your machine\n",
    "import os\n",
    "os.environ[\"CRDS_PATH\"] = \"/path/to/my/folder/\"\n",
    "os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds.stsci.edu\"\n",
    "import shutil\n",
    "\n",
    "# Numpy library:\n",
    "import numpy as np\n",
    "\n",
    "# To read association file\n",
    "import json\n",
    "\n",
    "# To download data\n",
    "import requests\n",
    "\n",
    "# To examine parameter reference files\n",
    "import asdf\n",
    "\n",
    "# Astropy tools:\n",
    "from astropy.io import fits\n",
    "from astropy.utils.data import download_file\n",
    "from astropy.visualization import ImageNormalize, ManualInterval, LogStretch, LinearStretch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up matplotlib for plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "\n",
    "# Use this version for non-interactive plots (easier scrolling of the notebook)\n",
    "%matplotlib inline\n",
    "\n",
    "# Use this version (outside of Jupyter Lab) if you want interactive plots\n",
    "#%matplotlib notebook\n",
    "\n",
    "# These gymnastics are needed to make the sizes of the figures\n",
    "# be the same in both the inline and notebook versions\n",
    "%config InlineBackend.print_figure_kwargs = {'bbox_inches': None}\n",
    "\n",
    "mpl.rcParams['savefig.dpi'] = 80\n",
    "mpl.rcParams['figure.dpi'] = 80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import JWST pipeline-related modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The entire calwebb_image2 pipeline\n",
    "from jwst.pipeline import calwebb_image2\n",
    "\n",
    "# Individual steps that make up calwebb_image2\n",
    "from jwst.background import BackgroundStep\n",
    "from jwst.assign_wcs import AssignWcsStep\n",
    "from jwst.flatfield import FlatFieldStep\n",
    "from jwst.photom import PhotomStep\n",
    "from jwst.resample import ResampleStep\n",
    "from jwst import datamodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check which version of the pipeline we are running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jwst\n",
    "print(jwst.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='convenience_functions'></a>\n",
    "## Define convenience functions and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define some functions that we will use repeatedly throughout the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Files created in this notebook will be saved\n",
    "# in the current working directory\n",
    "output_dir = './'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_files(files, output_directory, force=False):\n",
    "    \"\"\"Given a tuple or list of tuples containing (URL, filename),\n",
    "    download the given files into the current working directory.\n",
    "    Downloading is done via astropy's download_file. A symbolic link\n",
    "    is created in the specified output dirctory that points to the\n",
    "    downloaded file.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    files : tuple or list of tuples\n",
    "        Each 2-tuple should contain (URL, filename), where\n",
    "        URL is the URL from which to download the file, and\n",
    "        filename will be the name of the symlink pointing to\n",
    "        the downloaded file.\n",
    "        \n",
    "    output_directory : str\n",
    "        Name of the directory in which to create the symbolic\n",
    "        links to the downloaded files\n",
    "        \n",
    "    force : bool\n",
    "        If True, the file will be downloaded regarless of whether\n",
    "        it is already present or not.\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    filenames : list\n",
    "        List of filenames corresponding to the symbolic links\n",
    "        of the downloaded files\n",
    "    \"\"\"\n",
    "    # In the case of a single input tuple, make it a\n",
    "    # 1 element list, for consistency.\n",
    "    filenames = []\n",
    "    if isinstance(files, tuple):\n",
    "        files = [files]\n",
    "        \n",
    "    for file in files:\n",
    "        filenames.append(file[1])\n",
    "        if force:\n",
    "            print('Downloading {}...'.format(file[1]))\n",
    "            demo_file = download_file(file[0], cache='update')\n",
    "            # Make a symbolic link using a local name for convenience\n",
    "            if not os.path.islink(os.path.join(output_directory, file[1])):\n",
    "                os.symlink(demo_file, os.path.join(output_directory, file[1]))\n",
    "        else:\n",
    "            if not os.path.isfile(os.path.join(output_directory, file[1])):\n",
    "                print('Downloading {}...'.format(file[1]))\n",
    "                demo_file = download_file(file[0], cache=True)\n",
    "                # Make a symbolic link using a local name for convenience\n",
    "                os.symlink(demo_file, os.path.join(output_directory, file[1]))\n",
    "            else:\n",
    "                print('{} already exists, skipping download...'.format(file[1]))\n",
    "                continue\n",
    "    return filenames    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_image(data_2d, vmin, vmax, xpixel=None, ypixel=None, title=None,\n",
    "               scale='log', units='MJy/str'):\n",
    "    \"\"\"Function to generate a 2D, log-scaled image of the data, \n",
    "    with an option to highlight a specific pixel.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    data_2d : numpy.ndarray\n",
    "        2D image to be displayed\n",
    "        \n",
    "    vmin : float\n",
    "        Minimum signal value to use for scaling\n",
    "        \n",
    "    vmax : float\n",
    "        Maximum signal value to use for scaling\n",
    "        \n",
    "    xpixel : int\n",
    "        X-coordinate of pixel to highlight\n",
    "        \n",
    "    ypixel : int\n",
    "        Y-coordinate of pixel to highlight\n",
    "        \n",
    "    title : str\n",
    "        String to use for the plot title\n",
    "        \n",
    "    scale : str\n",
    "        Specify scaling of the image. Can be 'log' or 'linear'\n",
    "        \n",
    "    units : str\n",
    "        Units of the data. Used for the annotation in the\n",
    "        color bar\n",
    "    \"\"\"\n",
    "    if scale == 'log':\n",
    "        norm = ImageNormalize(data_2d, interval=ManualInterval(vmin=vmin, vmax=vmax),\n",
    "                              stretch=LogStretch())\n",
    "    elif scale == 'linear':\n",
    "        norm = ImageNormalize(data_2d, interval=ManualInterval(vmin=vmin, vmax=vmax),\n",
    "                              stretch=LinearStretch())\n",
    "    fig = plt.figure(figsize=(8, 8))\n",
    "    ax = fig.add_subplot(1, 1, 1)\n",
    "    im = ax.imshow(data_2d, origin='lower', norm=norm)\n",
    "    \n",
    "    if xpixel and ypixel:\n",
    "        plt.plot(xpixel, ypixel, marker='o', color='red', label='Selected Pixel')\n",
    "\n",
    "    fig.colorbar(im, label=units)\n",
    "    plt.xlabel('Pixel column')\n",
    "    plt.ylabel('Pixel row')\n",
    "    if title:\n",
    "        plt.title(title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of Notebook](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='download_data'></a>\n",
    "## Download Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this module, we will use rate files from a NIRCam simulated imaging exposure that is stored in Box. Let's grab them:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the rate files, association file, and parameter reference file, so that we have inputs to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nircam_info = [('https://stsci.box.com/shared/static/g6316wjr4mv936rlouzdjeq065s7ou6g.fits',\n",
    "                'jw98765001001_01101_00001_nrcb5_rate.fits'),\n",
    "               ('https://stsci.box.com/shared/static/z2xunff1d2g3m3fjxc1fixoz8rjfpl7h.fits',\n",
    "                'jw98765001001_01101_00002_nrcb5_rate.fits'),\n",
    "               ('https://stsci.box.com/shared/static/4xuvt56kr7gix7dx3tntek6wc9kockef.fits',\n",
    "                'jw98765001001_01101_00003_nrcb5_rate.fits'),\n",
    "               ('https://stsci.box.com/shared/static/lzhcnzds2l7mpf92oet1u69uof788u3l.json',\n",
    "                'level2_lw_asn.json'),\n",
    "               ('https://stsci.box.com/shared/static/d4pu8ieyjc27wzoe0of3ajb9vjtvc80g.asdf',\n",
    "                'image2_pipeline_params.asdf')]\n",
    "nircam_files = download_files(nircam_info, output_dir, force=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also download the example MIRI files if you wish to try the exercise of running it through the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "miri_info = [('https://stsci.box.com/shared/static/1ginadxu3c2srg6c0bbt76vh8anzq5lj.json',\n",
    "              'miri_level2_asn.json'),\n",
    "             ('https://stsci.box.com/shared/static/hx63gq4r7zr8lmgxv82ydv56o6h7zerd.fits',\n",
    "              'miri_F770W_exp1_rate.fits'),\n",
    "             ('https://stsci.box.com/shared/static/9ya4qwkoqm4e6y3r1sp58gelgiuup7qk.fits',\n",
    "              'miri_F770W_exp2_rate.fits'),\n",
    "             ('https://stsci.box.com/shared/static/owe2b1k7zu68otx4g3ovj2vbbyjugi5n.fits',\n",
    "              'miri_F770W_exp3_rate.fits')]\n",
    "miri_files = download_files(miri_info, output_dir, force=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='associations'></a>\n",
    "## Association Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Stage 2 pipeline can be called on a single fits file, or a collection of fits files. When calling on multiple files, the input is a json-formatted file called an [\"association\" file](https://jwst-pipeline.readthedocs.io/en/stable/jwst/associations/index.html) that lists each of the fits files to be processed. \n",
    "\n",
    "* An association is a means of identifying a set of exposures that belong together and may be dependent upon one another.\n",
    "* The association concept permits exposures to be calibrated, archived, retrieved, and reprocessed as a set rather than as individual objects.\n",
    "\n",
    "When retrieving your observations from MAST, you will be able to download the association files for your data along with the fits files containing the observations.\n",
    "\n",
    "The association file presents your data files in organized groups. Let's open the level 2 association file for the data to be processed in this notebook and look at its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the file and load into json object\n",
    "asn_file = os.path.join(output_dir, 'level2_lw_asn.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(asn_file) as f_obj:\n",
    "  asn_data = json.load(f_obj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asn_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that the association file begins with a few lines of data that give high-level information about the association. The most important entry here is the `asn_rule` field. Association files have different formats for the different stages of the pipeline. You should be sure that the `asn_rule` matches the pipeline that you will be running. In this case we'll be running the Stage 2 pipeline, and we see that the `asn_rule` mentions \"Level2b\", which is what we want.\n",
    "\n",
    "Beneath these lines, we see the `products` field. This field contains a list of dictionaries that specify the files that belong to this association, and the types of those files. When the Stage 2 pipeline is run on this association file, all files listed here will be run through the calibration steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='calling_methods'></a>\n",
    "## Methods for calling steps/pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three common methods by which the pipeline or pipeline steps can be called. From within python, the `run()` and `call()` methods of the pipeline or step classes can be used. Alternatively, the `strun` command can be used from the command line. Within this notebook, in the section where we [call the entire pipeline](#image2_at_once), we show examples of all three methods. For the remainder of the pipeline steps, we will focus on using the `run()` method.\n",
    "\n",
    "When using the `call()` method or `strun`, optional input parameters can be specified via [parameter reference files](#parameter_reffiles). When using the `run()` method, these parameters are instead specified within python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='parameter_reffiles'></a>\n",
    "## Parameter Reference Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When calling a pipeline or pipeline step using the `call()` method or the command line, [parameter reference files](https://jwst-pipeline.readthedocs.io/en/stable/jwst/stpipe/config_asdf.html#config-asdf-files) can be used to specify values for input parameters. These reference files are [asdf](https://asdf.readthedocs.io/en/stable/) format and appear somewhat similar to json files when examined in a text editor. \n",
    "\n",
    "Versions of parameter reference files containing default parameter values for each step and pipeline are available in CRDS. When using the `call()` method, if you do not specify a parameter reference file name in the call, the pipeline or step will retrieve and use the appropriate file from CRDS, which will then run the pipeline or step with the parameter values in that file. If you provide the name of a parameter reference file, then the parameter values in that file will take precedence. For any parameter not specified in your parameter reference file, the pipeline will use the default value.\n",
    "\n",
    "When using `strun`, the parameter reference file is a required input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the contents of a parameter reference file. We'll open it using the asdf package, and use the tree attribute to see what's inside:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image2_param_reffile = os.path.join(output_dir, 'image2_pipeline_params.asdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im2_reffile = asdf.open(image2_param_reffile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "im2_reffile.tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top part of the file contains various metadata entries about the file itself. Below that, you'll see a `'name'` entry, which lists `Image2Pipeline` as the class to which these parameters apply. The next line contains the `parameters` entry, which lists parameters and values attached to the pipeline itself. Below this is the `steps` entry, which contains a list of dictionaries. Each dictionary refers to one step within the pipeline, and specifies parameters and values that apply to that step. If you look through these entries, you'll see the same parameters and values that we specified manually when using the `run()` method above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Don't forget to close the file\n",
    "im2_reffile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of notebook](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='image2'></a>\n",
    "## The calwebb_image2 pipeline: Calibrated slope images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the sections below, we will run the Stage 2 pipeline using an association file containing several NIRCam exposures. We will first call the entire *calwebb_image2* pipeline itself. The pipeline is a wrapper which will string together all of the appropriate steps in the proper order. The final outputs from this call are a calibrated slope image which is ready to go into the Stage 3 pipeline (with a suffix of `_cal.fits`), as well as a calibrated slope image which has been resampled in order to remove distortion effects (with a suffix of `_i2d.fits`). The latter is only for user-examination. The `_cal.fits` file is used as input to the Stage 3 pipeline. Note that the units in these output images are now physical units (MJy/str), rather than DN/sec.\n",
    "\n",
    "After running the entire pipeline, we will go back to the original uncalibrated slope images and manually run them through each of the steps that comprise the Stage 2 pipeline. For each step we will describe in more detail what is going on and examine how the exposure files have changed.\n",
    "\n",
    "See [Figure 1](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_image2) on the calwebb_image2 algorithm page for a map of the steps are performed on the input data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='image2_at_once'></a>\n",
    "### Run the entire `calwebb_image2` pipeline\n",
    "\n",
    "In this section we show how to run the entire calwebb_image2 pipeline with a single call. \n",
    "\n",
    "We set parameter values for some of the individual steps, save some outputs, etc, and then call the pipeline.\n",
    "\n",
    "We will call the pipeline using the `run()` method and while that is running, we will go over the equivalent `call()` and `strun` commands, examine some of the pipeline log entries that are printed to the screen, and then look at the pipeline output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='run_method'></a>\n",
    "#### Call the pipeline using the run() method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using the `run()` method to execute a pipeline (or step), the pipeline class is first instantiated without the data to be processed. Optional input parameters are specified using attributes of the class instance. Finally, the call to the `run()` method is made and the data are supplied.  See here for [more examples of the run() method](https://jwst-pipeline.readthedocs.io/en/stable/jwst/stpipe/call_via_run.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `run()` method does not take any kind of parameter reference file as input. If you wish to set values for various parameters, you must do that manually. Below, we set several paramaters in order to show how it's done. \n",
    "\n",
    "How do you know what parameters are available to be set and what their default values are? The `spec` property for individual steps will list them. The property is less useful for the pipelines themselves, as it does not show the parameters for the steps compirising the pipeline.\n",
    "\n",
    "All steps and pipelines have several common parameters that can be set. \n",
    "\n",
    "* `save_results` specifies whether or not to save the output of that step/pipeline to a file. The default is False.\n",
    "* `output_dir` is the directory into which the output files will be saved.\n",
    "* `output_file` is the base filename to use for the saved result. Note that each step/pipeline will add a custom suffix onto output_file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the available parameters for the resample step, and manually set some of these in our call to `run()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ResampleStep.spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='detector1_using_run'></a>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "Finally, let's run the pipeline. The output can be a little overwhelming. There will be multiple log entries printed to the screen for each step.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create an instance of the pipeline class\n",
    "image2 = calwebb_image2.Image2Pipeline()\n",
    "\n",
    "# Set some parameters that pertain to the\n",
    "# entire pipeline\n",
    "image2.output_dir = output_dir\n",
    "image2.save_results = True\n",
    "\n",
    "# Set some parameters that pertain to some of\n",
    "# the individual steps\n",
    "image2.resample.pixfrac = 1.0    # this is the default. Set here as an example\n",
    "\n",
    "# Call the run() method\n",
    "image2.run(asn_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell will take a few seconds to run. While we're waiting, let's look at the other two methods that can be used to call the pipeline. Then we'll come back here and look at the log meessages output by this cell so we can see what happened."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='call_method'></a>\n",
    "#### Call the pipeline using the call() method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using the `call()` method, a single command will instantiate and run the pipeline (or step). The input data and optional parameter reference files are supplied in this single command. In this case, any desired input parameters cannot be set after instantiation, as with the `run()` method. See here for [example usage of call() method](https://jwst-pipeline.readthedocs.io/en/stable/jwst/stpipe/call_via_call.html).\n",
    "\n",
    "The commands below will call the pipeline using the `call()` method and will supply the parameter reference file. Since we just ran the pipeline with the `run()` method above, we won't actually execute the call to `call()`. But if you wish to try it out, use the pull-down menu above to change the cell to be 'Code', and then execute it. (Or, Click 'Cell' > 'Cell Type' > 'Code')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "<b>Method #1:</b>\n",
    "Provide the name of the observation file, the pipeline-specific input paramters, and the name of the parameter reference file that specifies step-specific parameters\n",
    "</div>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# NOTE: This is a \"raw\" cell and will not run unless you change to \"code\"\n",
    "call_output = calwebb_image2.Image2Pipeline.call(asn_file, output_dir=output_dir,\n",
    "                                                  save_results=True,\n",
    "                                                  config_file=image2_param_reffile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "<b>Method #2:</b>\n",
    "In this case, build a nested dictionary that specifies parameter values for various steps, and provide it in the call to call().\n",
    "</div>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# NOTE: This is a \"raw\" cell and will not run unless you change to \"code\"\n",
    "parameter_dict = {\"bkg_subtract\": {\"sigma\": 3.0},\n",
    "                  \"resample\": {\"pixfrac\": 1.0}\n",
    "                 }\n",
    "call_output = calwebb_image2.Image2Pipeline.call(asn_file, output_dir=output_dir, save_results=True,\n",
    "                                                 config_file=image2_param_reffile, steps=parameter_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='command_line'></a>\n",
    "#### Call the pipeline from the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling a pipeline or step from the command line is similar to using the `call()` method. The data file to be processed, along with an optional parameter reference file and optional parameter/value pairs can be provided to the `strun` command. See here for [additional examples of command line calls](https://jwst-pipeline.readthedocs.io/en/stable/jwst/introduction.html?highlight=%22command%20line%22#running-from-the-command-line).\n",
    "\n",
    "The cells below contains two different command line commands that use `strun` to call the calwebb_image2 pipeline. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "<b>Method #1:</b>\n",
    "We provide the name of the pipeline class, the association file, and explicitly set pipeline- and step-specific parameters. \n",
    "    \n",
    "```\n",
    "    strun jwst.pipeline.Image2Pipeline level2_lw_asn.json --steps.resample.pixfrac=1.0\n",
    "```\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "<b>Method #2:</b>\n",
    "This version of the command is much more succinct, as the parameter values to be set are all contained within the parameter reference file. The pipeline class is also contained in the parameter reference file, so there is no need to specify it in the command itself.\n",
    "    \n",
    "```\n",
    "    strun image2_pipeline_params.asdf level2_lw_asn.json\n",
    "```\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the filenames from the association file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a list of input file names from the association file\n",
    "input_files = [item['members'][0]['expname'] for item in asn_data['products']]       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a list of the output file names\n",
    "output_files = sorted(glob(os.path.join(output_dir, '*_cal.fits')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the first calibrated output file\n",
    "cal_data = fits.open(output_files[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the contents of the calibrated file\n",
    "cal_data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the header of the SCI extension, to see the information that has been added by the assign WCS and flux calibration steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cal_data['SCI'].header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the name of the `i2d` file associated with the first output file\n",
    "i2d_file = output_files[0].replace('cal.fits', 'i2d.fits')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the data from the i2d file so we can look at it.\n",
    "i2d_data = fits.getdata(i2d_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pipeline_output_view'></a>\n",
    "Display the calibrated slope image and the distortion-free output file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(cal_data['SCI'].data, 0., 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "show_image(i2d_data, 0, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of Notebook](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='exercise'></a>\n",
    "### Exercise: Run calwebb_image2 on MIRI data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try running the pipeline on the uncalibrated MIRI slope files downloaded earlier. We downloaded an association file along with them (NAME HERE), so you can try using that as input to either the `run()` or `call()` methods. Note that MIRI does not currently have a parameter reference file for the Stage 2 pipeline which means you will not be able to use it as input to the `call()` method nor `strun`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See [Exercise Solution](#exercise_solution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "miri_asn_file = 'miri_level2_asn.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the run() method\n",
    "# Create an instance of the pipeline class\n",
    "\n",
    "# Save the pipeline output, and specify the output directory\n",
    "\n",
    "# Set any step-related paramters\n",
    "\n",
    "# Call the run() method and provide the MIRI association file as input\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the call() method:\n",
    "# Specify the output directory and that the outputs should be saved.\n",
    "# Provide the MIRI association file as input.\n",
    "# Since there is no parameter reference file, there is no need to \n",
    "# specify it here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examine one of the output calibrated images\n",
    "# HINT: Open the file and use show_image with \n",
    "# min and max values of 0-100 MJy/str\n",
    "miri_cal_file = 'miri_F770W_exp1_cal.fits'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='image2_step_by_step'></a>\n",
    "## Run the individual pipeline steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the sections below we run the steps contained within calwebb_image2 one at a time, in order to more clearly see what each step is doing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='assign_wcs'></a>\n",
    "### The `WCS creation` step\n",
    "\n",
    "#### Summary\n",
    "\n",
    "This step adds a World Coordinate System (WCS) object to the observation. The WCS object contains transformations between positions on the detector to positions in a world coordinate frame.\n",
    "\n",
    "#### Documentation\n",
    "\n",
    "[Full description](https://jwst-pipeline.readthedocs.io/en/stable/jwst/assign_wcs/main.html) of the step.\n",
    "\n",
    "#### Arguments\n",
    "\n",
    "There are no optional arguments for this step.\n",
    "\n",
    "#### Reference files used\n",
    "\n",
    "The [reference files used](https://jwst-pipeline.readthedocs.io/en/stable/jwst/assign_wcs/reference_files.html) in this step depend on the instrument used. The primary reference file used is the `DISTORTION` reference file, which contains coefficients that can be used to translate between various coordinate systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The assign_wcs step expects an instance of an ImageModel as input, rather than an association file or fits file. So in this case we'll loop over the input files, read them into ImageModel instances, and call the step. Results will be saved to fits files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for filename in input_files:\n",
    "    image = datamodels.ImageModel(filename)\n",
    "    \n",
    "    assign_wcs_step = AssignWcsStep()\n",
    "    assign_wcs_step.output_dir = output_dir\n",
    "    assign_wcs_step.save_results = True\n",
    "    assign_wcs_step.run(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When the output is saved, the assign_wcs step will\n",
    "# attach a suffix of 'assignwcsstep' to the input filename.\n",
    "assign_wcs_output_files = sorted(glob(os.path.join(output_dir, '*_assignwcsstep.fits')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look into the WCS information that this step added to the files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the output file using datamodels, so we have access to the full WCS\n",
    "model = datamodels.ImageModel(assign_wcs_output_files[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full GWCS model is contained in the ASDF extension of the file, and can be seen through the `meta` property. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the WCS info in the calibrated image model \n",
    "model.meta.wcs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several world coordinate systems available in the file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What coordinate frames are available?\n",
    "model.meta.wcs.available_frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the input frame of the WCS object?\n",
    "model.meta.wcs.input_frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the output frame of the WCS object?\n",
    "model.meta.wcs.output_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a transformation function to go from detector pixels to location on the sky."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the transform to go from detector to world coordinates\n",
    "detector_to_world = model.meta.wcs.get_transform('detector', 'world')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a function for the inverse transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "world_to_detector = model.meta.wcs.get_transform('world', 'detector')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the calwebb_image2 pipeline output image from before (the _cal.fits files), and zoom in on an interesting area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "show_image(cal_data[1].data[1000:1150, 860:1010] , 0.3, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Close the file\n",
    "cal_data.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the transformation functions we defined above, we can now easily determine the RA and Dec of some randomly-chosen sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sources_x = [925., 945., 940.]\n",
    "sources_y = [1045, 1183.2, 1120.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call the transform function\n",
    "sources_ra, sources_dec = detector_to_world(sources_x, sources_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sources_ra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sources_dec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now the opposite case: My target is at a given RA and Dec, so where is it in this image?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "targ_ra = 12.012546822378457\n",
    "targ_dec = 12.018984533659786"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call the inverse transform function\n",
    "targ_x, targ_y = world_to_detector(targ_ra, targ_dec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Target located at (x, y) = ({}, {})'.format(targ_x, targ_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='flatfield'></a>\n",
    "## The `Flat Fielding` step\n",
    "\n",
    "#### Summary\n",
    "\n",
    "This step divides the data by a flat field in order to correct for pixel-to-pixel sensitivity variations.\n",
    "\n",
    "#### Documentation\n",
    "\n",
    "[Full description](https://jwst-pipeline.readthedocs.io/en/stable/jwst/flatfield/main.html) of the step.\n",
    "\n",
    "#### Arguments\n",
    "\n",
    "There is a [single optional argument](https://jwst-pipeline.readthedocs.io/en/stable/jwst/flatfield/arguments.html) for this step, which applies only to NIRSpec data.\n",
    "\n",
    "#### Reference files used\n",
    "\n",
    "This step uses the [`FLAT`](https://jwst-pipeline.readthedocs.io/en/stable/jwst/flatfield/reference_files.html) reference file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this and the subsequent steps, we will loop over the files output by the prceding step and run the step. \n",
    "\n",
    "Why not use an association file as input? Because we would need a separate association file for each step since the filenames to be used as input are different in each step. So in order to avoid dealing with many association files, we simply loop over the filenames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for filename in assign_wcs_output_files:\n",
    "    flatfield_step = FlatFieldStep()\n",
    "    flatfield_step.output_dir = output_dir\n",
    "    flatfield_step.save_results = True\n",
    "\n",
    "    flatfield_step.run(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When the output is saved, the flat field step will\n",
    "# attach a suffix of 'flatfieldstep' to the input filename.\n",
    "flatfield_output_files = sorted(glob(os.path.join(output_dir, '*_flatfieldstep.fits')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "before_flat = fits.getdata(assign_wcs_output_files[-1])\n",
    "after_flat = fits.getdata(flatfield_output_files[-1])\n",
    "\n",
    "# Some pixels were saturated in all groups of the integration.\n",
    "# This caused them to have a value of 0.0 in the slope image.\n",
    "# For this display, let's set those pixels equal to 1.0, just\n",
    "# to get a clearer picture.\n",
    "zeros = after_flat == 0\n",
    "before_flat[zeros] = 1.0\n",
    "after_flat[zeros] = 1.0\n",
    "\n",
    "# Recover the flat by taking the ratio of the data before and after\n",
    "# the flat field step\n",
    "flat_ratio = before_flat / after_flat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(after_flat , 0.3, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking the ratio of the data before and after the flat field step, we recover the flat field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(flat_ratio, 0.9, 1.1, scale='linear', units='Flat Field Value')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='photom'> </a>\n",
    "## The `Photometric calibration` step\n",
    "\n",
    "#### Summary\n",
    "\n",
    "This step applies flux (photometric) calibration to the data, converting it from units of ADU/sec to surface brightness. A conversion factor is retrieved from the `PHOTOM` reference file, and the pixel values in the science data are multiplied by this factor. The factor is also saved in the `PHOTMJSR` keyword within the header of the exposure file. The map of relative pixel areas is also appended to the exposure in a new extension called `AREA`. The average pixel area in units of steradians and square arcseconds is also saved in the science extension header, in the `PIXAR_SR` and `PIXAR_A2` keywords.\n",
    "\n",
    "\n",
    "#### Documentation\n",
    "\n",
    "[Full description](https://jwst-pipeline.readthedocs.io/en/stable/jwst/photom/main.html) of the step.\n",
    "\n",
    "#### Arguments\n",
    "\n",
    "There are no optional arguments for this step\n",
    "\n",
    "#### Reference files used\n",
    "\n",
    "This step uses the [`PHOTOM` and `AREA`](https://jwst-pipeline.readthedocs.io/en/stable/jwst/photom/reference_files.html) reference files. The `PHOTOM` reference file contains a table of conversion factors that depend on filter. The `AREA` reference file contains a map of the relative pixel areas across the detector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for filename in flatfield_output_files:\n",
    "    photom_step = PhotomStep()\n",
    "    photom_step.output_dir = output_dir\n",
    "    photom_step.save_results = True\n",
    "    photom_step.run(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the new information that was added to the output file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When the output is saved, the photom step will\n",
    "# attach a suffix of 'photomstep' to the input filename.\n",
    "photom_output_files = sorted(glob(os.path.join(output_dir, '*_photomstep.fits')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open one of the output files and look at the contents\n",
    "hdulist = fits.open(photom_output_files[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdulist.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The primary extension header is updated by the photom step\n",
    "sci_header = hdulist['SCI'].header"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the mean pixel area:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Mean pixel area in steradians: {}, and square arcseconds: {}'\n",
    "      .format(sci_header['PIXAR_SR'], sci_header['PIXAR_A2']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's pull out the science data and the newly-attached AREA extension\n",
    "area_map = hdulist['AREA'].data\n",
    "photom_science_data = hdulist['SCI'].data\n",
    "hdulist.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the new `AREA` extension. Let's have a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "show_image(area_map, 0.95, 1.05, scale='linear', units='Relative Pixel Area')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(photom_science_data, 0.2, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='resample'> </a>\n",
    "## The `Resample` step\n",
    "\n",
    "#### Summary\n",
    "\n",
    "This step resamples the calibrated slope image onto a distortion-free pixel grid. The output is a file with the suffix `_i2d.fits`. This file is for user-examination only. In the Stage 3 pipeline, the resample step will be called again when combining multiple images and creating the final, distortion-free mosaic image.\n",
    "\n",
    "#### Documentation\n",
    "\n",
    "[Full description](https://jwst-pipeline.readthedocs.io/en/stable/jwst/resample/main.html) of the step.\n",
    "\n",
    "#### Arguments\n",
    "\n",
    "There is a list of [optional Astrodrizzle-style](https://jwst-pipeline.readthedocs.io/en/stable/jwst/resample/arguments.html) input parameters that can be used to customize the resampling process.\n",
    "\n",
    "#### Reference files used\n",
    "\n",
    "This step uses the [`DRIZPARS`](https://jwst-pipeline.readthedocs.io/en/stable/jwst/resample/reference_files.html) reference file. This file contains Astrodrizzle-style keywords that can be used to control the details of the resampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what parameters are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ResampleStep.spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for filename in photom_output_files:\n",
    "    resample_step = ResampleStep()\n",
    "    resample_step.output_dir = output_dir\n",
    "    resample_step.save_results = True\n",
    "    resample_step.run(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When the output is saved, the resample step will\n",
    "# attach a suffix of 'resamplestep' to the input filename.\n",
    "resample_output_files = sorted(glob(os.path.join(output_dir, '*_resamplestep.fits')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the data from the three resampled output files so we can look\n",
    "# at the data\n",
    "resample_data_0 = fits.getdata(resample_output_files[0])\n",
    "resample_data_1 = fits.getdata(resample_output_files[1])\n",
    "resample_data_2 = fits.getdata(resample_output_files[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(resample_data_0, 0.2, 1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(resample_data_1, 0.2, 1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(resample_data_2, 0.2, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's compare this to the data immediately prior to the resample step, in order to highlight the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(photom_science_data, 0.2, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the array size has changed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(photom_science_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(resample_data_0.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of Notebook](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='exercise_solution'></a>\n",
    "## Exercise Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run calwebb_image2 pipeline on MIRI data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try running the pipeline on the downloaded MIRI data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "miri_asn_file = 'miri_level2_asn.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the run() method\n",
    "# Create an instance of the pipeline class\n",
    "miri_im2 = calwebb_image2.Image2Pipeline()\n",
    "\n",
    "# Save the pipeline output, and specify the output directory\n",
    "miri_im2.output_dir = output_dir\n",
    "miri_im2.save_results = True\n",
    "\n",
    "# Set any step-related paramters\n",
    "miri_im2.resample.pixfrac = 1.0\n",
    "\n",
    "# Call the run() method and provide the MIRI association file as input\n",
    "miri_im2.run(miri_asn_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the call() method:\n",
    "# Specify the output directory and that the outputs should be saved.\n",
    "# Provide the MIRI association file as input.\n",
    "# Since there is no parameter reference file, there is no need to \n",
    "# specify it here.\n",
    "call_output = calwebb_image2.Image2Pipeline.call(miri_asn_file, output_dir=output_dir,\n",
    "                                                 save_results=True,\n",
    "                                                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examine one of the output calibrated images\n",
    "# HINT: Open the file and use show_image with \n",
    "# min and max values of 0-100 MJy/str\n",
    "miri_cal_file = 'miri_F770W_exp1_cal.fits'\n",
    "miri_cal = datamodels.open(miri_cal_file)\n",
    "show_image(miri_cal.data, 0, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return to [Exercise](#exercise)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the end of the Stage 2 pipeline for imaging. The outputs from this, along with a level 3 association file, can now be used as input to the Stage 3 pipeline, where they will be combined into a single mosaic image. This will be shown in the third notebook for this module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of notebook](#top)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}