notebooks/MAST/TESS/interm_gi_query/interm_gi_query.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"intro_ID\"></a>\n",
    "# Intermediate:  Search and Download GI Program Light Curves\n",
    "\n",
    "## Introduction\n",
    "This notebook uses the MAST Portal's advanced search options to retrieve the light curves for a single guest investigator program.  The notbook will show how to do an advanced query on the MAST's database of holdings, determine which data files are associated with those observations and then download the files of interest.  \n",
    "\n",
    "For more information about the TESS mission and collected data, visit the [MAST's TESS homepage](http://archive.stsci.edu/tess/). To read more details about TESS Data Products, look in the [TESS Science Product Description Document](https://archive.stsci.edu/missions/tess/doc/EXP-TESS-ARC-ICD-TM-0014.pdf). A list of Guest Investigator programs can be found at the [TESS GI List of Approved Programs](https://heasarc.gsfc.nasa.gov/docs/tess/approved-programs.html).\n",
    "\n",
    "### Table of Contents \n",
    "<br> [Query MAST](#query_mast)  <br> [Retrieve Product List](#product_list) <br> [Download Data](#download) <br> [Summarize Code](#summary) <br> [About this Notebook](#about_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"imports_ID\"></a>\n",
    "## Imports\n",
    "- *Observations* module from astroquery.mast is needed to make the query and download.\n",
    "- *astropy.io.fits* module is needed to view the downloaded fits file. \n",
    "<p>\n",
    "For information on how to install astroquery see their [instructions](https://astroquery.readthedocs.io/en/latest/index.html). At the time of writing this requires the latest version on pip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from astroquery.mast import Observations\n",
    "from astropy.io import fits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"query_mast\"></a>\n",
    "## Query the MAST CAOM Database\n",
    "\n",
    "We want to retrieve TESS timeseries data (lightcurve and target pixel files) for the Guest Investigator program G011183 from Sector 1 (PI: Stephen Kane). We will need to query the MAST holdings database for the observations.  First we will simply count the number of observations and then we will request a table of those observations. \n",
    "\n",
    "All the filter names are listed on the [MAST CAOM Field Description Page](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). Or, can be found by hovering over the column names after doing a search in the MAST portal. \n",
    "\n",
    "The filters we will need to use are, \n",
    "- **obs_collection** to specify that we want TESS data\n",
    "- **dataproduct_type** to specify that we want timeseries data\n",
    "- **sequence_number** to specify that we want sector 1 \n",
    "- **propsal_id** to specify the GI program number.  \n",
    "\n",
    "Remember that more than one GI can propose for the same target so we need wild cards around the name of the program in our query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obsCount = Observations.query_criteria_count(obs_collection = \"TESS\",\n",
    "                                             dataproduct_type = [\"timeseries\"],\n",
    "                                             sequence_number = 1,\n",
    "                                             proposal_id = \"*G011183*\")\n",
    "print(\"Number of Observations: %i\" % obsCount)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obsTable = Observations.query_criteria(obs_collection = \"TESS\",\n",
    "                                       dataproduct_type = [\"timeseries\"],\n",
    "                                       sequence_number = 1,\n",
    "                                       proposal_id = \"*G011183*\")\n",
    "obsTable[0:5]['obsid','proposal_id','obs_id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"product_list\"></a>\n",
    "## Retrieve the list of Data Products\n",
    "\n",
    "Next we use astroquery to retrieve the list of data products that are associated with each observation.  \n",
    "\n",
    "We will only ask for the data products associated with the first five.  The [0:6] can be removed from the code below to get all the observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataProducts = Observations.get_product_list(obsTable[0:6])\n",
    "dataProducts.colnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataProducts['obsID', 'productFilename', 'description']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Light curves\n",
    "<a id=\"download\"></a>\n",
    "We limit our list of data products to just those with the **description** set to \"Light curves\" because we just want this type of data file. We then download those products and print out the manifest to show the download location. Finally we use fits.info to show the structure of the fits file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "want = dataProducts['description'] == \"Light curves\"\n",
    "print(dataProducts[want])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "manifest = Observations.download_products(dataProducts[want])\n",
    "print(manifest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fits.info(manifest['Local Path'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summarize Code\n",
    "<a id=\"summary\"></a>\n",
    "Here is a summary of the code required to do the problem described above, without all the outputs and investigations getting in the way.  The query information is brought to the top so that it can easily be changed. For instance you might want to do a create a query on a different GI program number. One thing you will notice is that astroquery is smart enough to know that you have already downloaded the files and so you will get the message \"Found cached file\" instead of \"Downloading URL\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query Information\n",
    "mission = \"TESS\"\n",
    "dataProdType = [\"timeseries\"]\n",
    "GiProgram = \"*G011183*\"\n",
    "fileType = \"Light curves\"\n",
    "sector = 1\n",
    "\n",
    "# Query Mast Holdings\n",
    "obsTable = Observations.query_criteria(obs_collection = mission,\n",
    "                                       dataproduct_type = dataProdType,\n",
    "                                       sequence_number = sector,\n",
    "                                       proposal_id = GiProgram)\n",
    "\n",
    "# Get Product List\n",
    "dataProducts = Observations.get_product_list(obsTable[0:2])\n",
    "want = dataProducts['description'] == fileType\n",
    "\n",
    "# Download Data\n",
    "manifest = Observations.download_products(dataProducts[want])\n",
    "print(manifest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"about_id\"></a>\n",
    "## About this Notebook\n",
    "**Author:** Susan E. Mullally, STScI Archive Scientist\n",
    "<br>**Updated On:** 2018-11-29"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Top of Page](#title_ID)\n",
    "<img style=\"float: right;\" src=\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\" alt=\"STScI logo\" width=\"200px\"/> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}