notebooks/MAST/TESS/beginner_tess_tap_search/beginner_tess_tap_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download MAST TESS Light Curves Within an FFI Footprint Using TAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> This notebook is a demo for accessing Transiting Exoplanet Survey Satellite (TESS) data in the Common Archive Observation Model (CAOM) catalog at MAST, using a Virtual Observatory standard Table Access Protocol (TAP) service.\n",
    "***\n",
    "### Table of Contents\n",
    "\n",
    "1. [TAP Service Introduction](#TAP-Service-Introduction)\n",
    "2. [Imports](#Imports)\n",
    "3. [Service Specific Configuration](#Service-Specific-Configuration)\n",
    "4. [Connecting to the TAP Service](#Connecting-to-the-TAP-Service)\n",
    "5. [Use Case: Getting light curves from a sector, camera, and chip](#Use-Case:-Getting-images-from-a-sector,-camera,-and-chip)\n",
    "    - [Step 1: Getting the footprint](#Step-1:-Getting-the-footprint)\n",
    "    - [Step 2: Getting an inventory of TESS lightcurves within the footprint](#Step-2:-Getting-an-inventory-of-TESS-lightcurves-within-the-footprint)\n",
    "5. [Additional Resources](#Additional-Resources)\n",
    "6. [About This Notebook](#About-this-Notebook)\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TAP Service Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Table Access Protocol (TAP) services allow more direct and flexible access to astronomical data than the simpler types of IVOA standard data services. Queries are built with the SQL-like Astronomical Data Query Language (ADQL), and can include geographic / spatial queries as well as filtering on other characteristics of the data. This also allows the user fine-grained control over the returned columns, unlike the fixed set of coumns retunred from cone, image, and spectral services.\n",
    "\n",
    "For this example, we'll be using the astropy affiliated PyVO client, which is interoperable with other valid TAP services, including those at MAST. PyVO documentation is available at ReadTheDocs: https://pyvo.readthedocs.io\n",
    "\n",
    "We'll be using PyVO to call the CAOM Catalog TAP service at MAST and filter the results for TESS-related information. The schema for this catalog is an IVOA standard, and is also described within the service itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the pyvo library as our client to the data service.\n",
    "import pyvo as vo\n",
    "\n",
    "# For handling ordinary astropy Tables in responses\n",
    "from astropy.table import Table\n",
    "\n",
    "# For displaying and manipulating some types of results\n",
    "import requests\n",
    "import astropy\n",
    "import time\n",
    "\n",
    "# To allow display tweaks for wider response tables\n",
    "from IPython.display import display\n",
    "from IPython.core.display import HTML\n",
    "\n",
    "# suppress unimportant unit warnings from many TAP services\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", module=\"astropy.io.votable.*\")\n",
    "warnings.filterwarnings(\"ignore\", module=\"pyvo.utils.xml.elements\")\n",
    "warnings.filterwarnings(\"ignore\", module=\"pyvo.io.vosi.vodataservice\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Service Specific Configuration\n",
    "\n",
    "Every TAP service has a \"Base URL\" plus associated endpoints for synchronous and asynchronous queries, as well as status and capability information, and sometimes service-provided sample queries. The endpoints are predefined in the TAP standard, so clients can infer them using the base. We therefore only have to provide astroquery that base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TAP_URL = \"https://vao.stsci.edu/caomtap/tapservice.aspx\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting to the TAP Service\n",
    "\n",
    "The PyVO library is able to connect to any TAP service, given the \"base\" URL as noted in metadata registry resources describing the service. The CAOM TAP service at MAST has access to TESS FFI and time series, including file URLs for download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TAP_service = vo.dal.TAPService(TAP_URL)\n",
    "TAP_service.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List available tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TAP_tables = TAP_service.tables\n",
    "for tablename in TAP_tables.keys():\n",
    "    if not \"tap_schema\" in tablename:  \n",
    "        TAP_tables[tablename].describe()\n",
    "        print(\"Columns={}\".format(sorted([k.name for k in TAP_tables[tablename].columns ])))\n",
    "        print(\"----\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Case: Getting images from a sector, camera, and chip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Getting the footprint\n",
    "\n",
    "For our purposes, any one footprint related to a sector, camera, and chip combination is good enough. We are not currently accounting for small movements of the spacecraft to form a composite footprint. Observation IDs for this mission are constructed based on sector, camera, and chip combination, and we can use this to launch our footprint search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sector = '1'\n",
    "camera = '1'\n",
    "chip = '2'\n",
    "observationIDwildcard = 'tess%-s000{}-{}-{}-%-s'.format(sector, camera, chip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we query for a single ID and footprint to start with. For filters, we use the TESS mission, target types associated with light curves instead of full frame images, and the sector number. Note that the sector ID (sequence number) is in a different table in the CAOM database than most of the metadata we want, so we have to join these tables based on the shared observation ID.\n",
    "\n",
    "This query uses an asynchronous job with longer timeouts in case of connection issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = TAP_service.run_async(f\"\"\"\n",
    "            SELECT top 1 obs_id, s_region  \n",
    "            FROM dbo.caomobservation JOIN ivoa.obscore on dbo.caomobservation.observationID = ivoa.obscore.obs_id \n",
    "            WHERE collection = 'TESS' and trgType = 'FIELD' and \n",
    "            sequenceNumber = {sector} and\n",
    "            observationID like '{observationIDwildcard}'\n",
    "            \"\"\")\n",
    "\n",
    "footprint_results = job.to_table()\n",
    "footprint_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Getting an inventory of TESS lightcurves within the footprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we take the footprint from the above query and find all lightcurves intersecting with this footprint, in ALL sectors. Depending on where this is in the sky, there could be responses only in the original sector, or there could be overlaps with other sectors. There would be more sector overlap near the poles, for instance. By filtering on obs_collection = TESS, we filter based on the TESS mission and exclude High Level Science Products (including those based on TESS).\n",
    "\n",
    "The format must be reformatted for our next query. We separate the shape from the list of vertices, which themselves must be comma-separated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "footprint = footprint_results['s_region'][0]\n",
    "footprintShape = footprint[0:footprint.find(' ')]\n",
    "footprintVertices = footprint[footprint.find(' '):].strip().replace(' ', ', ')\n",
    "\n",
    "print(footprintShape)\n",
    "print(footprintVertices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the footprint has been isolated and reformatted, we perform another query listing all lightcurves (minus the data validation timeseries, whose files end in _dvt) by their target name, sector, RA and Dec, as well as returning the access url for each FITS file and its estimated file size. We are doing this as an asynchronous query, which handles longer response times, just in case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = TAP_service.run_async(\"\"\"\n",
    "            SELECT target_name, sequenceNumber as sector, s_ra, s_dec, access_url, access_estsize, obs_id \n",
    "            FROM dbo.caomobservation JOIN ivoa.obscore on dbo.caomobservation.observationID = ivoa.obscore.obs_id \n",
    "            WHERE \n",
    "            obs_collection = 'TESS' and dataproduct_type = 'timeseries' \n",
    "            and access_url like '%lc.fits' and \n",
    "            CONTAINS(POINT('ICRS',s_ra,s_dec),{}('ICRS', {}))=1\n",
    "            ORDER BY obs_id\n",
    "            \"\"\".format(footprintShape, footprintVertices))\n",
    "\n",
    "TAP_results = job.to_table()\n",
    "TAP_results          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned data is in an astropy table; you can manipulate it to do more ordering or filtering. To download individual files or the whole set, you can use the <i>access_url</i> column, as below.\n",
    "\n",
    "Python's requests library lets you download files from a URL. The downloads will appear in the directory where your notebook is running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: first result row:\n",
    "single_url = TAP_results['access_url'][0]\n",
    "filename = TAP_results['obs_id'][0] + \"_lc.fits\"\n",
    "r = requests.get(single_url, allow_redirects=True)\n",
    "open(filename, 'wb').write(r.content)\n",
    "print('File downloaded: {} bytes'.format(r.headers['Content-length']))\n",
    "\n",
    "\n",
    "# Uncomment the code below to download every file in a loop. \n",
    "# Warning: this can take some time as there are ~900 2megabyte files listed from the notebook's original sample query \n",
    "# (see \"True length\" of masked table above).\n",
    "\n",
    "#for rows in TAP_results:\n",
    "#    single_url = rows['access_url']\n",
    "#    filename = rows['obs_id'] + \"_lc.fits\"\n",
    "#    r = requests.get(single_url, allow_redirects=True)\n",
    "#    open(filename, 'wb').write(r.content)\n",
    "#print('All files downloaded.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have problems running requests or would rather save individual files through your browser, you can simply print clickable links, instead, or wrap them in curl or wget calls, which may be different based on your operating system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: first result row:\n",
    "single_url = TAP_results['access_url'][0]\n",
    "print(single_url)\n",
    "\n",
    "# Uncomment the code below to display clickable links for every file in a loop.\n",
    "#for rows in TAP_results:\n",
    "#    single_url = rows['access_url'].decode('UTF-8')\n",
    "#    print(single_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# Additional Resources\n",
    "\n",
    "\n",
    "## Table Access Protocol \n",
    "* IVOA standard for RESTful web service access to tabular data\n",
    "* http://www.ivoa.net/documents/TAP/\n",
    "\n",
    "\n",
    "## Astronomical Query Data Language (2.0)\n",
    "* IVOA standard for querying astronomical data in tabular format, with geometric search support\n",
    "* http://www.ivoa.net/documents/latest/ADQL.html\n",
    "\n",
    "\n",
    "## Common Archive Observation Model (2)\n",
    "* IVOA standard data model whose relational representation this catalog follows\n",
    "* http://www.opencadc.org/caom2/\n",
    "\n",
    "\n",
    "## PyVO\n",
    "* an affiliated package for astropy\n",
    "* find and retrieve astronomical data available from archives that support standard IVOA virtual observatory service protocols.\n",
    "* https://pyvo.readthedocs.io/en/latest/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About this Notebook\n",
    "**Authors:** Scott Fleming & Theresa Dower, STScI Archive Scientists & Software Engineer\n",
    "\n",
    "**Last Updated**: Dec 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: right;\" src=\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\" alt=\"STScI logo\" width=\"200px\"/> "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}