docs/source/tutorials/2-creating-light-curves/2-2-kepler-noise-1-data-gaps-and-quality-flags.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NpbbMzECbvTv"
   },
   "source": [
    "# Instrumental Noise in _Kepler_ and _K2_ #1: Data Gaps and Quality Flags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L57_f2QscO85"
   },
   "source": [
    "## Learning Goals\n",
    "\n",
    "By the end of this tutorial, you will:\n",
    "\n",
    "- Have a working knowledge of _Kepler_ quality flags and be able to access them in light curve and TPF data.\n",
    "- Be able to identify the cause of various types of gaps in the data.\n",
    "- Understand the most common reasons for individual cadence exclusions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D0ixWhW-cTfm"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook is the first part of a series on identifying instrumental and systematic sources of noise in _Kepler_ and _K2_ data. In this tutorial, we will look at practical examples of data gaps and single-cadence quality issues in _Kepler_ data, and learn to identify their causes. Assumed knowledge for this tutorial is a good familiarity with light curve and target pixel file (TPF) data products, and the ability to work with their metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f0yfyjrD2ERM"
   },
   "source": [
    "## Imports\n",
    "\n",
    "We'll use **[Lightkurve](https://docs.lightkurve.org/)** for downloading and handling _Kepler_ data throughout this tutorial. We'll also use **[NumPy](https://numpy.org/)** to perform a few auxiliary functions, and **[Matplotlib](https://matplotlib.org/)** for plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LQ6s2Mlwc4l9"
   },
   "outputs": [],
   "source": [
    "import lightkurve as lk\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fInD_fMu2pmx"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XMPMikC-cq3N"
   },
   "source": [
    "## 1. Background\n",
    "\n",
    "The _Kepler_ space telescope observed the same patch of sky for four continuous years, between 2009 and 2013. Over the course of 18 observing quarters, it collected light curves and pixel data for 150,000 stars in 30-minute Long Cadence mode, and 512 stars per quarter in one-minute Short Cadence mode. Following the failure of two of the telescope's four reaction wheels, the telescope continued as the _K2_ mission, which observed along the ecliptic plane for 20 campaigns.\n",
    "\n",
    "In this tutorial, we'll explore some quality issues that arose during both the _Kepler_ and _K2_ missions, and learn how to identify and mitigate them. It's important to note that many of these issues only appear in the calibrated pixels or simple aperture photometry (SAP) data, and more still are removed by a quality masking process which we'll discuss in the next section. If you're working with presearch data conditioning SAP (PDCSAP) light curves, you won't run into many of these issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_v8jLrQBctUM"
   },
   "source": [
    "### 1.1 Quality flags\n",
    "\n",
    "Before we look at some practical examples in time series data, let's familiarize ourselves with how data gaps and single-cadence quality events are identified in _Kepler_ data files. Every _Kepler_ FITS file has a [QUALITY](https://docs.lightkurve.org/reference/api/lightkurve.KeplerTargetPixelFile.quality.html?highlight=quality#lightkurve.KeplerTargetPixelFile.quality) column, which contains a quality flag for each individual data cadence. These flags comprise of one or more binary bits, which are expressed as an integer. A handy reference document for _Kepler_'s quality flags is Table 2-3 in the [MAST Kepler Archive Manual](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/k2/_documents/MAST_Kepler_Archive_Manual_2020.pdf).\n",
    "\n",
    "Let's start by downloading the data we'll work with throughout this tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KORJOZglc-Lx"
   },
   "outputs": [],
   "source": [
    "lc = lk.search_lightcurve('KIC 2436365', author='Kepler', quarter=2).download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5ceFMui-ASy2"
   },
   "source": [
    "By default, Lightkurve downloads quality-masked data. This means that a variety of cadences with a non-zero quality flag will already be removed from the light curves or TPFs you download using the instruction above.\n",
    "\n",
    "For this tutorial, we are also going to download some TPF data with no quality mask applied.  We can do this by passing the optional `quality_bitmask=0` argument to the [download()](https://docs.lightkurve.org/reference/search.html?highlight=download) method as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "h695JEhsASF7"
   },
   "outputs": [],
   "source": [
    "tpf = lk.search_targetpixelfile('KIC 2436365', author='Kepler', quarter=2).download(quality_bitmask=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BRbY5HdRDlZ6"
   },
   "source": [
    "Passing `quality_bitmask=0` has the effect of including every cadence in the data, even those with serious quality issues or NaN ([not a number](https://en.wikipedia.org/wiki/NaN)) values in the flux. This is not necessarily recommended when using Lightkurve, but here it will allow us to explore a wide variety of data quality issues, many of which will be useful to know about if you're working directly with FITS files from MAST.\n",
    "\n",
    "Now that we have our data, let's have a look at the range of quality flags present. Remember, this is the unmasked data, so every flagged cadence is included.\n",
    "\n",
    "Here, we're using the NumPy function [`unique()`](https://numpy.org/doc/stable/reference/generated/numpy.unique.html), which takes an array as its input and returns every unique value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20120,
     "status": "ok",
     "timestamp": 1600725930587,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "E8QMKBgHetS_",
    "outputId": "945b812b-fee4-4ee1-dc2d-c065937852ad"
   },
   "outputs": [],
   "source": [
    "np.unique(tpf.quality)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B4Jrg5JMEBM-"
   },
   "source": [
    "You'll notice that some of these appear to be integers that correspond to bits — that is, powers of two — but others are additive. This indicates that multiple quality issues are present in a particular cadence.\n",
    "\n",
    "We can use Lightkurve's `decode()` function for accessing the information stored in each quality flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 459
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20109,
     "status": "ok",
     "timestamp": 1600725930588,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "m4VWYkUyevyC",
    "outputId": "b60a4dc9-8489-45b4-e567-3ff7705cb05c"
   },
   "outputs": [],
   "source": [
    "for flag in np.unique(tpf.quality):\n",
    "    print(flag, lk.KeplerQualityFlags.decode(flag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "haTYda77r4X1"
   },
   "source": [
    "We can use Python's \"bitwise and\" operator (&) to select the cadence numbers affected by a specific quality flag as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20098,
     "status": "ok",
     "timestamp": 1600725930589,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "XH6tC4x5rwZa",
    "outputId": "33b27a6f-a12b-4154-e1e4-25103b804b14"
   },
   "outputs": [],
   "source": [
    "tpf.cadenceno[(tpf.quality & 64) > 0]  # cadence numbers flagged for \"Argabrightening\" (flag 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pd3VEVtyFLzh"
   },
   "source": [
    "In the following sections, we'll walk through the majority of these data quality events, and look at practical examples in the light curve and TPF we downloaded above.\n",
    "\n",
    "Some of these flags are not covered in this tutorial, such as rolling bands. For more information on other data quality issues, see other tutorials in this series, as well as the various [_Kepler_ Data Handbooks](https://archive.stsci.edu/missions-and-data/kepler/documents)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HXfCyGQSeiNi"
   },
   "source": [
    "## 2. Common Data Gaps\n",
    "\n",
    "The nominal _Kepler_ mission observed one area continuously between 2009 and 2013 — but of course, there were various necessary breaks in that continuity. For example, the telescope rotated at the end of each quarter, which necessitated a break in data collection.\n",
    "\n",
    "We can use quality flags to identify these various events, but often it's more convenient to check the _Kepler_ Mission Timeline, from the [_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf):\n",
    "\n",
    "![Kepler Mission Timeline from the Data Characteristics Handbook, a calendar showing Quarters 0–17 from 2009–13, with major events and data gaps marked.](kepler-mission-timeline.png)\n",
    "\n",
    "Throughout this section we'll explore the most common reasons for gaps in _Kepler_ data: monthly data downlinks, safe modes, and coarse pointing/loss of fine pointing. Though we're looking at _Kepler_ data here, these data gaps can also be found in _K2_ mission data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LCcNWywldDnY"
   },
   "source": [
    "### 2.1 Monthly data downlink\n",
    "\n",
    "During the _Kepler_ mission, the telescope changed its orientation once a month to point at Earth and downlink the last month of data. This caused a gap in the time series data, as the telescope could not collect while it was downlinking. When the telescope returned to regular pointing, its motion induced what is known as a \"thermal transient.\" This means that the telescope components and detector electronics underwent a temperature change, and the electron count reading was temporarily increased. A change in temperature can slightly change the telescope focus. This manifests in the simple aperture photometry (SAP) data as a downward slope caused by \"reheating,\" while the flux returns to its previous level. The thermal transient is corrected in PDCSAP data; there is no quality flag for data affected by a thermal transient. ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 5.5.)\n",
    "\n",
    "The following code uses Matplotlib to zoom in on an Earth pointing event and highlight the affected section of the SAP light curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20086,
     "status": "ok",
     "timestamp": 1600725930590,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "2V7CICdpdIMP",
    "outputId": "e8d18550-9698-49e0-8fdd-54a71f95205c"
   },
   "outputs": [],
   "source": [
    "ax = lc.plot(column='sap_flux')\n",
    "ax.set_xlim(225, 238)\n",
    "ax.set_ylim(5300, 5600)\n",
    "ax.fill_betweenx(ax.get_ylim(), 230.3, 231.4, facecolor='r', alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JGo9-atEdBXX"
   },
   "source": [
    "### 2.2 Safe modes\n",
    "\n",
    "Safe modes ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 4.2) are another type of thermal transient that appears in _Kepler_ data. A safe mode occured when the telescope temporarily shut off operation due to an unexpected event, usually caused by an issue with the detector electronics.\n",
    "\n",
    "There were eleven safe mode events throughout the *Kepler* mission, and three during *K2*, in Campaigns 0, 9, and 12. The following code zooms in on the Quarter 2 safe mode, in the same data we've used above; note the similarity in appearance between the two thermal transient events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20074,
     "status": "ok",
     "timestamp": 1600725930591,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "7YrgNiVcc--I",
    "outputId": "58cdbe08-ee05-4d12-f9d2-47c36628d6e7"
   },
   "outputs": [],
   "source": [
    "ax = lc.plot(column='sap_flux')\n",
    "ax.set_xlim(170, 200)\n",
    "ax.set_ylim(5450, 5800)\n",
    "ax.fill_betweenx(ax.get_ylim(), 181.5, 183.8, facecolor='r', alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MfNEX_H-dMIr"
   },
   "source": [
    "### 2.3 Coarse pointing and loss of fine pointing\n",
    "\n",
    "Running the code below, you'll see two highlighted regions where there is no thermal transient, but still a gap in the data. Often, gaps like this are caused by a loss of fine pointing ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 4.3). Because there is lower photometric precision when this occurs, these cadences are not suggested for use in photometry, and are replaced with NaNs.\n",
    "\n",
    "In the `QUALITY` column, there are two flags for this situation: a 4 (bit 3) for coarse point, and 32768 (bit 16) for a loss of fine point. Coarse point is a manual flag, based on preprocessing, and coarse point cadences are removed based on an expected loss of fine pointing. Cadences marked as \"no fine point\" are due to unexpected events; in practice, these two quality issues manifest in the same manner as gapped data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20831,
     "status": "ok",
     "timestamp": 1600725931360,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "emnOroqhdSCz",
    "outputId": "02d3dadb-12c1-4750-8c13-b75dca4504d3"
   },
   "outputs": [],
   "source": [
    "ax = lc.plot(column='sap_flux')\n",
    "ax.set_xlim(215,260)\n",
    "ax.set_ylim(ymax=5550)\n",
    "ax.fill_betweenx(ax.get_ylim(), 223.5, 224, facecolor='r', alpha=0.3)\n",
    "ax.fill_betweenx(ax.get_ylim(), 255.3, 256.4, facecolor='r', alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0zmJFIsRtcIO"
   },
   "source": [
    "Let's also look at a TPF cadence affected by a loss of fine pointing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 488
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20819,
     "status": "ok",
     "timestamp": 1600725931361,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "jo1w9SJleTAi",
    "outputId": "90a77067-a271-4e58-f415-1249e651a218"
   },
   "outputs": [],
   "source": [
    "print(tpf.flux[np.argwhere((tpf.quality & 32804) > 0)][0])\n",
    "tpf.plot(cadenceno=tpf.cadenceno[np.argwhere(tpf.quality == 32804)[0]]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5yAF6AkfWuOp"
   },
   "source": [
    "As you can see, there's no data available for this cadence at the TPF level. The vast majority of data affected by a loss of fine pointing/coarse pointing during the _Kepler_ and _K2_ missions is \"gapped\" like this. When performing photometry or light curve corrections, it's important to pay attention to the data on either side of these gaps and make sure you don't overcorrect it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LgFJ0va4c2B6"
   },
   "source": [
    "## 3. Single-Cadence Events\n",
    "\n",
    "We've now seen some situations in _Kepler_ data where you'll observe a gap in the data — or NaN flux — and what the time series looks like on either side of those events. But there are many quality issues which can have an impact on one or more cadences at a time, which aren't necessarily as evident as data gaps. In the following sections, we'll look at some of the causes behind these single-cadence quality issues, where you'll find them, and how to mitigate them in a practical context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g-xkp3qHd7yF"
   },
   "source": [
    "### 3.1 Cosmic rays\n",
    "\n",
    "Cosmic rays on the detector ([_Kepler_ Instrument Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/KSCI-19033-002-instrument-hb.pdf), Section 4.16) are an unavoidable source of quality issues in all space based data. Depending on the severity of the event, an incidence of a cosmic ray can lead to sudden pixel sensitivity dropouts (SPSD) — covered in another tutorial in this series — or even long-term damage. Here, we'll only look at the short-term events.\n",
    "\n",
    "The cell below plots a section of our light curve for KIC 2436365 from above, showing a spike caused by a cosmic ray hitting the detector. Note that this is PDCSAP flux: most cosmic rays are removed by  presearch data conditioning (PDC), but those that aren't can be caught by outlier clipping or, in this case, may not be a large enough spike to cause problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 411
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 21491,
     "status": "ok",
     "timestamp": 1600725932047,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "FpVWBd3yd-e-",
    "outputId": "5e6cfdd7-dd20-4b55-e112-ec39000ba8ce"
   },
   "outputs": [],
   "source": [
    "ax = lc.plot()\n",
    "ax.set_xlim(243, 257)\n",
    "ax.set_ylim(ymax=6540)\n",
    "ax.fill_betweenx(ax.get_ylim(), 250, 250.8, facecolor='r', alpha=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GNS-eoO-aRj3"
   },
   "source": [
    "### 3.2 Argabrightening\n",
    "\n",
    "Argabrightening ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 5.8) is a single-cadence quality issue that looks similar to a cosmic ray on the detector. In fact, Argabrightening is thought to be caused by debris hitting the instrument, causing a brief increase in flux. This is distinct from the electronic event caused by a cosmic ray corrupting the pixel readout: Argabrightening is the result of physical illumination.\n",
    "\n",
    "Using Lightkurve's `KeplerQualityFlags.decode()` function in [Section 1.1](#1.1-Quality-flags), we saw that a quality flag of 4160 indicates an Argabrightening event on the charge-coupled device (CCD) and in the optimal aperture used for photometry. Let's see where that shows up in our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 21479,
     "status": "ok",
     "timestamp": 1600725932049,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "OtlWFxAfcAvJ",
    "outputId": "3ccb8055-21c1-4068-ab8f-2578c696b547"
   },
   "outputs": [],
   "source": [
    "tpf.time.value[(tpf.quality & 4160) > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uGfL2HFucOsm"
   },
   "source": [
    "Argabrightening events are removed by the _Kepler_ pipeline for both SAP and PDCSAP data. To see this Argabrightening event, let's create our own light curve from the TPF, using custom aperture photometry. We can confirm that the spike in the data is right where we expect it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 21468,
     "status": "ok",
     "timestamp": 1600725932050,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "06j5du1taw2y",
    "outputId": "09344263-c178-4496-c1c1-f9d087104b35"
   },
   "outputs": [],
   "source": [
    "ax = tpf.to_lightcurve().plot()\n",
    "ax.set_xlim(235,245)\n",
    "ax.set_ylim(5250,5600)\n",
    "ax.fill_betweenx(ax.get_ylim(), 240.1, 240.6, facecolor='r', alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ufFOqaMPezzr"
   },
   "source": [
    "### 3.3 Attitude tweaks\n",
    "\n",
    "*Kepler*'s orientation, or attitude, was adjusted every few days during Quarters 0, 1, and 2 of the nominal mission. Specifically, attitude tweaks ensured that no star would ever move more than 1/100th of a pixel from its expected location in each cadence ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 4.4). From Quarter 3 onwards, changes to the telescope's Fine Guidance Sensor (FGS) system (see the [_Kepler_ Instrument Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/KSCI-19033-002-instrument-hb.pdf), particularly sections 2.1 and 2.5.1) led to reduced drift, which meant that attitude tweaks were no longer necessary.\n",
    "\n",
    "Because the drift distances in *Kepler*'s first three quarters were so small, they're hard to detect without checking the quality flags. Additionally, the discontinuities were for the most part corrected by the data processing pipeline, with only a few remaining:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 21883,
     "status": "ok",
     "timestamp": 1600725932480,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "2vTG1GxyfICa",
    "outputId": "8b2cc64d-0db3-4f93-a623-151580172180"
   },
   "outputs": [],
   "source": [
    "ax = lc.plot(column='sap_flux')\n",
    "ax.set_xlim(241, 251)\n",
    "ax.set_ylim(5300, 5400)\n",
    "ax.fill_betweenx(ax.get_ylim(), 246, 246.5, facecolor='r', alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L8sFFU_ffJGC"
   },
   "source": [
    "### 3.4 Reaction wheel events\n",
    "\n",
    "*Kepler*'s attitude was controlled initially by four reaction wheels. Bits 5 and 6 are allocated to two data quality events caused by the reaction wheels: zero crossings and momentum desaturation.\n",
    "\n",
    "Zero crossings (bit 5/integer 16) occur when the reaction wheels have zero angular velocity ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 5.4). This caused the telescope's point to degrade for a few minutes at a time. Because of this short timescale, reaction wheel zero crossings are mostly an issue in Short Cadence data, where they manifest as negative spikes in the flux data. Zero crossing events became less prominent after the failure of one of *Kepler*'s reaction wheels in Quarter 14, due to an increase in speed of the remaining reaction wheels. There were no reaction wheel zero crossings in the *K2* mission.\n",
    "\n",
    "Momentum desaturation (bit 6/integer 32) was a consequence of a build up of torque on the reaction wheels ([_Kepler_ Data Characteristics Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/Data_Characteristics.pdf), Section 5.3). Desaturation events occurred every 146 Long Cadences during the _Kepler_ mission, leading to coarse pointing mode (as above) and NaN values in the light curves for one Long Cadence or several Short Cadences at a time.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SXmKokoefOLj"
   },
   "source": [
    "### 3.5 Manual exclusions\n",
    "\n",
    "For various reasons, some cadences were manually excluded during the data processing stage. In general, these cadences were those on either side of gaps and discontinuities, but sometimes manual exclusions were used to cover specific events that didn't fall under any other category. For example, during Quarter 12, a series of three coronal mass ejections from the Sun led to multiple cadences being manually excluded ([_Kepler_ Data Release 25 Notes](https://archive.stsci.edu/kepler/release_notes/release_notes25/KSCI-19065-002DRN25.pdf), Section 12.2). Further solar flares and coronal mass ejections led to manual exclusions in Campaign 15 of _K2_ ([_K2_ Data Release 22 Notes](https://archive.stsci.edu/missions/k2/doc/drn/KSCI-19138-001_K2-DRN22_C15.pdf), Section 2.2). On both occasions, these incidences led to increased noise and reduced accuracy in pointing.\n",
    "\n",
    "The code below plots the SAP light curve for Quarter 12 of KIC 8805616 with no quality-flagged data excluded. Here, you can clearly see the data quality issues caused by the coronal mass ejections, which are highlighted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 411
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23097,
     "status": "ok",
     "timestamp": 1600725933712,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "BcRBI3E786Qf",
    "outputId": "f6431102-1400-402a-fd26-6c67da360563"
   },
   "outputs": [],
   "source": [
    "lc_12 = lk.search_lightcurve('KIC 8805616', author='Kepler', quarter=12).download(quality_bitmask=0)\n",
    "ax = lc_12.plot(column='sap_flux')\n",
    "ax.set_ylim(47200,48200)\n",
    "ax.fill_betweenx(ax.get_ylim(), 1116.2, 1118.7, facecolor='r', alpha=0.3)\n",
    "ax.fill_betweenx(ax.get_ylim(), 1121.1, 1122.3, facecolor='r', alpha=0.3)\n",
    "ax.fill_betweenx(ax.get_ylim(), 1160.2, 1164.0, facecolor='r', alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IxHd6lK0h9Yu"
   },
   "source": [
    "## 4. _K2_\n",
    "\n",
    "The *K2* mission relied on only two of *Kepler*'s reaction wheels, meaning it required an additional thruster firing every six hours to maintain pointing. This led to a six-hour drift in *K2* data; while the drift is corrected in Long Cadence data by PDC, data taken during the thruster firings is treated the same way as gaps or single-cadence quality issues in *Kepler* data. There are two quality bits allocated to thruster firings: bit 21 for a thruster firing, and bit 20 for a possible thruster firing.\n",
    "\n",
    "To get an idea of what the six-hour drift looks like, let's start by downloading some *K2* data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-HlO71iRwOub"
   },
   "outputs": [],
   "source": [
    "lc_k2 = lk.search_lightcurve('EPIC 211414081', author='K2', cadence='long', campaign=5).download(quality_bitmask=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mFQjgP2yw0PN"
   },
   "source": [
    "Even with `quality_bitmask=0`, the data is gapped at all cadences flagged with bit 20 or 21. You can test this yourself by running the following code, and noting that there are no `time` values provided for any cadences with these quality flags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4094,
     "status": "ok",
     "timestamp": 1600725985052,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "Q97bmof9v0P3",
    "outputId": "bf1a9bac-89ec-4e6f-ae74-dadd41b890e5"
   },
   "outputs": [],
   "source": [
    "for val in lc_k2.quality.value[0:10]:\n",
    "  print(lc_k2.time[np.argwhere((val & 20) > 0 or (val & 21) > 0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1XIy0VZjyCP_"
   },
   "source": [
    "Now let's look at the SAP light curve for this star, and see what the six-hour drift looks like in practice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 28451,
     "status": "ok",
     "timestamp": 1600725939138,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "_3FcQsoAu-yt",
    "outputId": "ae14b3d5-191a-41ea-d8c8-53cbe93b75cb"
   },
   "outputs": [],
   "source": [
    "lc_k2.plot(column='sap_flux');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k2Kws3VyyIZd"
   },
   "source": [
    "Fortunately, this systematic is well-represented by the cotrending basis vectors (CBVs) used in the PDC pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 29800,
     "status": "ok",
     "timestamp": 1600725940506,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "pi8NtzF2vBpy",
    "outputId": "36c75140-dcef-4608-ce0d-ba260642dc7a"
   },
   "outputs": [],
   "source": [
    "lc_k2.plot(marker='.', linewidth=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RKAQJwhuzL1a"
   },
   "source": [
    "The [_K2_ Handbook](https://archive.stsci.edu/files/live/sites/mast/files/home/missions-and-data/kepler/_documents/KSCI-19116-002.pdf) cautions that these effects can never be fully corrected — you can see this particularly to the left of the plot above. But overall, PDC successfully removes the six-hour drift from _K2_ Long Cadence data, providing high-quality time series data for all targets.\n",
    "\n",
    "In the next tutorial in this series, we'll revisit EPIC 211414081, and look at how the thruster firings lead to persistent systematics in Short Cadence data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vbge5j28DVeN"
   },
   "source": [
    "## About this Notebook\n",
    "\n",
    "**Author:** [Isabel Colman](http://orcid.org/0000-0001-8196-516X) (`isabel.colman@sydney.edu.au`)\n",
    "\n",
    "**Updated on:** 2020-09-29\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DPQvLP9jNZQT"
   },
   "source": [
    "## Citing Lightkurve and Astropy\n",
    "\n",
    "If you use `lightkurve` or `astropy` for published research, please cite the authors. Click the buttons below to copy BibTeX entries to your clipboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 144
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 29792,
     "status": "ok",
     "timestamp": 1600725940508,
     "user": {
      "displayName": "Geert Barentsen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj8sjdnDeqdejfe7OoouYPIclAQV0KSTpsU469Jyeo=s64",
      "userId": "05704237875861987058"
     },
     "user_tz": 420
    },
    "id": "7vUtrWVjnlY7",
    "outputId": "7b910665-211f-407d-db43-fcf8f70e2db0"
   },
   "outputs": [],
   "source": [
    "lk.show_citation_instructions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y1HbVfHwDVeN"
   },
   "source": [
    "<img style=\"float: right;\" src=\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\" alt=\"Space Telescope Logo\" width=\"200px\"/>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Instrumental noise in Kepler and K2 #1: data gaps and quality flags.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}