docs/source/tutorials/3-science-examples/exoplanets-machine-learning-preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine learning style preprocessing with lightkurve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine learning offers interesting avenues for automated lightcurve classification, and complex regression tasks.  The key idea is to hand a pile of data to a black box algorithm that learns labels on training data through gradient descent optimization, while mitigating overfitting by checking classification performance against held-out test data.  \n",
    "\n",
    "Machine learning algorithms work best when data are \"clean\", or devoid of missing values and spurious noise.  Data scientists and machine learning practitioners often quip that data preparation, cleaning, and preprocessing can occupy a large fraction of a machine learning project, so making this step easier will help faciliate machine learning projects.\n",
    "\n",
    "In this tutorial we replicate the machine learning *Input Representations* of Kepler lightcurves described by [Shallue & Vanderburg 2018](https://ui.adsabs.harvard.edu/abs/2018AJ....155...94S/abstract):\n",
    "\n",
    "> We generate a **global view** of the light curve [...]  All light curves are binned to the same length\n",
    "\n",
    "> We generate a **local view** of the transit [...] so that the transit occupies a fixed fraction of the resulting vector. \n",
    "\n",
    "The section concludes with a normalization procedure to make the transits all the same depth:\n",
    "\n",
    "> ...we normalize all light curves to have median 0 and minimum value –1 so that all TCEs have a fixed transit depth.\n",
    "\n",
    "The global view has a fixed length of 2001 bins, while the local view has a fixed length of 201 bins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial requires version 1.5 or greater of `lightkurve`, which added new binning features that offer more flexibility and user-control of where and how to assign bins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightkurve as lk\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at exoplanet host star KIC 757450, which has a period of $P=8.88492\\;d$, $T_0=134.452\\;d$, and transit duration $2.078$ hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lcs = lk.search_lightcurve('KIC 757450', author='Kepler', cadence='long').download_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "period, t0, duration_hours = 8.88492, 134.452, 2.078"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can stitch all of the quarters together in one fell swoop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_raw = lcs.stitch()\n",
    "lc_raw.flux.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean outliers, but only those that are above the mean level (e.g. attributable to stellar flares or cosmic rays)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_clean = lc_raw.remove_outliers(sigma=20, sigma_upper=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to mask the transit to avoid self-subtraction the genuine planet signal when we flatten the lightcurve.  We have to do a hack to find where the time series should be masked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp_fold = lc_clean.fold(period, epoch_time=t0)\n",
    "fractional_duration = (duration_hours / 24.0) / period\n",
    "phase_mask = np.abs(temp_fold.phase.value) < (fractional_duration * 1.5)\n",
    "transit_mask = np.in1d(lc_clean.time.value, temp_fold.time_original.value[phase_mask])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now flatten the mask, but interpolate the trend over the points possessing the transit signal of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_flat, trend_lc = lc_clean.flatten(return_trend=True, mask=transit_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now fold the cleaned, flattened lightcurve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_fold = lc_flat.fold(period, epoch_time=t0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's generate the **global** input representation by binning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_global = lc_fold.bin(time_bin_size=0.005).normalize() - 1\n",
    "lc_global = (lc_global / np.abs(lc_global.flux.min()) ) * 2.0 + 1\n",
    "lc_global.flux.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_global.scatter();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent!  The pattern matches those seen the original publication figures.  What about the **local** view?  We need to *zoom in* on the transit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phase_mask = (lc_fold.phase > -4*fractional_duration) & (lc_fold.phase < 4.0*fractional_duration)\n",
    "lc_zoom = lc_fold[phase_mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_local = lc_zoom.bin(time_bin_size=0.0005).normalize() - 1\n",
    "lc_local = (lc_local / np.abs(np.nanmin(lc_local.flux)) ) * 2.0 + 1\n",
    "lc_local.flux.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lc_local.scatter();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voilà!  We've condensed the raw signal of over 65,000 time series data points into two cleaned, simplified input representation using `lightkurve`'s built-in functions.  These views could be passed into a neural network classifier to establish whether the signal is likely to be an exoplanet or not.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}