merging

notebooks/espresso.ipynb

@@ -0,0 +1,267 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import wobble\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "import glob\n",
+    "import os\n",
+    "from tqdm import tqdm\n",
+    "np.random.seed(0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### These cells are used to pre-process the data.\n",
+    "They only need to be run once, and after that the saved data file can be loaded up from disk."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = wobble.Data()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "filenames = glob.glob('/Users/mbedell/python/wobble/data/toi/TOI-*_CCF_A.fits')\n",
+    "for filename in tqdm(filenames):\n",
+    "    try:\n",
+    "        sp = wobble.Spectrum()\n",
+    "        sp.from_ESPRESSO(filename, process=True)\n",
+    "        data.append(sp)\n",
+    "    except Exception as e:\n",
+    "        print(\"File {0} failed; error: {1}\".format(filename, e))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data.write('../data/toi.hdf5')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load the data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = wobble.Data(filename='../data/toi.hdf5')\n",
+    "R = np.copy(data.R) # we'll need this later"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data.drop_bad_orders(min_snr=3)\n",
+    "data.drop_bad_epochs(min_snr=3)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data.orders"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "r = 0\n",
+    "good = data.ivars[r] > 0.\n",
+    "for e in [0,10,20]:\n",
+    "    plt.errorbar(data.xs[r][e][good[e]], data.ys[r][e][good[e]], \n",
+    "             1./np.sqrt(data.ivars[r][e][good[e]]), ls='', fmt='o', ms=2, alpha=0.5)\n",
+    "plt.title('Echelle order #{0}'.format(data.orders[r]), fontsize=14);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generate regularization parameter files & tune them"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Since we don't have any existing regularization parameter files for ESPRESSO, we have to make some new ones. \n",
+    "\n",
+    "This is needed because the default *wobble* regularization is tuned to HARPS, which has a different number of spectral orders and different wavelength coverage - if we try to run with those files, the optimization will *(a)* be non-optimal and *(b)* eventually crash when we try to access an order than does not exist for HARPS."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "star_filename = '../wobble/regularization/toi_star.hdf5'\n",
+    "tellurics_filename = '../wobble/regularization/toi_tellurics.hdf5'\n",
+    "wobble.generate_regularization_file(star_filename, R, type='star')\n",
+    "wobble.generate_regularization_file(tellurics_filename, R, type='telluric')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plot_dir = '../regularization/toi/'\n",
+    "if not os.path.exists(plot_dir):\n",
+    "    os.makedirs(plot_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We'll tune the regularization using a train-and-validate approach, so let's set aside some epochs to be the validation set:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validation_epochs = np.random.choice(data.N, data.N//6, replace=False) # 3 epochs for validation set"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "r = 100\n",
+    "for e in [validation_epochs[0]]:\n",
+    "    plt.errorbar(data.xs[r][e][good[e]], data.ys[r][e][good[e]], \n",
+    "             1./np.sqrt(data.ivars[r][e][good[e]]), ls='', fmt='o', ms=2, alpha=0.5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here's an example of how this regularization tuning will go for one order:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "r = 100\n",
+    "o = data.orders[r]\n",
+    "objs = wobble.setup_for_order(r, data, validation_epochs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "wobble.improve_order_regularization(o, star_filename, tellurics_filename,\n",
+    "                                         *objs,\n",
+    "                                         verbose=False, plot=False, \n",
+    "                                         basename='{0}o{1}'.format(plot_dir, o), \n",
+    "                                         K_t=0, L1=True, L2=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**This only does one order.** To save time & print statements, we'll do the full loop over all orders from a script. See wobble/scripts/tune_regularization.py for an example."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Get RVs!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "OK, let's assume that we have run the regularization tuning script and we're satisfied with the settings there. Now we can finally get some solutions!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##### (this part hasn't yet been filled in, see demo.ipynb or the scripts directory for examples)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python3 (wobble)",
+   "language": "python",
+   "name": "wobble"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}