Cloud ML - Financial Time Series.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Structured data prediction using Cloud ML Engine \n",
    "\n",
    "This notebook illustrates:\n",
    "\n",
    "1. Exploring a BigQuery dataset using Datalab\n",
    "2. Creating datasets for Machine Learning using Dataflow\n",
    "3. Creating a model using the high-level Estimator API \n",
    "4. Training on Cloud ML Engine\n",
    "5. Deploying the model\n",
    "6. Predicting with the model\n",
    "\n",
    "\n",
    "We will create a toy binary classifier to predict if the Standard&Poor 500 index will close positively or negatively. We will build our features using the close values of these indexes:\n",
    "\n",
    "|Index|Country|Closing Time (EST)|Hours Before S&P Close|\n",
    "|---|---|---|---|\n",
    "|[All Ords](https://en.wikipedia.org/wiki/All_Ordinaries)|Australia|0100|15|\n",
    "|[Nikkei 225](https://en.wikipedia.org/wiki/Nikkei_225)|Japan|0200|14|\n",
    "|[Hang Seng](https://en.wikipedia.org/wiki/Hang_Seng_Index)|Hong Kong|0400|12|\n",
    "|[DAX](https://en.wikipedia.org/wiki/DAX)|Germany|1130|4.5|\n",
    "|[FTSE 100](https://en.wikipedia.org/wiki/FTSE_100_Index)|UK|1130|4.5|\n",
    "|[NYSE Composite](https://en.wikipedia.org/wiki/NYSE_Composite)|US|1600|0|\n",
    "|[Dow Jones Industrial Average](https://en.wikipedia.org/wiki/Dow_Jones_Industrial_Average)|US|1600|0|\n",
    "|[S&P 500](https://en.wikipedia.org/wiki/S%26P_500_Index)|US|1600|0|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Housekeeping "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%bash\n",
    "pip uninstall -y google-cloud-dataflow\n",
    "pip install --upgrade --force apache-beam[gcp]==2.2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Restart the session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "BUCKET = # insert your bucket name (no underscore)\n",
    "PROJECT = #insert your project name\n",
    "REGION = #insert your region (us-central1, europe-west1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "gcs_data_dir = 'gs://{0}/data/financialtimeseries/'.format(BUCKET)\n",
    "gcs_model_dir = 'gs://{0}/ml-models/financialtimeseries/'.format(BUCKET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "gsutil -m rm -rf gs://${BUCKET}/ml-models/financialtimeseries/*\n",
    "gsutil -m rm -rf gs://${BUCKET}/data/financialtimeseries/big_data/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the rest of the code you need to import the data in *data/financialtimeseries/all_data.csv* in [BigQuery](https://bigquery.cloud.google.com/project/lf-gcp-demo)\n",
    "\n",
    "The first step is to copy the file in [Google Cloud Storage](https://cloud.google.com/storage/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gsutil -m cp data/financialtimeseries/all_data.csv gs://${BUCKET}/data/financialtimeseries/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to create from the user interface of BigQuery a *dataset* and a *table* by importing the file stored at *gs://${BUCKET}/data/financialtimeseries/all_data.csv*.\n",
    "\n",
    "Finally in the reminder of the code replace &lt;project&gt; by the project name, &lt;dataset&gt; by the dataset name and &lt;table&gt; by the table name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Query data in BigQuery\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bq query --name data\n",
    "\n",
    "SELECT\n",
    "  Date as date,\n",
    "  Close as close,\n",
    "  Market as market\n",
    "FROM\n",
    "  `<project>.<dataset>.<table>`\n",
    "WHERE\n",
    "  market = 'snp'\n",
    "ORDER BY\n",
    "  market, date "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Visualise with Datalab commands \n",
    "http://googledatalab.github.io/pydatalab/google.datalab%20Commands.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%chart line --data data --fields date,close\n",
    "title: Close Value\n",
    "height: 400\n",
    "width: 900\n",
    "hAxis:\n",
    "  title: Date\n",
    "vAxis:\n",
    "  title: Close Value of S&P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Fetch data from BigQuery as a pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%sql --module query \n",
    "\n",
    "SELECT\n",
    "  Date as date,\n",
    "  Close as close,\n",
    "  Market as market\n",
    "FROM\n",
    "  `<project>.<dataset>.<table>`\n",
    "WHERE\n",
    "  Close > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import datalab.bigquery as bq\n",
    "import sys\n",
    "data = bq.Query(query).to_dataframe(dialect='standard')\n",
    "data = data.pivot_table(index='date', columns='market', aggfunc='sum')\n",
    "print('Row count:{}'.format(data.shape[0]))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data.fillna(method='ffill', inplace=True)\n",
    "data.fillna(method='bfill', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Explore & Visualise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from pandas.plotting import autocorrelation_plot\n",
    "from pandas.plotting import scatter_matrix\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "_ = data.plot(figsize=(20, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data_scaled = data.apply(lambda x: x/x.max(), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "_ = data_scaled.plot(figsize=(20, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(20, 15))\n",
    "ax = fig.add_axes()\n",
    "\n",
    "for col in data_scaled.columns.tolist():\n",
    "  ax = autocorrelation_plot(\n",
    "          data_scaled[col],\n",
    "          label='{1}_{0}'.format(*col),\n",
    "          axes=ax)\n",
    "  \n",
    "_ = ax.legend(loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "_ = sns.pairplot(data_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "log_return_data = data.apply(lambda x : x/x.shift(), axis=0).apply(np.log)\n",
    "log_return_data.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "log_return_data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "_ = log_return_data.plot(figsize=(20, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(20, 15))\n",
    "ax = fig.add_axes()\n",
    "\n",
    "for col in log_return_data.columns.tolist():\n",
    "  ax = autocorrelation_plot(\n",
    "          log_return_data[col],\n",
    "          label='{1}_{0}'.format(*col),\n",
    "          axes=ax)\n",
    "  \n",
    "_ = ax.legend(loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "_ = sns.pairplot(log_return_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "log_return_data.corr()[(u'close', u'snp')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "log_return_data[(u'close', u'nyse_1')] = log_return_data[(u'close', u'nyse')].shift(-1)\n",
    "log_return_data[(u'close', u'djia_1')] = log_return_data[(u'close', u'djia')].shift(-1)\n",
    "log_return_data[(u'close', u'nyse_2')] = log_return_data[(u'close', u'nyse')].shift(-2)\n",
    "log_return_data[(u'close', u'djia_2')] = log_return_data[(u'close', u'djia')].shift(-2)\n",
    "log_return_data[(u'close', u'nyse_3')] = log_return_data[(u'close', u'nyse')].shift(-3)\n",
    "log_return_data[(u'close', u'djia_3')] = log_return_data[(u'close', u'djia')].shift(-3)\n",
    "log_return_data[(u'close', u'ftse_2')] = log_return_data[(u'close', u'djia')].shift(-2)\n",
    "log_return_data[(u'close', u'dax_2')] = log_return_data[(u'close', u'nyse')].shift(-2)\n",
    "log_return_data[(u'close', u'hangseng_2')] = log_return_data[(u'close', u'djia')].shift(-2)\n",
    "log_return_data[(u'close', u'nikkei_2')] = log_return_data[(u'close', u'nyse')].shift(-2)\n",
    "log_return_data[(u'close', u'aord_2')] = log_return_data[(u'close', u'djia')].shift(-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "log_return_data.corr()[(u'close', u'snp')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will use as features:\n",
    "1. the close value of S&P 500 of the day before\n",
    "1. the close value of S&P 500 of two day before\n",
    "1. the close value of Dow Jones Industrial Average of the day before\n",
    "1. the close value of Dow Jones Industrial Average of two day before\n",
    "1. the close value of NYSE Composite of the day before\n",
    "1. the close value of NYSE Composite of two day before\n",
    "1. the close value of FTSE 100 of the same day\n",
    "1. the close value of FTSE 100 of the day before\n",
    "1. the close value of DAX of the same day\n",
    "1. the close value of DAX of the day before\n",
    "1. the close value of Hang Seng of the same day\n",
    "1. the close value of Hang Seng of the day before\n",
    "1. the close value of Nikkei 225 of the same day\n",
    "1. the close value of Nikkei 225 of the day before\n",
    "1. the close value of All Ords of the same day\n",
    "1. the close value of All Ords of the day before"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### BEST PRACTICE : Average Weight as a Baseline Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "positive_log_returns = log_return_data[(u'close',u'snp')].apply(lambda x : 1 if x > 0 else 0).sum()\n",
    "total_log_returns = log_return_data[(u'close',u'snp')].dropna().shape[0]\n",
    "\n",
    "print(\"Fraction of positive: {}\\nFraction of negative: {}\".format(\n",
    "         round(1.0*positive_log_returns/total_log_returns,3),\n",
    "         round(1.0-1.0*positive_log_returns/total_log_returns,3)\n",
    "       )\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create ML dataset using Dataflow\n",
    "\n",
    "Let's use Cloud Dataflow to read in the BigQuery data and write it out as CSV files. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "with warnings.catch_warnings():\n",
    "    warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "    import apache_beam as beam\n",
    "    print(beam.__version__)\n",
    "\n",
    "import datetime\n",
    "\n",
    "\n",
    "query = \"\"\"\n",
    "SELECT\n",
    "  date,\n",
    "  next_date,\n",
    "  nnext_date,\n",
    "  market,\n",
    "  IFNULL(LOG(close/previous_close),0) log_return,\n",
    "  ABS(FARM_FINGERPRINT(CAST(date AS STRING))) AS hashdate\n",
    "FROM\n",
    "(\n",
    "  SELECT\n",
    "    Date as date,\n",
    "    Close as close,\n",
    "    Market as market,\n",
    "    LAG(close) OVER (PARTITION BY market ORDER BY date) as previous_close,\n",
    "    LEAD(date) OVER (PARTITION BY market ORDER BY date) as next_date,\n",
    "    LEAD(date, 2) OVER (PARTITION BY market ORDER BY date) as nnext_date\n",
    "  FROM\n",
    "    ``<project>.<dataset>.<table>``\n",
    ")\n",
    "    \"\"\"\n",
    "\n",
    "out_dir = gcs_data_dir + \"big_data\"\n",
    "\n",
    "  \n",
    "  \n",
    "class EmitShiftedValues(beam.DoFn):\n",
    "  \n",
    "  def process(self, element):\n",
    "    output = []\n",
    "    \n",
    "    if (\n",
    "      element['date'] != None and \n",
    "      element['next_date'] != None and \n",
    "      element['nnext_date'] != None ):\n",
    "      if element['market'] in ['snp']:\n",
    "        output.append(\n",
    "          (element['date'],('snp_label', '+' if element['log_return'] > 0 else '-'))\n",
    "        )\n",
    "        output += [\n",
    "          (element['next_date'], ('snp_m1', element['log_return'])),\n",
    "          (element['nnext_date'], ('snp_m2', element['log_return']))\n",
    "        ]\n",
    "      elif element['market'] in ['nyse', 'djia']:\n",
    "        output += [\n",
    "          (element['next_date'], ('{0}_m1'.format(element['market']), element['log_return'])),\n",
    "          (element['nnext_date'], ('{0}_m2'.format(element['market']), element['log_return']))\n",
    "        ]\n",
    "      elif element['market'] in ['nikkei', 'hangseng', 'ftse', 'dax', 'aord']:\n",
    "        output += [\n",
    "          (element['date'], ('{0}_m0'.format(element['market']), element['log_return'])),\n",
    "          (element['next_date'], ('{0}_m1'.format(element['market']), element['log_return']))\n",
    "        ]\n",
    "    for pair in output:\n",
    "      yield pair\n",
    "\n",
    "def create_output_row(pair):\n",
    "  (date, data) = pair\n",
    "  data = dict(data)\n",
    "  OUTPUT_ROWS=[\n",
    "    'snp_label', 'snp_m1', 'snp_m2', \n",
    "    'nyse_m1', 'nyse_m2',  \n",
    "    'djia_m1', 'djia_m2',  \n",
    "    'nikkei_m0', 'nikkei_m1', \n",
    "    'hangseng_m0', 'hangseng_m1',  \n",
    "    'ftse_m0', 'ftse_m1', \n",
    "    'dax_m0', 'dax_m1',  \n",
    "    'aord_m0', 'aord_m1']\n",
    "  \n",
    "  \n",
    "  #check that all values are present\n",
    "  if all(k in data for k in OUTPUT_ROWS):\n",
    "    output_row = ','.join([str(data[k]) for k in OUTPUT_ROWS])\n",
    "    yield output_row\n",
    "  \n",
    "      \n",
    "def run_pipeline():\n",
    "    \n",
    "    job_name = 'preprocess-financialtimeseries-data' + '-' + datetime.datetime.now().strftime('%y%m%d-%H%M%S')\n",
    "    print 'Launching Dataflow job {} ... hang on'.format(job_name)\n",
    "\n",
    "    options = {\n",
    "        'staging_location': os.path.join(out_dir, 'tmp', 'staging'),\n",
    "        'temp_location': os.path.join(out_dir, 'tmp'),\n",
    "        'job_name': job_name,\n",
    "        'project': PROJECT,\n",
    "        'region' : REGION,\n",
    "        'teardown_policy': 'TEARDOWN_ALWAYS',\n",
    "        'no_save_main_session': True\n",
    "    }\n",
    "  \n",
    "    opts = beam.pipeline.PipelineOptions(flags=[], **options)\n",
    "    RUNNER = 'DataflowRunner'\n",
    "  \n",
    "    with beam.Pipeline(RUNNER, options=opts) as pipeline:  \n",
    "      \n",
    "      for step in ['train', 'eval']:\n",
    "          if step == 'train':\n",
    "              source_query = 'SELECT * FROM ({}) WHERE MOD(hashdate,5) < 4'.format(query)\n",
    "          else:\n",
    "              source_query = 'SELECT * FROM ({}) WHERE MOD(hashdate,5) = 4'.format(query)\n",
    "\n",
    "          sink_location = os.path.join(out_dir, '{}-data'.format(step))\n",
    "\n",
    "          (pipeline \n",
    "             | '{} - Read from BigQuery'.format(step) >> beam.io.Read(beam.io.BigQuerySource(query=source_query, use_standard_sql=True))\n",
    "             | '{} - Process single row'.format(step) >> beam.ParDo(EmitShiftedValues())\n",
    "             | '{} - Group by date'.format(step) >> beam.GroupByKey()\n",
    "             | '{} - Create output row'.format(step) >> beam.FlatMap(create_output_row)\n",
    "             | '{} - Write to GCS '.format(step) >> beam.io.Write(beam.io.WriteToText(sink_location,\n",
    "                                                                  file_name_suffix='.csv',\n",
    "                                                                  num_shards=5))\n",
    "          )\n",
    "    \n",
    "   \n",
    "    job = pipeline.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Run Dataflow Preprocessing Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "run_pipeline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "gsutil ls gs://${BUCKET}/data/financialtimeseries/big_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Create TensorFlow Models using Estimator API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import data\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train Linear Regression Model\n",
    "\n",
    "1. Define dataset metadata + input function (to read and parse the data files)\n",
    "\n",
    "2. Create feature columns based on metadata\n",
    "\n",
    "3. Instantiate the model with feature columns \n",
    "\n",
    "4. Train, evaluate, and predict using the model and the data input function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1 - Define Metadata &  Input Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "HEADER = [\n",
    "  'snp_label', 'snp_m1', 'snp_m2', 'nyse_m1', 'nyse_m2',  \n",
    "  'djia_m1', 'djia_m2', 'nikkei_m0', 'nikkei_m1', \n",
    "  'hangseng_m0', 'hangseng_m1', 'ftse_m0', 'ftse_m1', \n",
    "  'dax_m0', 'dax_m1', 'aord_m0', 'aord_m1'\n",
    "]\n",
    "TARGET_NAME = 'snp_label'\n",
    "TARGET_VALUES = ['+','-']\n",
    "DEFAULTS = [\n",
    "  ['null'], [0.0], [0.0], [0.0], [0.0],\n",
    "  [0.0], [0.0], [0.0], [0.0], \n",
    "  [0.0], [0.0], [0.0], [0.0], \n",
    "  [0.0], [0.0], [0.0], [0.0]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def parse_csv_row(csv_row):\n",
    "  columns = tf.decode_csv(tf.expand_dims(csv_row, -1), record_defaults=DEFAULTS)\n",
    "  features = dict(zip(HEADER, columns))\n",
    "  target = features.pop(TARGET_NAME)\n",
    "  return features, target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def csv_input_fn(file_name, mode=tf.estimator.ModeKeys.EVAL, \n",
    "                 skip_header_lines=0, \n",
    "                 num_epochs=1, \n",
    "                 batch_size=500):\n",
    "    \n",
    "  shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False\n",
    "\n",
    "  file_names = tf.matching_files(file_name)\n",
    "\n",
    "  dataset = data.TextLineDataset(filenames=file_names)\n",
    "  dataset = dataset.skip(skip_header_lines)\n",
    "\n",
    "  if shuffle:\n",
    "      dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)\n",
    "\n",
    "  dataset = dataset.batch(batch_size)\n",
    "  dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))\n",
    "  dataset = dataset.repeat(num_epochs)\n",
    "  iterator = dataset.make_one_shot_iterator()\n",
    "\n",
    "  features, target = iterator.get_next()\n",
    "  return features, target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2 - Create Feature Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def create_feature_columns():\n",
    "  \n",
    "  snp_m1=tf.feature_column.numeric_column('snp_m1')\n",
    "  snp_m2=tf.feature_column.numeric_column('snp_m2')\n",
    "  nyse_m1=tf.feature_column.numeric_column('nyse_m1')\n",
    "  nyse_m2=tf.feature_column.numeric_column('nyse_m2')\n",
    "  djia_m1=tf.feature_column.numeric_column('djia_m1')\n",
    "  djia_m2=tf.feature_column.numeric_column('djia_m2')\n",
    "  nikkei_m0=tf.feature_column.numeric_column('nikkei_m0')\n",
    "  nikkei_m1=tf.feature_column.numeric_column('nikkei_m1')    \n",
    "  hangseng_m0=tf.feature_column.numeric_column('hangseng_m0')\n",
    "  hangseng_m1=tf.feature_column.numeric_column('hangseng_m1')\n",
    "  ftse_m0=tf.feature_column.numeric_column('ftse_m0')\n",
    "  ftse_m1=tf.feature_column.numeric_column('ftse_m1') \n",
    "  dax_m0=tf.feature_column.numeric_column('dax_m0')\n",
    "  dax_m1=tf.feature_column.numeric_column('dax_m1') \n",
    "  aord_m0=tf.feature_column.numeric_column('aord_m0')\n",
    "  aord_m1=tf.feature_column.numeric_column('aord_m1')   \n",
    "\n",
    "  features_columns = [\n",
    "     snp_m1, snp_m2, nyse_m1, nyse_m2,\n",
    "     djia_m1, djia_m2, nikkei_m0, nikkei_m1,\n",
    "     hangseng_m0, hangseng_m1, ftse_m0, ftse_m1,\n",
    "     dax_m0, dax_m1, aord_m0, aord_m1\n",
    "  ]\n",
    "  \n",
    "  return features_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3 - Instantiate a Regression Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "local_model_dir = \"trained_models/financialtimeseries_lc\"\n",
    "\n",
    "feature_columns = create_feature_columns()\n",
    "\n",
    "lc_estimator = tf.estimator.LinearClassifier(\n",
    "  feature_columns=feature_columns,\n",
    "  model_dir=local_model_dir,\n",
    "  label_vocabulary=TARGET_VALUES\n",
    "  )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 4 - Train, Evaluate, and Predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "ls data/financialtimeseries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### a. train the model with the"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "train_data_files = \"data/financialtimeseries/train-data.csv\"\n",
    "\n",
    "train_input_fn = lambda: csv_input_fn(train_data_files, \n",
    "                                              mode=tf.estimator.ModeKeys.TRAIN, \n",
    "                                              num_epochs=None,\n",
    "                                              batch_size = 200\n",
    "                                         )\n",
    "\n",
    "# remove the following line of code to resume training\n",
    "shutil.rmtree(local_model_dir, ignore_errors=True)\n",
    "\n",
    "lc_estimator.train(train_input_fn, max_steps=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "ls trained_models/financialtimeseries_lc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### b. evaluate the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "eval_data_files = \"data/financialtimeseries/train-data.csv\"\n",
    "\n",
    "eval_input_fn =lambda: csv_input_fn(eval_data_files)\n",
    "\n",
    "lc_estimator.evaluate(eval_input_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### c. predict using the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "\n",
    "predictions = lc_estimator.predict(eval_input_fn)\n",
    "\n",
    "values = [item['classes'][0] for item in itertools.islice(predictions, 5)]\n",
    "\n",
    "print(\"Predicted Classes: {}\".format(values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train a DNN Liner Combined Classification Model + Feature Engineering\n",
    "\n",
    "1. Define dataset metadata + input function (to read and parse the data files, + **process features**) \n",
    "\n",
    "2. Create feature columns based on metadata + **Extended Feature Columns**\n",
    "\n",
    "3. Initialise the Estimator + **Wide & Deep Columns for the combined DNN model**\n",
    "\n",
    "4. Run **train_and_evaluate** experiment: Supply TrainSpec, EvalSepc, config, and params\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1. Define input function with process features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "HEADER = [\n",
    "  'snp_label', 'snp_m1', 'snp_m2', 'nyse_m1', 'nyse_m2',  \n",
    "  'djia_m1', 'djia_m2', 'nikkei_m0', 'nikkei_m1', \n",
    "  'hangseng_m0', 'hangseng_m1', 'ftse_m0', 'ftse_m1', \n",
    "  'dax_m0', 'dax_m1', 'aord_m0', 'aord_m1'\n",
    "]\n",
    "TARGET_NAME = 'snp_label'\n",
    "TARGET_VALUES = ['+', '-']\n",
    "DEFAULTS = [\n",
    "  ['null'], [0.0], [0.0], [0.0], [0.0],\n",
    "  [0.0], [0.0], [0.0], [0.0], \n",
    "  [0.0], [0.0], [0.0], [0.0], \n",
    "  [0.0], [0.0], [0.0], [0.0]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def parse_csv_row(csv_row):\n",
    "  columns = tf.decode_csv(tf.expand_dims(csv_row, -1), record_defaults=DEFAULTS)\n",
    "  features = dict(zip(HEADER, columns))\n",
    "  target = features.pop(TARGET_NAME)\n",
    "  return features, target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# to be applied in traing and serving\n",
    "def process_features(features):\n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def csv_input_fn(file_name, mode=tf.estimator.ModeKeys.EVAL, \n",
    "                 skip_header_lines=0, \n",
    "                 num_epochs=1, \n",
    "                 batch_size=500):\n",
    "    \n",
    "  shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False\n",
    "\n",
    "  file_names = tf.matching_files(file_name)\n",
    "\n",
    "  dataset = data.TextLineDataset(filenames=file_names)\n",
    "  dataset = dataset.skip(skip_header_lines)\n",
    "\n",
    "  if shuffle:\n",
    "      dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)\n",
    "\n",
    "  dataset = dataset.batch(batch_size)\n",
    "  dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))\n",
    "  dataset = dataset.repeat(num_epochs)\n",
    "  iterator = dataset.make_one_shot_iterator()\n",
    "\n",
    "  features, target = iterator.get_next()\n",
    "  return features, target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2. Create Feature Columns with Extensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_deep_and_wide_columns():\n",
    "  snp_m1=tf.feature_column.numeric_column('snp_m1')\n",
    "  snp_m2=tf.feature_column.numeric_column('snp_m2')\n",
    "  nyse_m1=tf.feature_column.numeric_column('nyse_m1')\n",
    "  nyse_m2=tf.feature_column.numeric_column('nyse_m2')\n",
    "  djia_m1=tf.feature_column.numeric_column('djia_m1')\n",
    "  djia_m2=tf.feature_column.numeric_column('djia_m2')\n",
    "  nikkei_m0=tf.feature_column.numeric_column('nikkei_m0')\n",
    "  nikkei_m1=tf.feature_column.numeric_column('nikkei_m1')    \n",
    "  hangseng_m0=tf.feature_column.numeric_column('hangseng_m0')\n",
    "  hangseng_m1=tf.feature_column.numeric_column('hangseng_m1')\n",
    "  ftse_m0=tf.feature_column.numeric_column('ftse_m0')\n",
    "  ftse_m1=tf.feature_column.numeric_column('ftse_m1') \n",
    "  dax_m0=tf.feature_column.numeric_column('dax_m0')\n",
    "  dax_m1=tf.feature_column.numeric_column('dax_m1') \n",
    "  aord_m0=tf.feature_column.numeric_column('aord_m0')\n",
    "  aord_m1=tf.feature_column.numeric_column('aord_m1')   \n",
    "  \n",
    "  snp_m1_bucketized = tf.feature_column.bucketized_column(\n",
    "      snp_m1, \n",
    "      boundaries=[-.05, -.04, -.03, -.02, -.01, .00, .01, .02, .03, 0.4, 0.5]\n",
    "  )\n",
    "  \n",
    "  nyse_m1_bucketized = tf.feature_column.bucketized_column(\n",
    "      nyse_m1, \n",
    "      boundaries=[-.05, -.04, -.03, -.02, -.01, .00, .01, .02, .03, 0.4, 0.5]\n",
    "  )\n",
    "  \n",
    "  snp_m1_x_nyse_m1 = tf.feature_column.crossed_column(\n",
    "      [snp_m1_bucketized, nyse_m1_bucketized],\n",
    "      hash_bucket_size=10\n",
    "  )\n",
    "  \n",
    "  wide_columns = [snp_m1_bucketized, nyse_m1_bucketized, snp_m1_x_nyse_m1]\n",
    "  \n",
    "  deep_columns = [\n",
    "     snp_m1, snp_m2, nyse_m1, nyse_m2,\n",
    "    djia_m1, djia_m2, nikkei_m0, nikkei_m1,\n",
    "    hangseng_m0, hangseng_m1, ftse_m0, ftse_m1,\n",
    "    dax_m0, dax_m1, aord_m0, aord_m1\n",
    "  ]\n",
    "  \n",
    "  return wide_columns, deep_columns\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3 - Create a DNN Regression Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def create_DNNLinearCombinedClassifier(run_config, hparams):\n",
    "  \n",
    "    wide_columns, deep_columns = get_deep_and_wide_columns()\n",
    "\n",
    "    dnn_optimizer = tf.train.AdamOptimizer(learning_rate=hparams.learning_rate)\n",
    "    \n",
    "    estimator = tf.estimator.DNNLinearCombinedClassifier(\n",
    "        linear_feature_columns = wide_columns,\n",
    "        dnn_feature_columns = deep_columns,\n",
    "        dnn_optimizer=dnn_optimizer,\n",
    "        dnn_hidden_units=hparams.hidden_units,\n",
    "        label_vocabulary=TARGET_VALUES,\n",
    "        config = run_config\n",
    "                )\n",
    "    \n",
    "    return estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 4. Run Local Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### a. RunConfig and Hyper-params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Hyper-parameters\n",
    "hparams  = tf.contrib.training.HParams(num_epochs = 10,\n",
    "                                       batch_size = 500,\n",
    "                                       hidden_units=[32, 16],\n",
    "                                       max_steps = 2000,\n",
    "                                       learning_rate = 0.1,\n",
    "                                       evaluate_after_sec=10)\n",
    "\n",
    "# RunConfig\n",
    "local_model_dir = \"trained_models/financialtimeseries_dnn\"\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=19830610,\n",
    "    model_dir=local_model_dir\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### b. Serving Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def csv_serving_input_fn():\n",
    "  \n",
    "  SERVING_HEADER = [\n",
    "    'snp_m1', 'snp_m2', 'nyse_m1', 'nyse_m2',  \n",
    "    'djia_m1', 'djia_m2', 'nikkei_m0', 'nikkei_m1', \n",
    "    'hangseng_m0', 'hangseng_m1', 'ftse_m0', 'ftse_m1', \n",
    "    'dax_m0', 'dax_m1', 'aord_m0', 'aord_m1'\n",
    "  ]\n",
    "  SERVING_HEADER_DEFAULTS = [\n",
    "    [0.0], [0.0], [0.0], [0.0], \n",
    "    [0.0], [0.0], [0.0], [0.0], \n",
    "    [0.0], [0.0], [0.0], [0.0], \n",
    "    [0.0], [0.0], [0.0], [0.0]\n",
    "  ]\n",
    "\n",
    "  rows_string_tensor = tf.placeholder(dtype=tf.string,\n",
    "                                         shape=[None],\n",
    "                                         name='csv_rows')\n",
    "    \n",
    "  receiver_tensor = {'csv_rows': rows_string_tensor}\n",
    "\n",
    "  row_columns = tf.expand_dims(rows_string_tensor, -1)\n",
    "  columns = tf.decode_csv(row_columns, record_defaults=SERVING_HEADER_DEFAULTS)\n",
    "  features = dict(zip(SERVING_HEADER, columns))\n",
    "  \n",
    "  # apply feature preprocessing used input_fn\n",
    "  features = process_features(features)\n",
    "  \n",
    "  return tf.estimator.export.ServingInputReceiver(\n",
    "        features, receiver_tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### c. TrainSpec and EvalSpec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "train_data_files = \"data/financialtimeseries/train-data.csv\"\n",
    "eval_data_files = \"data/financialtimeseries/eval-data.csv\"\n",
    "\n",
    "# TrainSpec\n",
    "train_spec = tf.estimator.TrainSpec(\n",
    "  input_fn = lambda: csv_input_fn(\n",
    "    train_data_files,\n",
    "    mode=tf.estimator.ModeKeys.TRAIN,\n",
    "    num_epochs= hparams.num_epochs,\n",
    "    batch_size = hparams.batch_size\n",
    "  ),\n",
    "  max_steps=hparams.max_steps,\n",
    ")\n",
    "\n",
    "# EvalSpec\n",
    "eval_spec = tf.estimator.EvalSpec(\n",
    "  input_fn =lambda: csv_input_fn(eval_data_files),\n",
    "  exporters=[tf.estimator.LatestExporter(\n",
    "      name=\"estimate\",  # the name of the folder in which the model will be exported to under export\n",
    "      serving_input_receiver_fn=csv_serving_input_fn,\n",
    "      exports_to_keep=1,\n",
    "      as_text=True)],\n",
    "  steps = None,\n",
    "  throttle_secs = hparams.evaluate_after_sec # evalute after each 10 training seconds!\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### d. Run train_and_evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "# remove the following line of code to resume training\n",
    "shutil.rmtree(local_model_dir, ignore_errors=True)\n",
    "\n",
    "dnn_estimator = create_DNNLinearCombinedClassifier(run_config, hparams)\n",
    "\n",
    "# run train and evaluate experiment\n",
    "tf.estimator.train_and_evaluate(\n",
    "  dnn_estimator,\n",
    "  train_spec,\n",
    "  eval_spec\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "ls trained_models/financialtimeseries_dnn/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### >> TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from google.datalab.ml import TensorBoard\n",
    "TensorBoard().start(\"trained_models/financialtimeseries_dnn\")\n",
    "TensorBoard().list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#to stop TensorBoard\n",
    "TensorBoard().stop(3843)\n",
    "print('stopped TensorBoard')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train the Model on Cloud ML Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"Submitting a Cloud ML Engine job...\"\n",
    "\n",
    "REGION=${REGION}\n",
    "TIER=BASIC # BASIC | BASIC_GPU | STANDARD_1 | PREMIUM_1\n",
    "BUCKET=${BUCKET}\n",
    "\n",
    "MODEL_NAME=\"financialtimeseries\"\n",
    "\n",
    "PACKAGE_PATH=ml-packages/financialtimeseries-tf1.4/trainer\n",
    "TRAIN_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/train-*csv\n",
    "VALID_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/eval-*csv\n",
    "MODEL_DIR=gs://${BUCKET}/models/${MODEL_NAME}\n",
    "\n",
    "#remove model directory, if you don't want to resume training, or if you have changed the model structure\n",
    "#gsutil -m rm -r ${MODEL_DIR}\n",
    "\n",
    "CURRENT_DATE=`date +%Y%m%d_%H%M%S`\n",
    "JOB_NAME=train_${MODEL_NAME}_${TIER}_${CURRENT_DATE}\n",
    "\n",
    "gcloud ml-engine jobs submit training ${JOB_NAME} \\\n",
    "       --job-dir=${MODEL_DIR} \\\n",
    "       --runtime-version=1.4 \\\n",
    "       --region=${REGION} \\\n",
    "       --scale-tier=${TIER} \\\n",
    "       --module-name=trainer.task \\\n",
    "       --package-path=${PACKAGE_PATH} \\\n",
    "       -- \\\n",
    "       --train-files=${TRAIN_FILES} \\\n",
    "       --max-steps=5000 \\\n",
    "       --train-batch-size=500 \\\n",
    "       --eval-files=${VALID_FILES} \\\n",
    "       --eval-batch-size=500 \\\n",
    "       --learning-rate=0.01 \\\n",
    "       --layer-sizes-scale-factor=0.5 \\\n",
    "       --num-layers=3 \\\n",
    "       --job-dir=${MODEL_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train the Model on Cloud ML Engine + GPUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"Submitting a Cloud ML Engine job...\"\n",
    "\n",
    "REGION=${REGION}\n",
    "TIER=BASIC_GPU # BASIC | BASIC_GPU | STANDARD_1 | PREMIUM_1\n",
    "BUCKET=${BUCKET}\n",
    "\n",
    "MODEL_NAME=\"financialtimeseries\"\n",
    "\n",
    "PACKAGE_PATH=ml-packages/financialtimeseries-tf1.4/trainer\n",
    "TRAIN_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/train-*csv\n",
    "VALID_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/eval-*csv\n",
    "MODEL_DIR=gs://${BUCKET}/models/${MODEL_NAME}\n",
    "\n",
    "#remove model directory, if you don't want to resume training, or if you have changed the model structure\n",
    "#gsutil -m rm -r ${MODEL_DIR}\n",
    "\n",
    "CURRENT_DATE=`date +%Y%m%d_%H%M%S`\n",
    "JOB_NAME=train_${MODEL_NAME}_${TIER}_${CURRENT_DATE}\n",
    "\n",
    "gcloud ml-engine jobs submit training ${JOB_NAME} \\\n",
    "       --job-dir=${MODEL_DIR} \\\n",
    "       --runtime-version=1.4 \\\n",
    "       --region=${REGION} \\\n",
    "       --scale-tier=${TIER} \\\n",
    "       --module-name=trainer.task \\\n",
    "       --package-path=${PACKAGE_PATH} \\\n",
    "       -- \\\n",
    "       --train-files=${TRAIN_FILES} \\\n",
    "       --max-steps=5000 \\\n",
    "       --train-batch-size=500 \\\n",
    "       --eval-files=${VALID_FILES} \\\n",
    "       --eval-batch-size=500 \\\n",
    "       --learning-rate=0.01 \\\n",
    "       --layer-sizes-scale-factor=0.5 \\\n",
    "       --num-layers=3 \\\n",
    "       --job-dir=${MODEL_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train the Model on Cloud ML Engine + Custom GPUs Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"Submitting a Cloud ML Engine job...\"\n",
    "\n",
    "REGION=${REGION}\n",
    "TIER=CUSTOM # BASIC | BASIC_GPU | STANDARD_1 | PREMIUM_1\n",
    "BUCKET=${BUCKET}\n",
    "\n",
    "MODEL_NAME=\"financialtimeseries\"\n",
    "\n",
    "PACKAGE_PATH=ml-packages/financialtimeseries-tf1.4/trainer\n",
    "TRAIN_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/train-*csv\n",
    "VALID_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/eval-*csv\n",
    "MODEL_DIR=gs://${BUCKET}/models/${MODEL_NAME}\n",
    "\n",
    "#remove model directory, if you don't want to resume training, or if you have changed the model structure\n",
    "#gsutil -m rm -r ${MODEL_DIR}\n",
    "\n",
    "CURRENT_DATE=`date +%Y%m%d_%H%M%S`\n",
    "JOB_NAME=train_${MODEL_NAME}_${TIER}_${CURRENT_DATE}\n",
    "\n",
    "gcloud ml-engine jobs submit training ${JOB_NAME} \\\n",
    "       --job-dir=${MODEL_DIR} \\\n",
    "       --runtime-version=1.4 \\\n",
    "       --region=${REGION} \\\n",
    "       --module-name=trainer.task \\\n",
    "       --package-path=${PACKAGE_PATH} \\\n",
    "       --config=ml-packages/financialtimeseries-tf1.4/custom.yaml \\\n",
    "       -- \\\n",
    "       --train-files=${TRAIN_FILES} \\\n",
    "       --max-steps=5000 \\\n",
    "       --train-batch-size=500 \\\n",
    "       --eval-files=${VALID_FILES} \\\n",
    "       --eval-batch-size=500 \\\n",
    "       --learning-rate=0.01 \\\n",
    "       --layer-sizes-scale-factor=0.5 \\\n",
    "       --num-layers=3 \\\n",
    "       --job-dir=${MODEL_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Hyper-parameters Tuning on Cloud ML Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"Submitting a Cloud ML Engine job...\"\n",
    "\n",
    "REGION=${REGION}\n",
    "BUCKET=${BUCKET}\n",
    "\n",
    "MODEL_NAME=\"financialtimeseries\"\n",
    "\n",
    "PACKAGE_PATH=ml-packages/financialtimeseries-tf1.4/trainer\n",
    "TRAIN_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/train-*csv\n",
    "VALID_FILES=gs://${BUCKET}/data/financialtimeseries/big_data/eval-*csv\n",
    "MODEL_DIR=gs://${BUCKET}/models/${MODEL_NAME}_tune\n",
    "\n",
    "CURRENT_DATE=`date +%Y%m%d_%H%M%S`\n",
    "JOB_NAME=tune_${MODEL_NAME}_TUNE_${CURRENT_DATE}\n",
    "\n",
    "gcloud ml-engine jobs submit training ${JOB_NAME} \\\n",
    "        --job-dir=${MODEL_DIR} \\\n",
    "        --runtime-version=1.4 \\\n",
    "        --region=${REGION} \\\n",
    "        --module-name=trainer.task \\\n",
    "        --package-path=${PACKAGE_PATH} \\\n",
    "        --config=ml-packages/financialtimeseries-tf1.4/hyperparams.yaml \\\n",
    "        -- \\\n",
    "        --train-files=${TRAIN_FILES} \\\n",
    "        --max-steps=5000 \\\n",
    "        --train-batch-size=1000 \\\n",
    "        --eval-files=${VALID_FILES} \\\n",
    "        --eval-batch-size=1000 \\\n",
    "        --job-dir=${MODEL_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Deploy the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "REGION=${REGION}\n",
    "BUCKET=${BUCKET}\n",
    "\n",
    "MODEL_NAME=\"financialtimeseries\"\n",
    "MODEL_VERSION=\"v1\"\n",
    "\n",
    "MODEL_BINARIES=$(gsutil ls gs://${BUCKET}/models/${MODEL_NAME}/export/classifier | tail -1)\n",
    "\n",
    "\n",
    "# delete model version\n",
    "#gcloud ml-engine versions delete ${MODEL_VERSION} --model=${MODEL_NAME}\n",
    "\n",
    "# delete model\n",
    "#gcloud ml-engine models delete ${MODEL_NAME}\n",
    "\n",
    "# deploy model to GCP\n",
    "#gcloud ml-engine models create ${MODEL_NAME} --regions=${REGION}\n",
    "\n",
    "#deploy model version\n",
    "gcloud ml-engine versions create ${MODEL_VERSION} --model=${MODEL_NAME} --origin=${MODEL_BINARIES} --runtime-version=1.4\n",
    "\n",
    "echo  ${MODEL_NAME} ${MODEL_VERSION} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "MODEL_NAME=\"financialtimeseries\"\n",
    "MODEL_VERSION=\"v1\"\n",
    "\n",
    "# invoke deployed model to make prediction given new data instances\n",
    "gcloud ml-engine predict --model=${MODEL_NAME} --version=${MODEL_VERSION} --json-instances=data/financialtimeseries/new-data.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Consume the Model as API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from googleapiclient import discovery\n",
    "from oauth2client.client import GoogleCredentials\n",
    "\n",
    "def estimate(project, model_name, version, instances):\n",
    "\n",
    "    credentials = GoogleCredentials.get_application_default()\n",
    "    api = discovery.build('ml', 'v1', credentials=credentials,\n",
    "                discoveryServiceUrl='https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json')\n",
    "\n",
    "    request_data = {'instances': instances}\n",
    "\n",
    "    model_url = 'projects/{}/models/{}/versions/{}'.format(project, model_name, version)\n",
    "    response = api.projects().predict(body=request_data, name=model_url).execute()\n",
    "    \n",
    "    estimates = [item[u'classes'] for item in response[u'predictions']]\n",
    "\n",
    "\n",
    "    return estimates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "MODEL_NAME='financialtimeseries'\n",
    "MODEL_VERSION=\"v1\"\n",
    "\n",
    "instances =  [\n",
    "      {\n",
    "        'snp_m1' : -0.011381397443,\n",
    "        'snp_m2' : -0.00318472489586,\n",
    "        'nyse_m1' : -0.0122561131386,\n",
    "        'nyse_m2' : -0.00311830464655,\n",
    "        'djia_m1' : -0.00814669586107,\n",
    "        'djia_m2' : -0.00327516443468,\n",
    "        'nikkei_m0' : -0.0112286842277,\n",
    "        'nikkei_m1' : -0.00618448662269,\n",
    "        'hangseng_m0' : -0.00516025557739,\n",
    "        'hangseng_m1' : -0.0172446189673,\n",
    "        'ftse_m0' : -0.0096349683829,\n",
    "        'ftse_m1' : -0.0023942318244,\n",
    "        'dax_m0' : -0.00664421427352,\n",
    "        'dax_m1' : -0.00410411743843,\n",
    "        'aord_m0' : -0.00791849937675,\n",
    "        'aord_m1' : -0.0071570640642\n",
    "      },\n",
    "      {\n",
    "        'snp_m1' : -0.00262898268279,\n",
    "        'snp_m2' : 0.00519789154606,\n",
    "        'nyse_m1' : -0.00238490313864,\n",
    "        'nyse_m2' : 0.00336377324257,\n",
    "        'djia_m1' : -0.00193058871584,\n",
    "        'djia_m2' : 0.00387061058697,\n",
    "        'nikkei_m0' : -0.0162761526936,\n",
    "        'nikkei_m1' : -0.00319560688,\n",
    "        'hangseng_m0' : -0.0131692520561,\n",
    "        'hangseng_m1' : -0.0143661769443,\n",
    "        'ftse_m0' : -0.0189954213177,\n",
    "        'ftse_m1' : -0.00367068315366,\n",
    "        'dax_m0' : -0.021648106156,\n",
    "        'dax_m1' : -0.00223370158645,\n",
    "        'aord_m0' : 0.0131723464534,\n",
    "        'aord_m1' : -0.0110883690885\n",
    "      },\n",
    "      {\n",
    "        'snp_m1' : 0.00426055324048,\n",
    "        'snp_m2' : 0.00782533125554,\n",
    "        'nyse_m1' : 0.00591736960864,\n",
    "        'nyse_m2' : 0.00691895179585,\n",
    "        'djia_m1' : 0.00282309241802,\n",
    "        'djia_m2' : 0.00827282886371,\n",
    "        'nikkei_m0' : 0.00478911588037,\n",
    "        'nikkei_m1' : 0.00989148467924,\n",
    "        'hangseng_m0' : 0.00489156719444,\n",
    "        'hangseng_m1' : 0.00980530371273,\n",
    "        'ftse_m0' : 0.000596399488808,\n",
    "        'ftse_m1' : 0.0114790189729,\n",
    "        'dax_m0' : 0.000613251128797,\n",
    "        'dax_m1' : 0.00780638261224,\n",
    "        'aord_m0' : 0.00619442299894,\n",
    "        'aord_m1' : 0.00614341726831\n",
    "      }\n",
    "  ]\n",
    "\n",
    "estimates = estimate(\n",
    "    instances=instances,\n",
    "    project=PROJECT,\n",
    "    model_name=MODEL_NAME,\n",
    "    version=MODEL_VERSION)\n",
    "\n",
    "print(estimates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### the end ..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}