steering-models/community-models/komanda/solution-komanda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This jupyter notebook describes the solution approach of the team komanda to udacity self-driving car challenge 2: predicting steering angle from images.\n",
    "\n",
    "Author: Ilya Edrenkin, `ilya.edrenkin@gmail.com`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "slim = tf.contrib.slim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outline\n",
    "\n",
    "The presented model performs a mapping from sequences of images to sequences of steering angle measurements. The mapping is causal, i.e. there is no \"looking into future\" -- only past frames are used to predict the future steering decisions.\n",
    "\n",
    "The model is based on three key components:\n",
    "1) The input image sequences are processed with a 3D convolution stack, where the discrete time axis is interpreted as the first \"depth\" dimension. That allows the model to learn motion detectors and understand the dynamics of driving.\n",
    "2) The model predicts not only the steering angle, but also the vehicle speed and the torque applied to the steering wheel. \n",
    "3) The model is stateful: the two upper layers are a LSTM and a simple RNN, respectively. The predicted angle, torque and speed serve as the input to the next timestep. \n",
    "\n",
    "The model is optimized jointly for the autoregressive and ground truth modes: in the former, model's own outputs are fed into next timestep, in the latter, real targets are used as the context. Naturally, only autoregressive mode is used at the test time.\n",
    "\n",
    "I used a single GTX 1080 to train the model. In the training phase there was a constraint to fit into the memory of the card (8 GB). For the evaluation phase the model was performing nearly twice as fast as real-time in this setup.\n",
    "\n",
    "Data extraction from rosbags is performed using Ross Wightman's scripts, because these were also used for the test data in this challenge; for real-life scenarios (and not for the challenge) it would make sense to read data directly into the model from the rosbags. Another concern about real-life is that the steering angle sequence that is to be predicted should be probably delayed by the actuator's latency.\n",
    "\n",
    "No data augmentation (except for aggressive regularization via dropout) is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define some constants\n",
    "\n",
    "# RNNs are typically trained using (truncated) backprop through time. SEQ_LEN here is the length of BPTT. \n",
    "# Batch size specifies the number of sequence fragments used in a sigle optimization step.\n",
    "# (Actually we can use variable SEQ_LEN and BATCH_SIZE, they are set to constants only for simplicity).\n",
    "# LEFT_CONTEXT is the number of extra frames from the past that we append to the left of our input sequence.\n",
    "# We need to do it because 3D convolution with \"VALID\" padding \"eats\" frames from the left, decreasing the sequence length.\n",
    "# One should be careful here to maintain the model's causality.\n",
    "SEQ_LEN = 10 \n",
    "BATCH_SIZE = 4 \n",
    "LEFT_CONTEXT = 5\n",
    "\n",
    "# These are the input image parameters.\n",
    "HEIGHT = 480\n",
    "WIDTH = 640\n",
    "CHANNELS = 3 # RGB\n",
    "\n",
    "# The parameters of the LSTM that keeps the model state.\n",
    "RNN_SIZE = 32\n",
    "RNN_PROJ = 32\n",
    "\n",
    "# Our training data follows the \"interpolated.csv\" format from Ross Wightman's scripts.\n",
    "CSV_HEADER = \"index,timestamp,width,height,frame_id,filename,angle,torque,speed,lat,long,alt\".split(\",\")\n",
    "OUTPUTS = CSV_HEADER[-6:-3] # angle,torque,speed\n",
    "OUTPUT_DIM = len(OUTPUTS) # predict all features: steering angle, torque and vehicle speed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input/output format\n",
    "\n",
    "Our data is presented as a long sequence of observations (several concatenated rosbags).\n",
    "We need to chunk it into a number of batches: for this, we will create BATCH_SIZE cursors. Let their starting points be uniformly spaced in our long sequence. We will advance them by SEQ_LEN at each step, creating a BATCH_SIZE x SEQ_LEN matrix of training examples.\n",
    "Boundary effects when one rosbag ends and the next starts are simply ignored.\n",
    "\n",
    "(Actually, LEFT_CONTEXT frames are also added to the left of the input sequence; see code below for details)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class BatchGenerator(object):\n",
    "    def __init__(self, sequence, seq_len, batch_size):\n",
    "        self.sequence = sequence\n",
    "        self.seq_len = seq_len\n",
    "        self.batch_size = batch_size\n",
    "        chunk_size = 1 + (len(sequence) - 1) / batch_size\n",
    "        self.indices = [(i*chunk_size) % len(sequence) for i in range(batch_size)]\n",
    "        \n",
    "    def next(self):\n",
    "        while True:\n",
    "            output = []\n",
    "            for i in range(self.batch_size):\n",
    "                idx = self.indices[i]\n",
    "                left_pad = self.sequence[idx - LEFT_CONTEXT:idx]\n",
    "                if len(left_pad) < LEFT_CONTEXT:\n",
    "                    left_pad = [self.sequence[0]] * (LEFT_CONTEXT - len(left_pad)) + left_pad\n",
    "                assert len(left_pad) == LEFT_CONTEXT\n",
    "                leftover = len(self.sequence) - idx\n",
    "                if leftover >= self.seq_len:\n",
    "                    result = self.sequence[idx:idx + self.seq_len]\n",
    "                else:\n",
    "                    result = self.sequence[idx:] + self.sequence[:self.seq_len - leftover]\n",
    "                assert len(result) == self.seq_len\n",
    "                self.indices[i] = (idx + self.seq_len) % len(self.sequence)\n",
    "                images, targets = zip(*result)\n",
    "                images_left_pad, _ = zip(*left_pad)\n",
    "                output.append((np.stack(images_left_pad + images), np.stack(targets)))\n",
    "            output = zip(*output)\n",
    "            output[0] = np.stack(output[0]) # batch_size x (LEFT_CONTEXT + seq_len)\n",
    "            output[1] = np.stack(output[1]) # batch_size x seq_len x OUTPUT_DIM\n",
    "            return output\n",
    "        \n",
    "def read_csv(filename):\n",
    "    with open(filename, 'r') as f:\n",
    "        lines = [ln.strip().split(\",\")[-7:-3] for ln in f.readlines()]\n",
    "        lines = map(lambda x: (x[0], np.float32(x[1:])), lines) # imagefile, outputs\n",
    "        return lines\n",
    "\n",
    "def process_csv(filename, val=5):\n",
    "    sum_f = np.float128([0.0] * OUTPUT_DIM)\n",
    "    sum_sq_f = np.float128([0.0] * OUTPUT_DIM)\n",
    "    lines = read_csv(filename)\n",
    "    # leave val% for validation\n",
    "    train_seq = []\n",
    "    valid_seq = []\n",
    "    cnt = 0\n",
    "    for ln in lines:\n",
    "        if cnt < SEQ_LEN * BATCH_SIZE * (100 - val): \n",
    "            train_seq.append(ln)\n",
    "            sum_f += ln[1]\n",
    "            sum_sq_f += ln[1] * ln[1]\n",
    "        else:\n",
    "            valid_seq.append(ln)\n",
    "        cnt += 1\n",
    "        cnt %= SEQ_LEN * BATCH_SIZE * 100\n",
    "    mean = sum_f / len(train_seq)\n",
    "    var = sum_sq_f / len(train_seq) - mean * mean\n",
    "    std = np.sqrt(var)\n",
    "    print len(train_seq), len(valid_seq)\n",
    "    print mean, std # we will need these statistics to normalize the outputs (and ground truth inputs)\n",
    "    return (train_seq, valid_seq), (mean, std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "(train_seq, valid_seq), (mean, std) = process_csv(filename=\"output/interpolated_concat.csv\", val=5) # concatenated interpolated.csv from rosbags \n",
    "test_seq = read_csv(\"challenge_2/exampleSubmissionInterpolatedFinal.csv\") # interpolated.csv for testset filled with dummy values "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Key tricks\n",
    "\n",
    "Now we are ready to build the model.\n",
    "In the next cell we will define the vision module and the recurrent stateful cell.\n",
    "\n",
    "The vision module takes a tensor of shape `[BATCH_SIZE, LEFT_CONTEXT + SEQ_LEN, HEIGHT, WIDTH, CHANNELS]` and outputs a tensor of shape `[BATCH_SIZE, SEQ_LEN, 128]`. The entire LEFT_CONTEXT is eaten by the 3D convolutions. Well-known tricks like residual connections and layer normalization are used to improve the convergence of the vision module. Dropout between each pair of layers serves as a regularizer.\n",
    "\n",
    "We also need to define our own recurrent cell because we need to train our model jointly in two conditions: when it uses ground truth history and when it uses its own past predictions as the context for the future predictions.\n",
    "\n",
    "In addition, we define two helper functions: a layer normalizer with trainable gain/offset and a gradient-clipping optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "layer_norm = lambda x: tf.contrib.layers.layer_norm(inputs=x, center=True, scale=True, activation_fn=None, trainable=True)\n",
    "\n",
    "def get_optimizer(loss, lrate):\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=lrate)\n",
    "    gradvars = optimizer.compute_gradients(loss)\n",
    "    gradients, v = zip(*gradvars)\n",
    "    print [x.name for x in v]\n",
    "    gradients, _ = tf.clip_by_global_norm(gradients, 15.0)\n",
    "    return optimizer.apply_gradients(zip(gradients, v))\n",
    "\n",
    "def apply_vision_simple(image, keep_prob, batch_size, seq_len, scope=None, reuse=None):\n",
    "    video = tf.reshape(image, shape=[batch_size, LEFT_CONTEXT + seq_len, HEIGHT, WIDTH, CHANNELS])\n",
    "    with tf.variable_scope(scope, 'Vision', [image], reuse=reuse):\n",
    "        net = slim.convolution(video, num_outputs=64, kernel_size=[3,12,12], stride=[1,6,6], padding=\"VALID\")\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        aux1 = slim.fully_connected(tf.reshape(net[:, -seq_len:, :, :, :], [batch_size, seq_len, -1]), 128, activation_fn=None)\n",
    "        \n",
    "        net = slim.convolution(net, num_outputs=64, kernel_size=[2,5,5], stride=[1,2,2], padding=\"VALID\")\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        aux2 = slim.fully_connected(tf.reshape(net[:, -seq_len:, :, :, :], [batch_size, seq_len, -1]), 128, activation_fn=None)\n",
    "        \n",
    "        net = slim.convolution(net, num_outputs=64, kernel_size=[2,5,5], stride=[1,1,1], padding=\"VALID\")\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        aux3 = slim.fully_connected(tf.reshape(net[:, -seq_len:, :, :, :], [batch_size, seq_len, -1]), 128, activation_fn=None)\n",
    "        \n",
    "        net = slim.convolution(net, num_outputs=64, kernel_size=[2,5,5], stride=[1,1,1], padding=\"VALID\")\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        # at this point the tensor 'net' is of shape batch_size x seq_len x ...\n",
    "        aux4 = slim.fully_connected(tf.reshape(net, [batch_size, seq_len, -1]), 128, activation_fn=None)\n",
    "        \n",
    "        net = slim.fully_connected(tf.reshape(net, [batch_size, seq_len, -1]), 1024, activation_fn=tf.nn.relu)\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        net = slim.fully_connected(net, 512, activation_fn=tf.nn.relu)\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        net = slim.fully_connected(net, 256, activation_fn=tf.nn.relu)\n",
    "        net = tf.nn.dropout(x=net, keep_prob=keep_prob)\n",
    "        net = slim.fully_connected(net, 128, activation_fn=None)\n",
    "        return layer_norm(tf.nn.elu(net + aux1 + aux2 + aux3 + aux4)) # aux[1-4] are residual connections (shortcuts)\n",
    "\n",
    "class SamplingRNNCell(tf.nn.rnn_cell.RNNCell):\n",
    "  \"\"\"Simple sampling RNN cell.\"\"\"\n",
    "\n",
    "  def __init__(self, num_outputs, use_ground_truth, internal_cell):\n",
    "    \"\"\"\n",
    "    if use_ground_truth then don't sample\n",
    "    \"\"\"\n",
    "    self._num_outputs = num_outputs\n",
    "    self._use_ground_truth = use_ground_truth # boolean\n",
    "    self._internal_cell = internal_cell # may be LSTM or GRU or anything\n",
    "  \n",
    "  @property\n",
    "  def state_size(self):\n",
    "    return self._num_outputs, self._internal_cell.state_size # previous output and bottleneck state\n",
    "\n",
    "  @property\n",
    "  def output_size(self):\n",
    "    return self._num_outputs # steering angle, torque, vehicle speed\n",
    "\n",
    "  def __call__(self, inputs, state, scope=None):\n",
    "    (visual_feats, current_ground_truth) = inputs\n",
    "    prev_output, prev_state_internal = state\n",
    "    context = tf.concat(1, [prev_output, visual_feats])\n",
    "    new_output_internal, new_state_internal = internal_cell(context, prev_state_internal) # here the internal cell (e.g. LSTM) is called\n",
    "    new_output = tf.contrib.layers.fully_connected(\n",
    "        inputs=tf.concat(1, [new_output_internal, prev_output, visual_feats]),\n",
    "        num_outputs=self._num_outputs,\n",
    "        activation_fn=None,\n",
    "        scope=\"OutputProjection\")\n",
    "    # if self._use_ground_truth == True, we pass the ground truth as the state; otherwise, we use the model's predictions\n",
    "    return new_output, (current_ground_truth if self._use_ground_truth else new_output, new_state_internal)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Model\n",
    "\n",
    "Let's build the main graph. Code is mostly self-explanatory.\n",
    "\n",
    "A few comments:\n",
    "\n",
    "1) PNG images were used as the input only because this was the format for round1 testset. In practice, raw images should be fed directly from the rosbags.\n",
    "\n",
    "2) We define get_initial_state and deep_copy_initial_state functions to be able to preserve the state of our recurrent net between batches. The backpropagation is still truncated by SEQ_LEN.\n",
    "\n",
    "3) The loss is composed of two components. The first is the MSE of the steering angle prediction in the autoregressive setting -- that is exactly what interests us in the test time. The second components, weighted by the term aux_cost_weight, is the sum of MSEs for all outputs both in autoregressive and ground truth settings. \n",
    "\n",
    "Note: if the saver definition doesn't work for you please make sure you are using tensorflow 0.12rc0 or newer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "graph = tf.Graph()\n",
    "\n",
    "with graph.as_default():\n",
    "    # inputs  \n",
    "    learning_rate = tf.placeholder_with_default(input=1e-4, shape=())\n",
    "    keep_prob = tf.placeholder_with_default(input=1.0, shape=())\n",
    "    aux_cost_weight = tf.placeholder_with_default(input=0.1, shape=())\n",
    "    \n",
    "    inputs = tf.placeholder(shape=(BATCH_SIZE,LEFT_CONTEXT+SEQ_LEN), dtype=tf.string) # pathes to png files from the central camera\n",
    "    targets = tf.placeholder(shape=(BATCH_SIZE,SEQ_LEN,OUTPUT_DIM), dtype=tf.float32) # seq_len x batch_size x OUTPUT_DIM\n",
    "    targets_normalized = (targets - mean) / std\n",
    "    \n",
    "    input_images = tf.pack([tf.image.decode_png(tf.read_file(x))\n",
    "                            for x in tf.unpack(tf.reshape(inputs, shape=[(LEFT_CONTEXT+SEQ_LEN) * BATCH_SIZE]))])\n",
    "    input_images = -1.0 + 2.0 * tf.cast(input_images, tf.float32) / 255.0\n",
    "    input_images.set_shape([(LEFT_CONTEXT+SEQ_LEN) * BATCH_SIZE, HEIGHT, WIDTH, CHANNELS])\n",
    "    visual_conditions_reshaped = apply_vision_simple(image=input_images, keep_prob=keep_prob, \n",
    "                                                     batch_size=BATCH_SIZE, seq_len=SEQ_LEN)\n",
    "    visual_conditions = tf.reshape(visual_conditions_reshaped, [BATCH_SIZE, SEQ_LEN, -1])\n",
    "    visual_conditions = tf.nn.dropout(x=visual_conditions, keep_prob=keep_prob)\n",
    "    \n",
    "    rnn_inputs_with_ground_truth = (visual_conditions, targets_normalized)\n",
    "    rnn_inputs_autoregressive = (visual_conditions, tf.zeros(shape=(BATCH_SIZE, SEQ_LEN, OUTPUT_DIM), dtype=tf.float32))\n",
    "    \n",
    "    internal_cell = tf.nn.rnn_cell.LSTMCell(num_units=RNN_SIZE, num_proj=RNN_PROJ)\n",
    "    cell_with_ground_truth = SamplingRNNCell(num_outputs=OUTPUT_DIM, use_ground_truth=True, internal_cell=internal_cell)\n",
    "    cell_autoregressive = SamplingRNNCell(num_outputs=OUTPUT_DIM, use_ground_truth=False, internal_cell=internal_cell)\n",
    "    \n",
    "    def get_initial_state(complex_state_tuple_sizes):\n",
    "        flat_sizes = tf.nn.rnn_cell.nest.flatten(complex_state_tuple_sizes)\n",
    "        init_state_flat = [tf.tile(\n",
    "            multiples=[BATCH_SIZE, 1], \n",
    "            input=tf.get_variable(\"controller_initial_state_%d\" % i, initializer=tf.zeros_initializer, shape=([1, s]), dtype=tf.float32))\n",
    "         for i,s in enumerate(flat_sizes)]\n",
    "        init_state = tf.nn.rnn_cell.nest.pack_sequence_as(complex_state_tuple_sizes, init_state_flat)\n",
    "        return init_state\n",
    "    def deep_copy_initial_state(complex_state_tuple):\n",
    "        flat_state = tf.nn.rnn_cell.nest.flatten(complex_state_tuple)\n",
    "        flat_copy = [tf.identity(s) for s in flat_state]\n",
    "        deep_copy = tf.nn.rnn_cell.nest.pack_sequence_as(complex_state_tuple, flat_copy)\n",
    "        return deep_copy\n",
    "    \n",
    "    controller_initial_state_variables = get_initial_state(cell_autoregressive.state_size)\n",
    "    controller_initial_state_autoregressive = deep_copy_initial_state(controller_initial_state_variables)\n",
    "    controller_initial_state_gt = deep_copy_initial_state(controller_initial_state_variables)\n",
    "\n",
    "    with tf.variable_scope(\"predictor\"):\n",
    "        out_gt, controller_final_state_gt = tf.nn.dynamic_rnn(cell=cell_with_ground_truth, inputs=rnn_inputs_with_ground_truth, \n",
    "                          sequence_length=[SEQ_LEN]*BATCH_SIZE, initial_state=controller_initial_state_gt, dtype=tf.float32,\n",
    "                          swap_memory=True, time_major=False)\n",
    "    with tf.variable_scope(\"predictor\", reuse=True):\n",
    "        out_autoregressive, controller_final_state_autoregressive = tf.nn.dynamic_rnn(cell=cell_autoregressive, inputs=rnn_inputs_autoregressive, \n",
    "                          sequence_length=[SEQ_LEN]*BATCH_SIZE, initial_state=controller_initial_state_autoregressive, dtype=tf.float32,\n",
    "                          swap_memory=True, time_major=False)\n",
    "    \n",
    "    mse_gt = tf.reduce_mean(tf.squared_difference(out_gt, targets_normalized))\n",
    "    mse_autoregressive = tf.reduce_mean(tf.squared_difference(out_autoregressive, targets_normalized))\n",
    "    mse_autoregressive_steering = tf.reduce_mean(tf.squared_difference(out_autoregressive[:, :, 0], targets_normalized[:, :, 0]))\n",
    "    steering_predictions = (out_autoregressive[:, :, 0] * std[0]) + mean[0]\n",
    "    \n",
    "    total_loss = mse_autoregressive_steering + aux_cost_weight * (mse_gt + mse_autoregressive)\n",
    "    \n",
    "    optimizer = get_optimizer(total_loss, learning_rate)\n",
    "\n",
    "    tf.scalar_summary(\"MAIN TRAIN METRIC: rmse_autoregressive_steering\", tf.sqrt(mse_autoregressive_steering))\n",
    "    tf.scalar_summary(\"rmse_gt\", tf.sqrt(mse_gt))\n",
    "    tf.scalar_summary(\"rmse_autoregressive\", tf.sqrt(mse_autoregressive))\n",
    "    \n",
    "    summaries = tf.merge_all_summaries()\n",
    "    train_writer = tf.train.SummaryWriter('v3/train_summary', graph=graph)\n",
    "    valid_writer = tf.train.SummaryWriter('v3/valid_summary', graph=graph)\n",
    "    saver = tf.train.Saver(write_version=tf.train.SaverDef.V2)\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "\n",
    "At this point we can start the training procedure.\n",
    "\n",
    "We will perform optimization for 100 epochs, doing validation after each epoch. We will keep the model's version that obtains the best performance in terms of the primary loss (autoregressive steering MSE) on the validation set.\n",
    "An aggressive regularization is used (`keep_prob=0.25` for dropout), and the validation loss is highly non-monotonical.\n",
    "\n",
    "For each version of the model that beats the previous best validation score we will overwrite the checkpoint file and obtain predictions for the challenge test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=1.0)\n",
    "\n",
    "checkpoint_dir = os.getcwd() + \"/v3\"\n",
    "\n",
    "global_train_step = 0\n",
    "global_valid_step = 0\n",
    "\n",
    "KEEP_PROB_TRAIN = 0.25\n",
    "\n",
    "def do_epoch(session, sequences, mode):\n",
    "    global global_train_step, global_valid_step\n",
    "    test_predictions = {}\n",
    "    valid_predictions = {}\n",
    "    batch_generator = BatchGenerator(sequence=sequences, seq_len=SEQ_LEN, batch_size=BATCH_SIZE)\n",
    "    total_num_steps = 1 + (batch_generator.indices[1] - 1) / SEQ_LEN\n",
    "    controller_final_state_gt_cur, controller_final_state_autoregressive_cur = None, None\n",
    "    acc_loss = np.float128(0.0)\n",
    "    for step in range(total_num_steps):\n",
    "        feed_inputs, feed_targets = batch_generator.next()\n",
    "        feed_dict = {inputs : feed_inputs, targets : feed_targets}\n",
    "        if controller_final_state_autoregressive_cur is not None:\n",
    "            feed_dict.update({controller_initial_state_autoregressive : controller_final_state_autoregressive_cur})\n",
    "        if controller_final_state_gt_cur is not None:\n",
    "            feed_dict.update({controller_final_state_gt : controller_final_state_gt_cur})\n",
    "        if mode == \"train\":\n",
    "            feed_dict.update({keep_prob : KEEP_PROB_TRAIN})\n",
    "            summary, _, loss, controller_final_state_gt_cur, controller_final_state_autoregressive_cur = \\\n",
    "                session.run([summaries, optimizer, mse_autoregressive_steering, controller_final_state_gt, controller_final_state_autoregressive],\n",
    "                           feed_dict = feed_dict)\n",
    "            train_writer.add_summary(summary, global_train_step)\n",
    "            global_train_step += 1\n",
    "        elif mode == \"valid\":\n",
    "            model_predictions, summary, loss, controller_final_state_autoregressive_cur = \\\n",
    "                session.run([steering_predictions, summaries, mse_autoregressive_steering, controller_final_state_autoregressive],\n",
    "                           feed_dict = feed_dict)\n",
    "            valid_writer.add_summary(summary, global_valid_step)\n",
    "            global_valid_step += 1  \n",
    "            feed_inputs = feed_inputs[:, LEFT_CONTEXT:].flatten()\n",
    "            steering_targets = feed_targets[:, :, 0].flatten()\n",
    "            model_predictions = model_predictions.flatten()\n",
    "            stats = np.stack([steering_targets, model_predictions, (steering_targets - model_predictions)**2])\n",
    "            for i, img in enumerate(feed_inputs):\n",
    "                valid_predictions[img] = stats[:, i]\n",
    "        elif mode == \"test\":\n",
    "            model_predictions, controller_final_state_autoregressive_cur = \\\n",
    "                session.run([steering_predictions, controller_final_state_autoregressive],\n",
    "                           feed_dict = feed_dict)           \n",
    "            feed_inputs = feed_inputs[:, LEFT_CONTEXT:].flatten()\n",
    "            model_predictions = model_predictions.flatten()\n",
    "            for i, img in enumerate(feed_inputs):\n",
    "                test_predictions[img] = model_predictions[i]\n",
    "        if mode != \"test\":\n",
    "            acc_loss += loss\n",
    "            print '\\r', step + 1, \"/\", total_num_steps, np.sqrt(acc_loss / (step+1)),\n",
    "    print\n",
    "    return (np.sqrt(acc_loss / total_num_steps), valid_predictions) if mode != \"test\" else (None, test_predictions)\n",
    "    \n",
    "\n",
    "NUM_EPOCHS=100\n",
    "\n",
    "best_validation_score = None\n",
    "with tf.Session(graph=graph, config=tf.ConfigProto(gpu_options=gpu_options)) as session:\n",
    "    session.run(tf.initialize_all_variables())\n",
    "    print 'Initialized'\n",
    "    ckpt = tf.train.latest_checkpoint(checkpoint_dir)\n",
    "    if ckpt:\n",
    "        print \"Restoring from\", ckpt\n",
    "        saver.restore(sess=session, save_path=ckpt)\n",
    "    for epoch in range(NUM_EPOCHS):\n",
    "        print \"Starting epoch %d\" % epoch\n",
    "        print \"Validation:\"\n",
    "        valid_score, valid_predictions = do_epoch(session=session, sequences=valid_seq, mode=\"valid\")\n",
    "        if best_validation_score is None: \n",
    "            best_validation_score = valid_score\n",
    "        if valid_score < best_validation_score:\n",
    "            saver.save(session, 'v3/checkpoint-sdc-ch2')\n",
    "            best_validation_score = valid_score\n",
    "            print '\\r', \"SAVED at epoch %d\" % epoch,\n",
    "            with open(\"v3/valid-predictions-epoch%d\" % epoch, \"w\") as out:\n",
    "                result = np.float128(0.0)\n",
    "                for img, stats in valid_predictions.items():\n",
    "                    print >> out, img, stats\n",
    "                    result += stats[-1]\n",
    "            print \"Validation unnormalized RMSE:\", np.sqrt(result / len(valid_predictions))\n",
    "            with open(\"v3/test-predictions-epoch%d\" % epoch, \"w\") as out:\n",
    "                _, test_predictions = do_epoch(session=session, sequences=test_seq, mode=\"test\")\n",
    "                print >> out, \"frame_id,steering_angle\"\n",
    "                for img, pred in test_predictions.items():\n",
    "                    img = img.replace(\"challenge_2/Test-final/center/\", \"\")\n",
    "                    print >> out, \"%s,%f\" % (img, pred)\n",
    "        if epoch != NUM_EPOCHS - 1:\n",
    "            print \"Training\"\n",
    "            do_epoch(session=session, sequences=train_seq, mode=\"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically that's it.\n",
    "\n",
    "The model can be further fine-tuned for the challenge purposes by subsetting the training set and setting the aux_cost_weight to zero. It improves the result slightly, but the improvement is marginal (doesn't affect the challenge ranking). For real-life usage it would be probably harmful because of the risk of overfitting to the dev- or even testset.\n",
    "\n",
    "Of course, speaking of realistic models, we don't need to constrain our input only to the central camera -- other cameras and sensors can dramatically improve the performance. Also it is useful to think of a realistic delay for the target sequence to make an actual non-zero-latency control possible.\n",
    "\n",
    "If something in this writeup is unclear, please write me a e-mail so that I can add the necessary comments/clarifications."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}