rnn_baseline/advanced_baseline/tf_baseline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. Imports and requirements\n",
    "\n",
    "* В данном соревновании мы имеем дело с последовательностями, один из интуитивных способов работы с ними - использование рекуррентных сетей. Данный бейзлайн посвящен тому, чтобы показать, как можно строить хорошие решения без использования сложного и трудоемкого feature engineering-а (чтобы эффективно решать ту же задачу с высоким качеством с помощью бустингов нужно несколько тысяч признаков), благодаря рекуррентным сетям. В этом ноутбуке мы построим решение с использованием фреймфорка `tensorflow`. Для комфортной работы Вам понадобится машина с `GPU` (хватит ресурсов `google colab` или `kaggle`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "import sys\n",
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers as L\n",
    "from tensorflow.keras import Model\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tqdm import tqdm\n",
    "\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = '1'\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "# добавим корневую папку, в ней лежат все необходимые полезные функции для обработки данных\n",
    "sys.path.append('../../')\n",
    "sys.path.append('../')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* В базовом решении на нейронных сетях приведена вся обработка данных и все стадии препроцесснга. В данном ноутбуке опустим данный раздел и используем уже готовые данные."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../../../val_buckets/processed_chunk_000.pkl',\n",
       " '../../../val_buckets/processed_chunk_001.pkl',\n",
       " '../../../val_buckets/processed_chunk_002.pkl',\n",
       " '../../../val_buckets/processed_chunk_003.pkl',\n",
       " '../../../val_buckets/processed_chunk_004.pkl']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_to_dataset = '../../../val_buckets'\n",
    "dir_with_datasets = os.listdir(path_to_dataset)\n",
    "dataset_val = sorted([os.path.join(path_to_dataset, x) for x in dir_with_datasets])\n",
    "dataset_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../../../train_buckets/processed_chunk_000.pkl',\n",
       " '../../../train_buckets/processed_chunk_001.pkl',\n",
       " '../../../train_buckets/processed_chunk_002.pkl',\n",
       " '../../../train_buckets/processed_chunk_003.pkl',\n",
       " '../../../train_buckets/processed_chunk_004.pkl',\n",
       " '../../../train_buckets/processed_chunk_005.pkl',\n",
       " '../../../train_buckets/processed_chunk_006.pkl',\n",
       " '../../../train_buckets/processed_chunk_007.pkl',\n",
       " '../../../train_buckets/processed_chunk_008.pkl',\n",
       " '../../../train_buckets/processed_chunk_009.pkl']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_to_dataset = '../../../train_buckets'\n",
    "dir_with_datasets = os.listdir(path_to_dataset)\n",
    "dataset_train = sorted([os.path.join(path_to_dataset, x) for x in dir_with_datasets])\n",
    "dataset_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Для создания модели будем использовать фреймворк `tensorflow`. В нем есть все, чтобы писать произвольные сложные архитектуры и быстро эксперементировать.\n",
    "\n",
    "* В бейзлайне мы предлагаем базовые компоненты, чтобы можно было обучать нейронную сеть и отслеживать ее качество. Для этого вам предоставлены следующие функции:\n",
    "    * `data_generators.batches_generator` - функция-генератор, итеративно возвращает батчи, поддерживает батчи для `tensorflow.keras` и `torch.nn.module` моделей. В зависимости от флага `is_train` может быть использована для генерации батчей на train/val/test стадию.\n",
    "    * функция `tf_training.train_epoch` - обучает модель одну эпоху.\n",
    "    * функция `tf_training.eval_model` - проверяет качество модели на отложенной выборке и возвращает roc_auc_score.\n",
    "    * функция `tf_training.inference` - делает предикты на новых данных и готовит фрейм для проверяющей системы.\n",
    "    * класс `training_aux.EarlyStopping` - реализует early_stopping, сохраняя лучшую модель. Совместим с `tensorflow/torch`. Пример использования приведен ниже."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from data_generators import batches_generator, transaction_features\n",
    "from tf_training import train_epoch, eval_model, inference\n",
    "from training_aux import EarlyStopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Все признаки в нашей модели будут категориальными. Для их представления в модели используем категориальные эмбеддинги. Для этого нужно каждому категориальному признаку задать размерность латентного пространства. Используем [формулу](https://forums.fast.ai/t/size-of-embedding-for-categorical-variables/42608) из библиотеки `fast.ai`. Все отображения хранятся в файле `embedding_projections.pkl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../constants/embedding_projections.pkl', 'rb') as f:\n",
    "    embedding_projections = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Реализуем модель. Все входные признаки представим в виде эмбеддингов, сконкатенируем, чтобы получить векторное представление транзакции. Используем SpatialDropout, чтобы регуляризовать эмбеддинги. Подадим последовательности в `BiGRU` рекуррентную сеть. Используем все скрытые состояния сети, чтобы получить агрегированное представление об истории транзакции - пропустим все скрытые состояния `BiGRU` через `AvgPooling` и черерз `MaxPooling`. Представим признак `product` в виде отдельного эмбеддинга. Сконкатенируем его с результатами пулингов. На основе такого входа построим небольшой `MLP`, выступающий классификатором для целевой задачи. Используем градиентный спуск, чтобы решить оптимизационную задачу."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_transactions_rnn(transactions_cat_features, embedding_projections, product_col_name='product', \n",
    "                          rnn_units=128, classifier_units=32, optimizer=None):\n",
    "    if not optimizer:\n",
    "        optimizer = keras.optimizers.Adam(lr=1e-3)\n",
    "        \n",
    "    inputs = []\n",
    "    cat_embeds = []\n",
    "    \n",
    "    for feature_name in transactions_cat_features:\n",
    "        inp = L.Input(shape=(None, ), dtype='uint32', name=f'input_{feature_name}')\n",
    "        inputs.append(inp)\n",
    "        source_size, projection = embedding_projections[feature_name]\n",
    "        emb = L.Embedding(source_size+1, projection, trainable=True, mask_zero=False, name=f'embedding_{feature_name}')(inp)\n",
    "        cat_embeds.append(emb)\n",
    "    \n",
    "    # product feature\n",
    "    inp = L.Input(shape=(1, ), dtype='uint32', name=f'input_product')\n",
    "    inputs.append(inp)\n",
    "    source_size, projection = embedding_projections['product']\n",
    "    product_emb = L.Embedding(source_size+1, projection, trainable=True, mask_zero=False, name=f'embedding_product')(inp)\n",
    "    product_emb_reshape = L.Reshape((projection, ))(product_emb)\n",
    "    \n",
    "    concated_cat_embeds = L.concatenate(cat_embeds)\n",
    "    dropout_embeds = L.SpatialDropout1D(0.05)(concated_cat_embeds)\n",
    "    \n",
    "    sequences = L.Bidirectional(L.GRU(units=rnn_units, return_sequences=True))(dropout_embeds)\n",
    "    \n",
    "    pooled_avg_sequences = L.GlobalAveragePooling1D()(sequences)\n",
    "    pooled_max_sequences = L.GlobalMaxPooling1D()(sequences)\n",
    "    \n",
    "    concated = L.concatenate([pooled_avg_sequences, pooled_max_sequences, product_emb_reshape])\n",
    "    \n",
    "    dense_intermediate = L.Dense(classifier_units, activation='relu', \n",
    "                                 kernel_regularizer=keras.regularizers.L1L2(1e-7, 1e-5))(concated)\n",
    "    \n",
    "    proba = L.Dense(1, activation='sigmoid')(dense_intermediate)\n",
    "    \n",
    "    model = Model(inputs=inputs, outputs=proba)\n",
    "    model.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "! rm -r ../../rnn_baseline/checkpoints/tf_advanced_baseline\n",
    "! mkdir ../../rnn_baseline/checkpoints/tf_advanced_baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Для того, чтобы детектировать переобучение используем EarlyStopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_checkpoints = '../../rnn_baseline/checkpoints/tf_advanced_baseline/'\n",
    "es = EarlyStopping(patience=3, mode='max', verbose=True, save_path=os.path.join(path_to_checkpoints, 'best_checkpoint.pt'), \n",
    "                   metric_name='ROC-AUC', save_format='tf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 20\n",
    "train_batch_size = 128\n",
    "val_batch_szie = 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_transactions_rnn(transaction_features, embedding_projections, classifier_units=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_currency (InputLayer)     [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_operation_kind (InputLaye [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_card_type (InputLayer)    [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_operation_type (InputLaye [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_operation_type_group (Inp [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_ecommerce_flag (InputLaye [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_payment_system (InputLaye [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_income_flag (InputLayer)  [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_mcc (InputLayer)          [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_country (InputLayer)      [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_city (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_mcc_category (InputLayer) [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_day_of_week (InputLayer)  [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_hour (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_weekofyear (InputLayer)   [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_amnt (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_days_before (InputLayer)  [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_hour_diff (InputLayer)    [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_currency (Embedding)  (None, None, 6)      72          input_currency[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_operation_kind (Embed (None, None, 5)      40          input_operation_kind[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_card_type (Embedding) (None, None, 29)     5104        input_card_type[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_operation_type (Embed (None, None, 9)      207         input_operation_type[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_operation_type_group  (None, None, 3)      15          input_operation_type_group[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "embedding_ecommerce_flag (Embed (None, None, 3)      12          input_ecommerce_flag[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_payment_system (Embed (None, None, 5)      40          input_payment_system[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_income_flag (Embeddin (None, None, 3)      12          input_income_flag[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_mcc (Embedding)       (None, None, 22)     2398        input_mcc[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "embedding_country (Embedding)   (None, None, 9)      225         input_country[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "embedding_city (Embedding)      (None, None, 28)     4592        input_city[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "embedding_mcc_category (Embeddi (None, None, 10)     290         input_mcc_category[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "embedding_day_of_week (Embeddin (None, None, 5)      40          input_day_of_week[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_hour (Embedding)      (None, None, 9)      225         input_hour[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "embedding_weekofyear (Embedding (None, None, 15)     810         input_weekofyear[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedding_amnt (Embedding)      (None, None, 6)      66          input_amnt[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "embedding_days_before (Embeddin (None, None, 9)      216         input_days_before[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_hour_diff (Embedding) (None, None, 6)      66          input_hour_diff[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, None, 182)    0           embedding_currency[0][0]         \n",
      "                                                                 embedding_operation_kind[0][0]   \n",
      "                                                                 embedding_card_type[0][0]        \n",
      "                                                                 embedding_operation_type[0][0]   \n",
      "                                                                 embedding_operation_type_group[0]\n",
      "                                                                 embedding_ecommerce_flag[0][0]   \n",
      "                                                                 embedding_payment_system[0][0]   \n",
      "                                                                 embedding_income_flag[0][0]      \n",
      "                                                                 embedding_mcc[0][0]              \n",
      "                                                                 embedding_country[0][0]          \n",
      "                                                                 embedding_city[0][0]             \n",
      "                                                                 embedding_mcc_category[0][0]     \n",
      "                                                                 embedding_day_of_week[0][0]      \n",
      "                                                                 embedding_hour[0][0]             \n",
      "                                                                 embedding_weekofyear[0][0]       \n",
      "                                                                 embedding_amnt[0][0]             \n",
      "                                                                 embedding_days_before[0][0]      \n",
      "                                                                 embedding_hour_diff[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d (SpatialDropo (None, None, 182)    0           concatenate[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "input_product (InputLayer)      [(None, 1)]          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional (Bidirectional)   (None, None, 256)    239616      spatial_dropout1d[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_product (Embedding)   (None, 1, 4)         24          input_product[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d (Globa (None, 256)          0           bidirectional[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d (GlobalMax (None, 256)          0           bidirectional[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "reshape (Reshape)               (None, 4)            0           embedding_product[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 516)          0           global_average_pooling1d[0][0]   \n",
      "                                                                 global_max_pooling1d[0][0]       \n",
      "                                                                 reshape[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 128)          66176       concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 1)            129         dense[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 320,375\n",
      "Trainable params: 320,375\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Запустим цикл обучения, каждую эпоху будем логировать лосс, а так же roc-auc на валидации и на обучении. Будем сохрнаять веса после каждой эпохи, а так же лучшие с помощью early_stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 1\n",
      "7270/7270 [==============================] - 302s 41ms/step - loss: 0.1290\n",
      "Validation ROC-AUC improved (-inf --> 0.777933).  Saving model ...\n",
      "Epoch 1 completed. Train roc-auc: 0.7795548925869114, Val roc-auc: 0.7779327052405376\n",
      "Starting epoch 2\n",
      "Epoch 2/2\n",
      "7270/7270 [==============================] - 297s 41ms/step - loss: 0.1113\n",
      "Validation ROC-AUC improved (0.777933 --> 0.793945).  Saving model ...\n",
      "Epoch 2 completed. Train roc-auc: 0.7994083479474852, Val roc-auc: 0.7939452959186554\n",
      "Starting epoch 3\n",
      "Epoch 3/3\n",
      "7270/7270 [==============================] - 296s 41ms/step - loss: 0.1093\n",
      "Validation ROC-AUC improved (0.793945 --> 0.800401).  Saving model ...\n",
      "Epoch 3 completed. Train roc-auc: 0.81264856481915, Val roc-auc: 0.8004008509739607\n",
      "Starting epoch 4\n",
      "Epoch 4/4\n",
      "7270/7270 [==============================] - 297s 41ms/step - loss: 0.1075\n",
      "Validation ROC-AUC improved (0.800401 --> 0.800723).  Saving model ...\n",
      "Epoch 4 completed. Train roc-auc: 0.8241673851708469, Val roc-auc: 0.8007232405560614\n",
      "Starting epoch 5\n",
      "Epoch 5/5\n",
      "7270/7270 [==============================] - 297s 41ms/step - loss: 0.1059\n",
      "Validation ROC-AUC improved (0.800723 --> 0.802880).  Saving model ...\n",
      "Epoch 5 completed. Train roc-auc: 0.8354671322329961, Val roc-auc: 0.8028799480090277\n",
      "Starting epoch 6\n",
      "Epoch 6/6\n",
      "7270/7270 [==============================] - 295s 41ms/step - loss: 0.1038\n",
      "Validation ROC-AUC improved (0.802880 --> 0.804206).  Saving model ...\n",
      "Epoch 6 completed. Train roc-auc: 0.8500102276951634, Val roc-auc: 0.8042058282383981\n",
      "Starting epoch 7\n",
      "Epoch 7/7\n",
      "7270/7270 [==============================] - 297s 41ms/step - loss: 0.1013\n",
      "No imporvement in Validation ROC-AUC. Current: 0.800783. Current best: 0.804206\n",
      "EarlyStopping counter: 1 out of 3\n",
      "Epoch 7 completed. Train roc-auc: 0.8686440805493696, Val roc-auc: 0.8007829662979375\n",
      "Starting epoch 8\n",
      "Epoch 8/8\n",
      "7270/7270 [==============================] - 298s 41ms/step - loss: 0.0981\n",
      "No imporvement in Validation ROC-AUC. Current: 0.796300. Current best: 0.804206\n",
      "EarlyStopping counter: 2 out of 3\n",
      "Epoch 8 completed. Train roc-auc: 0.886690365073565, Val roc-auc: 0.7963000566256508\n",
      "Starting epoch 9\n",
      "Epoch 9/9\n",
      "7270/7270 [==============================] - 297s 41ms/step - loss: 0.0954\n",
      "No imporvement in Validation ROC-AUC. Current: 0.791652. Current best: 0.804206\n",
      "EarlyStopping counter: 3 out of 3\n",
      "Epoch 9 completed. Train roc-auc: 0.89001976660625, Val roc-auc: 0.7916524842879937\n",
      "Early stopping reached. Stop training...\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(num_epochs):\n",
    "    print(f'Starting epoch {epoch+1}')\n",
    "    train_epoch(model, dataset_train, batch_size=train_batch_size, shuffle=True, cur_epoch=epoch, \n",
    "                steps_per_epoch=7270)\n",
    "    \n",
    "    val_roc_auc = eval_model(model, dataset_val, batch_size=val_batch_szie)\n",
    "    model.save_weights(os.path.join(path_to_checkpoints, f'epoch_{epoch+1}_val_{val_roc_auc:.3f}.hdf5'))\n",
    "    \n",
    "    es(val_roc_auc, model)\n",
    "    \n",
    "    train_roc_auc = eval_model(model, dataset_train, batch_size=val_batch_szie)\n",
    "    print(f'Epoch {epoch+1} completed. Train roc-auc: {train_roc_auc}, Val roc-auc: {val_roc_auc}')\n",
    "    \n",
    "    if es.early_stop:\n",
    "        print('Early stopping reached. Stop training...')\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Все готово, чтобы сделать предсказания для тестовой выборки. Нужно только подготовить данные в том же формате, как и для train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1063620</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1063621</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1063622</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1063623</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1063624</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    app_id  product\n",
       "0  1063620        0\n",
       "1  1063621        0\n",
       "2  1063622        1\n",
       "3  1063623        1\n",
       "4  1063624        2"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_frame = pd.read_csv('../../../test_target_contest.csv')\n",
    "test_frame.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../../../test_buckets/processed_chunk_000.pkl',\n",
       " '../../../test_buckets/processed_chunk_001.pkl',\n",
       " '../../../test_buckets/processed_chunk_002.pkl',\n",
       " '../../../test_buckets/processed_chunk_003.pkl',\n",
       " '../../../test_buckets/processed_chunk_004.pkl']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_to_test_dataset = '../../../test_buckets/'\n",
    "dir_with_test_datasets = os.listdir(path_to_test_dataset)\n",
    "dataset_test = sorted([os.path.join(path_to_test_dataset, x) for x in dir_with_test_datasets])\n",
    "\n",
    "dataset_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Отдельный вопрос, какую из построенных моделей использовать для того, чтобы делать предсказания на тест. Можно выбирать лучшую по early_stopping. В таком случае есть риск, что мы подгонимся под валидационную выборку, особенно если она не является очень репрезентативной, однако это самый базовый вариант (используем его). Можно делать разные версии ансамблирования, используя веса с разных эпох. Такой подход требует дополнительного кода (обязательно попробуйте его!). Наконец, можно выбирать такую модель, которая показывает хорошие результаты на валидации и в то же время, не слишком переучена под train выборку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best_checkpoint.pt.data-00000-of-00001\tepoch_4_val_0.801.hdf5\r\n",
      "best_checkpoint.pt.index\t\tepoch_5_val_0.803.hdf5\r\n",
      "checkpoint\t\t\t\tepoch_6_val_0.804.hdf5\r\n",
      "epoch_1_val_0.778.hdf5\t\t\tepoch_7_val_0.801.hdf5\r\n",
      "epoch_2_val_0.794.hdf5\t\t\tepoch_8_val_0.796.hdf5\r\n",
      "epoch_3_val_0.800.hdf5\t\t\tepoch_9_val_0.792.hdf5\r\n"
     ]
    }
   ],
   "source": [
    "! ls $path_to_checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f7fb4205090>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_weights(os.path.join(path_to_checkpoints, 'best_checkpoint.pt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = inference(model, dataset_test, batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1063655</td>\n",
       "      <td>0.012428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1063672</td>\n",
       "      <td>0.021236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1063694</td>\n",
       "      <td>0.013795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1063709</td>\n",
       "      <td>0.038398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1063715</td>\n",
       "      <td>0.021840</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    app_id     score\n",
       "0  1063655  0.012428\n",
       "1  1063672  0.021236\n",
       "2  1063694  0.013795\n",
       "3  1063709  0.038398\n",
       "4  1063715  0.021840"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds.to_csv('rnn_advanced_baseline_submission.csv', index=None) # ~ 0.760 на public test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}