Ludwig_x_aim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6YuWG703vf6O"
   },
   "source": [
    "# Aim x Ludwig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AVGL9YQcvpA9"
   },
   "source": [
    "<a href=\"https://user-images.githubusercontent.com/13848158/154338760-edfe1885-06f3-4e02-87fe-4b13a403516b.png\"><img src=\"https://user-images.githubusercontent.com/13848158/154338760-edfe1885-06f3-4e02-87fe-4b13a403516b.png\" title=\"source: imgur.com\" /></a>\n",
    "\n",
    "Ludwig is a toolbox built on top of TensorFlow that allows users to train and test deep learning models without the need to write code.\n",
    "\n",
    "All you need to provide is a dataset file containing your data, a list of columns to use as inputs, and a list of columns to use as outputs, Ludwig will do the rest. Simple commands can be used to train models both locally and in a distributed way, and to use them to predict new data.\n",
    "\n",
    "A programmatic API is also available in order to use Ludwig from your python code. A suite of visualization tools allows you to analyze models' training and test performance and to compare them.\n",
    "\n",
    "### Why use Aim\n",
    "\n",
    "`Aim` is an open-source, self-hosted ML experiment tracking tool. It's good at tracking lots (1000s) of training runs and it allows you to compare them with a performant and beautiful UI.\n",
    "\n",
    "You can use not only the great Aim UI but also its SDK to query your runs' metadata programmatically. That's especially useful for automations and additional analysis on a Jupyter Notebook.\n",
    "\n",
    "Aim's mission is to democratize AI dev tools.\n",
    "\n",
    "### Using Aim with Ludwig\n",
    "\n",
    "Aim allows for seamless integration into the inner backend of ludwig (training/inference pipelines) and allows you to track your experiments with in-depth granularity\n",
    "\n",
    "Training is easy:\n",
    "`ludwig train --dataset DATA_PATH --config_file CONFIG_PATH --aim`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ymCLdZvUwmX8"
   },
   "source": [
    "# Installation and demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "wdI2UgZbEePT"
   },
   "outputs": [],
   "source": [
    "# ! pip install git+http://github.com/uber/ludwig.git -qq\n",
    "# ! pip install ludwig[serve] -qq\n",
    "# ! pip install aim -qq "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/erik/anaconda3/envs/ludwig/bin/python\r\n"
     ]
    }
   ],
   "source": [
    "!which python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "62KTErutC2SN",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found GPU at: /device:GPU:0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-06-03 18:19:06.152093: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:06.152409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: \n",
      "pciBusID: 0000:01:00.0 name: GeForce GTX 1060 computeCapability: 6.1\n",
      "coreClock: 1.6705GHz coreCount: 10 deviceMemorySize: 5.94GiB deviceMemoryBandwidth: 178.99GiB/s\n",
      "2022-06-03 18:19:06.152456: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2022-06-03 18:19:06.152492: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10\n",
      "2022-06-03 18:19:06.152510: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10\n",
      "2022-06-03 18:19:06.152527: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10\n",
      "2022-06-03 18:19:06.152542: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10\n",
      "2022-06-03 18:19:06.152558: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10\n",
      "2022-06-03 18:19:06.152575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7\n",
      "2022-06-03 18:19:06.152638: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:06.152922: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:06.153160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0\n",
      "2022-06-03 18:19:06.153200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2022-06-03 18:19:06.153206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 \n",
      "2022-06-03 18:19:06.153211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N \n",
      "2022-06-03 18:19:06.153299: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:06.153580: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:06.153841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/device:GPU:0 with 4401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "device_name = tf.test.gpu_device_name()\n",
    "if device_name != '/device:GPU:0':\n",
    "  raise SystemError('GPU device not found')\n",
    "print('Found GPU at: {}'.format(device_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/device:CPU:0', '/device:XLA_CPU:0', '/device:XLA_GPU:0', '/device:GPU:0']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-06-03 18:19:07.098395: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:07.099397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: \n",
      "pciBusID: 0000:01:00.0 name: GeForce GTX 1060 computeCapability: 6.1\n",
      "coreClock: 1.6705GHz coreCount: 10 deviceMemorySize: 5.94GiB deviceMemoryBandwidth: 178.99GiB/s\n",
      "2022-06-03 18:19:07.099532: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2022-06-03 18:19:07.099663: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10\n",
      "2022-06-03 18:19:07.099774: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10\n",
      "2022-06-03 18:19:07.099835: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10\n",
      "2022-06-03 18:19:07.099920: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10\n",
      "2022-06-03 18:19:07.100011: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10\n",
      "2022-06-03 18:19:07.100098: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7\n",
      "2022-06-03 18:19:07.100323: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:07.101492: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:07.102194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0\n",
      "2022-06-03 18:19:07.102253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2022-06-03 18:19:07.102271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 \n",
      "2022-06-03 18:19:07.102285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N \n",
      "2022-06-03 18:19:07.102530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:07.103839: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-06-03 18:19:07.104532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/device:GPU:0 with 4401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "def get_available_devices():\n",
    "    local_device_protos = device_lib.list_local_devices()\n",
    "    return [x.name for x in local_device_protos]\n",
    "print(get_available_devices()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G8HIcSwSwtLs"
   },
   "source": [
    "### Text Classification\n",
    "\n",
    "Text classification also known as text tagging or text categorization is the process of categorizing text into organized groups. By using Natural Language Processing (NLP), text classifiers can automatically analyze text and then assign a set of pre-defined tags or categories based on its content.\n",
    "\n",
    "Unstructured text is everywhere, such as emails, chat conversations, websites, and social media but it’s hard to extract value from this data unless it’s organized in a certain way. Doing so used to be a difficult and expensive process since it required spending time and resources to manually sort the data or creating handcrafted rules that are difficult to maintain. \n",
    "\n",
    "Let's build a text classifier using ludwig.\n",
    "\n",
    "### Kaggle's AGNews Dataset\n",
    "AG is a collection of more than 1 million news articles. News articles have been gathered from more than 2000  news sources by ComeToMyHead in more than 1 year of activity. ComeToMyHead is an academic news search engine which has been running since July, 2004. The dataset is provided by the academic comunity for research purposes in data mining (clustering, classification, etc), information retrieval (ranking, search, etc), xml, data compression, data streaming, and any other non-commercial activity. For more information, please refer to the link http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html .\n",
    "\n",
    "The articles are divided into 4 classes:\n",
    "```\n",
    "World\n",
    "Sports\n",
    "Business\n",
    "Sci/Tech\n",
    "```\n",
    "Let's download the dataset.\n",
    "\n",
    "Important to note that we will use only a sample of dataset to train (for showing off the tracking functional), however the training can be complete on the entire dataset as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "g4gOi3oBLKmA"
   },
   "outputs": [],
   "source": [
    "import aim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "jC1PZ7AA1RjC"
   },
   "outputs": [],
   "source": [
    "id_to_label = {\n",
    "   1: 'World', 2: 'Sports', 3: 'Business', 4: 'Sci/Tech'\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QNoybfJ-o5-y"
   },
   "source": [
    "## Experiment Tracking\n",
    "\n",
    "\n",
    "### Train \n",
    "This command lets you train a model from your data. You can call it with:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "4-jIkcuk0K17",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumExpr defaulting to 8 threads.\n",
      "2022-06-03 18:19:11.469307: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "ray.init() failed: Could not find any running Ray instance. Please specify the one to connect to by setting `--address` flag or `RAY_ADDRESS` environment variable.\n",
      "███████████████████████\n",
      "█ █ █ █  ▜█ █ █ █ █   █\n",
      "█ █ █ █ █ █ █ █ █ █ ███\n",
      "█ █   █ █ █ █ █ █ █ ▌ █\n",
      "█ █████ █ █ █ █ █ █ █ █\n",
      "█     █  ▟█     █ █   █\n",
      "███████████████████████\n",
      "ludwig v0.5rc2 - Train\n",
      "\n",
      "\n",
      "╒════════════════════════╕\n",
      "│ EXPERIMENT DESCRIPTION │\n",
      "╘════════════════════════╛\n",
      "\n",
      "╒══════════════════╤═════════════════════════════════════════════════════════════════════════════════╕\n",
      "│ Experiment name  │ Classification                                                                  │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ Model name       │ run                                                                             │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ Output directory │ /home/erik/Documents/UCPH/aim_projs/aim-ludwig-demo/results/Classification_run  │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ ludwig_version   │ '0.5rc2'                                                                        │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ command          │ ('/home/erik/anaconda3/envs/ludwig/bin/ludwig train --dataset final_train.csv ' │\n",
      "│                  │  '--config train_conf.yaml -g 0 --aim --experiment_name Classification')        │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ commit_hash      │ '56ad78d74a9f'                                                                  │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ random_seed      │ 42                                                                              │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ dataset          │ 'final_train.csv'                                                               │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ data_format      │ 'csv'                                                                           │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ torch_version    │ '1.11.0+cu102'                                                                  │\n",
      "├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┤\n",
      "│ compute          │ {'gpu_type': 'GeForce GTX 1060', 'gpus_per_node': 1, 'num_nodes': 1}            │\n",
      "╘══════════════════╧═════════════════════════════════════════════════════════════════════════════════╛\n",
      "\n",
      "╒═══════════════╕\n",
      "│ LUDWIG CONFIG │\n",
      "╘═══════════════╛\n",
      "\n",
      "{   'combiner': {'type': 'concat'},\n",
      "    'input_features': [   {   'column': 'Title',\n",
      "                              'encoder': 'parallel_cnn',\n",
      "                              'name': 'Title',\n",
      "                              'proc_column': 'Title_mZFLky',\n",
      "                              'tied': None,\n",
      "                              'type': 'text'},\n",
      "                          {   'column': 'Description',\n",
      "                              'encoder': 'parallel_cnn',\n",
      "                              'name': 'Description',\n",
      "                              'proc_column': 'Description_mZFLky',\n",
      "                              'tied': None,\n",
      "                              'type': 'text'}],\n",
      "    'output_features': [   {   'column': 'ClassIndex',\n",
      "                               'dependencies': [],\n",
      "                               'loss': {   'class_similarities_temperature': 0,\n",
      "                                           'class_weights': 1,\n",
      "                                           'confidence_penalty': 0,\n",
      "                                           'robust_lambda': 0,\n",
      "                                           'type': 'softmax_cross_entropy',\n",
      "                                           'weight': 1},\n",
      "                               'name': 'ClassIndex',\n",
      "                               'preprocessing': {   'missing_value_strategy': 'drop_row'},\n",
      "                               'proc_column': 'ClassIndex_mZFLky',\n",
      "                               'reduce_dependencies': 'sum',\n",
      "                               'reduce_input': 'sum',\n",
      "                               'top_k': 3,\n",
      "                               'type': 'category'}],\n",
      "    'preprocessing': {   'audio': {   'audio_feature': {'type': 'raw'},\n",
      "                                      'audio_file_length_limit_in_s': 7.5,\n",
      "                                      'in_memory': True,\n",
      "                                      'missing_value_strategy': 'backfill',\n",
      "                                      'norm': None,\n",
      "                                      'padding_value': 0},\n",
      "                         'bag': {   'fill_value': '<UNK>',\n",
      "                                    'lowercase': False,\n",
      "                                    'missing_value_strategy': 'fill_with_const',\n",
      "                                    'most_common': 10000,\n",
      "                                    'tokenizer': 'space'},\n",
      "                         'binary': {   'missing_value_strategy': 'fill_with_false'},\n",
      "                         'category': {   'fill_value': '<UNK>',\n",
      "                                         'lowercase': False,\n",
      "                                         'missing_value_strategy': 'fill_with_const',\n",
      "                                         'most_common': 10000},\n",
      "                         'date': {   'datetime_format': None,\n",
      "                                     'fill_value': '',\n",
      "                                     'missing_value_strategy': 'fill_with_const'},\n",
      "                         'force_split': False,\n",
      "                         'h3': {   'fill_value': 576495936675512319,\n",
      "                                   'missing_value_strategy': 'fill_with_const'},\n",
      "                         'image': {   'in_memory': True,\n",
      "                                      'infer_image_dimensions': True,\n",
      "                                      'infer_image_max_height': 256,\n",
      "                                      'infer_image_max_width': 256,\n",
      "                                      'infer_image_num_channels': True,\n",
      "                                      'infer_image_sample_size': 100,\n",
      "                                      'missing_value_strategy': 'backfill',\n",
      "                                      'num_processes': 1,\n",
      "                                      'resize_method': 'interpolate',\n",
      "                                      'scaling': 'pixel_normalization'},\n",
      "                         'number': {   'fill_value': 0,\n",
      "                                       'missing_value_strategy': 'fill_with_const',\n",
      "                                       'normalization': None},\n",
      "                         'oversample_minority': None,\n",
      "                         'sample_ratio': 1.0,\n",
      "                         'sequence': {   'fill_value': '<UNK>',\n",
      "                                         'lowercase': False,\n",
      "                                         'max_sequence_length': 256,\n",
      "                                         'missing_value_strategy': 'fill_with_const',\n",
      "                                         'most_common': 20000,\n",
      "                                         'padding': 'right',\n",
      "                                         'padding_symbol': '<PAD>',\n",
      "                                         'tokenizer': 'space',\n",
      "                                         'unknown_symbol': '<UNK>',\n",
      "                                         'vocab_file': None},\n",
      "                         'set': {   'fill_value': '<UNK>',\n",
      "                                    'lowercase': False,\n",
      "                                    'missing_value_strategy': 'fill_with_const',\n",
      "                                    'most_common': 10000,\n",
      "                                    'tokenizer': 'space'},\n",
      "                         'split_probabilities': (0.7, 0.1, 0.2),\n",
      "                         'stratify': None,\n",
      "                         'text': {   'fill_value': '<UNK>',\n",
      "                                     'lowercase': True,\n",
      "                                     'max_sequence_length': 256,\n",
      "                                     'missing_value_strategy': 'fill_with_const',\n",
      "                                     'most_common': 20000,\n",
      "                                     'padding': 'right',\n",
      "                                     'padding_symbol': '<PAD>',\n",
      "                                     'pretrained_model_name_or_path': None,\n",
      "                                     'tokenizer': 'space_punct',\n",
      "                                     'unknown_symbol': '<UNK>',\n",
      "                                     'vocab_file': None},\n",
      "                         'timeseries': {   'fill_value': '',\n",
      "                                           'missing_value_strategy': 'fill_with_const',\n",
      "                                           'padding': 'right',\n",
      "                                           'padding_value': 0,\n",
      "                                           'timeseries_length_limit': 256,\n",
      "                                           'tokenizer': 'space'},\n",
      "                         'undersample_majority': None,\n",
      "                         'vector': {   'fill_value': '',\n",
      "                                       'missing_value_strategy': 'fill_with_const'}},\n",
      "    'trainer': {   'batch_size': 128,\n",
      "                   'decay': False,\n",
      "                   'decay_rate': 0.96,\n",
      "                   'decay_steps': 10000,\n",
      "                   'early_stop': 1000,\n",
      "                   'epochs': 20,\n",
      "                   'eval_batch_size': None,\n",
      "                   'gradient_clipping': None,\n",
      "                   'increase_batch_size_on_plateau': 0,\n",
      "                   'increase_batch_size_on_plateau_max': 512,\n",
      "                   'increase_batch_size_on_plateau_patience': 5,\n",
      "                   'increase_batch_size_on_plateau_rate': 2,\n",
      "                   'learning_rate': 0.001,\n",
      "                   'learning_rate_warmup_epochs': 1,\n",
      "                   'optimizer': {   'betas': (0.9, 0.999),\n",
      "                                    'eps': 1e-08,\n",
      "                                    'type': 'adam'},\n",
      "                   'reduce_learning_rate_on_plateau': 0,\n",
      "                   'reduce_learning_rate_on_plateau_patience': 5,\n",
      "                   'reduce_learning_rate_on_plateau_rate': 0.5,\n",
      "                   'regularization_lambda': 0,\n",
      "                   'regularization_type': 'l2',\n",
      "                   'staircase': False,\n",
      "                   'steps_per_checkpoint': 0,\n",
      "                   'validation_field': 'combined',\n",
      "                   'validation_metric': 'loss'}}\n",
      "\n",
      "╒═══════════════╕\n",
      "│ PREPROCESSING │\n",
      "╘═══════════════╛\n",
      "\n",
      "Found cached dataset and meta.json with the same filename of the dataset, but checksum don't match, if saving of processed input is not skipped they will be overridden\n",
      "Using full raw dataset, no hdf5 and json file with the same name have been found\n",
      "Building dataset (it may take a while)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building dataset: DONE\n",
      "Writing preprocessed training set cache\n",
      "Writing preprocessed test set cache\n",
      "Writing preprocessed validation set cache\n",
      "Writing train set metadata\n",
      "\n",
      "Dataset sizes:\n",
      "╒════════════╤════════╕\n",
      "│ Dataset    │   Size │\n",
      "╞════════════╪════════╡\n",
      "│ Training   │    209 │\n",
      "├────────────┼────────┤\n",
      "│ Validation │     28 │\n",
      "├────────────┼────────┤\n",
      "│ Test       │     63 │\n",
      "╘════════════╧════════╛\n",
      "aim.on_train_init() called...\n",
      "base config  {'input_features': [{'name': 'Title', 'type': 'text'}, {'name': 'Description', 'type': 'text'}], 'output_features': [{'name': 'ClassIndex', 'type': 'category'}], 'trainer': {'epochs': 20}}\n",
      "experiment directory  /home/erik/Documents/UCPH/aim_projs/aim-ludwig-demo/results/Classification_run\n",
      "experiment name  Classification\n",
      "model name  run\n",
      "output directory  /home/erik/Documents/UCPH/aim_projs/aim-ludwig-demo/results/Classification_run\n",
      "{'name': 'run', 'dir': '/home/erik/Documents/UCPH/aim_projs/aim-ludwig-demo/results/Classification_run'}\n",
      "\n",
      "╒═══════╕\n",
      "│ MODEL │\n",
      "╘═══════╛\n",
      "\n",
      "Warnings and other logs:\n",
      "/home/erik/anaconda3/envs/ludwig/lib/python3.8/site-packages/torch/nn/modules/conv.py:298: UserWarning: Using padding='same' with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at  ../aten/src/ATen/native/Convolution.cpp:744.)\n",
      "  return F.conv1d(input, weight, bias, self.stride,\n",
      "\n",
      "╒══════════╕\n",
      "│ TRAINING │\n",
      "╘══════════╛\n",
      "\n",
      "aim.on_train_start() called...\n",
      "{'trainer': {'epochs': 20, 'optimizer': {'type': 'adam', 'betas': (0.9, 0.999), 'eps': 1e-08}, 'regularization_lambda': 0, 'regularization_type': 'l2', 'learning_rate': 0.001, 'batch_size': 128, 'eval_batch_size': None, 'early_stop': 1000, 'steps_per_checkpoint': 0, 'reduce_learning_rate_on_plateau': 0, 'reduce_learning_rate_on_plateau_patience': 5, 'reduce_learning_rate_on_plateau_rate': 0.5, 'increase_batch_size_on_plateau': 0, 'increase_batch_size_on_plateau_patience': 5, 'increase_batch_size_on_plateau_rate': 2, 'increase_batch_size_on_plateau_max': 512, 'decay': False, 'decay_steps': 10000, 'decay_rate': 0.96, 'staircase': False, 'gradient_clipping': None, 'validation_field': 'combined', 'validation_metric': 'loss', 'learning_rate_warmup_epochs': 1}, 'preprocessing': {'force_split': False, 'split_probabilities': (0.7, 0.1, 0.2), 'stratify': None, 'undersample_majority': None, 'oversample_minority': None, 'sample_ratio': 1.0, 'text': {'tokenizer': 'space_punct', 'pretrained_model_name_or_path': None, 'vocab_file': None, 'max_sequence_length': 256, 'most_common': 20000, 'padding_symbol': '<PAD>', 'unknown_symbol': '<UNK>', 'padding': 'right', 'lowercase': True, 'missing_value_strategy': 'fill_with_const', 'fill_value': '<UNK>'}, 'category': {'most_common': 10000, 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'fill_value': '<UNK>'}, 'set': {'tokenizer': 'space', 'most_common': 10000, 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'fill_value': '<UNK>'}, 'bag': {'tokenizer': 'space', 'most_common': 10000, 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'fill_value': '<UNK>'}, 'binary': {'missing_value_strategy': 'fill_with_false'}, 'number': {'missing_value_strategy': 'fill_with_const', 'fill_value': 0, 'normalization': None}, 'sequence': {'max_sequence_length': 256, 'most_common': 20000, 'padding_symbol': '<PAD>', 'unknown_symbol': '<UNK>', 'padding': 'right', 'tokenizer': 'space', 'lowercase': False, 'vocab_file': None, 'missing_value_strategy': 'fill_with_const', 'fill_value': '<UNK>'}, 'timeseries': {'timeseries_length_limit': 256, 'padding_value': 0, 'padding': 'right', 'tokenizer': 'space', 'missing_value_strategy': 'fill_with_const', 'fill_value': ''}, 'image': {'missing_value_strategy': 'backfill', 'in_memory': True, 'resize_method': 'interpolate', 'scaling': 'pixel_normalization', 'num_processes': 1, 'infer_image_num_channels': True, 'infer_image_dimensions': True, 'infer_image_max_height': 256, 'infer_image_max_width': 256, 'infer_image_sample_size': 100}, 'audio': {'audio_file_length_limit_in_s': 7.5, 'missing_value_strategy': 'backfill', 'in_memory': True, 'padding_value': 0, 'norm': None, 'audio_feature': {'type': 'raw'}}, 'h3': {'missing_value_strategy': 'fill_with_const', 'fill_value': 576495936675512319}, 'date': {'missing_value_strategy': 'fill_with_const', 'fill_value': '', 'datetime_format': None}, 'vector': {'missing_value_strategy': 'fill_with_const', 'fill_value': ''}}, 'combiner': {'type': 'concat'}}\n",
      "Note: steps_per_checkpoint (was 2) is now set to the number of steps per epoch: 2.\n",
      "\n",
      "Training for 40 step(s), approximately 20 epoch(s).\n",
      "Starting with step 0, epoch: 0\n",
      "Training:   2%|▊                                 | 1/40 [00:00<00:04,  9.65it/s]\n",
      "Running evaluation for step: 2, epoch: 0\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 34.14it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 111.64it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 60.69it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.6514 │     0.7416 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.7260 │     0.7143 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.6978 │     0.7460 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.6514 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.7260 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.6978 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 0, 'steps': 2, 'tune_checkpoint_num': 0, 'last_improvement_steps': 2, 'learning_rate': 0.001, 'best_valid_metric': 0.7260150909423828, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.6513936519622803, 'train_metrics.ClassIndex.accuracy': 0.7416267991065979, 'train_metrics.combined.loss': 0.6513936519622803, 'validation_metrics.ClassIndex.loss': 0.7260150909423828, 'validation_metrics.ClassIndex.accuracy': 0.7142857313156128, 'validation_metrics.combined.loss': 0.7260150909423828, 'test_metrics.ClassIndex.loss': 0.697843611240387, 'test_metrics.ClassIndex.accuracy': 0.7460317611694336, 'test_metrics.combined.loss': 0.697843611240387}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:   8%|██▌                               | 3/40 [00:00<00:10,  3.56it/s]\n",
      "Running evaluation for step: 4, epoch: 1\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 33.51it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 104.59it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 61.71it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.5078 │     0.7416 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.6625 │     0.7143 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.6084 │     0.7460 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.5078 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.6625 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.6084 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 1, 'steps': 4, 'tune_checkpoint_num': 0, 'last_improvement_steps': 4, 'learning_rate': 0.001, 'best_valid_metric': 0.662549614906311, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.5077632069587708, 'train_metrics.ClassIndex.accuracy': 0.7416267991065979, 'train_metrics.combined.loss': 0.5077632069587708, 'validation_metrics.ClassIndex.loss': 0.662549614906311, 'validation_metrics.ClassIndex.accuracy': 0.7142857313156128, 'validation_metrics.combined.loss': 0.662549614906311, 'test_metrics.ClassIndex.loss': 0.6084316372871399, 'test_metrics.ClassIndex.accuracy': 0.7460317611694336, 'test_metrics.combined.loss': 0.6084316372871399}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training:  12%|████▎                             | 5/40 [00:01<00:09,  3.62it/s]\n",
      "Running evaluation for step: 6, epoch: 2\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 34.81it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 95.87it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 59.33it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.4670 │     0.7416 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5670 │     0.7143 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.5450 │     0.7460 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.4670 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5670 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.5450 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 2, 'steps': 6, 'tune_checkpoint_num': 0, 'last_improvement_steps': 6, 'learning_rate': 0.001, 'best_valid_metric': 0.5669645667076111, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.46695542335510254, 'train_metrics.ClassIndex.accuracy': 0.7416267991065979, 'train_metrics.combined.loss': 0.46695542335510254, 'validation_metrics.ClassIndex.loss': 0.5669645667076111, 'validation_metrics.ClassIndex.accuracy': 0.7142857313156128, 'validation_metrics.combined.loss': 0.5669645667076111, 'test_metrics.ClassIndex.loss': 0.5449604392051697, 'test_metrics.ClassIndex.accuracy': 0.7460317611694336, 'test_metrics.combined.loss': 0.5449604392051697}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  18%|█████▉                            | 7/40 [00:01<00:09,  3.46it/s]\n",
      "Running evaluation for step: 8, epoch: 3\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 39.51it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 105.48it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 63.44it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.4177 │     0.7416 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5442 │     0.7143 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.5287 │     0.7460 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.4177 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5442 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.5287 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 3, 'steps': 8, 'tune_checkpoint_num': 0, 'last_improvement_steps': 8, 'learning_rate': 0.001, 'best_valid_metric': 0.5441716313362122, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.41765373945236206, 'train_metrics.ClassIndex.accuracy': 0.7416267991065979, 'train_metrics.combined.loss': 0.41765373945236206, 'validation_metrics.ClassIndex.loss': 0.5441716313362122, 'validation_metrics.ClassIndex.accuracy': 0.7142857313156128, 'validation_metrics.combined.loss': 0.5441716313362122, 'test_metrics.ClassIndex.loss': 0.5287315845489502, 'test_metrics.ClassIndex.accuracy': 0.7460317611694336, 'test_metrics.combined.loss': 0.5287315845489502}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  22%|███████▋                          | 9/40 [00:02<00:08,  3.48it/s]\n",
      "Running evaluation for step: 10, epoch: 4\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 35.92it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 82.85it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 61.75it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.3344 │     0.7416 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5566 │     0.7143 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.5365 │     0.7460 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.3344 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5566 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.5365 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 2 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 4, 'steps': 10, 'tune_checkpoint_num': 0, 'last_improvement_steps': 8, 'learning_rate': 0.001, 'best_valid_metric': 0.5441716313362122, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.3344016671180725, 'train_metrics.ClassIndex.accuracy': 0.7416267991065979, 'train_metrics.combined.loss': 0.3344016671180725, 'validation_metrics.ClassIndex.loss': 0.5565897226333618, 'validation_metrics.ClassIndex.accuracy': 0.7142857313156128, 'validation_metrics.combined.loss': 0.5565897226333618, 'test_metrics.ClassIndex.loss': 0.5364888310432434, 'test_metrics.ClassIndex.accuracy': 0.7460317611694336, 'test_metrics.combined.loss': 0.5364888310432434}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  28%|█████████                        | 11/40 [00:03<00:08,  3.59it/s]\n",
      "Running evaluation for step: 12, epoch: 5\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 34.70it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 100.05it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 53.79it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.2478 │     0.9378 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5023 │     0.7500 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.5028 │     0.7460 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.2478 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5023 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.5028 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 5, 'steps': 12, 'tune_checkpoint_num': 0, 'last_improvement_steps': 12, 'learning_rate': 0.001, 'best_valid_metric': 0.5023441910743713, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.24780306220054626, 'train_metrics.ClassIndex.accuracy': 0.9377990365028381, 'train_metrics.combined.loss': 0.24780306220054626, 'validation_metrics.ClassIndex.loss': 0.5023441910743713, 'validation_metrics.ClassIndex.accuracy': 0.75, 'validation_metrics.combined.loss': 0.5023441910743713, 'test_metrics.ClassIndex.loss': 0.5028085112571716, 'test_metrics.ClassIndex.accuracy': 0.7460317611694336, 'test_metrics.combined.loss': 0.5028085112571716}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training:  32%|██████████▋                      | 13/40 [00:03<00:07,  3.72it/s]\n",
      "Running evaluation for step: 14, epoch: 6\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 36.66it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 124.55it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 62.71it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.1965 │     0.9904 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4860 │     0.6429 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.5091 │     0.7143 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.1965 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4860 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.5091 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 6, 'steps': 14, 'tune_checkpoint_num': 0, 'last_improvement_steps': 14, 'learning_rate': 0.001, 'best_valid_metric': 0.4860207736492157, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.1964854747056961, 'train_metrics.ClassIndex.accuracy': 0.9904305934906006, 'train_metrics.combined.loss': 0.1964854747056961, 'validation_metrics.ClassIndex.loss': 0.4860207736492157, 'validation_metrics.ClassIndex.accuracy': 0.6428571343421936, 'validation_metrics.combined.loss': 0.4860207736492157, 'test_metrics.ClassIndex.loss': 0.5090556740760803, 'test_metrics.ClassIndex.accuracy': 0.7142857313156128, 'test_metrics.combined.loss': 0.5090556740760803}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  38%|████████████▍                    | 15/40 [00:04<00:06,  3.63it/s]\n",
      "Running evaluation for step: 16, epoch: 7\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 38.00it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 114.73it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 55.19it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0802 │     0.9904 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4225 │     0.7500 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4588 │     0.7937 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0802 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4225 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4588 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 7, 'steps': 16, 'tune_checkpoint_num': 0, 'last_improvement_steps': 16, 'learning_rate': 0.001, 'best_valid_metric': 0.422544926404953, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.08017656207084656, 'train_metrics.ClassIndex.accuracy': 0.9904305934906006, 'train_metrics.combined.loss': 0.08017656207084656, 'validation_metrics.ClassIndex.loss': 0.422544926404953, 'validation_metrics.ClassIndex.accuracy': 0.75, 'validation_metrics.combined.loss': 0.422544926404953, 'test_metrics.ClassIndex.loss': 0.45881181955337524, 'test_metrics.ClassIndex.accuracy': 0.7936508059501648, 'test_metrics.combined.loss': 0.45881181955337524}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  42%|██████████████                   | 17/40 [00:04<00:06,  3.61it/s]\n",
      "Running evaluation for step: 18, epoch: 8\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 34.94it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 92.21it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 46.53it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0325 │     0.9904 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4597 │     0.7500 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.5101 │     0.7619 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0325 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4597 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.5101 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 2 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 8, 'steps': 18, 'tune_checkpoint_num': 0, 'last_improvement_steps': 16, 'learning_rate': 0.001, 'best_valid_metric': 0.422544926404953, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.032489974051713943, 'train_metrics.ClassIndex.accuracy': 0.9904305934906006, 'train_metrics.combined.loss': 0.032489974051713943, 'validation_metrics.ClassIndex.loss': 0.45972874760627747, 'validation_metrics.ClassIndex.accuracy': 0.75, 'validation_metrics.combined.loss': 0.45972874760627747, 'test_metrics.ClassIndex.loss': 0.5100890398025513, 'test_metrics.ClassIndex.accuracy': 0.761904776096344, 'test_metrics.combined.loss': 0.5100890398025513}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  48%|███████████████▋                 | 19/40 [00:05<00:05,  3.65it/s]\n",
      "Running evaluation for step: 20, epoch: 9\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 33.19it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 120.66it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 56.06it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0072 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.3876 │     0.7857 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4249 │     0.8095 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0072 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.3876 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4249 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 9, 'steps': 20, 'tune_checkpoint_num': 0, 'last_improvement_steps': 20, 'learning_rate': 0.001, 'best_valid_metric': 0.3875730335712433, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.0071991137228906155, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 0.0071991137228906155, 'validation_metrics.ClassIndex.loss': 0.3875730335712433, 'validation_metrics.ClassIndex.accuracy': 0.7857142686843872, 'validation_metrics.combined.loss': 0.3875730335712433, 'test_metrics.ClassIndex.loss': 0.4248819649219513, 'test_metrics.ClassIndex.accuracy': 0.8095238208770752, 'test_metrics.combined.loss': 0.4248819649219513}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training:  52%|█████████████████▎               | 21/40 [00:05<00:05,  3.55it/s]\n",
      "Running evaluation for step: 22, epoch: 10\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 38.10it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 103.80it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 60.87it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0026 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.3737 │     0.8214 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.3858 │     0.8095 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0026 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.3737 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.3858 │\n",
      "╘════════════╧════════╛\n",
      "Validation loss on combined improved, model saved.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 10, 'steps': 22, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.002632095944136381, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 0.002632095944136381, 'validation_metrics.ClassIndex.loss': 0.37374797463417053, 'validation_metrics.ClassIndex.accuracy': 0.8214285969734192, 'validation_metrics.combined.loss': 0.37374797463417053, 'test_metrics.ClassIndex.loss': 0.3858174681663513, 'test_metrics.ClassIndex.accuracy': 0.8095238208770752, 'test_metrics.combined.loss': 0.3858174681663513}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  57%|██████████████████▉              | 23/40 [00:06<00:04,  3.55it/s]\n",
      "Running evaluation for step: 24, epoch: 11\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 37.52it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 96.92it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 58.51it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0011 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.3872 │     0.8571 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.3807 │     0.8095 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0011 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.3872 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.3807 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 2 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 11, 'steps': 24, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.0010602909605950117, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 0.0010602909605950117, 'validation_metrics.ClassIndex.loss': 0.3872045576572418, 'validation_metrics.ClassIndex.accuracy': 0.8571428656578064, 'validation_metrics.combined.loss': 0.3872045576572418, 'test_metrics.ClassIndex.loss': 0.38070347905158997, 'test_metrics.ClassIndex.accuracy': 0.8095238208770752, 'test_metrics.combined.loss': 0.38070347905158997}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  62%|████████████████████▋            | 25/40 [00:06<00:03,  3.81it/s]\n",
      "Running evaluation for step: 26, epoch: 12\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 38.18it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 110.22it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 59.75it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0004 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4039 │     0.8571 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.3837 │     0.7937 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0004 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4039 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.3837 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 4 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 12, 'steps': 26, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.0004003221110906452, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 0.0004003221110906452, 'validation_metrics.ClassIndex.loss': 0.40388011932373047, 'validation_metrics.ClassIndex.accuracy': 0.8571428656578064, 'validation_metrics.combined.loss': 0.40388011932373047, 'test_metrics.ClassIndex.loss': 0.3837473392486572, 'test_metrics.ClassIndex.accuracy': 0.7936508059501648, 'test_metrics.combined.loss': 0.3837473392486572}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  68%|██████████████████████▎          | 27/40 [00:07<00:03,  4.07it/s]\n",
      "Running evaluation for step: 28, epoch: 13\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 37.34it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 100.40it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 54.58it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0001 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4213 │     0.8571 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.3899 │     0.8254 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0001 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4213 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.3899 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 6 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 13, 'steps': 28, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 0.0001411950506735593, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 0.0001411950506735593, 'validation_metrics.ClassIndex.loss': 0.42134714126586914, 'validation_metrics.ClassIndex.accuracy': 0.8571428656578064, 'validation_metrics.combined.loss': 0.42134714126586914, 'test_metrics.ClassIndex.loss': 0.3899196684360504, 'test_metrics.ClassIndex.accuracy': 0.8253968358039856, 'test_metrics.combined.loss': 0.3899196684360504}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training:  72%|███████████████████████▉         | 29/40 [00:07<00:02,  4.27it/s]\n",
      "Running evaluation for step: 30, epoch: 14\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 38.72it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 81.70it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 61.66it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0001 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4413 │     0.8571 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4002 │     0.8095 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0001 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4413 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4002 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 8 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 14, 'steps': 30, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 5.1510804041754454e-05, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 5.1510804041754454e-05, 'validation_metrics.ClassIndex.loss': 0.4412993788719177, 'validation_metrics.ClassIndex.accuracy': 0.8571428656578064, 'validation_metrics.combined.loss': 0.4412993788719177, 'test_metrics.ClassIndex.loss': 0.40022823214530945, 'test_metrics.ClassIndex.accuracy': 0.8095238208770752, 'test_metrics.combined.loss': 0.40022823214530945}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  78%|█████████████████████████▌       | 31/40 [00:08<00:02,  4.49it/s]\n",
      "Running evaluation for step: 32, epoch: 15\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 32.11it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 110.02it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 60.10it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0000 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4647 │     0.8214 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4156 │     0.8095 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0000 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4647 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4156 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 10 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 15, 'steps': 32, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 2.0669032892328687e-05, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 2.0669032892328687e-05, 'validation_metrics.ClassIndex.loss': 0.4647095799446106, 'validation_metrics.ClassIndex.accuracy': 0.8214285969734192, 'validation_metrics.combined.loss': 0.4647095799446106, 'test_metrics.ClassIndex.loss': 0.41558510065078735, 'test_metrics.ClassIndex.accuracy': 0.8095238208770752, 'test_metrics.combined.loss': 0.41558510065078735}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  82%|███████████████████████████▏     | 33/40 [00:08<00:01,  4.49it/s]\n",
      "Running evaluation for step: 34, epoch: 16\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 37.69it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 113.23it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 55.90it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0000 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.4906 │     0.8214 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4346 │     0.8413 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0000 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.4906 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4346 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 12 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 16, 'steps': 34, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 9.521041647531092e-06, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 9.521041647531092e-06, 'validation_metrics.ClassIndex.loss': 0.4906318783760071, 'validation_metrics.ClassIndex.accuracy': 0.8214285969734192, 'validation_metrics.combined.loss': 0.4906318783760071, 'test_metrics.ClassIndex.loss': 0.4346058666706085, 'test_metrics.ClassIndex.accuracy': 0.841269850730896, 'test_metrics.combined.loss': 0.4346058666706085}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  88%|████████████████████████████▉    | 35/40 [00:08<00:01,  4.56it/s]\n",
      "Running evaluation for step: 36, epoch: 17\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 37.93it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 91.98it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 61.19it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0000 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5175 │     0.8214 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4555 │     0.8254 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0000 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5175 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4555 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 14 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 17, 'steps': 36, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 5.043408691562945e-06, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 5.043408691562945e-06, 'validation_metrics.ClassIndex.loss': 0.5175368189811707, 'validation_metrics.ClassIndex.accuracy': 0.8214285969734192, 'validation_metrics.combined.loss': 0.5175368189811707, 'test_metrics.ClassIndex.loss': 0.4555448591709137, 'test_metrics.ClassIndex.accuracy': 0.8253968358039856, 'test_metrics.combined.loss': 0.4555448591709137}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training:  92%|██████████████████████████████▌  | 37/40 [00:09<00:00,  4.64it/s]\n",
      "Running evaluation for step: 38, epoch: 18\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 35.88it/s]\n",
      "Evaluation vali : 100%|███████████████████████████| 1/1 [00:00<00:00, 66.57it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 59.98it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0000 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5443 │     0.8571 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4768 │     0.8413 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0000 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5443 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4768 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 16 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 18, 'steps': 38, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 3.0138639885990415e-06, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 3.0138639885990415e-06, 'validation_metrics.ClassIndex.loss': 0.5443124175071716, 'validation_metrics.ClassIndex.accuracy': 0.8571428656578064, 'validation_metrics.combined.loss': 0.5443124175071716, 'test_metrics.ClassIndex.loss': 0.4768122136592865, 'test_metrics.ClassIndex.accuracy': 0.841269850730896, 'test_metrics.combined.loss': 0.4768122136592865}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training:  98%|████████████████████████████████▏| 39/40 [00:09<00:00,  4.64it/s]\n",
      "Running evaluation for step: 40, epoch: 19\n",
      "Evaluation train: 100%|███████████████████████████| 2/2 [00:00<00:00, 38.26it/s]\n",
      "Evaluation vali : 100%|██████████████████████████| 1/1 [00:00<00:00, 111.09it/s]\n",
      "Evaluation test : 100%|███████████████████████████| 1/1 [00:00<00:00, 58.62it/s]\n",
      "╒══════════════╤════════╤════════════╕\n",
      "│ ClassIndex   │   loss │   accuracy │\n",
      "╞══════════════╪════════╪════════════╡\n",
      "│ train        │ 0.0000 │     1.0000 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ vali         │ 0.5697 │     0.8571 │\n",
      "├──────────────┼────────┼────────────┤\n",
      "│ test         │ 0.4972 │     0.8413 │\n",
      "╘══════════════╧════════╧════════════╛\n",
      "╒════════════╤════════╕\n",
      "│ combined   │   loss │\n",
      "╞════════════╪════════╡\n",
      "│ train      │ 0.0000 │\n",
      "├────────────┼────────┤\n",
      "│ vali       │ 0.5697 │\n",
      "├────────────┼────────┤\n",
      "│ test       │ 0.4972 │\n",
      "╘════════════╧════════╛\n",
      "Last improvement of combined validation loss happened 18 step(s) ago.\n",
      "\n",
      "END OF EPOCH\n",
      "LogMetrics are being tracked... {'batch_size': 128, 'epoch': 19, 'steps': 40, 'tune_checkpoint_num': 0, 'last_improvement_steps': 22, 'learning_rate': 0.001, 'best_valid_metric': 0.37374797463417053, 'num_reductions_lr': 0, 'num_increases_bs': 0, 'train_metrics.ClassIndex.loss': 1.9722046999959275e-06, 'train_metrics.ClassIndex.accuracy': 1.0, 'train_metrics.combined.loss': 1.9722046999959275e-06, 'validation_metrics.ClassIndex.loss': 0.5697073340415955, 'validation_metrics.ClassIndex.accuracy': 0.8571428656578064, 'validation_metrics.combined.loss': 0.5697073340415955, 'test_metrics.ClassIndex.loss': 0.49723130464553833, 'test_metrics.ClassIndex.accuracy': 0.841269850730896, 'test_metrics.combined.loss': 0.49723130464553833}\n",
      "train_metrics.ClassIndex.loss\n",
      "train_metrics.ClassIndex.accuracy\n",
      "train_metrics.combined.loss\n",
      "validation_metrics.ClassIndex.loss\n",
      "validation_metrics.ClassIndex.accuracy\n",
      "validation_metrics.combined.loss\n",
      "test_metrics.ClassIndex.loss\n",
      "test_metrics.ClassIndex.accuracy\n",
      "test_metrics.combined.loss\n",
      "_________________\n",
      "Training: 100%|█████████████████████████████████| 40/40 [00:10<00:00,  3.93it/s]\n",
      "\n",
      "╒═════════════════╕\n",
      "│ TRAINING REPORT │\n",
      "╘═════════════════╛\n",
      "\n",
      "Best validation model step: 22, epoch: 11\n",
      "Best validation model loss on validation set combined: 0.37374797463417053\n",
      "Best validation model loss on test set combined: 0.3858174681663513\n",
      "\n",
      "Finished: Classification_run\n",
      "Saved to: /home/erik/Documents/UCPH/aim_projs/aim-ludwig-demo/results/Classification_run\n",
      "END OF TRAINING\n",
      "{}\n",
      "_________________\n",
      "\n",
      "╒══════════╕\n",
      "│ FINISHED │\n",
      "╘══════════╛\n",
      "\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!ludwig train --dataset final_train.csv --config train_conf.yaml -g 0 \\\n",
    "--aim --experiment_name \"Classification\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6hhIq3A5tBF_"
   },
   "source": [
    "You get all of these detailed insights about the training process in the Aim dashboard:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┌------------------------------------------------------------------------┐\r\n",
      "                Aim UI collects anonymous usage analytics.                \r\n",
      "                        Read how to opt-out here:                         \r\n",
      "    https://aimstack.readthedocs.io/en/latest/community/telemetry.html    \r\n",
      "└------------------------------------------------------------------------┘\r\n",
      "\u001b[33mRunning Aim UI on repo `<Repo#-6691639633609373110 path=/home/erik/Documents/UCPH/aim_projs/aim-ludwig-demo/results/Classification_run/.aim read_only=None>`\u001b[0m\r\n",
      "Open http://127.0.0.1:43800\r\n",
      "Press Ctrl+C to exit\r\n"
     ]
    }
   ],
   "source": [
    "!aim up --repo \"results/Classification_run\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Ludwig x W&B.ipynb",
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "ludwig",
   "language": "python",
   "name": "ludwig"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}