SNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About\n",
    "\n",
    "This is a reproduction of the NIPS paper \"Self-Normalizing Neural Networks\". The NIPS version of the paper can be found here: https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf\n",
    "\n",
    "Supplementary material can be found here: https://papers.nips.cc/paper/6698-self-normalizing-neural-networks-supplemental.zip\n",
    "\n",
    "Alternatively, the arxiv version contains both, but has some slight differences with the NIPS paper (and is a bit older): https://arxiv.org/abs/1706.02515\n",
    "\n",
    "This implementation is under the MIT license: https://github.com/AdrienLE/SNN_repro/blob/master/LICENSE\n",
    "\n",
    "As well as implementing a custom version of SNN, it reproduces the UCI experiments present in the original paper. Due to lack of time and computational resources, it does not reproduce the HTRU2 part, and the Tox21 is slightly incomplete, but UCI is arguably the most significant anyway since it covers a staggering 121 datasets!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.io.arff import loadarff\n",
    "from scipy.io import mmread\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torch.autograd as autograd\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "import glob\n",
    "import os\n",
    "import numpy as np\n",
    "import sklearn.model_selection\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import math\n",
    "import pickle\n",
    "import random\n",
    "import multiprocessing\n",
    "import collections\n",
    "import pandas as pd\n",
    "import scipy.stats\n",
    "import hashlib\n",
    "import gzip\n",
    "from IPython.display import display\n",
    "\n",
    "# Note: this is just a progress bar library, if you don't have it,\n",
    "# you can either install it or simply replace tqdm by the identity\n",
    "# function.\n",
    "SHOW_TQDM = True\n",
    "\n",
    "def tqdm(*args, **kwargs):\n",
    "    try:\n",
    "        from tqdm import tqdm_notebook\n",
    "    except Exception:\n",
    "        return args[0]\n",
    "    if SHOW_TQDM:\n",
    "        return tqdm_notebook(*args, **kwargs)\n",
    "    else:\n",
    "        return args[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set some global variables which will allow us to parallelize some of the work involved. If you only want to use a single notebook to do the processing, use BUCKET = 0 and N_BUCKETS = 1, otherwise change these variables to create as many buckets as you want and clone the notebook to create copies that will process the different buckets, potentially on different cuda devices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET = 0\n",
    "N_BUCKETS = 1\n",
    "CUDA_DEVICE = 0\n",
    "\n",
    "USE_CUDA = True\n",
    "\n",
    "def w(v):\n",
    "    if USE_CUDA:\n",
    "        return v.cuda()\n",
    "    return v\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## UCI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We used data sets and preprocessing scripts by Fernández-Delgado et al. [10] for data preparation and defining training and test sets. With several flaws in the method comparison[37] that we avoided, the authors compared 179 machine learning methods of 17 groups in their experiments.\n",
    "\n",
    "[10] http://jmlr.org/papers/volume15/delgado14a/delgado14a.pdf\n",
    "\n",
    "[37] http://www.jmlr.org/papers/volume17/15-374/15-374.pdf\n",
    "\n",
    "Data available at: http://persoal.citius.usc.es/manuel.fernandez.delgado/papers/jmlr/data.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘data.tar.gz’ already there; not retrieving.\n",
      "\n",
      "File ‘data.tar.gz’ already there; not retrieving.\n",
      "\n",
      "tar: abalone/conxuntos_kfold.dat: Cannot open: File exists\n",
      "tar: abalone/abalone.arff: Cannot open: File exists\n",
      "tar: abalone/abalone.data: Cannot open: File exists\n",
      "tar: abalone/abalone.names: Cannot open: File exists\n",
      "tar: abalone/abalone.cost: Cannot open: File exists\n",
      "tar: abalone/conxuntos_kfold.dat: Cannot open: File exists\n",
      "tar: abalone/abalone.arff: Cannot open: File exists\n",
      "tar: abalone/abalone.data: Cannot open: File exists\n",
      "tar: abalone/abalone.names: Cannot open: File exists\n",
      "tar: abalone/abalone.cost: Cannot open: File exists\n",
      "...\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "# Download the uci datasets\n",
    "!mkdir -p datasets/uci\n",
    "!cd datasets/uci && wget -nc http://persoal.citius.usc.es/manuel.fernandez.delgado/papers/jmlr/data.tar.gz\n",
    "!cd datasets/uci && tar xkf data.tar.gz 2>&1 | head -n 5 && echo ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(121, 'breast-cancer-wisc')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(121, 'breast-cancer-wisc')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def uci_folder_to_name(f):\n",
    "    return f.split('/')[-1]\n",
    "\n",
    "uci_folders = sorted(set([\n",
    "    '/'.join(f.split('/')[:-1]) for f in glob.glob('datasets/uci/*/*.arff')\n",
    "]))\n",
    "\n",
    "random.shuffle(uci_folders)\n",
    "\n",
    "# We get 121 datasets, the expected amount\n",
    "len(uci_folders), uci_folder_to_name(uci_folders[14])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'datasets/uci/data.tar.gz', 'datasets/uci/molec-biol-protein-second'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "{'datasets/uci/data.tar.gz', 'datasets/uci/molec-biol-protein-second'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Seems like molec-biol-protein-second hasn't been preprocessed... oh well.\n",
    "set(glob.glob('datasets/uci/*')) - set(uci_folders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[10] says:\n",
    "\n",
    ">  The indexes of the training and test patterns (i.e., the\n",
    "data partitioning) are given by the file conxuntos.dat for each data set, and are the same\n",
    "for all the classifiers. Then, using the selected values for the tunable parameters, a 4-fold\n",
    "cross validation is developed using the whole available data. The indexes of the training\n",
    "and test patterns for each fold are the same for all the classifiers, and they are listed in\n",
    "the file conxuntos kfold.dat for each data set. The test results is the average over the 4\n",
    "test sets. However, for some data sets, which provide separate data for training and\n",
    "testing (data sets annealing and audiology-std, among others), the classifier (with the\n",
    "tuned parameter values) is trained and tested on the respective data sets.\n",
    "\n",
    "This is ambiguous because it isn't clear whether to use the 4-fold version or the training and test set. Based on [37] and my understanding of what is said in the main paper, I will assume that we should use the training/test set defined in `conxuntos.dat` and ignore the 4-fold cross-validation. Eg the main paper says:\n",
    "\n",
    "> For the UCI data sets, the best hyperparameter setting was determined by a grid-search over all hyperparameter combinations using 15% of the training data as validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def line_to_idx(l):\n",
    "    return np.array([int(e) for e in l.split()], dtype=np.int32)\n",
    "\n",
    "def load_uci_dataset(folder, train=True):\n",
    "    full_file = f'{folder}/{uci_folder_to_name(folder)}.arff'\n",
    "    if os.path.exists(full_file):\n",
    "        data = loadarff(full_file)\n",
    "        train_idx, test_idx = [line_to_idx(l) for l in open(f'{folder}/conxuntos.dat').readlines()]\n",
    "        assert len(set(train_idx) & set(test_idx)) == 0\n",
    "        all_idx = list(train_idx) + list(test_idx)\n",
    "        assert len(all_idx) == np.max(all_idx) + 1\n",
    "        assert np.min(all_idx) == 0\n",
    "        if train:\n",
    "            data = (data[0][train_idx], data[1])\n",
    "        else:\n",
    "            data = (data[0][test_idx], data[1])\n",
    "    else:\n",
    "        typename = 'train' if train else 'test'\n",
    "        filename = f'{folder}/{uci_folder_to_name(folder)}_{typename}.arff'\n",
    "        data = loadarff(filename)\n",
    "    assert data[1].types() == ['numeric'] * (len(data[1].types()) - 1) + ['nominal']\n",
    "    X = np.array(data[0][data[1].names()[:-1]].tolist())\n",
    "    y = np.array([int(e) for e in data[0][data[1].names()[-1]]])\n",
    "    return X.astype(np.float32), y\n",
    "\n",
    "def uci_validation_set(X, y):\n",
    "    return sklearn.model_selection.train_test_split(X, y, test_size=0.15, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tox21"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For the Tox21 data set, the best hyperparameter setting was determined by a grid-search over all hyperparameter combinations using the validation set defined by the challenge winners (Mayr et al., 2016). \n",
    "\n",
    "Paper: https://arxiv.org/pdf/1503.01445.pdf\n",
    "\n",
    "I assume this also means that they used the processed dataset provided by the winners as well, available at: http://bioinf.jku.at/research/DeepTox/tox21.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ‘tox21.zip’ already there; not retrieving.\n",
      "\n",
      "File ‘tox21.zip’ already there; not retrieving.\n",
      "\n",
      "Archive:  tox21.zip\n",
      "Archive:  tox21.zip\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p datasets/tox21\n",
    "!cd datasets/tox21 && wget -nc http://bioinf.jku.at/research/DeepTox/tox21.zip\n",
    "!cd datasets/tox21 && unzip -f tox21.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/Test/Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paper is ambiguous because it talks about using \"the validation set defined by the challenge winners\", but the challenge winners used *cross-validation*. Two possibilities on what this means:\n",
    "\n",
    "- The self-norm paper uses a single fold of the cross-validation as its validation set or simply created its own validation set based on the principles outlined in the paper, in which case the type of validation set is what really matters.\n",
    "- The self-norm paper actually does use the same cross-validation but doesn't mention it in those words. This is weird because it mentions cross-validation elsewhere.\n",
    "\n",
    "I will assume the former possibility, in part because it is simpler to implement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILE_CACHE = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def c_csv(*args, **kwargs):\n",
    "    if args[0] not in FILE_CACHE:\n",
    "        FILE_CACHE[args[0]] = pd.read_csv(*args, **kwargs)\n",
    "    return FILE_CACHE[args[0]]\n",
    "\n",
    "def c_mmread(*args, **kwargs):\n",
    "    if args[0] not in FILE_CACHE:\n",
    "        FILE_CACHE[args[0]] = mmread(*args, **kwargs)\n",
    "    return FILE_CACHE[args[0]]\n",
    "\n",
    "def load_tox21(effect_label, typ='train', cvfold=0.0):\n",
    "    pref = 'datasets/tox21/'\n",
    "    # Code based on the sample code given here: http://bioinf.jku.at/research/DeepTox/sampleCode.py\n",
    "    y_tr = c_csv(pref + 'tox21_labels_train.csv.gz', index_col=0, compression=\"gzip\")\n",
    "    y_te = c_csv(pref + 'tox21_labels_test.csv.gz', index_col=0, compression=\"gzip\")\n",
    "    x_tr_dense = c_csv(pref + 'tox21_dense_train.csv.gz', index_col=0, compression=\"gzip\").values\n",
    "    x_te_dense = c_csv(pref + 'tox21_dense_test.csv.gz', index_col=0, compression=\"gzip\").values\n",
    "    x_tr_sparse = c_mmread(pref + 'tox21_sparse_train.mtx.gz').tocsc()\n",
    "    x_te_sparse = c_mmread(pref + 'tox21_sparse_test.mtx.gz').tocsc()\n",
    "    \n",
    "    compound_data = c_csv(pref + 'tox21_compoundData.csv', index_col=0)\n",
    "    tr_cv = c_csv(pref + 'tox21_dense_train.csv.gz', index_col=0, compression=\"gzip\").join(compound_data)\n",
    "\n",
    "    # filter out very sparse features\n",
    "    sparse_col_idx = ((x_tr_sparse > 0).mean(0) > 0.05).A.ravel()\n",
    "    x_tr = np.hstack([x_tr_dense, x_tr_sparse[:, sparse_col_idx].A])\n",
    "    x_te = np.hstack([x_te_dense, x_te_sparse[:, sparse_col_idx].A])\n",
    "\n",
    "    target = y_tr.columns[effect_label]\n",
    "    rows_tr = np.isfinite(y_tr[target]).values\n",
    "    rows_val = tr_cv['CVfold'] == cvfold\n",
    "    rows_te = np.isfinite(y_te[target]).values\n",
    "\n",
    "    if typ == 'train':\n",
    "        res = x_tr[rows_tr & (~rows_val)], y_tr[target][rows_tr & (~rows_val)]\n",
    "    elif typ == 'val':\n",
    "        res = x_tr[rows_tr & (rows_val)], y_tr[target][rows_tr & (rows_val)]\n",
    "    else:\n",
    "        res = x_te[rows_te], y_te[target][rows_te]\n",
    "    X, y = [np.array(r) for r in res]\n",
    "    return X.astype(np.float32), y.astype(np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7748, 1644)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(7748, 1644)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = load_tox21(5, typ='train')\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given in supplement S4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "UCI_HYPER = {\n",
    "    'SNN': {\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 3, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'dropout': [0.05, 0.0],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['selu']\n",
    "    },\n",
    "    'MSRAinit': {  # Note: same as above except dropout\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 3, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'dropout': [0.5, 0.0],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']\n",
    "    },\n",
    "    'BatchNorm': { # Note: no dropout, batchnorm\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 3, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'norm': ['batchnorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'WeightNorm' : { # Note: same as above, weightnorm\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 3, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'norm': ['weightnorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'LayerNorm': { # Note: same as above, layernorm\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 3, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'norm': ['layernorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'Highway': {\n",
    "        'type': ['highway'],\n",
    "        'layers': [2, 3, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'dropout': [0.5, 0.0],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'ResNet': {\n",
    "        'type': ['resnet'],\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 3, 4, 8, 16],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'bottleneck': [0.25, 0.5],\n",
    "        'form': ['rect', 'diavolo'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "}\n",
    "\n",
    "TOX21_HYPER = {\n",
    "    'SNN': {\n",
    "        'units': [1024, 2048],\n",
    "        'layers': [2, 3, 4, 6, 8, 16, 32],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'dropout': [0.05, 0.1],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['selu'],\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "    'MSRAinit': {  # Note: same as above except dropout\n",
    "        'units': [1024, 2048],\n",
    "        'layers': [2, 3, 4, 6, 8, 16, 32],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'dropout': [0.5, 0.0],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu'],\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "    'BatchNorm': { # Note: no dropout, batchnorm\n",
    "        'units': [1024, 2048],\n",
    "        'layers': [2, 3, 4, 6, 8, 16, 32],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'norm': ['batchnorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu'],  # ?\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "    'WeightNorm' : { # Note: same as above, weightnorm\n",
    "        'units': [1024, 2048],\n",
    "        'layers': [2, 3, 4, 6, 8, 16, 32],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'norm': ['weightnorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu'],  # ?\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "    'LayerNorm': { # Note: same as above, layernorm\n",
    "        'units': [1024, 2048],\n",
    "        'layers': [2, 3, 4, 6, 8, 16, 32],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'norm': ['layernorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu'],  # ?\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "    'Highway': {\n",
    "        'type': ['highway'],\n",
    "        'layers': [2, 3, 4, 6, 8, 16, 32],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'dropout': [0.5, 0.0],\n",
    "        'activation': ['relu'],  # ?\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "    'ResNet': {\n",
    "        'type': ['resnet'],\n",
    "        'units': [1024, 2048],\n",
    "        'layers': [2, 3, 4, 6, 8, 16],\n",
    "        'lr': [0.01, 0.05, 0.1],\n",
    "        'bottleneck': [0.25, 0.5],\n",
    "        'form': ['rect', 'diavolo'],\n",
    "        'activation': ['relu'],  # ?\n",
    "        'l2': [0.001, 0.0001, 0.00001],\n",
    "    },\n",
    "}\n",
    "\n",
    "HTRU2_HYPER = {\n",
    "    'SNN': {\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'dropout': [0.05, 0.0],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['selu']\n",
    "    },\n",
    "    'MSRAinit': {  # Note: same as above except dropout\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'dropout': [0.5, 0.0],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']\n",
    "    },\n",
    "    'BatchNorm': { # Note: no dropout, batchnorm\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'norm': ['batchnorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'WeightNorm' : { # Note: same as above, weightnorm\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'norm': ['weightnorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'LayerNorm': { # Note: same as above, layernorm\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'norm': ['layernorm'],\n",
    "        'form': ['rect', 'conic'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'Highway': {\n",
    "        'type': ['highway'],\n",
    "        'layers': [2, 4, 8, 16, 32],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'dropout': [0.5, 0.0],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "    'ResNet': {\n",
    "        'type': ['resnet'],\n",
    "        'units': [1024, 512, 256],\n",
    "        'layers': [2, 4, 8, 16],\n",
    "        'lr': [0.01, 0.1, 1.0],\n",
    "        'bottleneck': [0.25, 0.5],\n",
    "        'form': ['rect', 'diavolo'],\n",
    "        'activation': ['relu']  # ?\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Initialization. Since SNNs have a fixed point at zero mean and unit variance for normalized weights ω =?n\n",
    "i=1 wi = 0 and τ =?n i=1 w2 i = 1 (see above), we initialize SNNs such that these constraints\n",
    "are fulfilled in expectation. We draw the weights from a Gaussian distribution with E(wi) = 0 and variance Var(wi) = 1/n. Uniform and truncated Gaussian distributions with these moments led to networks with similar behavior. The “MSRA initialization” is similar since it uses zero mean and variance 2/n to initialize the weights [17]. The additional factor 2 counters the effect of rectified linear units.\n",
    "\n",
    "Initialization names are all over the place, but at least it is clear what initialization they use for their own networks and for ReLU networks. The initialization given here is more or less the one given by kaiming_normal in torch.nn.init, but we either need to call it in counterintuitive ways or to modify it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_tensor(tensor, nonlinearity, mode='fan_in'):\n",
    "    \"\"\"\n",
    "    Modified version of kaiming_normal from torch.nn.init that can handle the selu nonlinearity properly.\n",
    "    \"\"\"\n",
    "    if isinstance(tensor, autograd.Variable):\n",
    "        init_tensor(tensor.data, nonlinearity, mode=mode)\n",
    "        return tensor\n",
    "\n",
    "    if nonlinearity == 'selu':\n",
    "        nonlinearity = 'linear'\n",
    "    fan = nn.init._calculate_correct_fan(tensor, mode)\n",
    "    gain = nn.init.calculate_gain(nonlinearity)\n",
    "    std = gain / math.sqrt(fan)\n",
    "    return tensor.normal_(0, std)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feedforward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paper does not discuss the following:\n",
    "- Batch size (the concept does not appear at all)\n",
    "- Optimizers: the NIPS version does mention optimizers a little, but does not explain which one was used for the experiments.\n",
    "\n",
    "Specifically this is all the paper says on optimizers:\n",
    "\n",
    "> Empirically, we found that SGD, momentum, Adadelta and Adamax worked well for training SNNs, whereas for Adam we had to adjust the parameters (β2 = 0.99, epsilon = 0.01) to obtain proficient networks.\n",
    "\n",
    "Basically it sounds like all these optimizers can work, but which one was actually used? This is highly relevant for the learning rate!\n",
    "\n",
    "For now, we will assume regular SGD as the optimizer (no momentum, since no momentum shows up in the hyperparameters), and 64 as the batch size.\n",
    "\n",
    "In addition, it is not fully clear what activation function is used by the different networks, except for SNN and MSRAinit, which are clearly using SeLU and ReLU, respectively. We will assume that all other networks are also using ReLU, as is most likely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SeLU(nn.Module):\n",
    "    # Custom implementation of SeLU. Note that an equivalent implementation\n",
    "    # exists in pytorch as nn.SELU\n",
    "    def __init__(self, lambd=1.0507, alpha=1.67326):\n",
    "        super().__init__()\n",
    "        self.lambd = lambd\n",
    "        self.alpha = alpha\n",
    "        \n",
    "    def forward(self, x):\n",
    "        mask = (x > 0).type(x.data.type())\n",
    "        return self.lambd * (\n",
    "            (x * mask) +\n",
    "            ((self.alpha * torch.exp(x) - self.alpha) * (1 - mask))\n",
    "        )\n",
    "\n",
    "class AlphaDropout(nn.Module):\n",
    "    # Custom implementation of alpha dropout. Note that an equivalent\n",
    "    # implementation exists in pytorch as nn.AlphaDropout\n",
    "    def __init__(self, dropout, lambd=1.0507, alpha=1.67326):\n",
    "        super().__init__()\n",
    "        self.lambd = lambd\n",
    "        self.alpha = alpha\n",
    "        self.aprime = -lambd * alpha\n",
    "        \n",
    "        self.q = 1 - dropout\n",
    "        self.p = dropout\n",
    "\n",
    "        self.a = (self.q + self.aprime**2 * self.q * self.p)**(-0.5)\n",
    "        self.b = -self.a * (self.p * self.aprime)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        if not self.training:\n",
    "            return x\n",
    "        ones = torch.ones(x.size())\n",
    "        if x.is_cuda:\n",
    "            ones = ones.cuda()\n",
    "        mask = torch.bernoulli(ones * self.p)\n",
    "        x = x.masked_fill(autograd.Variable(mask.byte()), self.aprime)\n",
    "        return x * self.a + self.b\n",
    "    \n",
    "ACTIVATIONS = {\n",
    "    'relu': nn.ReLU,\n",
    "    'selu': SeLU\n",
    "}\n",
    "\n",
    "OPTIMIZER_CLASS = optim.SGD\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "class LayerNorm(nn.Module):\n",
    "    # Stolen from https://github.com/pytorch/pytorch/issues/1959\n",
    "    def __init__(self, features, eps=1e-6):\n",
    "        super().__init__()\n",
    "        self.gamma = nn.Parameter(torch.ones(features))\n",
    "        self.beta = nn.Parameter(torch.zeros(features))\n",
    "        self.eps = eps\n",
    "\n",
    "    def forward(self, x):\n",
    "        mean = x.mean(-1, keepdim=True)\n",
    "        std = x.std(-1, keepdim=True, unbiased=False)  # Note: unbiased=False to prevent divide by 0 errors.\n",
    "        return self.gamma * (x - mean) / (std + self.eps) + self.beta\n",
    "\n",
    "\n",
    "class Layer(nn.Module):\n",
    "    def __init__(self, input_size, output_size, activation, normalization, dropout, is_highway):\n",
    "        super().__init__()\n",
    "\n",
    "        self.highway_lin = None\n",
    "        if is_highway:\n",
    "            assert input_size == output_size\n",
    "            self.highway_lin = nn.Linear(output_size, output_size)\n",
    "        \n",
    "        ident = lambda x: x\n",
    "        self.lin = nn.Linear(input_size, output_size)\n",
    "        init_tensor(self.lin.weight, activation)\n",
    "        self.activation = ACTIVATIONS[activation]()\n",
    "        if dropout > 0.0:\n",
    "            if activation != 'selu':\n",
    "                self.dropout = nn.Dropout(dropout)\n",
    "            else:\n",
    "                self.dropout = AlphaDropout(dropout)\n",
    "        else:\n",
    "            self.dropout = ident\n",
    "        if normalization == 'batchnorm':\n",
    "            self.normalization = nn.BatchNorm1d(output_size)\n",
    "        elif normalization == 'weightnorm':\n",
    "            self.normalization = ident\n",
    "            self.lin = nn.utils.weight_norm(self.lin)\n",
    "        elif normalization == 'layernorm':\n",
    "            self.normalization = LayerNorm(output_size)\n",
    "        elif normalization is None:\n",
    "            self.normalization = ident\n",
    "        else:\n",
    "            raise ValueError(f'Unknown normalization {normalization}')\n",
    "                    \n",
    "    def forward(self, x):\n",
    "        activations = self.activation(self.lin(x))\n",
    "        if self.highway_lin is not None:\n",
    "            act_mul = F.sigmoid(self.highway_lin(x))\n",
    "            activations = act_mul*activations + (1 - act_mul)*x\n",
    "        \n",
    "        # Note: we use normalization AFTER the activation. The reason for this is twofold:\n",
    "        # 1/ This is the proper way to use layernorm and although it is not the \"official\" way to\n",
    "        #    use batchnorm, it is known to work better.\n",
    "        # 2/ It is likely the way it was done in the paper since when I tried doing it before the\n",
    "        #    activation I found that layernorm and batchnorm had performances that differed\n",
    "        #    significantly from the paper.\n",
    "        return self.dropout(self.normalization(activations))\n",
    "\n",
    "class ResNetLayer(nn.Module):\n",
    "    def __init__(self, input_size, n_nodes, bottleneck):\n",
    "        super().__init__()\n",
    "        \n",
    "        # First if we are the first block, we need to make the input the same\n",
    "        # size as n_nodes.\n",
    "        self.init_layers = nn.ModuleList()\n",
    "        if input_size != n_nodes:\n",
    "            self.init_layers.append(nn.Linear(input_size, n_nodes))\n",
    "            self.init_layers.append(nn.BatchNorm1d(n_nodes))\n",
    "            self.init_layers.append(nn.ReLU())\n",
    "            input_size = n_nodes\n",
    "        \n",
    "        # Important note: here we assume that the \"number of neuron per block\"\n",
    "        # is in fact the number of neurons within a non-bottleneck layer of \n",
    "        # each block. Reasons to make this assumption include:\n",
    "        # 1/ The math doesn't work out if the numbers given are really the\n",
    "        #    total number of neurons across the 2 or 3 blocks! For instance,\n",
    "        #    a layer with 256 neurons and a bottleneck of 50% would need 102.4\n",
    "        #    neurons in the two outside layers and 51.2 in the bottleneck layer\n",
    "        #    (of course this could be done with some rounding, but the fact that\n",
    "        #    no such thing is mentioned indicates that this wasn't done).\n",
    "        # 2/ The maximum number of blocks for ResNet is 16, which is twice as\n",
    "        #    little as the maximum number of layers for other networks. This\n",
    "        #    squares well with the fact that Resnet contains 2 (or 3) layers per \n",
    "        #    block.\n",
    "        self.layers = nn.ModuleList()\n",
    "        \n",
    "        self.layers.append(nn.Linear(input_size, n_nodes))\n",
    "        input_size = n_nodes\n",
    "        # Note: unlike in the other layers, we put it before the activation to\n",
    "        # follow the ResNet paper more closely.\n",
    "        self.layers.append(nn.BatchNorm1d(n_nodes))\n",
    "        self.layers.append(nn.ReLU())\n",
    "        \n",
    "        if bottleneck is not None:\n",
    "            bottleneck_size = int(n_nodes * bottleneck)\n",
    "            self.layers.append(nn.Linear(input_size, bottleneck_size))\n",
    "            input_size = bottleneck_size\n",
    "            self.layers.append(nn.BatchNorm1d(bottleneck_size))\n",
    "            self.layers.append(nn.ReLU())\n",
    "        \n",
    "        self.layers.append(nn.Linear(input_size, n_nodes))\n",
    "        self.layers.append(nn.BatchNorm1d(n_nodes))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        for l in self.init_layers:\n",
    "            x = l(x)\n",
    "        y = x\n",
    "        for l in self.layers:\n",
    "            y = l(y)\n",
    "        y += x\n",
    "        return F.relu(y)\n",
    "    \n",
    "class FNN(nn.Module):\n",
    "    def __init__(self, n_features, n_classes, description):\n",
    "        super().__init__()\n",
    "        self.n_features = n_features\n",
    "        self.n_classes = n_classes\n",
    "        self.description = description\n",
    "        self.layers = nn.ModuleList()\n",
    "        if 'units' not in description:\n",
    "            assert description.get('type') == 'highway'\n",
    "        n_nodes = description.get('units', n_features)\n",
    "        input_size = n_features\n",
    "        for _ in range(description['layers']):\n",
    "            if description.get('type') == 'resnet':\n",
    "                if description['form'] == 'rect':\n",
    "                    assert description.get('bottleneck') is None\n",
    "                else:\n",
    "                    assert description.get('bottleneck') is not None\n",
    "                self.layers.append(\n",
    "                    ResNetLayer(\n",
    "                        input_size, n_nodes,\n",
    "                        description.get('bottleneck')\n",
    "                    )\n",
    "                )\n",
    "            else:\n",
    "                self.layers.append(\n",
    "                    Layer(\n",
    "                        input_size, n_nodes,\n",
    "                        description['activation'],\n",
    "                        description.get('norm'),\n",
    "                        description.get('dropout', 0.0),\n",
    "                        description.get('type') == 'highway'\n",
    "                    )\n",
    "                )\n",
    "            input_size = n_nodes\n",
    "            if description.get('form') == 'conic':\n",
    "                n_nodes = max(1, n_nodes // 2)  # TODO: figure out if this is correct\n",
    "        self.layers.append(nn.Linear(input_size, n_classes))\n",
    "        self.layers.append(nn.LogSoftmax())\n",
    "    \n",
    "    def forward(self, x):\n",
    "        for i, l in enumerate(self.layers):\n",
    "            x = l(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f2dce784438>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f2dce784438>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHsNJREFUeJzt3Xl8lOW99/HPL5OVEMKSIEsIi4CA7AaSHGu1LhUBURStC9a64Yaojz2t9pzTvp6e0/b0dBVRlGLdN7SIGwJq60NbSCDshn0JxCAmbAkhZJmZ6/wB7eM5VbbcyT3L9/16zYuZZLju70D4ennNfV9jzjlERCR2JPgdQEREvKViFxGJMSp2EZEYo2IXEYkxKnYRkRijYhcRiTEqdhGRGKNiFxGJMSp2EZEYk+jHQbOyslyvXr38OLSISNRasWLFXudc9ome50ux9+rVi5KSEj8OLSIStcxs58k8T0sxIiIxRsUuIhJjVOwiIjFGxS4iEmNU7CIiMUbFLiISY1TsIiIxRsUuItIK6mqrKX78NqoP7G3xY6nYRURaWG3NAcoeHUte5R8oW/Vhix/PlytPRUTiRc3BfeyeMY7+TZtYNfpX5F14XYsf05NiN7My4BAQAoLOuTwvxhURiWbV+6v4/Imx9Gnaxtp/epS8S7/dKsf1csb+Dedcyy8eiYhEgYN797B35lh6BXey/rzHGXnx9a12bC3FiIh4bN/nn1Lz1Dh6hCrYcMFTDP/GpFY9vldvnjpgkZmtMLMpHo0pIhJ19u7ZxaGnxtAltJvNFz3NsFYudfBuxv4151yFmXUGPjCzjc65xV98wrHCnwKQm5vr0WFFRCJHZcUOGmaPpXN4H9svfY4h/zTWlxyezNidcxXHfq0E3gRGf8lzZjnn8pxzednZJ9wnXkQkquzZtYXG2WPoGN7PzrEvcLZPpQ4eFLuZpZtZxt/uA98EPmnuuCIi0WJ32SbCz4wlM1xN+eUvMzD/Ul/zeLEUcwbwppn9bbyXnXMLPBhXRCTiVWwvJfD8BNpwhM+umMOAkV/3O1Lzi905tx0Y5kEWEZGosmvzalJfnkgSTVROfJ3+w871OxKg0x1FRE7Lzg0rSH/tKhIIc2DSH+g7ON/vSH+nvWJERE7RjtJiMl67EoBD35pHnwgqddCMXUTklGxbu4SOc6+liSTqb5xHz36RtxKtGbuIyEnasmoxWXMn0UAKjTe9S24Eljqo2EVETsrGko/oMu9aDls64e/MJ+fMs/2O9JW0FCMicgIbihfSY/7NHEzIJPHW9+jSo6/fkY5LM3YRkeMo/et79Jx/E/sTOpJ8+4KIL3VQsYuIfKV1i9+iz6LvUBnoTJs7F9C5e2+/I50UFbuIyJdY+6c36P/RbXwW6EbGXQvJ6hI9mxeq2EVE/pfVH77CgI/vpDwxl473LKRT5+5+RzolKnYRkS9YtfA5zv7zvZQl9SH73oW0z+rid6RTpmIXETlmxXuzGbLkAbYl9afL1AVkdozOLcZV7CIiQMnbMxm+7LtsSR5IzrT3ade+k9+RTpuKXUTi3rI3H2PkikfYmDKUnve/T9t2HfyO1CwqdhGJa8Wv/4rRa/6V0tQR9Ln/Pdq0zfQ7UrPpylMRiVvFr/0n+Rt+xpq00Zw1bR6pael+R/KEil1E4lLRSz+mYMuvWNWmkEHT5pKS2sbvSJ5RsYtI3Cl6/t8o2D6dlelfZ/C010lOSfU7kqdU7CISV5Y+830Kdz5JScaFDJv2GklJyX5H8pyKXUTigguHKXrmnyksn83yzG8yYupLJMZgqYOHZ8WYWcDMVpnZu16NKSLiBRcOUzT7AQrLZ7Os/VhG3vdKzJY6eHu64/3ABg/HExFpNhcOU/zUPRTufo7ijleQd9+LBBJje7HCk2I3sxxgHDDbi/FERLzgwmGWzbyDgs9foSh7EqOnPktCIOB3rBbn1Yz9t8D3gPBXPcHMpphZiZmVVFVVeXRYEZEvFw6FWPb4LeRXvcHSM64n/+7fYQnxcU1ms1+lmY0HKp1zK473POfcLOdcnnMuLzs7OjfWEZHoEAoGKXlsMvn75rG067cpuPOJuCl18GbGfi4wwczKgFeBC83sRQ/GFRE5ZaFgkJWPXc/og/NZmnM7BXc8GlelDh4Uu3PuEedcjnOuF3Ad8Efn3ORmJxMROUXBpkZWP3oNo6oXsbTnXRTe/qu4K3XQeewiEiOaGhtY9+gkzjm8mKV9plH47X/3O5JvPC1259zHwMdejikiciIN9XWsn341I+uWUNTvIQpv/KHfkXylGbuIRLX6I4fZNP1KRhxZRvHARyj41sN+R/Kdil1EotaRw4fYOn0CwxpWUnz2D8m/5iG/I0UEFbuIRKW62mp2TL+csxvWsmz4v5M/cZrfkSKGil1Eok5tzQF2PTaeAY2lrDznZ4yecLffkSKKil1EokrNwX3snjGO/k2bWDX6l+SNu93vSBFHxS4iUaN6fxWfPzGWM5u2sbbwt+SNudnvSBFJxS4iUeHg3j3snTmWXsGdfHLe44y8+Hq/I0UsFbuIRLz9lRUcfHIsPUIVbDj/SUZceI3fkSKail1EItrePbuonTWWrqHP2XzRbIZ9/Uq/I0U8FbuIRKyq3WUcmT2W7NBetn3zGYacO97vSFFBxS4iEWlP+VaCvx9Pp/ABdl72AoMLLvU7UtRQsYtIxNldtgmeu5zMcA3l419m0KiL/I4UVVTsIhJRKraXEnj+CtpQx2dXzGHAyK/7HSnqqNhFJGKUb1lDyktXkkQTlRNfp/+wc/2OFJVU7CISEXZuXEn6qxNJIMyBSX+g7+B8vyNFrfj7aBERiTg7SovJePUKAA59ax59VOrNohm7iPhq29oldJx7LU0kUX/Dm/TsP9zvSFFPM3YR8c2WVYvJmjuJBlJonPwOuSp1T6jYRcQXG0s+osu8azls6YRvfo+cvoP9jhQzml3sZpZqZsvMbI2ZlZrZ//UimIjErg3FC8l550aqEzJJuGU+3XoP8DtSTPFijb0BuNA5V2tmScBfzOx951yRB2OLSIwp/et79F50C/sSOpFy+3w6d+/td6SY0+wZuzuq9tjDpGM319xxRST2rFv8Fn0WfYfKQGfSpixUqbcQT9bYzSxgZquBSuAD51yxF+OKSOxY+6c36P/RbewJdCPjzgVkdc31O1LM8qTYnXMh59xwIAcYbWb/8C6ImU0xsxIzK6mqqvLisCISJVZ/+AoDPr6T8sRc2t+zkE5n5PgdKaZ5elaMc+4g8CdgzJd8b5ZzLs85l5edne3lYUUkgq1c+AKD/nwvZUl9yL53AR2yuvgdKeZ5cVZMtpm1P3Y/DbgE2NjccUUk+q2Y/zRDl0xje1J/ukxdQGbHzn5HigtenBXTFXjOzAIc/Q/FHOfcux6MKyJRrOTtJxmx4mE2Jw8iZ+q7ZGR29DtS3Gh2sTvn1gIjPMgiIjFi+bwZnLPqX1mfMpTe971Dekam35HiivaKERFPLXvj1+St+zGlqSPoO+1t0tIz/I4Ud1TsIuKZ4td+Tv6Gn7ImbRRn3f8WqWnpfkeKSyp2EfFE0cv/TsHmX7K6TSEDp80lJbWN35HilopdRJqt6IUfUrDtUVamn8fgaW+QnJLqd6S4pmIXkWZZ+uzDFJbNZEXGNxh632skJaf4HSnuqdhF5LS4cJiiZ/6ZwvLZlLS7hOH3vUxiUrLfsQQVu4icBhcOUzT7AQp3P8ey9mM5Z+oLBBJVJ5FCfxMickpcOEzxU/dQ+PkrFHecwKh7nyUhEPA7lnyBil1ETpoLh1k28w4Kqt6gKOtq8u+ZjSXog9gijYpdRE5KOBRi+RO3kr9vHkWdryP/rpkq9QilYheREwoFg6yYcRP5B+eztOtNFNwxXaUewVTsInJcoWCQlY9dz+jqRSzNuY2CW3+pUo9wKnYR+UrBpkZWT7+OUYc+YmnPuyi85ed+R5KToGIXkS/V1NjAuumTyKtdzNI+91H47f/wO5KcJBW7iPyDhvo61k+/mpF1Syjq9xCFN/7Q70hyClTsIvI/1B85zKbpExlxpJiiAY9QcN3DfkeSU6RiF5G/q6+rZfP0CQyrX0Hx2T+k4JqH/I4kp0HFLiIA1NVWs/2xCQyuX8PyYT8m/6r7/Y4kp0nFLiLU1hxg12PjGdhYyspzfsaoCXf7HUmaQcUuEudqDu5j94xx9G/axKrRvyBv3B1+R5JmavZVBmbWw8z+ZGbrzazUzPT/byJRonp/FXtmjOHMps2sLfyNSj1GeDFjDwIPOedWmlkGsMLMPnDOrfdgbBFpIQf37mHvzLH0CpZR+rUZjLzkBr8jiUeaPWN3zn3mnFt57P4hYAPQvbnjikjL2V9Zwf4nLqVHcBcbzn+S4Sr1mOLphg9m1gsYARR/yfemmFmJmZVUVVV5eVgROQV79+yi5slL6RaqYPNFv2PYhdf6HUk85lmxm1lb4A/AA865mv/9fefcLOdcnnMuLzs726vDisgpqNpdRt2sMXQOVbL1m88w5OsT/Y4kLcCTs2LMLImjpf6Sc26uF2OKiLf2lG8l+PvxdAofoOyy5xlcMMbvSNJCml3sZmbA08AG59yvmx9JRLy2u2wTPHc5meEayse/xKBRF/sdSVqQF0sx5wI3ARea2epjt7EejCsiHqjYXkrCs+No62rZfcVrDFCpx7xmz9idc38BzIMsIuKx8i1rSHnpSpJppHLi65w17Fy/I0kr0JWnIjFq58aVtHn1KgKE2D9pLn0H5/sdSVqJil0kBu1Yv5x2c67GYdR8ax59Bp7jdyRpRfrgQpEYs23tEtrPmUiIAHU3vEUvlXrcUbGLxJAtq/9M1txJNJBC4+R3yO0/3O9I4gMtxYjEiE0lf6TruzdymHTcze+S03uA35HEJ5qxi8SAjcWL6P7ODRyyDOzW9+mmUo9rKnaRKFe6ZD658ydzIKEDSbcvpEtuP78jic9U7CJR7JM/v0WfhTdTFehM2h0L6Ny9t9+RJAKo2EWi1No/vUHfD29jT6Abbe9cQFa3nn5HkgihYheJQqs/epUBH99JRWIP2t+9gE5n5PgdSSKIil0kyqxc+AKDFt/DzqTeZN2zgA7ZXf2OJBFGxS4SRVbMf5qhS6axI6kfZ0xdSGanM/yOJBFIxS4SJUrefpLhxQ+xJXkg3e57n3btO/kdSSKUil0kCiyfN4ORKx5mY8oQcqfNJyOzo9+RJILpylORCLfsjV+Tt+7HlKYOp++0d0hLz/A7kkQ4FbtIBCt+7efkb/gpa9JGcdb9b5Galu53JIkCKnaRCFX08n9QsPkXrG5TyMBpc0lJbeN3JIkSKnaRCFT04o8o2PpbVqafx+Bpb5Cckup3JIkiKnaRCFP07A8oKHucFW0vYOi0OSQlp/gdSaKMJ2fFmNnvzazSzD7xYjyReOTCYZY+/V0Kyh6npN3FDLv/dZW6nBavTnd8Fhjj0VgicceFwxQ9/SCF5b9jefvLGDHtNRKTkv2OJVHKk2J3zi0G9nsxlki8ceEwxbOmUljxLMs6Xs45971EIFGrpHL69NMj4iMXDlM8cwoFVa9TnHUVo+6eTUIg4HcsiXKtduWpmU0xsxIzK6mqqmqtw4pErHAoxLLHb6Gg6nWKOn+L0fc8rVIXT7RasTvnZjnn8pxzednZ2a11WJGIFA6FKJlxE/n75rG062Ty73oSS9AOH+INLcWItLJQMMjKx25gdPVClubcSsGtv1Kpi6e8Ot3xFWApcJaZfWpmt3kxrkisCTY1surRaxlVvZClPe+i8PbfqNTFc57M2J1z13sxjkgsa2psYN30SeTVLmZp76kU3vwTvyNJjNJSjEgraKivY/30qxlZt4Sifv+Hwht/5HckiWEqdpEWVn/kMJumT2TEkWKKBzxMwXWP+B1JYpyKXaQF1dfVsnn6BIbVr6D47H8j/5rv+h1J4oCKXaSF1NVWs/2xCQyuX8PyYT8m/6r7/Y4kcULFLtICamsOsGvG5Qxs+IQVI3/GqCvu9juSxBEVu4jHDlXvp2LGOPo3bmTVqF8wavwdfkeSOKNiF/FQ9YG9fP74ZZzZtI21Bb8m77Jb/I4kcUjFLuKR6n2fU/XEZfQKlvHJudMZ+c3JfkeSOKViF/HA/soKDjw1jh7BT9lw/pOMuPBavyNJHFOxizTT3j3l1M4aS/fQZ2y+cBbDzr/K70gS51TsIs1QtbuMutnj6ByqYuslv2fI1yb4HUlExS5yuvaUbyX4+/FkhQ9QdtnzDC7Qp0NKZFCxi5yG3WWb4LnLyQzXUD7uRQaNvsTvSCJ/p2IXOUUV2zcQeP5y2lDHZ1e8yoCRF/gdSeR/ULGLnILyretIeXECyTRSOXEO/Yd9ze9IIv9AxS5yknZuXEmbV68ikRD7J82l7+B8vyOJfCkVu8hJ2LF+Oe3mXI3DqP7Wm/QZmOd3JJGvpM/kEjmBbWuX0H7OREIEqLvhLXqp1CXCqdhFjmPL6j+TNXcSDaTQOPkdcvsP9zuSyAlpKUbkK2wq+SNd372Rw6Tjbn6XnN4D/I4kclI8mbGb2Rgz22RmW83sYS/GFPHTxuJFdH/nBg5ZBtwyn24qdYkizS52MwsAjwOXAYOA681sUHPHFfHL+qXvkzt/MgcSOpB42/t07dnf70gip8SLGftoYKtzbrtzrhF4FbjCg3FFWl3pkvn0XvBtqgLZpN2xgDNyzvQ7ksgp86LYuwPlX3j86bGv/Q9mNsXMSsyspKqqyoPDinjrUPV+shfdS1Ugm/QpC8jq1tPvSCKnpdXOinHOzXLO5Tnn8rKzs1vrsCInrfTF75HlDlA//gmyuvTwO47IafOi2CuAL/4ryDn2NZGosbHkI0ZXvsHy7Kvor71fJMp5UezLgX5m1tvMkoHrgLc9GFekVRw5fIj096ZSaVkMuumXfscRabZmF7tzLghMBRYCG4A5zrnS5o4r0lrWPPcQPdxu9l78azIyO/odR6TZPLlAyTk3H5jvxVgiral0yXwKKl+jOOsq8vXpRxIjtKWAxK3amgN0+OABPrUuDPnOb/2OI+IZFbvEJRcOs3H27ZwRruTQmOm0aZvpdyQRz6jYJS4tf3M6eTUfsqzXnQzMv9TvOCKeUrFL3NmxfjlD1v6ET1KGM/qmn/gdR8RzKnaJK3W11SS8cQuHrQ1dbnmRQKI2OJXYo2KXuBEOhdg4czI5oU/Zc9F0XV0qMUvFLnGj+NnvM/LwYpb3f5DB52mfOoldKnaJCyvmP0Nh+e9Y3v4y8q//N7/jiLQoFbvEvC2rFjOo+HtsTBrE0Lt+jyXox15im37CJaaVb1lD1ls3ciChPVm3zSEltY3fkURanIpdYlbV7jISX7oaB4RunKs3SyVuqNglJlXvr6J29gTauUPsu/JlevQd4nckkVajYpeYU72/isrHx9A9VMGOi2fRb/h5fkcSaVUqdokp1fs+p+rxS+kZLGPD+TN1WqPEJV12JzHj4N497J05lh7BXWy84CmGfWOS35FEfKFil5iwZ9cWGp6dSI/QHjZ94ymGXnC135FEfKNil6i3o7SY9NevowP1bBvzAkMLL/M7koivtMYuUe2Tv7xNp9evBGDfNW8xSKUuomKX6OTCYYpe+SkDPriZ/QmdCN+6iN5nj/Y7lkhEaFaxm9k1ZlZqZmEzy/MqlMjxNNTXsXz6jRRs+jnr0gvodP9iuuT28zuWSMRo7hr7J8BVwFMeZBE5oYrtpdS9fDOjg1soyrmN0bf8goRAwO9YIhGlWcXunNsAYGbepBE5jpJ3ZzFg+Q/JsARWFs6g4NKb/I4kEpF0VoxEvOr9VWx+7l5GVS9kY/IgMic/y8ieZ/kdSyRinbDYzexDoMuXfOtfnHNvneyBzGwKMAUgNzf3pANKfFvzx1fpuvgRRriDFPW4jbyb/5PEpGS/Y4lEtBMWu3PuYi8O5JybBcwCyMvLc16MKbFr755dlL38IHk1H7IjoRfVE56nQHu+iJwULcVIRAk2NVLy+n9x9qYZDKWJpbm3c87kn5Cckup3NJGo0axiN7OJwGNANvCema12zl3qSTKJKy4cZu3Hc8j8y08oCO9ibVoeHSb9lkJttytyypp7VsybwJseZZE4tbHkI0KLfsSwxnWUWzdWFs5gxCU36iPsRE6TlmLEFy4cZv3S93GL/4vBDavZRybFA3/AyIkP0CM5xe94IlFNxS6t6uiSyxukLP0NZzetZy/tKTrzfgZf8SD57Tr4HU8kJqjYpVVUH9jLhgVP0W3LSwwLV7CHbIoHPsKwy6dS0Kat3/FEYoqKXVqMC4fZtm4p+//fTAbvW0SBNbAp8SyWD/0pwy67jS4600WkRajYxXO7yzax6+Pn6LLrbfqGyzniklnX8RI6nH83Z+lcdJEWp2IXT+wu28SupX8gc/u7DGwqpRuwIelsigf8KwMu/g6jO2b7HVEkbqjY5bSEQyG2rf0r+1bMo/Puj+gTLqMbUJaQS1Gve8m94GYG9tJ+LiJ+ULHLSXHhMBXb11Ox8n2Sdi2mT+1K+lFLH2dsSj6boj4P0r3ganr1HUIvv8OKxDkVu3yp+iOHKVu3hIOb/0rynhXk1H5CDvvJAfaQxZb252F9zufMwisZlN3V77gi8gUqduHI4UOUb1rJwR0rcZ+to8PBdfRq2sYACwGw2zqzK2MEO3Ly6TZyDDlnDqGLrgoViVgq9jhSV1vNZ9tLOfjpBho/30zKvg1k1W2le2g3/e3ohpt1LoWy5H6s6HY9qb0L6TH063Trkks3n7OLyMlTsceQYFMjVbt3cOCzHdRV7iC4vxyrKadt7U6yG8vpzH7O/MLzK+wMKtP6UtFpLKk5Q+jcL4+uPQcwSB81JxLVVOxRoKG+jgNVu6mpqqDuwGc0HtxD+FAlHK4kqX4vbeor6dD0OdluH13N8cUV7wO0ozKxGzszR7G9Q1+Sz+hH+x6D6Np7EN3TM+ju26sSkZaiYm8FoWCQI3WHqK+tob7uEPWHD1Jfs4/G2v0E6w4SrjuAO3KQhIZqEhtrSGqqITVYQ1qolkxXTTsO04V//BirQy6NgwkdqEnqRHnmOezI6E6gQy5pWT1p16UPnXPOpEN6BtqBRSS+xGyxu3CYUChIMNhEKNhEsKmJULCRcDBIMNREOBgkFGoi/LfvN9YTaqon2FhPuOkI4aYGQk0NhJvqcU0NuODRG8F6CDZioQYINZAQaiQhWE8gdJik0BGSQkdIDteT4o6Q5upJdfWkWSNtgePtiBJ0CRyyttRaW44E2lKfmMGhtO7sTulIOD2bQMYZJGd2Ia1DFzKyutOxc3cy2rQlo7X+QEUkakRVsS995vt0K3+XgAuRQIiACxHg/98SXYgA4aP3LUwiLfMCm1yAJhJpsGSaSKLRUmhISKMxIZWGxAwOBzoTSmxDODENl5SOS07Hjt0CqW0JpGaQktGJ1IyOtMnsREb7LNqkt6NDQoJm1yLSbFFV7IF2Xdjbpi8uIRFngaO3hEQ49piERFxCEiQcvf+3Xy1w9DmWkAiBJOzY1/7+ayCRhKRUAkkpBJLTSExOJZCUSlJKKonJaSSnppGUnEZySirJKWkkJSaSBLTx+w9ERORLRFWxj776QeBBv2OIiEQ0XWUiIhJjmlXsZvYLM9toZmvN7E0za+9VMBEROT3NnbF/AAx2zg0FNgOPND+SiIg0R7OK3Tm3yDkXPPawCMhpfiQREWkOL9fYbwXe93A8ERE5DSc8K8bMPuQfL3oE+Bfn3FvHnvMvQBB46TjjTAGmAOTm5p5WWBERObETFrtz7uLjfd/MvgOMBy5yzrnjjDMLmAWQl5f3lc8TEZHmadZ57GY2BvgecL5zrs6bSCIi0hx2nEn2iX+z2VYgBdh37EtFzrm7TuL3VQE7T/vA/skC9vodopXpNccHvebo0NM5d8JPhm9WsccbMytxzuX5naM16TXHB73m2KIrT0VEYoyKXUQkxqjYT80svwP4QK85Pug1xxCtsYuIxBjN2EVEYoyK/TSZ2UNm5swsy+8sLS2edvE0szFmtsnMtprZw37naWlm1sPM/mRm682s1Mzu9ztTazGzgJmtMrN3/c7iNRX7aTCzHsA3gV1+Z2klcbGLp5kFgMeBy4BBwPVmNsjfVC0uCDzknBsEFAD3xsFr/pv7gQ1+h2gJKvbT8xuOXnEbF29QxNEunqOBrc657c65RuBV4AqfM7Uo59xnzrmVx+4f4mjRdfc3VcszsxxgHDDb7ywtQcV+iszsCqDCObfG7yw+ieVdPLsD5V94/ClxUHJ/Y2a9gBFAsb9JWsVvOTo5C/sdpCVE1Weetpbj7WgJ/ICjyzAxxatdPCU6mVlb4A/AA865Gr/ztCQzGw9UOudWmNkFfudpCSr2L/FVO1qa2RCgN7DGzODoksRKMxvtnNvTihE959UunlGuAujxhcc5x74W08wsiaOl/pJzbq7feVrBucAEMxsLpALtzOxF59xkn3N5RuexN4OZlQF5zrlo20jolBzbxfPXHN3Fs8rvPC3FzBI5+ubwRRwt9OXADc65Ul+DtSA7OkN5DtjvnHvA7zyt7diM/bvOufF+Z/GS1tjlZMwAMoAPzGy1mT3pd6CWcOwN4qnAQo6+iTgnlkv9mHOBm4ALj/3drj42k5Uophm7iEiM0YxdRCTGqNhFRGKMil1EJMao2EVEYoyKXUQkxqjYRURijIpdRCTGqNhFRGLMfwO1npMiKcm3rQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2dce784780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHsNJREFUeJzt3Xl8lOW99/HPL5OVEMKSIEsIi4CA7AaSHGu1LhUBURStC9a64Yaojz2t9pzTvp6e0/b0dBVRlGLdN7SIGwJq60NbSCDshn0JxCAmbAkhZJmZ6/wB7eM5VbbcyT3L9/16zYuZZLju70D4ennNfV9jzjlERCR2JPgdQEREvKViFxGJMSp2EZEYo2IXEYkxKnYRkRijYhcRiTEqdhGRGKNiFxGJMSp2EZEYk+jHQbOyslyvXr38OLSISNRasWLFXudc9ome50ux9+rVi5KSEj8OLSIStcxs58k8T0sxIiIxRsUuIhJjVOwiIjFGxS4iEmNU7CIiMUbFLiISY1TsIiIxRsUuItIK6mqrKX78NqoP7G3xY6nYRURaWG3NAcoeHUte5R8oW/Vhix/PlytPRUTiRc3BfeyeMY7+TZtYNfpX5F14XYsf05NiN7My4BAQAoLOuTwvxhURiWbV+6v4/Imx9Gnaxtp/epS8S7/dKsf1csb+Dedcyy8eiYhEgYN797B35lh6BXey/rzHGXnx9a12bC3FiIh4bN/nn1Lz1Dh6hCrYcMFTDP/GpFY9vldvnjpgkZmtMLMpHo0pIhJ19u7ZxaGnxtAltJvNFz3NsFYudfBuxv4151yFmXUGPjCzjc65xV98wrHCnwKQm5vr0WFFRCJHZcUOGmaPpXN4H9svfY4h/zTWlxyezNidcxXHfq0E3gRGf8lzZjnn8pxzednZJ9wnXkQkquzZtYXG2WPoGN7PzrEvcLZPpQ4eFLuZpZtZxt/uA98EPmnuuCIi0WJ32SbCz4wlM1xN+eUvMzD/Ul/zeLEUcwbwppn9bbyXnXMLPBhXRCTiVWwvJfD8BNpwhM+umMOAkV/3O1Lzi905tx0Y5kEWEZGosmvzalJfnkgSTVROfJ3+w871OxKg0x1FRE7Lzg0rSH/tKhIIc2DSH+g7ON/vSH+nvWJERE7RjtJiMl67EoBD35pHnwgqddCMXUTklGxbu4SOc6+liSTqb5xHz36RtxKtGbuIyEnasmoxWXMn0UAKjTe9S24Eljqo2EVETsrGko/oMu9aDls64e/MJ+fMs/2O9JW0FCMicgIbihfSY/7NHEzIJPHW9+jSo6/fkY5LM3YRkeMo/et79Jx/E/sTOpJ8+4KIL3VQsYuIfKV1i9+iz6LvUBnoTJs7F9C5e2+/I50UFbuIyJdY+6c36P/RbXwW6EbGXQvJ6hI9mxeq2EVE/pfVH77CgI/vpDwxl473LKRT5+5+RzolKnYRkS9YtfA5zv7zvZQl9SH73oW0z+rid6RTpmIXETlmxXuzGbLkAbYl9afL1AVkdozOLcZV7CIiQMnbMxm+7LtsSR5IzrT3ade+k9+RTpuKXUTi3rI3H2PkikfYmDKUnve/T9t2HfyO1CwqdhGJa8Wv/4rRa/6V0tQR9Ln/Pdq0zfQ7UrPpylMRiVvFr/0n+Rt+xpq00Zw1bR6pael+R/KEil1E4lLRSz+mYMuvWNWmkEHT5pKS2sbvSJ5RsYtI3Cl6/t8o2D6dlelfZ/C010lOSfU7kqdU7CISV5Y+830Kdz5JScaFDJv2GklJyX5H8pyKXUTigguHKXrmnyksn83yzG8yYupLJMZgqYOHZ8WYWcDMVpnZu16NKSLiBRcOUzT7AQrLZ7Os/VhG3vdKzJY6eHu64/3ABg/HExFpNhcOU/zUPRTufo7ijleQd9+LBBJje7HCk2I3sxxgHDDbi/FERLzgwmGWzbyDgs9foSh7EqOnPktCIOB3rBbn1Yz9t8D3gPBXPcHMpphZiZmVVFVVeXRYEZEvFw6FWPb4LeRXvcHSM64n/+7fYQnxcU1ms1+lmY0HKp1zK473POfcLOdcnnMuLzs7OjfWEZHoEAoGKXlsMvn75rG067cpuPOJuCl18GbGfi4wwczKgFeBC83sRQ/GFRE5ZaFgkJWPXc/og/NZmnM7BXc8GlelDh4Uu3PuEedcjnOuF3Ad8Efn3ORmJxMROUXBpkZWP3oNo6oXsbTnXRTe/qu4K3XQeewiEiOaGhtY9+gkzjm8mKV9plH47X/3O5JvPC1259zHwMdejikiciIN9XWsn341I+uWUNTvIQpv/KHfkXylGbuIRLX6I4fZNP1KRhxZRvHARyj41sN+R/Kdil1EotaRw4fYOn0CwxpWUnz2D8m/5iG/I0UEFbuIRKW62mp2TL+csxvWsmz4v5M/cZrfkSKGil1Eok5tzQF2PTaeAY2lrDznZ4yecLffkSKKil1EokrNwX3snjGO/k2bWDX6l+SNu93vSBFHxS4iUaN6fxWfPzGWM5u2sbbwt+SNudnvSBFJxS4iUeHg3j3snTmWXsGdfHLe44y8+Hq/I0UsFbuIRLz9lRUcfHIsPUIVbDj/SUZceI3fkSKail1EItrePbuonTWWrqHP2XzRbIZ9/Uq/I0U8FbuIRKyq3WUcmT2W7NBetn3zGYacO97vSFFBxS4iEWlP+VaCvx9Pp/ABdl72AoMLLvU7UtRQsYtIxNldtgmeu5zMcA3l419m0KiL/I4UVVTsIhJRKraXEnj+CtpQx2dXzGHAyK/7HSnqqNhFJGKUb1lDyktXkkQTlRNfp/+wc/2OFJVU7CISEXZuXEn6qxNJIMyBSX+g7+B8vyNFrfj7aBERiTg7SovJePUKAA59ax59VOrNohm7iPhq29oldJx7LU0kUX/Dm/TsP9zvSFFPM3YR8c2WVYvJmjuJBlJonPwOuSp1T6jYRcQXG0s+osu8azls6YRvfo+cvoP9jhQzml3sZpZqZsvMbI2ZlZrZ//UimIjErg3FC8l550aqEzJJuGU+3XoP8DtSTPFijb0BuNA5V2tmScBfzOx951yRB2OLSIwp/et79F50C/sSOpFy+3w6d+/td6SY0+wZuzuq9tjDpGM319xxRST2rFv8Fn0WfYfKQGfSpixUqbcQT9bYzSxgZquBSuAD51yxF+OKSOxY+6c36P/RbewJdCPjzgVkdc31O1LM8qTYnXMh59xwIAcYbWb/8C6ImU0xsxIzK6mqqvLisCISJVZ/+AoDPr6T8sRc2t+zkE5n5PgdKaZ5elaMc+4g8CdgzJd8b5ZzLs85l5edne3lYUUkgq1c+AKD/nwvZUl9yL53AR2yuvgdKeZ5cVZMtpm1P3Y/DbgE2NjccUUk+q2Y/zRDl0xje1J/ukxdQGbHzn5HigtenBXTFXjOzAIc/Q/FHOfcux6MKyJRrOTtJxmx4mE2Jw8iZ+q7ZGR29DtS3Gh2sTvn1gIjPMgiIjFi+bwZnLPqX1mfMpTe971Dekam35HiivaKERFPLXvj1+St+zGlqSPoO+1t0tIz/I4Ud1TsIuKZ4td+Tv6Gn7ImbRRn3f8WqWnpfkeKSyp2EfFE0cv/TsHmX7K6TSEDp80lJbWN35HilopdRJqt6IUfUrDtUVamn8fgaW+QnJLqd6S4pmIXkWZZ+uzDFJbNZEXGNxh632skJaf4HSnuqdhF5LS4cJiiZ/6ZwvLZlLS7hOH3vUxiUrLfsQQVu4icBhcOUzT7AQp3P8ey9mM5Z+oLBBJVJ5FCfxMickpcOEzxU/dQ+PkrFHecwKh7nyUhEPA7lnyBil1ETpoLh1k28w4Kqt6gKOtq8u+ZjSXog9gijYpdRE5KOBRi+RO3kr9vHkWdryP/rpkq9QilYheREwoFg6yYcRP5B+eztOtNFNwxXaUewVTsInJcoWCQlY9dz+jqRSzNuY2CW3+pUo9wKnYR+UrBpkZWT7+OUYc+YmnPuyi85ed+R5KToGIXkS/V1NjAuumTyKtdzNI+91H47f/wO5KcJBW7iPyDhvo61k+/mpF1Syjq9xCFN/7Q70hyClTsIvI/1B85zKbpExlxpJiiAY9QcN3DfkeSU6RiF5G/q6+rZfP0CQyrX0Hx2T+k4JqH/I4kp0HFLiIA1NVWs/2xCQyuX8PyYT8m/6r7/Y4kp0nFLiLU1hxg12PjGdhYyspzfsaoCXf7HUmaQcUuEudqDu5j94xx9G/axKrRvyBv3B1+R5JmavZVBmbWw8z+ZGbrzazUzPT/byJRonp/FXtmjOHMps2sLfyNSj1GeDFjDwIPOedWmlkGsMLMPnDOrfdgbBFpIQf37mHvzLH0CpZR+rUZjLzkBr8jiUeaPWN3zn3mnFt57P4hYAPQvbnjikjL2V9Zwf4nLqVHcBcbzn+S4Sr1mOLphg9m1gsYARR/yfemmFmJmZVUVVV5eVgROQV79+yi5slL6RaqYPNFv2PYhdf6HUk85lmxm1lb4A/AA865mv/9fefcLOdcnnMuLzs726vDisgpqNpdRt2sMXQOVbL1m88w5OsT/Y4kLcCTs2LMLImjpf6Sc26uF2OKiLf2lG8l+PvxdAofoOyy5xlcMMbvSNJCml3sZmbA08AG59yvmx9JRLy2u2wTPHc5meEayse/xKBRF/sdSVqQF0sx5wI3ARea2epjt7EejCsiHqjYXkrCs+No62rZfcVrDFCpx7xmz9idc38BzIMsIuKx8i1rSHnpSpJppHLi65w17Fy/I0kr0JWnIjFq58aVtHn1KgKE2D9pLn0H5/sdSVqJil0kBu1Yv5x2c67GYdR8ax59Bp7jdyRpRfrgQpEYs23tEtrPmUiIAHU3vEUvlXrcUbGLxJAtq/9M1txJNJBC4+R3yO0/3O9I4gMtxYjEiE0lf6TruzdymHTcze+S03uA35HEJ5qxi8SAjcWL6P7ODRyyDOzW9+mmUo9rKnaRKFe6ZD658ydzIKEDSbcvpEtuP78jic9U7CJR7JM/v0WfhTdTFehM2h0L6Ny9t9+RJAKo2EWi1No/vUHfD29jT6Abbe9cQFa3nn5HkgihYheJQqs/epUBH99JRWIP2t+9gE5n5PgdSSKIil0kyqxc+AKDFt/DzqTeZN2zgA7ZXf2OJBFGxS4SRVbMf5qhS6axI6kfZ0xdSGanM/yOJBFIxS4SJUrefpLhxQ+xJXkg3e57n3btO/kdSSKUil0kCiyfN4ORKx5mY8oQcqfNJyOzo9+RJILpylORCLfsjV+Tt+7HlKYOp++0d0hLz/A7kkQ4FbtIBCt+7efkb/gpa9JGcdb9b5Galu53JIkCKnaRCFX08n9QsPkXrG5TyMBpc0lJbeN3JIkSKnaRCFT04o8o2PpbVqafx+Bpb5Cckup3JIkiKnaRCFP07A8oKHucFW0vYOi0OSQlp/gdSaKMJ2fFmNnvzazSzD7xYjyReOTCYZY+/V0Kyh6npN3FDLv/dZW6nBavTnd8Fhjj0VgicceFwxQ9/SCF5b9jefvLGDHtNRKTkv2OJVHKk2J3zi0G9nsxlki8ceEwxbOmUljxLMs6Xs45971EIFGrpHL69NMj4iMXDlM8cwoFVa9TnHUVo+6eTUIg4HcsiXKtduWpmU0xsxIzK6mqqmqtw4pErHAoxLLHb6Gg6nWKOn+L0fc8rVIXT7RasTvnZjnn8pxzednZ2a11WJGIFA6FKJlxE/n75rG062Ty73oSS9AOH+INLcWItLJQMMjKx25gdPVClubcSsGtv1Kpi6e8Ot3xFWApcJaZfWpmt3kxrkisCTY1surRaxlVvZClPe+i8PbfqNTFc57M2J1z13sxjkgsa2psYN30SeTVLmZp76kU3vwTvyNJjNJSjEgraKivY/30qxlZt4Sifv+Hwht/5HckiWEqdpEWVn/kMJumT2TEkWKKBzxMwXWP+B1JYpyKXaQF1dfVsnn6BIbVr6D47H8j/5rv+h1J4oCKXaSF1NVWs/2xCQyuX8PyYT8m/6r7/Y4kcULFLtICamsOsGvG5Qxs+IQVI3/GqCvu9juSxBEVu4jHDlXvp2LGOPo3bmTVqF8wavwdfkeSOKNiF/FQ9YG9fP74ZZzZtI21Bb8m77Jb/I4kcUjFLuKR6n2fU/XEZfQKlvHJudMZ+c3JfkeSOKViF/HA/soKDjw1jh7BT9lw/pOMuPBavyNJHFOxizTT3j3l1M4aS/fQZ2y+cBbDzr/K70gS51TsIs1QtbuMutnj6ByqYuslv2fI1yb4HUlExS5yuvaUbyX4+/FkhQ9QdtnzDC7Qp0NKZFCxi5yG3WWb4LnLyQzXUD7uRQaNvsTvSCJ/p2IXOUUV2zcQeP5y2lDHZ1e8yoCRF/gdSeR/ULGLnILyretIeXECyTRSOXEO/Yd9ze9IIv9AxS5yknZuXEmbV68ikRD7J82l7+B8vyOJfCkVu8hJ2LF+Oe3mXI3DqP7Wm/QZmOd3JJGvpM/kEjmBbWuX0H7OREIEqLvhLXqp1CXCqdhFjmPL6j+TNXcSDaTQOPkdcvsP9zuSyAlpKUbkK2wq+SNd372Rw6Tjbn6XnN4D/I4kclI8mbGb2Rgz22RmW83sYS/GFPHTxuJFdH/nBg5ZBtwyn24qdYkizS52MwsAjwOXAYOA681sUHPHFfHL+qXvkzt/MgcSOpB42/t07dnf70gip8SLGftoYKtzbrtzrhF4FbjCg3FFWl3pkvn0XvBtqgLZpN2xgDNyzvQ7ksgp86LYuwPlX3j86bGv/Q9mNsXMSsyspKqqyoPDinjrUPV+shfdS1Ugm/QpC8jq1tPvSCKnpdXOinHOzXLO5Tnn8rKzs1vrsCInrfTF75HlDlA//gmyuvTwO47IafOi2CuAL/4ryDn2NZGosbHkI0ZXvsHy7Kvor71fJMp5UezLgX5m1tvMkoHrgLc9GFekVRw5fIj096ZSaVkMuumXfscRabZmF7tzLghMBRYCG4A5zrnS5o4r0lrWPPcQPdxu9l78azIyO/odR6TZPLlAyTk3H5jvxVgiral0yXwKKl+jOOsq8vXpRxIjtKWAxK3amgN0+OABPrUuDPnOb/2OI+IZFbvEJRcOs3H27ZwRruTQmOm0aZvpdyQRz6jYJS4tf3M6eTUfsqzXnQzMv9TvOCKeUrFL3NmxfjlD1v6ET1KGM/qmn/gdR8RzKnaJK3W11SS8cQuHrQ1dbnmRQKI2OJXYo2KXuBEOhdg4czI5oU/Zc9F0XV0qMUvFLnGj+NnvM/LwYpb3f5DB52mfOoldKnaJCyvmP0Nh+e9Y3v4y8q//N7/jiLQoFbvEvC2rFjOo+HtsTBrE0Lt+jyXox15im37CJaaVb1lD1ls3ciChPVm3zSEltY3fkURanIpdYlbV7jISX7oaB4RunKs3SyVuqNglJlXvr6J29gTauUPsu/JlevQd4nckkVajYpeYU72/isrHx9A9VMGOi2fRb/h5fkcSaVUqdokp1fs+p+rxS+kZLGPD+TN1WqPEJV12JzHj4N497J05lh7BXWy84CmGfWOS35FEfKFil5iwZ9cWGp6dSI/QHjZ94ymGXnC135FEfKNil6i3o7SY9NevowP1bBvzAkMLL/M7koivtMYuUe2Tv7xNp9evBGDfNW8xSKUuomKX6OTCYYpe+SkDPriZ/QmdCN+6iN5nj/Y7lkhEaFaxm9k1ZlZqZmEzy/MqlMjxNNTXsXz6jRRs+jnr0gvodP9iuuT28zuWSMRo7hr7J8BVwFMeZBE5oYrtpdS9fDOjg1soyrmN0bf8goRAwO9YIhGlWcXunNsAYGbepBE5jpJ3ZzFg+Q/JsARWFs6g4NKb/I4kEpF0VoxEvOr9VWx+7l5GVS9kY/IgMic/y8ieZ/kdSyRinbDYzexDoMuXfOtfnHNvneyBzGwKMAUgNzf3pANKfFvzx1fpuvgRRriDFPW4jbyb/5PEpGS/Y4lEtBMWu3PuYi8O5JybBcwCyMvLc16MKbFr755dlL38IHk1H7IjoRfVE56nQHu+iJwULcVIRAk2NVLy+n9x9qYZDKWJpbm3c87kn5Cckup3NJGo0axiN7OJwGNANvCema12zl3qSTKJKy4cZu3Hc8j8y08oCO9ibVoeHSb9lkJttytyypp7VsybwJseZZE4tbHkI0KLfsSwxnWUWzdWFs5gxCU36iPsRE6TlmLEFy4cZv3S93GL/4vBDavZRybFA3/AyIkP0CM5xe94IlFNxS6t6uiSyxukLP0NZzetZy/tKTrzfgZf8SD57Tr4HU8kJqjYpVVUH9jLhgVP0W3LSwwLV7CHbIoHPsKwy6dS0Kat3/FEYoqKXVqMC4fZtm4p+//fTAbvW0SBNbAp8SyWD/0pwy67jS4600WkRajYxXO7yzax6+Pn6LLrbfqGyzniklnX8RI6nH83Z+lcdJEWp2IXT+wu28SupX8gc/u7DGwqpRuwIelsigf8KwMu/g6jO2b7HVEkbqjY5bSEQyG2rf0r+1bMo/Puj+gTLqMbUJaQS1Gve8m94GYG9tJ+LiJ+ULHLSXHhMBXb11Ox8n2Sdi2mT+1K+lFLH2dsSj6boj4P0r3ganr1HUIvv8OKxDkVu3yp+iOHKVu3hIOb/0rynhXk1H5CDvvJAfaQxZb252F9zufMwisZlN3V77gi8gUqduHI4UOUb1rJwR0rcZ+to8PBdfRq2sYACwGw2zqzK2MEO3Ly6TZyDDlnDqGLrgoViVgq9jhSV1vNZ9tLOfjpBho/30zKvg1k1W2le2g3/e3ohpt1LoWy5H6s6HY9qb0L6TH063Trkks3n7OLyMlTsceQYFMjVbt3cOCzHdRV7iC4vxyrKadt7U6yG8vpzH7O/MLzK+wMKtP6UtFpLKk5Q+jcL4+uPQcwSB81JxLVVOxRoKG+jgNVu6mpqqDuwGc0HtxD+FAlHK4kqX4vbeor6dD0OdluH13N8cUV7wO0ozKxGzszR7G9Q1+Sz+hH+x6D6Np7EN3TM+ju26sSkZaiYm8FoWCQI3WHqK+tob7uEPWHD1Jfs4/G2v0E6w4SrjuAO3KQhIZqEhtrSGqqITVYQ1qolkxXTTsO04V//BirQy6NgwkdqEnqRHnmOezI6E6gQy5pWT1p16UPnXPOpEN6BtqBRSS+xGyxu3CYUChIMNhEKNhEsKmJULCRcDBIMNREOBgkFGoi/LfvN9YTaqon2FhPuOkI4aYGQk0NhJvqcU0NuODRG8F6CDZioQYINZAQaiQhWE8gdJik0BGSQkdIDteT4o6Q5upJdfWkWSNtgePtiBJ0CRyyttRaW44E2lKfmMGhtO7sTulIOD2bQMYZJGd2Ia1DFzKyutOxc3cy2rQlo7X+QEUkakRVsS995vt0K3+XgAuRQIiACxHg/98SXYgA4aP3LUwiLfMCm1yAJhJpsGSaSKLRUmhISKMxIZWGxAwOBzoTSmxDODENl5SOS07Hjt0CqW0JpGaQktGJ1IyOtMnsREb7LNqkt6NDQoJm1yLSbFFV7IF2Xdjbpi8uIRFngaO3hEQ49piERFxCEiQcvf+3Xy1w9DmWkAiBJOzY1/7+ayCRhKRUAkkpBJLTSExOJZCUSlJKKonJaSSnppGUnEZySirJKWkkJSaSBLTx+w9ERORLRFWxj776QeBBv2OIiEQ0XWUiIhJjmlXsZvYLM9toZmvN7E0za+9VMBEROT3NnbF/AAx2zg0FNgOPND+SiIg0R7OK3Tm3yDkXPPawCMhpfiQREWkOL9fYbwXe93A8ERE5DSc8K8bMPuQfL3oE+Bfn3FvHnvMvQBB46TjjTAGmAOTm5p5WWBERObETFrtz7uLjfd/MvgOMBy5yzrnjjDMLmAWQl5f3lc8TEZHmadZ57GY2BvgecL5zrs6bSCIi0hx2nEn2iX+z2VYgBdh37EtFzrm7TuL3VQE7T/vA/skC9vodopXpNccHvebo0NM5d8JPhm9WsccbMytxzuX5naM16TXHB73m2KIrT0VEYoyKXUQkxqjYT80svwP4QK85Pug1xxCtsYuIxBjN2EVEYoyK/TSZ2UNm5swsy+8sLS2edvE0szFmtsnMtprZw37naWlm1sPM/mRm682s1Mzu9ztTazGzgJmtMrN3/c7iNRX7aTCzHsA3gV1+Z2klcbGLp5kFgMeBy4BBwPVmNsjfVC0uCDzknBsEFAD3xsFr/pv7gQ1+h2gJKvbT8xuOXnEbF29QxNEunqOBrc657c65RuBV4AqfM7Uo59xnzrmVx+4f4mjRdfc3VcszsxxgHDDb7ywtQcV+iszsCqDCObfG7yw+ieVdPLsD5V94/ClxUHJ/Y2a9gBFAsb9JWsVvOTo5C/sdpCVE1Weetpbj7WgJ/ICjyzAxxatdPCU6mVlb4A/AA865Gr/ztCQzGw9UOudWmNkFfudpCSr2L/FVO1qa2RCgN7DGzODoksRKMxvtnNvTihE959UunlGuAujxhcc5x74W08wsiaOl/pJzbq7feVrBucAEMxsLpALtzOxF59xkn3N5RuexN4OZlQF5zrlo20jolBzbxfPXHN3Fs8rvPC3FzBI5+ubwRRwt9OXADc65Ul+DtSA7OkN5DtjvnHvA7zyt7diM/bvOufF+Z/GS1tjlZMwAMoAPzGy1mT3pd6CWcOwN4qnAQo6+iTgnlkv9mHOBm4ALj/3drj42k5Uophm7iEiM0YxdRCTGqNhFRGKMil1EJMao2EVEYoyKXUQkxqjYRURijIpdRCTGqNhFRGLMfwO1npMiKcm3rQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2dce784780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's check that SeLU actually looks like it should\n",
    "s = SeLU()\n",
    "x = autograd.Variable(torch.from_numpy(np.arange(-5, 5, 0.05)))\n",
    "plt.plot(x.data.numpy(), s(x).data.numpy())\n",
    "plt.plot(x.data.numpy(), nn.SELU()(x).data.numpy())\n",
    "\n",
    "# If a single line shows, our implementation is roughly equivalent to that\n",
    "# in pytorch, which is a good sign"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the test below can sometimes fail due to some long encoding error. If it happens, you can try to restart the notebook and try again. If it keeps happening, just comment out the test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEST_SIZE = 1000000\n",
    "\n",
    "for p in [0.0, 0.01, 0.05, 0.1, 0.5, 0.99]:\n",
    "    # Let's check AlphaDropout vs Torch's AlphaDropout\n",
    "    inp = autograd.Variable(torch.normal(torch.zeros(TEST_SIZE), torch.ones(TEST_SIZE)))\n",
    "    init_seed = torch.initial_seed()\n",
    "    torch.manual_seed(0)\n",
    "    out1 = nn.AlphaDropout(p)(inp)\n",
    "    torch.manual_seed(0)\n",
    "    out2 = AlphaDropout(p)(inp)\n",
    "    torch.manual_seed(init_seed)\n",
    "\n",
    "\n",
    "    # If this is true, we are returning identical results to nn.AlphaDropout,\n",
    "    # which means this is probably correct.\n",
    "    # Note: the reason why we don't get exactly the same numbers is that\n",
    "    # this implementation uses the alpha and lambda given in the paper,\n",
    "    # whereas pytorch uses a lot more significant digits.\n",
    "    print(np.sum(np.abs(out1.data.numpy() - out2.data.numpy()) < 1e-4) == inp.size()[0]) \n",
    "\n",
    "    # In both cases the first number should be close to 0 and the second\n",
    "    # close to 1\n",
    "    print(out1.mean().data.numpy(), out1.std().data.numpy())\n",
    "    print(out2.mean().data.numpy(), out2.std().data.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## UCI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For the UCI data sets, the best hyperparameter setting was determined by a grid-search over all hyperparameter combinations using 15% of the training data as validation set. The early stopping parameter was determined on the smoothed learning curves of 100 epochs of the validation set. Smoothing was done using moving averages of 10 consecutive values.\n",
    "\n",
    "...\n",
    "\n",
    "> If multiple hyperparameters provided identical performance on the validation set, we preferred settings with a higher number of layers, lower learning rates and higher dropout rates.\n",
    "\n",
    "My understanding of the first statement: we run training with a validation set for 100 epochs, find the best hyperparameters and number of epochs based on the moving average of the last 10 epochs in validation accuracy (note: could be loss instead, not sure). If we have identical values, we bias towards more layers, lower learning rate, higher dropout rate and (assumption) fewer epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we write some quick code to handle pickling intermediate results in the `pickled/uci` folder. This is to not have to re-run everything if training crashes, since it takes days to finish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p pickled/uci\n",
    "\n",
    "def get_uci_cache_filename(folder, params, has_val, has_test, epochs, n_test, metric):\n",
    "    uci_name = uci_folder_to_name(folder)\n",
    "    filename = []\n",
    "    for k in sorted(params.keys()):\n",
    "        filename.append(f'{k}_{params[k]}')\n",
    "    filename += [str(has_val), str(has_test), str(epochs)]\n",
    "    if has_test:\n",
    "        filename = ['correct'] + filename\n",
    "    if n_test > 1:\n",
    "        filename = [f'tests_{n_test}'] + filename\n",
    "    if metric != 'acc':\n",
    "        filename.append(metric)\n",
    "    filename = ':'.join(filename)\n",
    "    if 'selu' in filename and has_test:\n",
    "        filename += 'v2'  # To handle a fix in the implementation of SeLU\n",
    "    return f'pickled/uci/{uci_name}/{filename}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the code to evaluate, fit and extract the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def score_key(score):\n",
    "    # Note: we assume that we sort in a descending way\n",
    "    param, epochs, score = score\n",
    "    return (score, param['layers'], -param['lr'], param.get('dropout', 0.0), -epochs)\n",
    "\n",
    "def evaluate(net, val_loader, metric='acc'):\n",
    "    net.eval()\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    all_labels = []\n",
    "    all_preds = []\n",
    "    for batch, labels in val_loader:\n",
    "        batch = w(autograd.Variable(batch))\n",
    "        labels = labels.cuda()\n",
    "        outputs = net(batch)\n",
    "        _, predictions = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predictions == labels).sum()\n",
    "        all_labels += list(labels)\n",
    "        all_preds += list(predictions)\n",
    "    if metric == 'acc':\n",
    "        return correct / total\n",
    "    if metric == 'auc':\n",
    "        return roc_auc_score(all_labels, all_preds)\n",
    "\n",
    "class Uncached(Exception):\n",
    "    pass\n",
    "\n",
    "N_TEST = 10\n",
    "\n",
    "def fit(folder, features, classes, params, train_loader, val_loader, test_loader, epochs=100, force_cached=False, metric='acc'):\n",
    "    if val_loader is None and test_loader is not None:\n",
    "        n_test = N_TEST\n",
    "    else:\n",
    "        n_test = 1\n",
    "    filename = get_uci_cache_filename(folder, params, val_loader is not None, test_loader is not None, epochs, n_test, metric)\n",
    "    if os.path.exists(filename):\n",
    "        try:\n",
    "            results = pickle.load(open(filename, 'rb'))\n",
    "        except Exception:\n",
    "            results = (1,)\n",
    "        if results[0] == 0:  # version number\n",
    "            comb, test_acc = results[1:]\n",
    "            if not isinstance(test_acc, list):\n",
    "                test_acc = [test_acc]\n",
    "            return comb, test_acc\n",
    "    if force_cached:\n",
    "        raise Uncached\n",
    "    \n",
    "    all_combinations = []\n",
    "    all_test_accs = []\n",
    "    \n",
    "    for _ in range(n_test):\n",
    "        net = w(FNN(features, classes, params))\n",
    "        criterion = w(nn.NLLLoss())\n",
    "        optimizer = optim.SGD(net.parameters(), lr=params['lr'], weight_decay=params.get('l2', 0.0))\n",
    "\n",
    "        for epoch in range(epochs):\n",
    "            net.train()\n",
    "            for batch, labels in train_loader:\n",
    "                batch = w(autograd.Variable(batch))\n",
    "                labels = w(autograd.Variable(labels))\n",
    "                optimizer.zero_grad()  # TODO: check if this is useful.\n",
    "                outputs = net(batch)\n",
    "                loss = criterion(outputs, labels)\n",
    "                loss.backward()\n",
    "                optimizer.step()\n",
    "            if val_loader is not None:\n",
    "                accuracy = evaluate(net, val_loader, metric)\n",
    "                all_combinations.append((params, epoch + 1, accuracy))\n",
    "\n",
    "        test_acc = 0.0\n",
    "        if test_loader is not None:\n",
    "            test_acc = evaluate(net, test_loader, metric)\n",
    "            all_test_accs.append(test_acc)\n",
    "    \n",
    "    pickle.dump((0, all_combinations, all_test_accs), open(filename, 'wb'))\n",
    "    return all_combinations, all_test_accs\n",
    "\n",
    "def make_loader(X, y, transformer=None, batch_size=BATCH_SIZE):\n",
    "    if transformer is None:\n",
    "        transformer = sklearn.preprocessing.StandardScaler()\n",
    "        transformer.fit(X)\n",
    "    X = transformer.transform(X)\n",
    "    return DataLoader(\n",
    "        dataset=TensorDataset(*[w(torch.from_numpy(e)) for e in [X, y]]),\n",
    "        batch_size=batch_size,\n",
    "        shuffle=transformer is None\n",
    "    ), transformer\n",
    "\n",
    "def get_performance(X, y, Xtest, ytest, parameters, folder, variant, force_cached=False, metric='acc', train_val=None, force_depth=None, return_combs=False):\n",
    "    \n",
    "    features = X.shape[-1]\n",
    "    classes = len(set(y))\n",
    "\n",
    "    if train_val is not None:\n",
    "        Xtrain, Xval, ytrain, yval = train_val\n",
    "    else:\n",
    "        Xtrain, Xval, ytrain, yval = uci_validation_set(X, y)\n",
    "    batch_size = BATCH_SIZE\n",
    "    if len(ytrain) > 1000:\n",
    "        batch_size = 512\n",
    "    \n",
    "    train_loader_small, transformer = make_loader(Xtrain, ytrain, batch_size=batch_size)\n",
    "    val_loader, _ = make_loader(Xval, yval, transformer, batch_size=batch_size)\n",
    "    train_loader_big, transformer = make_loader(X, y, batch_size=batch_size)\n",
    "    test_loader, _ = make_loader(Xtest, ytest, transformer, batch_size=batch_size)\n",
    "    \n",
    "    all_combinations = []\n",
    "    all_params = list(sklearn.model_selection.ParameterGrid(parameters))\n",
    "    random.shuffle(all_params)\n",
    "    for params in tqdm(all_params, desc=variant):\n",
    "        if force_depth is not None and params['layers'] != force_depth:\n",
    "            continue\n",
    "        if 'bottleneck' in params and params['form'] == 'rect':\n",
    "            del params['bottleneck']  # A rectangular resnet doesn't have a bottleneck\n",
    "        comb_list = fit(folder, features, classes, params, train_loader_small, val_loader, None, force_cached=force_cached, metric=metric)[0]\n",
    "        # The comb_list at this point does not have a smoothed learning curve, make it so\n",
    "        new_comb_list = []\n",
    "        last_10 = [0.0] * 10\n",
    "        for params, epochs, val_score in comb_list:\n",
    "            last_10.pop(0)\n",
    "            last_10.append(val_score)\n",
    "            assert len(last_10) == 10\n",
    "            new_comb_list.append((params, epochs, np.mean(last_10)))\n",
    "        all_combinations += new_comb_list\n",
    "\n",
    "    if len(all_combinations) == 0:\n",
    "        return {}, -1, -1, -1\n",
    "    best_params, best_epochs, val_score = max(all_combinations, key=score_key)\n",
    "    return (\n",
    "        best_params, best_epochs, val_score,\n",
    "        fit(folder, features, classes, best_params, train_loader_big, None, test_loader, best_epochs, metric=metric)[1]\n",
    "    )\n",
    "    \n",
    "\n",
    "def process_folder(folder, force_cached=False):\n",
    "    X, y = load_uci_dataset(folder)\n",
    "    \n",
    "    print(f'{uci_folder_to_name(folder)}: {len(y)} training samples')\n",
    "    Xtest, ytest = load_uci_dataset(folder, False)\n",
    "\n",
    "    try:\n",
    "        os.makedirs(f'pickled/uci/{uci_folder_to_name(folder)}')\n",
    "    except Exception:\n",
    "        pass\n",
    "        \n",
    "    perf = {}\n",
    "    for variant in tqdm(random.sample(list(UCI_HYPER), len(UCI_HYPER)), uci_folder_to_name(folder)):\n",
    "        if not force_cached:\n",
    "            cur_name = folder + '::' + variant\n",
    "            bucket = int(hashlib.md5(cur_name.encode('utf8')).hexdigest(), 16) % N_BUCKETS\n",
    "            if bucket != BUCKET:\n",
    "                continue\n",
    "        perf[variant] = get_performance(X, y, Xtest, ytest, UCI_HYPER[variant], folder, variant, force_cached)\n",
    "    return perf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SHOW_TQDM = True\n",
    "for folder in tqdm(uci_folders, 'Everything'):\n",
    "    with torch.cuda.device(CUDA_DEVICE):\n",
    "        SHOW_TQDM = False\n",
    "        process_folder(folder)\n",
    "        SHOW_TQDM = True\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we extract all the data from the pickle files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def processed_to_pandas_format(processed):\n",
    "    KEY_MAP = {\n",
    "        'BatchNorm': 'BN',\n",
    "        'LayerNorm': 'LN',\n",
    "        'MSRAinit': 'MS',\n",
    "        'SNN': 'SNN',\n",
    "        'WeightNorm': 'WN',\n",
    "        'Highway': 'HW',\n",
    "        'ResNet': 'ResNet'\n",
    "    }\n",
    "    res = {}\n",
    "    for k in processed:\n",
    "        res[KEY_MAP[k]] = processed[k][-1]\n",
    "    return res\n",
    "\n",
    "all_perf = {}\n",
    "SHOW_TQDM = True\n",
    "for folder in tqdm(uci_folders, 'Everything'):\n",
    "    SHOW_TQDM = False\n",
    "    try:\n",
    "        all_perf[uci_folder_to_name(folder)] = processed_to_pandas_format(process_folder(folder, True))\n",
    "    except Uncached:\n",
    "        pass\n",
    "    \n",
    "SHOW_TQDM = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`UCI_details.csv` is a file containing data extracted from table S8 from the paper supplement. This contains the details of the performance of the different algorithms as given by the paper. This will allow us to reproduce the other tables based on the original performance data as well as to show the amount of difference that our data gets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected = pd.read_csv('UCI_details.csv', index_col=0).transpose().to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the different metrics for both the old data and the newly computed data. We used 10 iterations on the test set in case I found out that the mean or max accuracy was needed, but in the end we only use the first accuracy, which is probably what was done in the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "overlap = {}\n",
    "old_ranks = collections.defaultdict(lambda: collections.defaultdict(list))\n",
    "new_ranks = collections.defaultdict(lambda: collections.defaultdict(list))\n",
    "\n",
    "AGGREGATOR = lambda x: x[0]\n",
    "\n",
    "for k in expected:\n",
    "    if k not in all_perf:\n",
    "        continue\n",
    "    tmp = {}\n",
    "    old_order = []\n",
    "    new_order = []\n",
    "    for l in expected[k]:\n",
    "        if l not in all_perf.get(k, {}):\n",
    "            tmp[l] = expected[k][l]\n",
    "        else:\n",
    "            new_order.append((AGGREGATOR(all_perf[k][l]), l))\n",
    "            cur_perf = AGGREGATOR(all_perf[k][l])\n",
    "            tmp[l + ' (old/new)'] = f'{expected[k][l]:.4f}/{cur_perf:.4f} ({100*(cur_perf/(expected[k][l]+1e-8) - 1):+.1f}%)'\n",
    "            old_order.append((expected[k][l], l))\n",
    "    overlap[k] = tmp\n",
    "    old_order.sort(reverse=True)\n",
    "    new_order.sort(reverse=True)\n",
    "    for i, (_, name) in enumerate(old_order):\n",
    "        old_ranks[expected[k]['N'] >= 1000][name].append(i + 1)\n",
    "    for i, (_, name) in enumerate(new_order):\n",
    "        new_ranks[expected[k]['N'] >= 1000][name].append(i + 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table S8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First final reproduction: a version of table S8, with some changes to show the difference in accuracy between the original paper and my reproduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BN (old/new)</th>\n",
       "      <th>HW (old/new)</th>\n",
       "      <th>LN (old/new)</th>\n",
       "      <th>M</th>\n",
       "      <th>MS (old/new)</th>\n",
       "      <th>N</th>\n",
       "      <th>ResNet (old/new)</th>\n",
       "      <th>SNN (old/new)</th>\n",
       "      <th>WN (old/new)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>abalone</th>\n",
       "      <td>0.6303/0.5575 (-11.6%)</td>\n",
       "      <td>0.6427/0.5383 (-16.2%)</td>\n",
       "      <td>0.6178/0.6049 (-2.1%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.6284/0.6130 (-2.4%)</td>\n",
       "      <td>4177</td>\n",
       "      <td>0.6466/0.3463 (-46.4%)</td>\n",
       "      <td>0.6657/0.3166 (-52.4%)</td>\n",
       "      <td>0.6351/0.5465 (-14.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acute-inflammation</th>\n",
       "      <td>1.0000/0.9667 (-3.3%)</td>\n",
       "      <td>1.0000/0.6167 (-38.3%)</td>\n",
       "      <td>0.9000/1.0000 (+11.1%)</td>\n",
       "      <td>7</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>120</td>\n",
       "      <td>1.0000/0.7000 (-30.0%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acute-nephritis</th>\n",
       "      <td>1.0000/0.9333 (-6.7%)</td>\n",
       "      <td>1.0000/0.5833 (-41.7%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>7</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>120</td>\n",
       "      <td>1.0000/0.5833 (-41.7%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>adult</th>\n",
       "      <td>0.8499/0.8522 (+0.3%)</td>\n",
       "      <td>0.8453/0.8524 (+0.8%)</td>\n",
       "      <td>0.8517/0.8522 (+0.1%)</td>\n",
       "      <td>15</td>\n",
       "      <td>0.8487/0.8511 (+0.3%)</td>\n",
       "      <td>48842</td>\n",
       "      <td>0.8484/0.8502 (+0.2%)</td>\n",
       "      <td>0.8476/0.8527 (+0.6%)</td>\n",
       "      <td>0.8453/0.8492 (+0.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>annealing</th>\n",
       "      <td>0.1200/0.0600 (-50.0%)</td>\n",
       "      <td>0.3600/0.6500 (+80.6%)</td>\n",
       "      <td>0.5000/0.4700 (-6.0%)</td>\n",
       "      <td>32</td>\n",
       "      <td>0.7300/0.7500 (+2.7%)</td>\n",
       "      <td>898</td>\n",
       "      <td>0.2600/0.7400 (+184.6%)</td>\n",
       "      <td>0.7600/0.1300 (-82.9%)</td>\n",
       "      <td>0.6500/0.7700 (+18.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>arrhythmia</th>\n",
       "      <td>0.5929/0.5044 (-14.9%)</td>\n",
       "      <td>0.6283/0.6062 (-3.5%)</td>\n",
       "      <td>0.5752/0.6106 (+6.2%)</td>\n",
       "      <td>263</td>\n",
       "      <td>0.6372/0.5929 (-6.9%)</td>\n",
       "      <td>452</td>\n",
       "      <td>0.6460/0.4159 (-35.6%)</td>\n",
       "      <td>0.6549/0.5973 (-8.8%)</td>\n",
       "      <td>0.6018/0.5796 (-3.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>audiology-std</th>\n",
       "      <td>0.6400/0.5200 (-18.8%)</td>\n",
       "      <td>0.7200/0.6400 (-11.1%)</td>\n",
       "      <td>0.8000/0.8000 (-0.0%)</td>\n",
       "      <td>60</td>\n",
       "      <td>0.6800/0.7200 (+5.9%)</td>\n",
       "      <td>196</td>\n",
       "      <td>0.8000/0.5200 (-35.0%)</td>\n",
       "      <td>0.8000/0.6800 (-15.0%)</td>\n",
       "      <td>0.7200/0.7200 (-0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balance-scale</th>\n",
       "      <td>0.9231/0.4712 (-49.0%)</td>\n",
       "      <td>0.9103/0.9679 (+6.3%)</td>\n",
       "      <td>0.9872/0.9359 (-5.2%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.9231/0.9615 (+4.2%)</td>\n",
       "      <td>625</td>\n",
       "      <td>0.9167/0.4744 (-48.3%)</td>\n",
       "      <td>0.9231/0.9679 (+4.9%)</td>\n",
       "      <td>0.9551/0.9295 (-2.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balloons</th>\n",
       "      <td>1.0000/0.3750 (-62.5%)</td>\n",
       "      <td>0.2500/0.6250 (+150.0%)</td>\n",
       "      <td>0.7500/0.5000 (-33.3%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.5000/0.5000 (-0.0%)</td>\n",
       "      <td>16</td>\n",
       "      <td>1.0000/0.8750 (-12.5%)</td>\n",
       "      <td>1.0000/0.3750 (-62.5%)</td>\n",
       "      <td>0.0000/0.7500 (+7499999900.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bank</th>\n",
       "      <td>0.8823/0.8850 (+0.3%)</td>\n",
       "      <td>0.8885/0.8221 (-7.5%)</td>\n",
       "      <td>0.8920/0.8735 (-2.1%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.8876/0.8854 (-0.2%)</td>\n",
       "      <td>4521</td>\n",
       "      <td>0.8796/0.8801 (+0.1%)</td>\n",
       "      <td>0.8903/0.8677 (-2.5%)</td>\n",
       "      <td>0.8850/0.8664 (-2.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>blood</th>\n",
       "      <td>0.7647/0.7567 (-1.0%)</td>\n",
       "      <td>0.7968/0.6765 (-15.1%)</td>\n",
       "      <td>0.7112/0.2380 (-66.5%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.7754/0.2380 (-69.3%)</td>\n",
       "      <td>748</td>\n",
       "      <td>0.8021/0.5963 (-25.7%)</td>\n",
       "      <td>0.7701/0.7674 (-0.4%)</td>\n",
       "      <td>0.7594/0.7086 (-6.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer</th>\n",
       "      <td>0.7324/0.2937 (-59.9%)</td>\n",
       "      <td>0.7465/0.6573 (-11.9%)</td>\n",
       "      <td>0.6620/0.6224 (-6.0%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.6901/0.6713 (-2.7%)</td>\n",
       "      <td>286</td>\n",
       "      <td>0.7465/0.7203 (-3.5%)</td>\n",
       "      <td>0.7183/0.5734 (-20.2%)</td>\n",
       "      <td>0.6197/0.5734 (-7.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer-wisc</th>\n",
       "      <td>0.9829/0.3438 (-65.0%)</td>\n",
       "      <td>0.9771/0.9284 (-5.0%)</td>\n",
       "      <td>0.9714/0.9599 (-1.2%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.9714/0.9599 (-1.2%)</td>\n",
       "      <td>699</td>\n",
       "      <td>0.9714/0.4298 (-55.8%)</td>\n",
       "      <td>0.9714/0.9484 (-2.4%)</td>\n",
       "      <td>0.9657/0.9570 (-0.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer-wisc-diag</th>\n",
       "      <td>0.9789/0.8873 (-9.4%)</td>\n",
       "      <td>0.9789/0.9754 (-0.4%)</td>\n",
       "      <td>0.9648/0.9683 (+0.4%)</td>\n",
       "      <td>31</td>\n",
       "      <td>0.9718/0.9472 (-2.5%)</td>\n",
       "      <td>569</td>\n",
       "      <td>0.9507/0.7430 (-21.9%)</td>\n",
       "      <td>0.9789/0.9859 (+0.7%)</td>\n",
       "      <td>0.9718/0.9824 (+1.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer-wisc-prog</th>\n",
       "      <td>0.7755/0.5556 (-28.4%)</td>\n",
       "      <td>0.8367/0.6869 (-17.9%)</td>\n",
       "      <td>0.7959/0.5758 (-27.7%)</td>\n",
       "      <td>34</td>\n",
       "      <td>0.7347/0.6162 (-16.1%)</td>\n",
       "      <td>198</td>\n",
       "      <td>0.8163/0.6162 (-24.5%)</td>\n",
       "      <td>0.6735/0.7071 (+5.0%)</td>\n",
       "      <td>0.8367/0.6768 (-19.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-tissue</th>\n",
       "      <td>0.4615/0.7358 (+59.4%)</td>\n",
       "      <td>0.6154/0.5849 (-5.0%)</td>\n",
       "      <td>0.5769/0.4528 (-21.5%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.4615/0.6792 (+47.2%)</td>\n",
       "      <td>106</td>\n",
       "      <td>0.4231/0.6792 (+60.5%)</td>\n",
       "      <td>0.7308/0.6981 (-4.5%)</td>\n",
       "      <td>0.5385/0.6038 (+12.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <td>0.9606/0.1146 (-88.1%)</td>\n",
       "      <td>0.9560/0.9664 (+1.1%)</td>\n",
       "      <td>0.9907/0.9826 (-0.8%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.9861/0.9213 (-6.6%)</td>\n",
       "      <td>1728</td>\n",
       "      <td>0.9282/0.1493 (-83.9%)</td>\n",
       "      <td>0.9838/0.7014 (-28.7%)</td>\n",
       "      <td>0.9769/0.9826 (+0.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cardiotocography-10clases</th>\n",
       "      <td>0.7910/0.0941 (-88.1%)</td>\n",
       "      <td>0.8456/0.7582 (-10.3%)</td>\n",
       "      <td>0.8362/0.8109 (-3.0%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.8418/0.5814 (-30.9%)</td>\n",
       "      <td>2126</td>\n",
       "      <td>0.8173/0.2427 (-70.3%)</td>\n",
       "      <td>0.8399/0.7846 (-6.6%)</td>\n",
       "      <td>0.8606/0.7752 (-9.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cardiotocography-3clases</th>\n",
       "      <td>0.9096/0.7789 (-14.4%)</td>\n",
       "      <td>0.9171/0.8965 (-2.2%)</td>\n",
       "      <td>0.9021/0.9040 (+0.2%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.8964/0.9106 (+1.6%)</td>\n",
       "      <td>2126</td>\n",
       "      <td>0.9021/0.7893 (-12.5%)</td>\n",
       "      <td>0.9153/0.9219 (+0.7%)</td>\n",
       "      <td>0.8945/0.1383 (-84.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chess-krvk</th>\n",
       "      <td>0.8781/0.0139 (-98.4%)</td>\n",
       "      <td>0.5255/0.1500 (-71.5%)</td>\n",
       "      <td>0.8938/0.1495 (-83.3%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.8606/0.1823 (-78.8%)</td>\n",
       "      <td>28056</td>\n",
       "      <td>0.8543/0.1647 (-80.7%)</td>\n",
       "      <td>0.8805/0.1625 (-81.5%)</td>\n",
       "      <td>0.7673/0.1625 (-78.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chess-krvkp</th>\n",
       "      <td>0.9862/0.5225 (-47.0%)</td>\n",
       "      <td>0.9900/0.9837 (-0.6%)</td>\n",
       "      <td>0.9875/0.9762 (-1.1%)</td>\n",
       "      <td>37</td>\n",
       "      <td>0.9900/0.5225 (-47.2%)</td>\n",
       "      <td>3196</td>\n",
       "      <td>0.9912/0.5763 (-41.9%)</td>\n",
       "      <td>0.9912/0.5225 (-47.3%)</td>\n",
       "      <td>0.9912/0.9781 (-1.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>congressional-voting</th>\n",
       "      <td>0.5872/0.3871 (-34.1%)</td>\n",
       "      <td>0.5872/0.5115 (-12.9%)</td>\n",
       "      <td>0.5780/0.5576 (-3.5%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.6055/0.5253 (-13.2%)</td>\n",
       "      <td>435</td>\n",
       "      <td>0.5963/0.4286 (-28.1%)</td>\n",
       "      <td>0.6147/0.3871 (-37.0%)</td>\n",
       "      <td>0.5872/0.4931 (-16.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>conn-bench-sonar-mines-rocks</th>\n",
       "      <td>0.7115/0.4712 (-33.8%)</td>\n",
       "      <td>0.8462/0.8750 (+3.4%)</td>\n",
       "      <td>0.6731/0.8077 (+20.0%)</td>\n",
       "      <td>61</td>\n",
       "      <td>0.8269/0.8365 (+1.2%)</td>\n",
       "      <td>208</td>\n",
       "      <td>0.8077/0.6731 (-16.7%)</td>\n",
       "      <td>0.7885/0.7692 (-2.4%)</td>\n",
       "      <td>0.8269/0.4615 (-44.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>conn-bench-vowel-deterding</th>\n",
       "      <td>0.9610/0.9004 (-6.3%)</td>\n",
       "      <td>0.9784/0.9870 (+0.9%)</td>\n",
       "      <td>0.9935/0.7468 (-24.8%)</td>\n",
       "      <td>12</td>\n",
       "      <td>0.9935/0.9978 (+0.4%)</td>\n",
       "      <td>990</td>\n",
       "      <td>0.9935/0.8485 (-14.6%)</td>\n",
       "      <td>0.9957/0.9978 (+0.2%)</td>\n",
       "      <td>0.9524/0.7273 (-23.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>connect-4</th>\n",
       "      <td>0.8729/0.2462 (-71.8%)</td>\n",
       "      <td>0.8599/0.3112 (-63.8%)</td>\n",
       "      <td>0.8856/0.2462 (-72.2%)</td>\n",
       "      <td>43</td>\n",
       "      <td>0.8831/0.3711 (-58.0%)</td>\n",
       "      <td>67557</td>\n",
       "      <td>0.8716/0.2940 (-66.3%)</td>\n",
       "      <td>0.8807/0.4176 (-52.6%)</td>\n",
       "      <td>0.8833/0.3747 (-57.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>contrac</th>\n",
       "      <td>0.4538/0.4348 (-4.2%)</td>\n",
       "      <td>0.5054/0.4321 (-14.5%)</td>\n",
       "      <td>0.4592/0.4864 (+5.9%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.5136/0.4946 (-3.7%)</td>\n",
       "      <td>1473</td>\n",
       "      <td>0.5136/0.3465 (-32.5%)</td>\n",
       "      <td>0.5190/0.5177 (-0.3%)</td>\n",
       "      <td>0.4755/0.4878 (+2.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit-approval</th>\n",
       "      <td>0.8721/0.7768 (-10.9%)</td>\n",
       "      <td>0.8547/0.8493 (-0.6%)</td>\n",
       "      <td>0.8547/0.8348 (-2.3%)</td>\n",
       "      <td>16</td>\n",
       "      <td>0.8430/0.8087 (-4.1%)</td>\n",
       "      <td>690</td>\n",
       "      <td>0.8430/0.7449 (-11.6%)</td>\n",
       "      <td>0.8430/0.8841 (+4.9%)</td>\n",
       "      <td>0.9070/0.7884 (-13.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinder-bands</th>\n",
       "      <td>0.7500/0.6172 (-17.7%)</td>\n",
       "      <td>0.7969/0.7656 (-3.9%)</td>\n",
       "      <td>0.7578/0.7461 (-1.5%)</td>\n",
       "      <td>36</td>\n",
       "      <td>0.7656/0.7031 (-8.2%)</td>\n",
       "      <td>512</td>\n",
       "      <td>0.7734/0.7305 (-5.6%)</td>\n",
       "      <td>0.7266/0.7461 (+2.7%)</td>\n",
       "      <td>0.7578/0.7305 (-3.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dermatology</th>\n",
       "      <td>0.9341/0.3716 (-60.2%)</td>\n",
       "      <td>0.9780/0.9563 (-2.2%)</td>\n",
       "      <td>0.9451/0.9563 (+1.2%)</td>\n",
       "      <td>35</td>\n",
       "      <td>0.9121/0.9672 (+6.0%)</td>\n",
       "      <td>366</td>\n",
       "      <td>0.9231/0.9016 (-2.3%)</td>\n",
       "      <td>0.9231/0.9781 (+6.0%)</td>\n",
       "      <td>0.9451/0.9727 (+2.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>echocardiogram</th>\n",
       "      <td>0.8485/0.6462 (-23.8%)</td>\n",
       "      <td>0.6061/0.3231 (-46.7%)</td>\n",
       "      <td>0.8182/0.7538 (-7.9%)</td>\n",
       "      <td>11</td>\n",
       "      <td>0.8485/0.6923 (-18.4%)</td>\n",
       "      <td>131</td>\n",
       "      <td>0.8485/0.6615 (-22.0%)</td>\n",
       "      <td>0.8182/0.7231 (-11.6%)</td>\n",
       "      <td>0.7879/0.3231 (-59.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ecoli</th>\n",
       "      <td>0.8214/0.7381 (-10.1%)</td>\n",
       "      <td>0.8690/0.8690 (+0.0%)</td>\n",
       "      <td>0.8571/0.8452 (-1.4%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8333/0.8333 (+0.0%)</td>\n",
       "      <td>336</td>\n",
       "      <td>0.8214/0.5833 (-29.0%)</td>\n",
       "      <td>0.8929/0.8631 (-3.3%)</td>\n",
       "      <td>0.8452/0.8571 (+1.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>energy-y1</th>\n",
       "      <td>0.8646/0.3516 (-59.3%)</td>\n",
       "      <td>0.8802/0.9349 (+6.2%)</td>\n",
       "      <td>0.9479/0.8021 (-15.4%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.9583/0.9089 (-5.2%)</td>\n",
       "      <td>768</td>\n",
       "      <td>0.8177/0.1667 (-79.6%)</td>\n",
       "      <td>0.9583/0.1771 (-81.5%)</td>\n",
       "      <td>0.9010/0.9219 (+2.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>energy-y2</th>\n",
       "      <td>0.8750/0.2448 (-72.0%)</td>\n",
       "      <td>0.9010/0.8438 (-6.4%)</td>\n",
       "      <td>0.8802/0.8880 (+0.9%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.8958/0.9089 (+1.5%)</td>\n",
       "      <td>768</td>\n",
       "      <td>0.8750/0.6849 (-21.7%)</td>\n",
       "      <td>0.9063/0.9323 (+2.9%)</td>\n",
       "      <td>0.8906/0.8958 (+0.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fertility</th>\n",
       "      <td>0.6800/0.8800 (+29.4%)</td>\n",
       "      <td>0.8800/0.8800 (-0.0%)</td>\n",
       "      <td>0.8800/0.8800 (-0.0%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.8800/0.1200 (-86.4%)</td>\n",
       "      <td>100</td>\n",
       "      <td>0.8400/0.8800 (+4.8%)</td>\n",
       "      <td>0.9200/0.8800 (-4.3%)</td>\n",
       "      <td>0.6800/0.8800 (+29.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flags</th>\n",
       "      <td>0.4167/0.4845 (+16.3%)</td>\n",
       "      <td>0.4375/0.4742 (+8.4%)</td>\n",
       "      <td>0.3542/0.1443 (-59.3%)</td>\n",
       "      <td>29</td>\n",
       "      <td>0.4583/0.5464 (+19.2%)</td>\n",
       "      <td>194</td>\n",
       "      <td>0.3750/0.4021 (+7.2%)</td>\n",
       "      <td>0.4583/0.4742 (+3.5%)</td>\n",
       "      <td>0.4167/0.5670 (+36.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>glass</th>\n",
       "      <td>0.5849/0.5888 (+0.7%)</td>\n",
       "      <td>0.6415/0.6075 (-5.3%)</td>\n",
       "      <td>0.6981/0.5421 (-22.4%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.6038/0.6449 (+6.8%)</td>\n",
       "      <td>214</td>\n",
       "      <td>0.6415/0.4766 (-25.7%)</td>\n",
       "      <td>0.7358/0.7103 (-3.5%)</td>\n",
       "      <td>0.6792/0.6449 (-5.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>haberman-survival</th>\n",
       "      <td>0.7368/0.6275 (-14.8%)</td>\n",
       "      <td>0.6447/0.2614 (-59.4%)</td>\n",
       "      <td>0.6842/0.6275 (-8.3%)</td>\n",
       "      <td>4</td>\n",
       "      <td>0.7237/0.6993 (-3.4%)</td>\n",
       "      <td>306</td>\n",
       "      <td>0.6842/0.3660 (-46.5%)</td>\n",
       "      <td>0.7368/0.6732 (-8.6%)</td>\n",
       "      <td>0.7500/0.6601 (-12.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hayes-roth</th>\n",
       "      <td>0.7500/0.0357 (-95.2%)</td>\n",
       "      <td>0.7857/0.2143 (-72.7%)</td>\n",
       "      <td>0.8929/0.4643 (-48.0%)</td>\n",
       "      <td>4</td>\n",
       "      <td>0.4643/0.0357 (-92.3%)</td>\n",
       "      <td>160</td>\n",
       "      <td>0.7143/0.0357 (-95.0%)</td>\n",
       "      <td>0.6786/0.0357 (-94.7%)</td>\n",
       "      <td>0.5714/0.4643 (-18.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-cleveland</th>\n",
       "      <td>0.5789/0.3510 (-39.4%)</td>\n",
       "      <td>0.6316/0.5497 (-13.0%)</td>\n",
       "      <td>0.5789/0.5364 (-7.3%)</td>\n",
       "      <td>14</td>\n",
       "      <td>0.6053/0.5232 (-13.6%)</td>\n",
       "      <td>303</td>\n",
       "      <td>0.5658/0.6093 (+7.7%)</td>\n",
       "      <td>0.6184/0.5828 (-5.8%)</td>\n",
       "      <td>0.5658/0.5960 (+5.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-hungarian</th>\n",
       "      <td>0.8493/0.7551 (-11.1%)</td>\n",
       "      <td>0.7945/0.7891 (-0.7%)</td>\n",
       "      <td>0.8493/0.7891 (-7.1%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.8356/0.8503 (+1.8%)</td>\n",
       "      <td>294</td>\n",
       "      <td>0.8082/0.7347 (-9.1%)</td>\n",
       "      <td>0.7945/0.7959 (+0.2%)</td>\n",
       "      <td>0.7534/0.8299 (+10.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-switzerland</th>\n",
       "      <td>0.3871/0.0656 (-83.1%)</td>\n",
       "      <td>0.5806/0.4098 (-29.4%)</td>\n",
       "      <td>0.5161/0.4754 (-7.9%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.3871/0.3607 (-6.8%)</td>\n",
       "      <td>123</td>\n",
       "      <td>0.3226/0.3607 (+11.8%)</td>\n",
       "      <td>0.3548/0.0656 (-81.5%)</td>\n",
       "      <td>0.2581/0.3115 (+20.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-va</th>\n",
       "      <td>0.2800/0.2800 (-0.0%)</td>\n",
       "      <td>0.4000/0.3100 (-22.5%)</td>\n",
       "      <td>0.2400/0.2900 (+20.8%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.2600/0.3000 (+15.4%)</td>\n",
       "      <td>200</td>\n",
       "      <td>0.2600/0.3400 (+30.8%)</td>\n",
       "      <td>0.3600/0.2600 (-27.8%)</td>\n",
       "      <td>0.2200/0.3100 (+40.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hepatitis</th>\n",
       "      <td>0.8718/0.7922 (-9.1%)</td>\n",
       "      <td>0.6667/0.7922 (+18.8%)</td>\n",
       "      <td>0.7436/0.7922 (+6.5%)</td>\n",
       "      <td>20</td>\n",
       "      <td>0.7692/0.7922 (+3.0%)</td>\n",
       "      <td>155</td>\n",
       "      <td>0.7692/0.7922 (+3.0%)</td>\n",
       "      <td>0.7692/0.7922 (+3.0%)</td>\n",
       "      <td>0.8462/0.7922 (-6.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hill-valley</th>\n",
       "      <td>0.5050/0.5693 (+12.7%)</td>\n",
       "      <td>0.5000/0.4934 (-1.3%)</td>\n",
       "      <td>0.5050/0.6287 (+24.5%)</td>\n",
       "      <td>101</td>\n",
       "      <td>0.5116/0.7030 (+37.4%)</td>\n",
       "      <td>1212</td>\n",
       "      <td>0.5396/0.5776 (+7.0%)</td>\n",
       "      <td>0.5248/0.5627 (+7.2%)</td>\n",
       "      <td>0.4934/0.5759 (+16.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horse-colic</th>\n",
       "      <td>0.8529/0.6029 (-29.3%)</td>\n",
       "      <td>0.7794/0.6029 (-22.6%)</td>\n",
       "      <td>0.7941/0.6324 (-20.4%)</td>\n",
       "      <td>26</td>\n",
       "      <td>0.8529/0.6029 (-29.3%)</td>\n",
       "      <td>368</td>\n",
       "      <td>0.8088/0.6029 (-25.5%)</td>\n",
       "      <td>0.8088/0.2794 (-65.5%)</td>\n",
       "      <td>0.7059/0.6029 (-14.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ilpd-indian-liver</th>\n",
       "      <td>0.5959/0.6529 (+9.6%)</td>\n",
       "      <td>0.6781/0.6667 (-1.7%)</td>\n",
       "      <td>0.6986/0.6942 (-0.6%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.6644/0.6701 (+0.9%)</td>\n",
       "      <td>583</td>\n",
       "      <td>0.6712/0.6976 (+3.9%)</td>\n",
       "      <td>0.6986/0.6976 (-0.1%)</td>\n",
       "      <td>0.6918/0.6392 (-7.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>image-segmentation</th>\n",
       "      <td>0.8481/0.1686 (-80.1%)</td>\n",
       "      <td>0.9024/0.2114 (-76.6%)</td>\n",
       "      <td>0.8838/0.4657 (-47.3%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.9090/0.2310 (-74.6%)</td>\n",
       "      <td>2310</td>\n",
       "      <td>0.8919/0.1881 (-78.9%)</td>\n",
       "      <td>0.9114/0.1724 (-81.1%)</td>\n",
       "      <td>0.8938/0.1429 (-84.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ionosphere</th>\n",
       "      <td>0.9432/0.7200 (-23.7%)</td>\n",
       "      <td>0.9432/0.6457 (-31.5%)</td>\n",
       "      <td>0.9432/0.6400 (-32.1%)</td>\n",
       "      <td>34</td>\n",
       "      <td>0.9091/0.9143 (+0.6%)</td>\n",
       "      <td>351</td>\n",
       "      <td>0.9545/0.7200 (-24.6%)</td>\n",
       "      <td>0.8864/0.8743 (-1.4%)</td>\n",
       "      <td>0.9318/0.9029 (-3.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>iris</th>\n",
       "      <td>0.9189/0.6267 (-31.8%)</td>\n",
       "      <td>0.8378/0.9467 (+13.0%)</td>\n",
       "      <td>0.9730/0.9333 (-4.1%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.9189/0.8933 (-2.8%)</td>\n",
       "      <td>150</td>\n",
       "      <td>0.9730/0.7467 (-23.3%)</td>\n",
       "      <td>0.9730/0.8800 (-9.6%)</td>\n",
       "      <td>1.0000/0.9333 (-6.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>led-display</th>\n",
       "      <td>0.6280/0.4980 (-20.7%)</td>\n",
       "      <td>0.7040/0.6080 (-13.6%)</td>\n",
       "      <td>0.6480/0.1100 (-83.0%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.7200/0.7000 (-2.8%)</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.7160/0.2080 (-70.9%)</td>\n",
       "      <td>0.7640/0.7180 (-6.0%)</td>\n",
       "      <td>0.6920/0.6840 (-1.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lenses</th>\n",
       "      <td>0.8333/0.1667 (-80.0%)</td>\n",
       "      <td>1.0000/0.6667 (-33.3%)</td>\n",
       "      <td>0.6667/0.1667 (-75.0%)</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0000/0.1667 (-83.3%)</td>\n",
       "      <td>24</td>\n",
       "      <td>0.6667/0.6667 (-0.0%)</td>\n",
       "      <td>0.6667/0.7500 (+12.5%)</td>\n",
       "      <td>0.8333/0.1667 (-80.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>letter</th>\n",
       "      <td>0.9796/0.0847 (-91.4%)</td>\n",
       "      <td>0.8984/0.8377 (-6.8%)</td>\n",
       "      <td>0.9742/0.0367 (-96.2%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.9712/0.0401 (-95.9%)</td>\n",
       "      <td>20000</td>\n",
       "      <td>0.9762/0.3086 (-68.4%)</td>\n",
       "      <td>0.9726/0.0401 (-95.9%)</td>\n",
       "      <td>0.9580/0.0401 (-95.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libras</th>\n",
       "      <td>0.7444/0.7500 (+0.8%)</td>\n",
       "      <td>0.8222/0.6056 (-26.3%)</td>\n",
       "      <td>0.8333/0.6667 (-20.0%)</td>\n",
       "      <td>91</td>\n",
       "      <td>0.8667/0.7944 (-8.3%)</td>\n",
       "      <td>360</td>\n",
       "      <td>0.7111/0.5333 (-25.0%)</td>\n",
       "      <td>0.7889/0.7667 (-2.8%)</td>\n",
       "      <td>0.8000/0.7500 (-6.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>low-res-spect</th>\n",
       "      <td>0.8571/0.6340 (-26.0%)</td>\n",
       "      <td>0.9023/0.5245 (-41.9%)</td>\n",
       "      <td>0.8947/0.8453 (-5.5%)</td>\n",
       "      <td>101</td>\n",
       "      <td>0.8496/0.8604 (+1.3%)</td>\n",
       "      <td>531</td>\n",
       "      <td>0.8647/0.3170 (-63.3%)</td>\n",
       "      <td>0.8571/0.8679 (+1.3%)</td>\n",
       "      <td>0.8872/0.8189 (-7.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lung-cancer</th>\n",
       "      <td>0.5000/0.4375 (-12.5%)</td>\n",
       "      <td>0.1250/0.4375 (+250.0%)</td>\n",
       "      <td>0.2500/0.5625 (+125.0%)</td>\n",
       "      <td>57</td>\n",
       "      <td>0.3750/0.3125 (-16.7%)</td>\n",
       "      <td>32</td>\n",
       "      <td>0.2500/0.5000 (+100.0%)</td>\n",
       "      <td>0.6250/0.5000 (-20.0%)</td>\n",
       "      <td>0.5000/0.5625 (+12.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lymphography</th>\n",
       "      <td>0.7568/0.7973 (+5.4%)</td>\n",
       "      <td>0.7297/0.5541 (-24.1%)</td>\n",
       "      <td>0.7838/0.7838 (-0.0%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.7297/0.8378 (+14.8%)</td>\n",
       "      <td>148</td>\n",
       "      <td>0.6757/0.7027 (+4.0%)</td>\n",
       "      <td>0.9189/0.7973 (-13.2%)</td>\n",
       "      <td>0.7568/0.7162 (-5.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>magic</th>\n",
       "      <td>0.8713/0.6485 (-25.6%)</td>\n",
       "      <td>0.8673/0.5792 (-33.2%)</td>\n",
       "      <td>0.8620/0.7728 (-10.4%)</td>\n",
       "      <td>11</td>\n",
       "      <td>0.8629/0.4118 (-52.3%)</td>\n",
       "      <td>19020</td>\n",
       "      <td>0.8723/0.3516 (-59.7%)</td>\n",
       "      <td>0.8692/0.6707 (-22.8%)</td>\n",
       "      <td>0.8690/0.7857 (-9.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mammographic</th>\n",
       "      <td>0.8167/0.6417 (-21.4%)</td>\n",
       "      <td>0.7917/0.7771 (-1.8%)</td>\n",
       "      <td>0.8208/0.7896 (-3.8%)</td>\n",
       "      <td>6</td>\n",
       "      <td>0.8083/0.7917 (-2.1%)</td>\n",
       "      <td>961</td>\n",
       "      <td>0.7833/0.4000 (-48.9%)</td>\n",
       "      <td>0.8250/0.8021 (-2.8%)</td>\n",
       "      <td>0.8292/0.7000 (-15.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>miniboone</th>\n",
       "      <td>0.9262/0.7194 (-22.3%)</td>\n",
       "      <td>0.9270/0.7355 (-20.7%)</td>\n",
       "      <td>0.9313/0.7450 (-20.0%)</td>\n",
       "      <td>51</td>\n",
       "      <td>0.9250/0.7641 (-17.4%)</td>\n",
       "      <td>130064</td>\n",
       "      <td>0.9254/0.7194 (-22.3%)</td>\n",
       "      <td>0.9307/0.7194 (-22.7%)</td>\n",
       "      <td>0.9272/0.7506 (-19.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>molec-biol-promoter</th>\n",
       "      <td>0.7692/0.6981 (-9.2%)</td>\n",
       "      <td>0.6923/0.8113 (+17.2%)</td>\n",
       "      <td>0.4615/0.7358 (+59.4%)</td>\n",
       "      <td>58</td>\n",
       "      <td>0.7692/0.6226 (-19.1%)</td>\n",
       "      <td>106</td>\n",
       "      <td>0.7692/0.6604 (-14.1%)</td>\n",
       "      <td>0.8462/0.8491 (+0.3%)</td>\n",
       "      <td>0.6923/0.6038 (-12.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>molec-biol-splice</th>\n",
       "      <td>0.8519/0.5273 (-38.1%)</td>\n",
       "      <td>0.8833/0.8401 (-4.9%)</td>\n",
       "      <td>0.8607/0.7197 (-16.4%)</td>\n",
       "      <td>61</td>\n",
       "      <td>0.8482/0.8031 (-5.3%)</td>\n",
       "      <td>3190</td>\n",
       "      <td>0.8557/0.6075 (-29.0%)</td>\n",
       "      <td>0.9009/0.2408 (-73.3%)</td>\n",
       "      <td>0.8494/0.8050 (-5.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monks-1</th>\n",
       "      <td>0.9074/0.5255 (-42.1%)</td>\n",
       "      <td>0.5833/0.6435 (+10.3%)</td>\n",
       "      <td>0.7014/0.6829 (-2.6%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.6551/0.7523 (+14.8%)</td>\n",
       "      <td>556</td>\n",
       "      <td>0.7546/0.5208 (-31.0%)</td>\n",
       "      <td>0.7523/0.6458 (-14.2%)</td>\n",
       "      <td>0.5000/0.7292 (+45.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monks-2</th>\n",
       "      <td>0.3287/0.6551 (+99.3%)</td>\n",
       "      <td>0.6389/0.6713 (+5.1%)</td>\n",
       "      <td>0.5162/0.6690 (+29.6%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.6343/0.6644 (+4.7%)</td>\n",
       "      <td>601</td>\n",
       "      <td>0.6273/0.6713 (+7.0%)</td>\n",
       "      <td>0.5926/0.6667 (+12.5%)</td>\n",
       "      <td>0.6644/0.6667 (+0.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monks-3</th>\n",
       "      <td>0.5278/0.5648 (+7.0%)</td>\n",
       "      <td>0.5880/0.5764 (-2.0%)</td>\n",
       "      <td>0.6991/0.5486 (-21.5%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.7454/0.5417 (-27.3%)</td>\n",
       "      <td>554</td>\n",
       "      <td>0.5833/0.6806 (+16.7%)</td>\n",
       "      <td>0.6042/0.6435 (+6.5%)</td>\n",
       "      <td>0.5231/0.7407 (+41.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mushroom</th>\n",
       "      <td>0.9990/0.4628 (-53.7%)</td>\n",
       "      <td>1.0000/0.4820 (-51.8%)</td>\n",
       "      <td>0.9995/0.9183 (-8.1%)</td>\n",
       "      <td>22</td>\n",
       "      <td>1.0000/0.9953 (-0.5%)</td>\n",
       "      <td>8124</td>\n",
       "      <td>1.0000/0.4820 (-51.8%)</td>\n",
       "      <td>1.0000/0.9983 (-0.2%)</td>\n",
       "      <td>0.9995/0.4828 (-51.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>musk-1</th>\n",
       "      <td>0.8235/0.7269 (-11.7%)</td>\n",
       "      <td>0.8992/0.6765 (-24.8%)</td>\n",
       "      <td>0.8992/0.8445 (-6.1%)</td>\n",
       "      <td>167</td>\n",
       "      <td>0.8655/0.7773 (-10.2%)</td>\n",
       "      <td>476</td>\n",
       "      <td>0.8739/0.7311 (-16.3%)</td>\n",
       "      <td>0.8739/0.8613 (-1.4%)</td>\n",
       "      <td>0.8992/0.8487 (-5.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>musk-2</th>\n",
       "      <td>0.9982/0.1540 (-84.6%)</td>\n",
       "      <td>0.9915/0.9900 (-0.2%)</td>\n",
       "      <td>0.9951/0.1540 (-84.5%)</td>\n",
       "      <td>167</td>\n",
       "      <td>0.9945/0.1540 (-84.5%)</td>\n",
       "      <td>6598</td>\n",
       "      <td>0.9964/0.9536 (-4.3%)</td>\n",
       "      <td>0.9891/0.1540 (-84.4%)</td>\n",
       "      <td>0.9927/0.1540 (-84.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nursery</th>\n",
       "      <td>0.9994/0.6267 (-37.3%)</td>\n",
       "      <td>1.0000/0.9647 (-3.5%)</td>\n",
       "      <td>0.9966/0.3120 (-68.7%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.9988/0.3120 (-68.8%)</td>\n",
       "      <td>12960</td>\n",
       "      <td>0.9994/0.5350 (-46.5%)</td>\n",
       "      <td>0.9978/0.9347 (-6.3%)</td>\n",
       "      <td>0.9966/0.9861 (-1.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_merluccius_nucleus_4d</th>\n",
       "      <td>0.8078/0.7065 (-12.5%)</td>\n",
       "      <td>0.7176/0.6712 (-6.5%)</td>\n",
       "      <td>0.7686/0.7847 (+2.1%)</td>\n",
       "      <td>42</td>\n",
       "      <td>0.8196/0.7730 (-5.7%)</td>\n",
       "      <td>1022</td>\n",
       "      <td>0.8000/0.6732 (-15.9%)</td>\n",
       "      <td>0.8235/0.8141 (-1.1%)</td>\n",
       "      <td>0.8078/0.7808 (-3.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_merluccius_states_2f</th>\n",
       "      <td>0.9333/0.6888 (-26.2%)</td>\n",
       "      <td>0.9490/0.9119 (-3.9%)</td>\n",
       "      <td>0.9412/0.9061 (-3.7%)</td>\n",
       "      <td>26</td>\n",
       "      <td>0.9490/0.8943 (-5.8%)</td>\n",
       "      <td>1022</td>\n",
       "      <td>0.9373/0.6849 (-26.9%)</td>\n",
       "      <td>0.9529/0.9119 (-4.3%)</td>\n",
       "      <td>0.9020/0.8826 (-2.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_trisopterus_nucleus_2f</th>\n",
       "      <td>0.7456/0.5811 (-22.1%)</td>\n",
       "      <td>0.8289/0.5789 (-30.2%)</td>\n",
       "      <td>0.8202/0.6031 (-26.5%)</td>\n",
       "      <td>26</td>\n",
       "      <td>0.8728/0.7654 (-12.3%)</td>\n",
       "      <td>912</td>\n",
       "      <td>0.7719/0.6162 (-20.2%)</td>\n",
       "      <td>0.7982/0.7632 (-4.4%)</td>\n",
       "      <td>0.7939/0.5899 (-25.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_trisopterus_states_5b</th>\n",
       "      <td>0.8947/0.7873 (-12.0%)</td>\n",
       "      <td>0.9342/0.8991 (-3.8%)</td>\n",
       "      <td>0.8991/0.9057 (+0.7%)</td>\n",
       "      <td>33</td>\n",
       "      <td>0.9430/0.4079 (-56.7%)</td>\n",
       "      <td>912</td>\n",
       "      <td>0.8947/0.6754 (-24.5%)</td>\n",
       "      <td>0.9342/0.9057 (-3.1%)</td>\n",
       "      <td>0.9254/0.8947 (-3.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>optical</th>\n",
       "      <td>0.9716/0.9644 (-0.7%)</td>\n",
       "      <td>0.9644/0.9738 (+1.0%)</td>\n",
       "      <td>0.9755/0.9699 (-0.6%)</td>\n",
       "      <td>63</td>\n",
       "      <td>0.9666/0.9688 (+0.2%)</td>\n",
       "      <td>5620</td>\n",
       "      <td>0.9627/0.9627 (+0.0%)</td>\n",
       "      <td>0.9711/0.9638 (-0.7%)</td>\n",
       "      <td>0.9638/0.9677 (+0.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ozone</th>\n",
       "      <td>0.9669/0.9685 (+0.2%)</td>\n",
       "      <td>0.9716/0.9685 (-0.3%)</td>\n",
       "      <td>0.9716/0.9669 (-0.5%)</td>\n",
       "      <td>73</td>\n",
       "      <td>0.9732/0.0284 (-97.1%)</td>\n",
       "      <td>2536</td>\n",
       "      <td>0.9669/0.9700 (+0.3%)</td>\n",
       "      <td>0.9700/0.9685 (-0.2%)</td>\n",
       "      <td>0.9748/0.9629 (-1.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page-blocks</th>\n",
       "      <td>0.9613/0.9419 (-2.0%)</td>\n",
       "      <td>0.9656/0.9401 (-2.6%)</td>\n",
       "      <td>0.9708/0.9550 (-1.6%)</td>\n",
       "      <td>11</td>\n",
       "      <td>0.9708/0.0599 (-93.8%)</td>\n",
       "      <td>5473</td>\n",
       "      <td>0.9605/0.9364 (-2.5%)</td>\n",
       "      <td>0.9583/0.9437 (-1.5%)</td>\n",
       "      <td>0.9730/0.7876 (-19.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parkinsons</th>\n",
       "      <td>0.8571/0.7526 (-12.2%)</td>\n",
       "      <td>0.8367/0.8763 (+4.7%)</td>\n",
       "      <td>0.8571/0.9175 (+7.1%)</td>\n",
       "      <td>23</td>\n",
       "      <td>0.9184/0.7526 (-18.1%)</td>\n",
       "      <td>195</td>\n",
       "      <td>0.9184/0.8866 (-3.5%)</td>\n",
       "      <td>0.8980/0.7732 (-13.9%)</td>\n",
       "      <td>0.8163/0.7526 (-7.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pendigits</th>\n",
       "      <td>0.9734/0.9720 (-0.1%)</td>\n",
       "      <td>0.9671/0.9605 (-0.7%)</td>\n",
       "      <td>0.9657/0.9720 (+0.7%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.9714/0.9777 (+0.6%)</td>\n",
       "      <td>10992</td>\n",
       "      <td>0.9708/0.9723 (+0.2%)</td>\n",
       "      <td>0.9706/0.9711 (+0.1%)</td>\n",
       "      <td>0.9620/0.9706 (+0.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pima</th>\n",
       "      <td>0.7188/0.6719 (-6.5%)</td>\n",
       "      <td>0.7188/0.7135 (-0.7%)</td>\n",
       "      <td>0.6927/0.7474 (+7.9%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.7656/0.7344 (-4.1%)</td>\n",
       "      <td>768</td>\n",
       "      <td>0.7135/0.3490 (-51.1%)</td>\n",
       "      <td>0.7552/0.7552 (+0.0%)</td>\n",
       "      <td>0.6979/0.6693 (-4.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-MATERIAL</th>\n",
       "      <td>0.8846/0.8679 (-1.9%)</td>\n",
       "      <td>0.9231/0.8679 (-6.0%)</td>\n",
       "      <td>0.9231/0.7170 (-22.3%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8462/0.8679 (+2.6%)</td>\n",
       "      <td>106</td>\n",
       "      <td>0.9231/0.8302 (-10.1%)</td>\n",
       "      <td>0.8846/0.8302 (-6.2%)</td>\n",
       "      <td>0.8077/0.7547 (-6.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-REL-L</th>\n",
       "      <td>0.7692/0.5686 (-26.1%)</td>\n",
       "      <td>0.6923/0.4510 (-34.9%)</td>\n",
       "      <td>0.7308/0.1373 (-81.2%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.7692/0.5686 (-26.1%)</td>\n",
       "      <td>103</td>\n",
       "      <td>0.8462/0.6863 (-18.9%)</td>\n",
       "      <td>0.6923/0.5294 (-23.5%)</td>\n",
       "      <td>0.6538/0.6667 (+2.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-SPAN</th>\n",
       "      <td>0.5652/0.5000 (-11.5%)</td>\n",
       "      <td>0.5652/0.7609 (+34.6%)</td>\n",
       "      <td>0.6087/0.6304 (+3.6%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.5217/0.7391 (+41.7%)</td>\n",
       "      <td>92</td>\n",
       "      <td>0.5652/0.6087 (+7.7%)</td>\n",
       "      <td>0.6957/0.6087 (-12.5%)</td>\n",
       "      <td>0.6522/0.6087 (-6.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-T-OR-D</th>\n",
       "      <td>0.8800/0.7059 (-19.8%)</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>102</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>0.8400/0.8627 (+2.7%)</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-TYPE</th>\n",
       "      <td>0.1154/0.5192 (+349.9%)</td>\n",
       "      <td>0.5385/0.5577 (+3.6%)</td>\n",
       "      <td>0.6538/0.5769 (-11.8%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.6538/0.5577 (-14.7%)</td>\n",
       "      <td>105</td>\n",
       "      <td>0.6538/0.6538 (+0.0%)</td>\n",
       "      <td>0.6538/0.5192 (-20.6%)</td>\n",
       "      <td>0.4615/0.6538 (+41.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>planning</th>\n",
       "      <td>0.6222/0.5165 (-17.0%)</td>\n",
       "      <td>0.6000/0.3956 (-34.1%)</td>\n",
       "      <td>0.6889/0.7143 (+3.7%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.6667/0.4615 (-30.8%)</td>\n",
       "      <td>182</td>\n",
       "      <td>0.7111/0.5385 (-24.3%)</td>\n",
       "      <td>0.6889/0.7143 (+3.7%)</td>\n",
       "      <td>0.6444/0.7143 (+10.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant-margin</th>\n",
       "      <td>0.7600/0.5375 (-29.3%)</td>\n",
       "      <td>0.8375/0.6663 (-20.4%)</td>\n",
       "      <td>0.8425/0.7688 (-8.8%)</td>\n",
       "      <td>65</td>\n",
       "      <td>0.8125/0.7950 (-2.2%)</td>\n",
       "      <td>1600</td>\n",
       "      <td>0.7975/0.4200 (-47.3%)</td>\n",
       "      <td>0.8125/0.7638 (-6.0%)</td>\n",
       "      <td>0.8175/0.7725 (-5.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant-shape</th>\n",
       "      <td>0.2850/0.2062 (-27.6%)</td>\n",
       "      <td>0.6325/0.3925 (-37.9%)</td>\n",
       "      <td>0.6775/0.6388 (-5.7%)</td>\n",
       "      <td>65</td>\n",
       "      <td>0.6350/0.5175 (-18.5%)</td>\n",
       "      <td>1600</td>\n",
       "      <td>0.5150/0.2575 (-50.0%)</td>\n",
       "      <td>0.7275/0.5637 (-22.5%)</td>\n",
       "      <td>0.6575/0.0100 (-98.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant-texture</th>\n",
       "      <td>0.8200/0.7134 (-13.0%)</td>\n",
       "      <td>0.7900/0.7860 (-0.5%)</td>\n",
       "      <td>0.8350/0.0100 (-98.8%)</td>\n",
       "      <td>65</td>\n",
       "      <td>0.7900/0.7972 (+0.9%)</td>\n",
       "      <td>1599</td>\n",
       "      <td>0.8000/0.5419 (-32.3%)</td>\n",
       "      <td>0.8125/0.7935 (-2.3%)</td>\n",
       "      <td>0.8175/0.7960 (-2.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>post-operative</th>\n",
       "      <td>0.5909/0.6222 (+5.3%)</td>\n",
       "      <td>0.5909/0.6889 (+16.6%)</td>\n",
       "      <td>0.7727/0.0222 (-97.1%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.7273/0.7111 (-2.2%)</td>\n",
       "      <td>90</td>\n",
       "      <td>0.7273/0.5556 (-23.6%)</td>\n",
       "      <td>0.7273/0.2667 (-63.3%)</td>\n",
       "      <td>0.5455/0.7111 (+30.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>primary-tumor</th>\n",
       "      <td>0.5122/0.2727 (-46.8%)</td>\n",
       "      <td>0.4512/0.3939 (-12.7%)</td>\n",
       "      <td>0.4512/0.3455 (-23.4%)</td>\n",
       "      <td>18</td>\n",
       "      <td>0.5000/0.4121 (-17.6%)</td>\n",
       "      <td>330</td>\n",
       "      <td>0.3902/0.2788 (-28.6%)</td>\n",
       "      <td>0.5244/0.4061 (-22.6%)</td>\n",
       "      <td>0.5000/0.3879 (-22.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ringnorm</th>\n",
       "      <td>0.9843/0.5049 (-48.7%)</td>\n",
       "      <td>0.9692/0.9673 (-0.2%)</td>\n",
       "      <td>0.9827/0.5049 (-48.6%)</td>\n",
       "      <td>21</td>\n",
       "      <td>0.9843/0.9762 (-0.8%)</td>\n",
       "      <td>7400</td>\n",
       "      <td>0.9811/0.5503 (-43.9%)</td>\n",
       "      <td>0.9751/0.9832 (+0.8%)</td>\n",
       "      <td>0.9719/0.9762 (+0.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seeds</th>\n",
       "      <td>0.8654/0.6667 (-23.0%)</td>\n",
       "      <td>0.9423/0.8952 (-5.0%)</td>\n",
       "      <td>0.8846/0.8952 (+1.2%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8654/0.5048 (-41.7%)</td>\n",
       "      <td>210</td>\n",
       "      <td>0.8654/0.6190 (-28.5%)</td>\n",
       "      <td>0.8846/0.9048 (+2.3%)</td>\n",
       "      <td>0.8846/0.8762 (-1.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>semeion</th>\n",
       "      <td>0.9372/0.0992 (-89.4%)</td>\n",
       "      <td>0.9447/0.9397 (-0.5%)</td>\n",
       "      <td>0.9447/0.0992 (-89.5%)</td>\n",
       "      <td>257</td>\n",
       "      <td>0.9296/0.9296 (+0.0%)</td>\n",
       "      <td>1593</td>\n",
       "      <td>0.9146/0.5452 (-40.4%)</td>\n",
       "      <td>0.9196/0.9209 (+0.1%)</td>\n",
       "      <td>0.9322/0.0992 (-89.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>soybean</th>\n",
       "      <td>0.8883/0.5053 (-43.1%)</td>\n",
       "      <td>0.8617/0.3856 (-55.2%)</td>\n",
       "      <td>0.8484/0.4468 (-47.3%)</td>\n",
       "      <td>36</td>\n",
       "      <td>0.8723/0.3590 (-58.8%)</td>\n",
       "      <td>683</td>\n",
       "      <td>0.8670/0.3750 (-56.7%)</td>\n",
       "      <td>0.8511/0.0239 (-97.2%)</td>\n",
       "      <td>0.8537/0.4149 (-51.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spambase</th>\n",
       "      <td>0.9426/0.8948 (-5.1%)</td>\n",
       "      <td>0.9435/0.9222 (-2.3%)</td>\n",
       "      <td>0.9513/0.9230 (-3.0%)</td>\n",
       "      <td>58</td>\n",
       "      <td>0.9461/0.3939 (-58.4%)</td>\n",
       "      <td>4601</td>\n",
       "      <td>0.9461/0.7070 (-25.3%)</td>\n",
       "      <td>0.9409/0.9243 (-1.8%)</td>\n",
       "      <td>0.9504/0.8109 (-14.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spect</th>\n",
       "      <td>0.6344/0.4624 (-27.1%)</td>\n",
       "      <td>0.6022/0.5484 (-8.9%)</td>\n",
       "      <td>0.6720/0.6237 (-7.2%)</td>\n",
       "      <td>23</td>\n",
       "      <td>0.6183/0.5484 (-11.3%)</td>\n",
       "      <td>265</td>\n",
       "      <td>0.6667/0.5484 (-17.7%)</td>\n",
       "      <td>0.6398/0.5645 (-11.8%)</td>\n",
       "      <td>0.6398/0.5484 (-14.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spectf</th>\n",
       "      <td>0.2299/0.6738 (+193.1%)</td>\n",
       "      <td>0.8930/0.8610 (-3.6%)</td>\n",
       "      <td>0.5561/0.9144 (+64.4%)</td>\n",
       "      <td>45</td>\n",
       "      <td>0.6043/0.9198 (+52.2%)</td>\n",
       "      <td>267</td>\n",
       "      <td>0.7005/0.9198 (+31.3%)</td>\n",
       "      <td>0.4973/0.9198 (+85.0%)</td>\n",
       "      <td>0.4545/0.9198 (+102.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-australian-credit</th>\n",
       "      <td>0.6802/0.6725 (-1.1%)</td>\n",
       "      <td>0.6802/0.6232 (-8.4%)</td>\n",
       "      <td>0.6279/0.6783 (+8.0%)</td>\n",
       "      <td>15</td>\n",
       "      <td>0.6802/0.6667 (-2.0%)</td>\n",
       "      <td>690</td>\n",
       "      <td>0.6395/0.6783 (+6.1%)</td>\n",
       "      <td>0.5988/0.6377 (+6.5%)</td>\n",
       "      <td>0.6860/0.6783 (-1.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-german-credit</th>\n",
       "      <td>0.7520/0.6980 (-7.2%)</td>\n",
       "      <td>0.7760/0.7120 (-8.2%)</td>\n",
       "      <td>0.7400/0.7420 (+0.3%)</td>\n",
       "      <td>25</td>\n",
       "      <td>0.7280/0.7000 (-3.8%)</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.7720/0.5840 (-24.4%)</td>\n",
       "      <td>0.7560/0.7720 (+2.1%)</td>\n",
       "      <td>0.7400/0.7000 (-5.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-heart</th>\n",
       "      <td>0.7910/0.4444 (-43.8%)</td>\n",
       "      <td>0.7761/0.4444 (-42.7%)</td>\n",
       "      <td>0.7910/0.8444 (+6.8%)</td>\n",
       "      <td>14</td>\n",
       "      <td>0.8358/0.8593 (+2.8%)</td>\n",
       "      <td>270</td>\n",
       "      <td>0.8657/0.6889 (-20.4%)</td>\n",
       "      <td>0.9254/0.8593 (-7.1%)</td>\n",
       "      <td>0.8657/0.8667 (+0.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-image</th>\n",
       "      <td>0.9671/0.1446 (-85.0%)</td>\n",
       "      <td>0.9584/0.9195 (-4.1%)</td>\n",
       "      <td>0.9757/0.1429 (-85.4%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.9757/0.8234 (-15.6%)</td>\n",
       "      <td>2310</td>\n",
       "      <td>0.9584/0.3281 (-65.8%)</td>\n",
       "      <td>0.9549/0.9498 (-0.5%)</td>\n",
       "      <td>0.9515/0.9437 (-0.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-landsat</th>\n",
       "      <td>0.9040/0.8755 (-3.2%)</td>\n",
       "      <td>0.9110/0.8920 (-2.1%)</td>\n",
       "      <td>0.9040/0.8945 (-1.1%)</td>\n",
       "      <td>37</td>\n",
       "      <td>0.9075/0.2305 (-74.6%)</td>\n",
       "      <td>6435</td>\n",
       "      <td>0.9055/0.8620 (-4.8%)</td>\n",
       "      <td>0.9100/0.8780 (-3.5%)</td>\n",
       "      <td>0.8925/0.8810 (-1.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-shuttle</th>\n",
       "      <td>0.9988/0.9961 (-0.3%)</td>\n",
       "      <td>0.9977/0.9977 (+0.0%)</td>\n",
       "      <td>0.9987/0.9991 (+0.0%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.9983/0.9989 (+0.1%)</td>\n",
       "      <td>58000</td>\n",
       "      <td>0.9992/0.9970 (-0.2%)</td>\n",
       "      <td>0.9990/0.0009 (-99.9%)</td>\n",
       "      <td>0.9988/0.9988 (+0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-vehicle</th>\n",
       "      <td>0.7583/0.6809 (-10.2%)</td>\n",
       "      <td>0.7962/0.6879 (-13.6%)</td>\n",
       "      <td>0.7915/0.7636 (-3.5%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.8294/0.7612 (-8.2%)</td>\n",
       "      <td>846</td>\n",
       "      <td>0.7583/0.5910 (-22.1%)</td>\n",
       "      <td>0.8009/0.7329 (-8.5%)</td>\n",
       "      <td>0.8009/0.7565 (-5.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>steel-plates</th>\n",
       "      <td>0.7031/0.6124 (-12.9%)</td>\n",
       "      <td>0.7608/0.7082 (-6.9%)</td>\n",
       "      <td>0.7588/0.3474 (-54.2%)</td>\n",
       "      <td>28</td>\n",
       "      <td>0.7567/0.6887 (-9.0%)</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.7629/0.4227 (-44.6%)</td>\n",
       "      <td>0.7835/0.7052 (-10.0%)</td>\n",
       "      <td>0.7856/0.6711 (-14.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>synthetic-control</th>\n",
       "      <td>0.9733/0.2133 (-78.1%)</td>\n",
       "      <td>0.9867/0.1667 (-83.1%)</td>\n",
       "      <td>0.9733/0.2067 (-78.8%)</td>\n",
       "      <td>61</td>\n",
       "      <td>0.9800/0.7033 (-28.2%)</td>\n",
       "      <td>600</td>\n",
       "      <td>0.9600/0.4433 (-53.8%)</td>\n",
       "      <td>0.9867/0.7833 (-20.6%)</td>\n",
       "      <td>0.9867/0.4467 (-54.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>teaching</th>\n",
       "      <td>0.5000/0.3467 (-30.7%)</td>\n",
       "      <td>0.5263/0.3867 (-26.5%)</td>\n",
       "      <td>0.6316/0.4400 (-30.3%)</td>\n",
       "      <td>6</td>\n",
       "      <td>0.6053/0.3333 (-44.9%)</td>\n",
       "      <td>151</td>\n",
       "      <td>0.5526/0.3467 (-37.3%)</td>\n",
       "      <td>0.5000/0.3333 (-33.3%)</td>\n",
       "      <td>0.3158/0.3200 (+1.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thyroid</th>\n",
       "      <td>0.9778/0.9787 (+0.1%)</td>\n",
       "      <td>0.9708/0.9770 (+0.6%)</td>\n",
       "      <td>0.9752/0.9761 (+0.1%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.9770/0.9767 (-0.0%)</td>\n",
       "      <td>7200</td>\n",
       "      <td>0.9799/0.9770 (-0.3%)</td>\n",
       "      <td>0.9816/0.9772 (-0.4%)</td>\n",
       "      <td>0.9807/0.9749 (-0.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tic-tac-toe</th>\n",
       "      <td>0.9833/0.7724 (-21.4%)</td>\n",
       "      <td>0.9749/0.9457 (-3.0%)</td>\n",
       "      <td>0.9791/0.9645 (-1.5%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.9833/0.9353 (-4.9%)</td>\n",
       "      <td>958</td>\n",
       "      <td>0.9623/0.8246 (-14.3%)</td>\n",
       "      <td>0.9665/0.9812 (+1.5%)</td>\n",
       "      <td>0.9707/0.6534 (-32.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>titanic</th>\n",
       "      <td>0.7800/0.7491 (-4.0%)</td>\n",
       "      <td>0.7927/0.3227 (-59.3%)</td>\n",
       "      <td>0.7891/0.3227 (-59.1%)</td>\n",
       "      <td>4</td>\n",
       "      <td>0.7909/0.5464 (-30.9%)</td>\n",
       "      <td>2201</td>\n",
       "      <td>0.7727/0.3227 (-58.2%)</td>\n",
       "      <td>0.7836/0.3300 (-57.9%)</td>\n",
       "      <td>0.7818/0.5236 (-33.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trains</th>\n",
       "      <td>0.5000/0.4000 (-20.0%)</td>\n",
       "      <td>nan/0.4000 (+nan%)</td>\n",
       "      <td>1.0000/0.6000 (-40.0%)</td>\n",
       "      <td>30</td>\n",
       "      <td>nan/0.4000 (+nan%)</td>\n",
       "      <td>10</td>\n",
       "      <td>nan/0.8000 (+nan%)</td>\n",
       "      <td>nan/0.4000 (+nan%)</td>\n",
       "      <td>0.5000/0.6000 (+20.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>twonorm</th>\n",
       "      <td>0.9757/0.9127 (-6.5%)</td>\n",
       "      <td>0.9708/0.9738 (+0.3%)</td>\n",
       "      <td>0.9724/0.9732 (+0.1%)</td>\n",
       "      <td>21</td>\n",
       "      <td>0.9778/0.9700 (-0.8%)</td>\n",
       "      <td>7400</td>\n",
       "      <td>0.9735/0.4995 (-48.7%)</td>\n",
       "      <td>0.9805/0.4995 (-49.1%)</td>\n",
       "      <td>0.9730/0.9743 (+0.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vertebral-column-2clases</th>\n",
       "      <td>0.8312/0.3226 (-61.2%)</td>\n",
       "      <td>0.8571/0.6581 (-23.2%)</td>\n",
       "      <td>0.8442/0.8000 (-5.2%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.8701/0.7548 (-13.2%)</td>\n",
       "      <td>310</td>\n",
       "      <td>0.8312/0.6452 (-22.4%)</td>\n",
       "      <td>0.8312/0.7806 (-6.1%)</td>\n",
       "      <td>0.6623/0.8065 (+21.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vertebral-column-3clases</th>\n",
       "      <td>0.7792/0.7935 (+1.8%)</td>\n",
       "      <td>0.7922/0.8194 (+3.4%)</td>\n",
       "      <td>0.8312/0.8065 (-3.0%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.8052/0.8452 (+5.0%)</td>\n",
       "      <td>310</td>\n",
       "      <td>0.7532/0.7097 (-5.8%)</td>\n",
       "      <td>0.8312/0.8129 (-2.2%)</td>\n",
       "      <td>0.7403/0.8452 (+14.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wall-following</th>\n",
       "      <td>0.9333/0.1514 (-83.8%)</td>\n",
       "      <td>0.9230/0.8904 (-3.5%)</td>\n",
       "      <td>0.9128/0.0601 (-93.4%)</td>\n",
       "      <td>25</td>\n",
       "      <td>0.9076/0.4637 (-48.9%)</td>\n",
       "      <td>5456</td>\n",
       "      <td>0.9223/0.5154 (-44.1%)</td>\n",
       "      <td>0.9098/0.1514 (-83.4%)</td>\n",
       "      <td>0.9274/0.1514 (-83.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>waveform</th>\n",
       "      <td>0.8360/0.7064 (-15.5%)</td>\n",
       "      <td>0.8320/0.8380 (+0.7%)</td>\n",
       "      <td>0.8448/0.8592 (+1.7%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.8312/0.8464 (+1.8%)</td>\n",
       "      <td>5000</td>\n",
       "      <td>0.8360/0.4664 (-44.2%)</td>\n",
       "      <td>0.8480/0.8304 (-2.1%)</td>\n",
       "      <td>0.8376/0.7972 (-4.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>waveform-noise</th>\n",
       "      <td>0.8480/0.6956 (-18.0%)</td>\n",
       "      <td>0.8696/0.8368 (-3.8%)</td>\n",
       "      <td>0.8504/0.8480 (-0.3%)</td>\n",
       "      <td>41</td>\n",
       "      <td>0.8328/0.8252 (-0.9%)</td>\n",
       "      <td>5000</td>\n",
       "      <td>0.8584/0.6572 (-23.4%)</td>\n",
       "      <td>0.8608/0.8336 (-3.2%)</td>\n",
       "      <td>0.8640/0.8424 (-2.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wine</th>\n",
       "      <td>0.9773/0.5056 (-48.3%)</td>\n",
       "      <td>0.9091/0.9663 (+6.3%)</td>\n",
       "      <td>0.9773/0.5393 (-44.8%)</td>\n",
       "      <td>14</td>\n",
       "      <td>0.9318/0.2809 (-69.9%)</td>\n",
       "      <td>178</td>\n",
       "      <td>0.9773/0.6517 (-33.3%)</td>\n",
       "      <td>0.9773/0.9888 (+1.2%)</td>\n",
       "      <td>0.9773/0.9213 (-5.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wine-quality-red</th>\n",
       "      <td>0.5450/0.4631 (-15.0%)</td>\n",
       "      <td>0.5625/0.5557 (-1.2%)</td>\n",
       "      <td>0.6100/0.5820 (-4.6%)</td>\n",
       "      <td>12</td>\n",
       "      <td>0.6250/0.5770 (-7.7%)</td>\n",
       "      <td>1599</td>\n",
       "      <td>0.6150/0.4456 (-27.6%)</td>\n",
       "      <td>0.6300/0.5081 (-19.3%)</td>\n",
       "      <td>0.5575/0.4693 (-15.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wine-quality-white</th>\n",
       "      <td>0.5335/0.4684 (-12.2%)</td>\n",
       "      <td>0.5564/0.4565 (-18.0%)</td>\n",
       "      <td>0.6544/0.5892 (-10.0%)</td>\n",
       "      <td>12</td>\n",
       "      <td>0.6479/0.4924 (-24.0%)</td>\n",
       "      <td>4898</td>\n",
       "      <td>0.6307/0.4945 (-21.6%)</td>\n",
       "      <td>0.6373/0.5517 (-13.4%)</td>\n",
       "      <td>0.5482/0.4973 (-9.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yeast</th>\n",
       "      <td>0.4906/0.0984 (-79.9%)</td>\n",
       "      <td>0.6065/0.4987 (-17.8%)</td>\n",
       "      <td>0.6092/0.5620 (-7.7%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.6173/0.5768 (-6.6%)</td>\n",
       "      <td>1484</td>\n",
       "      <td>0.5499/0.3976 (-27.7%)</td>\n",
       "      <td>0.6307/0.4704 (-25.4%)</td>\n",
       "      <td>0.5876/0.5391 (-8.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zoo</th>\n",
       "      <td>0.7200/0.8600 (+19.4%)</td>\n",
       "      <td>0.8800/0.2000 (-77.3%)</td>\n",
       "      <td>0.9600/0.8600 (-10.4%)</td>\n",
       "      <td>17</td>\n",
       "      <td>1.0000/0.9400 (-6.0%)</td>\n",
       "      <td>101</td>\n",
       "      <td>1.0000/0.9400 (-6.0%)</td>\n",
       "      <td>0.9200/0.9800 (+6.5%)</td>\n",
       "      <td>0.9600/0.8000 (-16.7%)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           BN (old/new)  \\\n",
       "abalone                          0.6303/0.5575 (-11.6%)   \n",
       "acute-inflammation                1.0000/0.9667 (-3.3%)   \n",
       "acute-nephritis                   1.0000/0.9333 (-6.7%)   \n",
       "adult                             0.8499/0.8522 (+0.3%)   \n",
       "annealing                        0.1200/0.0600 (-50.0%)   \n",
       "arrhythmia                       0.5929/0.5044 (-14.9%)   \n",
       "audiology-std                    0.6400/0.5200 (-18.8%)   \n",
       "balance-scale                    0.9231/0.4712 (-49.0%)   \n",
       "balloons                         1.0000/0.3750 (-62.5%)   \n",
       "bank                              0.8823/0.8850 (+0.3%)   \n",
       "blood                             0.7647/0.7567 (-1.0%)   \n",
       "breast-cancer                    0.7324/0.2937 (-59.9%)   \n",
       "breast-cancer-wisc               0.9829/0.3438 (-65.0%)   \n",
       "breast-cancer-wisc-diag           0.9789/0.8873 (-9.4%)   \n",
       "breast-cancer-wisc-prog          0.7755/0.5556 (-28.4%)   \n",
       "breast-tissue                    0.4615/0.7358 (+59.4%)   \n",
       "car                              0.9606/0.1146 (-88.1%)   \n",
       "cardiotocography-10clases        0.7910/0.0941 (-88.1%)   \n",
       "cardiotocography-3clases         0.9096/0.7789 (-14.4%)   \n",
       "chess-krvk                       0.8781/0.0139 (-98.4%)   \n",
       "chess-krvkp                      0.9862/0.5225 (-47.0%)   \n",
       "congressional-voting             0.5872/0.3871 (-34.1%)   \n",
       "conn-bench-sonar-mines-rocks     0.7115/0.4712 (-33.8%)   \n",
       "conn-bench-vowel-deterding        0.9610/0.9004 (-6.3%)   \n",
       "connect-4                        0.8729/0.2462 (-71.8%)   \n",
       "contrac                           0.4538/0.4348 (-4.2%)   \n",
       "credit-approval                  0.8721/0.7768 (-10.9%)   \n",
       "cylinder-bands                   0.7500/0.6172 (-17.7%)   \n",
       "dermatology                      0.9341/0.3716 (-60.2%)   \n",
       "echocardiogram                   0.8485/0.6462 (-23.8%)   \n",
       "ecoli                            0.8214/0.7381 (-10.1%)   \n",
       "energy-y1                        0.8646/0.3516 (-59.3%)   \n",
       "energy-y2                        0.8750/0.2448 (-72.0%)   \n",
       "fertility                        0.6800/0.8800 (+29.4%)   \n",
       "flags                            0.4167/0.4845 (+16.3%)   \n",
       "glass                             0.5849/0.5888 (+0.7%)   \n",
       "haberman-survival                0.7368/0.6275 (-14.8%)   \n",
       "hayes-roth                       0.7500/0.0357 (-95.2%)   \n",
       "heart-cleveland                  0.5789/0.3510 (-39.4%)   \n",
       "heart-hungarian                  0.8493/0.7551 (-11.1%)   \n",
       "heart-switzerland                0.3871/0.0656 (-83.1%)   \n",
       "heart-va                          0.2800/0.2800 (-0.0%)   \n",
       "hepatitis                         0.8718/0.7922 (-9.1%)   \n",
       "hill-valley                      0.5050/0.5693 (+12.7%)   \n",
       "horse-colic                      0.8529/0.6029 (-29.3%)   \n",
       "ilpd-indian-liver                 0.5959/0.6529 (+9.6%)   \n",
       "image-segmentation               0.8481/0.1686 (-80.1%)   \n",
       "ionosphere                       0.9432/0.7200 (-23.7%)   \n",
       "iris                             0.9189/0.6267 (-31.8%)   \n",
       "led-display                      0.6280/0.4980 (-20.7%)   \n",
       "lenses                           0.8333/0.1667 (-80.0%)   \n",
       "letter                           0.9796/0.0847 (-91.4%)   \n",
       "libras                            0.7444/0.7500 (+0.8%)   \n",
       "low-res-spect                    0.8571/0.6340 (-26.0%)   \n",
       "lung-cancer                      0.5000/0.4375 (-12.5%)   \n",
       "lymphography                      0.7568/0.7973 (+5.4%)   \n",
       "magic                            0.8713/0.6485 (-25.6%)   \n",
       "mammographic                     0.8167/0.6417 (-21.4%)   \n",
       "miniboone                        0.9262/0.7194 (-22.3%)   \n",
       "molec-biol-promoter               0.7692/0.6981 (-9.2%)   \n",
       "molec-biol-splice                0.8519/0.5273 (-38.1%)   \n",
       "monks-1                          0.9074/0.5255 (-42.1%)   \n",
       "monks-2                          0.3287/0.6551 (+99.3%)   \n",
       "monks-3                           0.5278/0.5648 (+7.0%)   \n",
       "mushroom                         0.9990/0.4628 (-53.7%)   \n",
       "musk-1                           0.8235/0.7269 (-11.7%)   \n",
       "musk-2                           0.9982/0.1540 (-84.6%)   \n",
       "nursery                          0.9994/0.6267 (-37.3%)   \n",
       "oocytes_merluccius_nucleus_4d    0.8078/0.7065 (-12.5%)   \n",
       "oocytes_merluccius_states_2f     0.9333/0.6888 (-26.2%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.7456/0.5811 (-22.1%)   \n",
       "oocytes_trisopterus_states_5b    0.8947/0.7873 (-12.0%)   \n",
       "optical                           0.9716/0.9644 (-0.7%)   \n",
       "ozone                             0.9669/0.9685 (+0.2%)   \n",
       "page-blocks                       0.9613/0.9419 (-2.0%)   \n",
       "parkinsons                       0.8571/0.7526 (-12.2%)   \n",
       "pendigits                         0.9734/0.9720 (-0.1%)   \n",
       "pima                              0.7188/0.6719 (-6.5%)   \n",
       "pittsburg-bridges-MATERIAL        0.8846/0.8679 (-1.9%)   \n",
       "pittsburg-bridges-REL-L          0.7692/0.5686 (-26.1%)   \n",
       "pittsburg-bridges-SPAN           0.5652/0.5000 (-11.5%)   \n",
       "pittsburg-bridges-T-OR-D         0.8800/0.7059 (-19.8%)   \n",
       "pittsburg-bridges-TYPE          0.1154/0.5192 (+349.9%)   \n",
       "planning                         0.6222/0.5165 (-17.0%)   \n",
       "plant-margin                     0.7600/0.5375 (-29.3%)   \n",
       "plant-shape                      0.2850/0.2062 (-27.6%)   \n",
       "plant-texture                    0.8200/0.7134 (-13.0%)   \n",
       "post-operative                    0.5909/0.6222 (+5.3%)   \n",
       "primary-tumor                    0.5122/0.2727 (-46.8%)   \n",
       "ringnorm                         0.9843/0.5049 (-48.7%)   \n",
       "seeds                            0.8654/0.6667 (-23.0%)   \n",
       "semeion                          0.9372/0.0992 (-89.4%)   \n",
       "soybean                          0.8883/0.5053 (-43.1%)   \n",
       "spambase                          0.9426/0.8948 (-5.1%)   \n",
       "spect                            0.6344/0.4624 (-27.1%)   \n",
       "spectf                          0.2299/0.6738 (+193.1%)   \n",
       "statlog-australian-credit         0.6802/0.6725 (-1.1%)   \n",
       "statlog-german-credit             0.7520/0.6980 (-7.2%)   \n",
       "statlog-heart                    0.7910/0.4444 (-43.8%)   \n",
       "statlog-image                    0.9671/0.1446 (-85.0%)   \n",
       "statlog-landsat                   0.9040/0.8755 (-3.2%)   \n",
       "statlog-shuttle                   0.9988/0.9961 (-0.3%)   \n",
       "statlog-vehicle                  0.7583/0.6809 (-10.2%)   \n",
       "steel-plates                     0.7031/0.6124 (-12.9%)   \n",
       "synthetic-control                0.9733/0.2133 (-78.1%)   \n",
       "teaching                         0.5000/0.3467 (-30.7%)   \n",
       "thyroid                           0.9778/0.9787 (+0.1%)   \n",
       "tic-tac-toe                      0.9833/0.7724 (-21.4%)   \n",
       "titanic                           0.7800/0.7491 (-4.0%)   \n",
       "trains                           0.5000/0.4000 (-20.0%)   \n",
       "twonorm                           0.9757/0.9127 (-6.5%)   \n",
       "vertebral-column-2clases         0.8312/0.3226 (-61.2%)   \n",
       "vertebral-column-3clases          0.7792/0.7935 (+1.8%)   \n",
       "wall-following                   0.9333/0.1514 (-83.8%)   \n",
       "waveform                         0.8360/0.7064 (-15.5%)   \n",
       "waveform-noise                   0.8480/0.6956 (-18.0%)   \n",
       "wine                             0.9773/0.5056 (-48.3%)   \n",
       "wine-quality-red                 0.5450/0.4631 (-15.0%)   \n",
       "wine-quality-white               0.5335/0.4684 (-12.2%)   \n",
       "yeast                            0.4906/0.0984 (-79.9%)   \n",
       "zoo                              0.7200/0.8600 (+19.4%)   \n",
       "\n",
       "                                           HW (old/new)  \\\n",
       "abalone                          0.6427/0.5383 (-16.2%)   \n",
       "acute-inflammation               1.0000/0.6167 (-38.3%)   \n",
       "acute-nephritis                  1.0000/0.5833 (-41.7%)   \n",
       "adult                             0.8453/0.8524 (+0.8%)   \n",
       "annealing                        0.3600/0.6500 (+80.6%)   \n",
       "arrhythmia                        0.6283/0.6062 (-3.5%)   \n",
       "audiology-std                    0.7200/0.6400 (-11.1%)   \n",
       "balance-scale                     0.9103/0.9679 (+6.3%)   \n",
       "balloons                        0.2500/0.6250 (+150.0%)   \n",
       "bank                              0.8885/0.8221 (-7.5%)   \n",
       "blood                            0.7968/0.6765 (-15.1%)   \n",
       "breast-cancer                    0.7465/0.6573 (-11.9%)   \n",
       "breast-cancer-wisc                0.9771/0.9284 (-5.0%)   \n",
       "breast-cancer-wisc-diag           0.9789/0.9754 (-0.4%)   \n",
       "breast-cancer-wisc-prog          0.8367/0.6869 (-17.9%)   \n",
       "breast-tissue                     0.6154/0.5849 (-5.0%)   \n",
       "car                               0.9560/0.9664 (+1.1%)   \n",
       "cardiotocography-10clases        0.8456/0.7582 (-10.3%)   \n",
       "cardiotocography-3clases          0.9171/0.8965 (-2.2%)   \n",
       "chess-krvk                       0.5255/0.1500 (-71.5%)   \n",
       "chess-krvkp                       0.9900/0.9837 (-0.6%)   \n",
       "congressional-voting             0.5872/0.5115 (-12.9%)   \n",
       "conn-bench-sonar-mines-rocks      0.8462/0.8750 (+3.4%)   \n",
       "conn-bench-vowel-deterding        0.9784/0.9870 (+0.9%)   \n",
       "connect-4                        0.8599/0.3112 (-63.8%)   \n",
       "contrac                          0.5054/0.4321 (-14.5%)   \n",
       "credit-approval                   0.8547/0.8493 (-0.6%)   \n",
       "cylinder-bands                    0.7969/0.7656 (-3.9%)   \n",
       "dermatology                       0.9780/0.9563 (-2.2%)   \n",
       "echocardiogram                   0.6061/0.3231 (-46.7%)   \n",
       "ecoli                             0.8690/0.8690 (+0.0%)   \n",
       "energy-y1                         0.8802/0.9349 (+6.2%)   \n",
       "energy-y2                         0.9010/0.8438 (-6.4%)   \n",
       "fertility                         0.8800/0.8800 (-0.0%)   \n",
       "flags                             0.4375/0.4742 (+8.4%)   \n",
       "glass                             0.6415/0.6075 (-5.3%)   \n",
       "haberman-survival                0.6447/0.2614 (-59.4%)   \n",
       "hayes-roth                       0.7857/0.2143 (-72.7%)   \n",
       "heart-cleveland                  0.6316/0.5497 (-13.0%)   \n",
       "heart-hungarian                   0.7945/0.7891 (-0.7%)   \n",
       "heart-switzerland                0.5806/0.4098 (-29.4%)   \n",
       "heart-va                         0.4000/0.3100 (-22.5%)   \n",
       "hepatitis                        0.6667/0.7922 (+18.8%)   \n",
       "hill-valley                       0.5000/0.4934 (-1.3%)   \n",
       "horse-colic                      0.7794/0.6029 (-22.6%)   \n",
       "ilpd-indian-liver                 0.6781/0.6667 (-1.7%)   \n",
       "image-segmentation               0.9024/0.2114 (-76.6%)   \n",
       "ionosphere                       0.9432/0.6457 (-31.5%)   \n",
       "iris                             0.8378/0.9467 (+13.0%)   \n",
       "led-display                      0.7040/0.6080 (-13.6%)   \n",
       "lenses                           1.0000/0.6667 (-33.3%)   \n",
       "letter                            0.8984/0.8377 (-6.8%)   \n",
       "libras                           0.8222/0.6056 (-26.3%)   \n",
       "low-res-spect                    0.9023/0.5245 (-41.9%)   \n",
       "lung-cancer                     0.1250/0.4375 (+250.0%)   \n",
       "lymphography                     0.7297/0.5541 (-24.1%)   \n",
       "magic                            0.8673/0.5792 (-33.2%)   \n",
       "mammographic                      0.7917/0.7771 (-1.8%)   \n",
       "miniboone                        0.9270/0.7355 (-20.7%)   \n",
       "molec-biol-promoter              0.6923/0.8113 (+17.2%)   \n",
       "molec-biol-splice                 0.8833/0.8401 (-4.9%)   \n",
       "monks-1                          0.5833/0.6435 (+10.3%)   \n",
       "monks-2                           0.6389/0.6713 (+5.1%)   \n",
       "monks-3                           0.5880/0.5764 (-2.0%)   \n",
       "mushroom                         1.0000/0.4820 (-51.8%)   \n",
       "musk-1                           0.8992/0.6765 (-24.8%)   \n",
       "musk-2                            0.9915/0.9900 (-0.2%)   \n",
       "nursery                           1.0000/0.9647 (-3.5%)   \n",
       "oocytes_merluccius_nucleus_4d     0.7176/0.6712 (-6.5%)   \n",
       "oocytes_merluccius_states_2f      0.9490/0.9119 (-3.9%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.8289/0.5789 (-30.2%)   \n",
       "oocytes_trisopterus_states_5b     0.9342/0.8991 (-3.8%)   \n",
       "optical                           0.9644/0.9738 (+1.0%)   \n",
       "ozone                             0.9716/0.9685 (-0.3%)   \n",
       "page-blocks                       0.9656/0.9401 (-2.6%)   \n",
       "parkinsons                        0.8367/0.8763 (+4.7%)   \n",
       "pendigits                         0.9671/0.9605 (-0.7%)   \n",
       "pima                              0.7188/0.7135 (-0.7%)   \n",
       "pittsburg-bridges-MATERIAL        0.9231/0.8679 (-6.0%)   \n",
       "pittsburg-bridges-REL-L          0.6923/0.4510 (-34.9%)   \n",
       "pittsburg-bridges-SPAN           0.5652/0.7609 (+34.6%)   \n",
       "pittsburg-bridges-T-OR-D          0.8800/0.8627 (-2.0%)   \n",
       "pittsburg-bridges-TYPE            0.5385/0.5577 (+3.6%)   \n",
       "planning                         0.6000/0.3956 (-34.1%)   \n",
       "plant-margin                     0.8375/0.6663 (-20.4%)   \n",
       "plant-shape                      0.6325/0.3925 (-37.9%)   \n",
       "plant-texture                     0.7900/0.7860 (-0.5%)   \n",
       "post-operative                   0.5909/0.6889 (+16.6%)   \n",
       "primary-tumor                    0.4512/0.3939 (-12.7%)   \n",
       "ringnorm                          0.9692/0.9673 (-0.2%)   \n",
       "seeds                             0.9423/0.8952 (-5.0%)   \n",
       "semeion                           0.9447/0.9397 (-0.5%)   \n",
       "soybean                          0.8617/0.3856 (-55.2%)   \n",
       "spambase                          0.9435/0.9222 (-2.3%)   \n",
       "spect                             0.6022/0.5484 (-8.9%)   \n",
       "spectf                            0.8930/0.8610 (-3.6%)   \n",
       "statlog-australian-credit         0.6802/0.6232 (-8.4%)   \n",
       "statlog-german-credit             0.7760/0.7120 (-8.2%)   \n",
       "statlog-heart                    0.7761/0.4444 (-42.7%)   \n",
       "statlog-image                     0.9584/0.9195 (-4.1%)   \n",
       "statlog-landsat                   0.9110/0.8920 (-2.1%)   \n",
       "statlog-shuttle                   0.9977/0.9977 (+0.0%)   \n",
       "statlog-vehicle                  0.7962/0.6879 (-13.6%)   \n",
       "steel-plates                      0.7608/0.7082 (-6.9%)   \n",
       "synthetic-control                0.9867/0.1667 (-83.1%)   \n",
       "teaching                         0.5263/0.3867 (-26.5%)   \n",
       "thyroid                           0.9708/0.9770 (+0.6%)   \n",
       "tic-tac-toe                       0.9749/0.9457 (-3.0%)   \n",
       "titanic                          0.7927/0.3227 (-59.3%)   \n",
       "trains                               nan/0.4000 (+nan%)   \n",
       "twonorm                           0.9708/0.9738 (+0.3%)   \n",
       "vertebral-column-2clases         0.8571/0.6581 (-23.2%)   \n",
       "vertebral-column-3clases          0.7922/0.8194 (+3.4%)   \n",
       "wall-following                    0.9230/0.8904 (-3.5%)   \n",
       "waveform                          0.8320/0.8380 (+0.7%)   \n",
       "waveform-noise                    0.8696/0.8368 (-3.8%)   \n",
       "wine                              0.9091/0.9663 (+6.3%)   \n",
       "wine-quality-red                  0.5625/0.5557 (-1.2%)   \n",
       "wine-quality-white               0.5564/0.4565 (-18.0%)   \n",
       "yeast                            0.6065/0.4987 (-17.8%)   \n",
       "zoo                              0.8800/0.2000 (-77.3%)   \n",
       "\n",
       "                                           LN (old/new)    M  \\\n",
       "abalone                           0.6178/0.6049 (-2.1%)    9   \n",
       "acute-inflammation               0.9000/1.0000 (+11.1%)    7   \n",
       "acute-nephritis                   1.0000/1.0000 (-0.0%)    7   \n",
       "adult                             0.8517/0.8522 (+0.1%)   15   \n",
       "annealing                         0.5000/0.4700 (-6.0%)   32   \n",
       "arrhythmia                        0.5752/0.6106 (+6.2%)  263   \n",
       "audiology-std                     0.8000/0.8000 (-0.0%)   60   \n",
       "balance-scale                     0.9872/0.9359 (-5.2%)    5   \n",
       "balloons                         0.7500/0.5000 (-33.3%)    5   \n",
       "bank                              0.8920/0.8735 (-2.1%)   17   \n",
       "blood                            0.7112/0.2380 (-66.5%)    5   \n",
       "breast-cancer                     0.6620/0.6224 (-6.0%)   10   \n",
       "breast-cancer-wisc                0.9714/0.9599 (-1.2%)   10   \n",
       "breast-cancer-wisc-diag           0.9648/0.9683 (+0.4%)   31   \n",
       "breast-cancer-wisc-prog          0.7959/0.5758 (-27.7%)   34   \n",
       "breast-tissue                    0.5769/0.4528 (-21.5%)   10   \n",
       "car                               0.9907/0.9826 (-0.8%)    7   \n",
       "cardiotocography-10clases         0.8362/0.8109 (-3.0%)   22   \n",
       "cardiotocography-3clases          0.9021/0.9040 (+0.2%)   22   \n",
       "chess-krvk                       0.8938/0.1495 (-83.3%)    7   \n",
       "chess-krvkp                       0.9875/0.9762 (-1.1%)   37   \n",
       "congressional-voting              0.5780/0.5576 (-3.5%)   17   \n",
       "conn-bench-sonar-mines-rocks     0.6731/0.8077 (+20.0%)   61   \n",
       "conn-bench-vowel-deterding       0.9935/0.7468 (-24.8%)   12   \n",
       "connect-4                        0.8856/0.2462 (-72.2%)   43   \n",
       "contrac                           0.4592/0.4864 (+5.9%)   10   \n",
       "credit-approval                   0.8547/0.8348 (-2.3%)   16   \n",
       "cylinder-bands                    0.7578/0.7461 (-1.5%)   36   \n",
       "dermatology                       0.9451/0.9563 (+1.2%)   35   \n",
       "echocardiogram                    0.8182/0.7538 (-7.9%)   11   \n",
       "ecoli                             0.8571/0.8452 (-1.4%)    8   \n",
       "energy-y1                        0.9479/0.8021 (-15.4%)    9   \n",
       "energy-y2                         0.8802/0.8880 (+0.9%)    9   \n",
       "fertility                         0.8800/0.8800 (-0.0%)   10   \n",
       "flags                            0.3542/0.1443 (-59.3%)   29   \n",
       "glass                            0.6981/0.5421 (-22.4%)   10   \n",
       "haberman-survival                 0.6842/0.6275 (-8.3%)    4   \n",
       "hayes-roth                       0.8929/0.4643 (-48.0%)    4   \n",
       "heart-cleveland                   0.5789/0.5364 (-7.3%)   14   \n",
       "heart-hungarian                   0.8493/0.7891 (-7.1%)   13   \n",
       "heart-switzerland                 0.5161/0.4754 (-7.9%)   13   \n",
       "heart-va                         0.2400/0.2900 (+20.8%)   13   \n",
       "hepatitis                         0.7436/0.7922 (+6.5%)   20   \n",
       "hill-valley                      0.5050/0.6287 (+24.5%)  101   \n",
       "horse-colic                      0.7941/0.6324 (-20.4%)   26   \n",
       "ilpd-indian-liver                 0.6986/0.6942 (-0.6%)   10   \n",
       "image-segmentation               0.8838/0.4657 (-47.3%)   19   \n",
       "ionosphere                       0.9432/0.6400 (-32.1%)   34   \n",
       "iris                              0.9730/0.9333 (-4.1%)    5   \n",
       "led-display                      0.6480/0.1100 (-83.0%)    8   \n",
       "lenses                           0.6667/0.1667 (-75.0%)    5   \n",
       "letter                           0.9742/0.0367 (-96.2%)   17   \n",
       "libras                           0.8333/0.6667 (-20.0%)   91   \n",
       "low-res-spect                     0.8947/0.8453 (-5.5%)  101   \n",
       "lung-cancer                     0.2500/0.5625 (+125.0%)   57   \n",
       "lymphography                      0.7838/0.7838 (-0.0%)   19   \n",
       "magic                            0.8620/0.7728 (-10.4%)   11   \n",
       "mammographic                      0.8208/0.7896 (-3.8%)    6   \n",
       "miniboone                        0.9313/0.7450 (-20.0%)   51   \n",
       "molec-biol-promoter              0.4615/0.7358 (+59.4%)   58   \n",
       "molec-biol-splice                0.8607/0.7197 (-16.4%)   61   \n",
       "monks-1                           0.7014/0.6829 (-2.6%)    7   \n",
       "monks-2                          0.5162/0.6690 (+29.6%)    7   \n",
       "monks-3                          0.6991/0.5486 (-21.5%)    7   \n",
       "mushroom                          0.9995/0.9183 (-8.1%)   22   \n",
       "musk-1                            0.8992/0.8445 (-6.1%)  167   \n",
       "musk-2                           0.9951/0.1540 (-84.5%)  167   \n",
       "nursery                          0.9966/0.3120 (-68.7%)    9   \n",
       "oocytes_merluccius_nucleus_4d     0.7686/0.7847 (+2.1%)   42   \n",
       "oocytes_merluccius_states_2f      0.9412/0.9061 (-3.7%)   26   \n",
       "oocytes_trisopterus_nucleus_2f   0.8202/0.6031 (-26.5%)   26   \n",
       "oocytes_trisopterus_states_5b     0.8991/0.9057 (+0.7%)   33   \n",
       "optical                           0.9755/0.9699 (-0.6%)   63   \n",
       "ozone                             0.9716/0.9669 (-0.5%)   73   \n",
       "page-blocks                       0.9708/0.9550 (-1.6%)   11   \n",
       "parkinsons                        0.8571/0.9175 (+7.1%)   23   \n",
       "pendigits                         0.9657/0.9720 (+0.7%)   17   \n",
       "pima                              0.6927/0.7474 (+7.9%)    9   \n",
       "pittsburg-bridges-MATERIAL       0.9231/0.7170 (-22.3%)    8   \n",
       "pittsburg-bridges-REL-L          0.7308/0.1373 (-81.2%)    8   \n",
       "pittsburg-bridges-SPAN            0.6087/0.6304 (+3.6%)    8   \n",
       "pittsburg-bridges-T-OR-D          0.8800/0.8627 (-2.0%)    8   \n",
       "pittsburg-bridges-TYPE           0.6538/0.5769 (-11.8%)    8   \n",
       "planning                          0.6889/0.7143 (+3.7%)   13   \n",
       "plant-margin                      0.8425/0.7688 (-8.8%)   65   \n",
       "plant-shape                       0.6775/0.6388 (-5.7%)   65   \n",
       "plant-texture                    0.8350/0.0100 (-98.8%)   65   \n",
       "post-operative                   0.7727/0.0222 (-97.1%)    9   \n",
       "primary-tumor                    0.4512/0.3455 (-23.4%)   18   \n",
       "ringnorm                         0.9827/0.5049 (-48.6%)   21   \n",
       "seeds                             0.8846/0.8952 (+1.2%)    8   \n",
       "semeion                          0.9447/0.0992 (-89.5%)  257   \n",
       "soybean                          0.8484/0.4468 (-47.3%)   36   \n",
       "spambase                          0.9513/0.9230 (-3.0%)   58   \n",
       "spect                             0.6720/0.6237 (-7.2%)   23   \n",
       "spectf                           0.5561/0.9144 (+64.4%)   45   \n",
       "statlog-australian-credit         0.6279/0.6783 (+8.0%)   15   \n",
       "statlog-german-credit             0.7400/0.7420 (+0.3%)   25   \n",
       "statlog-heart                     0.7910/0.8444 (+6.8%)   14   \n",
       "statlog-image                    0.9757/0.1429 (-85.4%)   19   \n",
       "statlog-landsat                   0.9040/0.8945 (-1.1%)   37   \n",
       "statlog-shuttle                   0.9987/0.9991 (+0.0%)   10   \n",
       "statlog-vehicle                   0.7915/0.7636 (-3.5%)   19   \n",
       "steel-plates                     0.7588/0.3474 (-54.2%)   28   \n",
       "synthetic-control                0.9733/0.2067 (-78.8%)   61   \n",
       "teaching                         0.6316/0.4400 (-30.3%)    6   \n",
       "thyroid                           0.9752/0.9761 (+0.1%)   22   \n",
       "tic-tac-toe                       0.9791/0.9645 (-1.5%)   10   \n",
       "titanic                          0.7891/0.3227 (-59.1%)    4   \n",
       "trains                           1.0000/0.6000 (-40.0%)   30   \n",
       "twonorm                           0.9724/0.9732 (+0.1%)   21   \n",
       "vertebral-column-2clases          0.8442/0.8000 (-5.2%)    7   \n",
       "vertebral-column-3clases          0.8312/0.8065 (-3.0%)    7   \n",
       "wall-following                   0.9128/0.0601 (-93.4%)   25   \n",
       "waveform                          0.8448/0.8592 (+1.7%)   22   \n",
       "waveform-noise                    0.8504/0.8480 (-0.3%)   41   \n",
       "wine                             0.9773/0.5393 (-44.8%)   14   \n",
       "wine-quality-red                  0.6100/0.5820 (-4.6%)   12   \n",
       "wine-quality-white               0.6544/0.5892 (-10.0%)   12   \n",
       "yeast                             0.6092/0.5620 (-7.7%)    9   \n",
       "zoo                              0.9600/0.8600 (-10.4%)   17   \n",
       "\n",
       "                                          MS (old/new)       N  \\\n",
       "abalone                          0.6284/0.6130 (-2.4%)    4177   \n",
       "acute-inflammation               1.0000/1.0000 (-0.0%)     120   \n",
       "acute-nephritis                  1.0000/1.0000 (-0.0%)     120   \n",
       "adult                            0.8487/0.8511 (+0.3%)   48842   \n",
       "annealing                        0.7300/0.7500 (+2.7%)     898   \n",
       "arrhythmia                       0.6372/0.5929 (-6.9%)     452   \n",
       "audiology-std                    0.6800/0.7200 (+5.9%)     196   \n",
       "balance-scale                    0.9231/0.9615 (+4.2%)     625   \n",
       "balloons                         0.5000/0.5000 (-0.0%)      16   \n",
       "bank                             0.8876/0.8854 (-0.2%)    4521   \n",
       "blood                           0.7754/0.2380 (-69.3%)     748   \n",
       "breast-cancer                    0.6901/0.6713 (-2.7%)     286   \n",
       "breast-cancer-wisc               0.9714/0.9599 (-1.2%)     699   \n",
       "breast-cancer-wisc-diag          0.9718/0.9472 (-2.5%)     569   \n",
       "breast-cancer-wisc-prog         0.7347/0.6162 (-16.1%)     198   \n",
       "breast-tissue                   0.4615/0.6792 (+47.2%)     106   \n",
       "car                              0.9861/0.9213 (-6.6%)    1728   \n",
       "cardiotocography-10clases       0.8418/0.5814 (-30.9%)    2126   \n",
       "cardiotocography-3clases         0.8964/0.9106 (+1.6%)    2126   \n",
       "chess-krvk                      0.8606/0.1823 (-78.8%)   28056   \n",
       "chess-krvkp                     0.9900/0.5225 (-47.2%)    3196   \n",
       "congressional-voting            0.6055/0.5253 (-13.2%)     435   \n",
       "conn-bench-sonar-mines-rocks     0.8269/0.8365 (+1.2%)     208   \n",
       "conn-bench-vowel-deterding       0.9935/0.9978 (+0.4%)     990   \n",
       "connect-4                       0.8831/0.3711 (-58.0%)   67557   \n",
       "contrac                          0.5136/0.4946 (-3.7%)    1473   \n",
       "credit-approval                  0.8430/0.8087 (-4.1%)     690   \n",
       "cylinder-bands                   0.7656/0.7031 (-8.2%)     512   \n",
       "dermatology                      0.9121/0.9672 (+6.0%)     366   \n",
       "echocardiogram                  0.8485/0.6923 (-18.4%)     131   \n",
       "ecoli                            0.8333/0.8333 (+0.0%)     336   \n",
       "energy-y1                        0.9583/0.9089 (-5.2%)     768   \n",
       "energy-y2                        0.8958/0.9089 (+1.5%)     768   \n",
       "fertility                       0.8800/0.1200 (-86.4%)     100   \n",
       "flags                           0.4583/0.5464 (+19.2%)     194   \n",
       "glass                            0.6038/0.6449 (+6.8%)     214   \n",
       "haberman-survival                0.7237/0.6993 (-3.4%)     306   \n",
       "hayes-roth                      0.4643/0.0357 (-92.3%)     160   \n",
       "heart-cleveland                 0.6053/0.5232 (-13.6%)     303   \n",
       "heart-hungarian                  0.8356/0.8503 (+1.8%)     294   \n",
       "heart-switzerland                0.3871/0.3607 (-6.8%)     123   \n",
       "heart-va                        0.2600/0.3000 (+15.4%)     200   \n",
       "hepatitis                        0.7692/0.7922 (+3.0%)     155   \n",
       "hill-valley                     0.5116/0.7030 (+37.4%)    1212   \n",
       "horse-colic                     0.8529/0.6029 (-29.3%)     368   \n",
       "ilpd-indian-liver                0.6644/0.6701 (+0.9%)     583   \n",
       "image-segmentation              0.9090/0.2310 (-74.6%)    2310   \n",
       "ionosphere                       0.9091/0.9143 (+0.6%)     351   \n",
       "iris                             0.9189/0.8933 (-2.8%)     150   \n",
       "led-display                      0.7200/0.7000 (-2.8%)    1000   \n",
       "lenses                          1.0000/0.1667 (-83.3%)      24   \n",
       "letter                          0.9712/0.0401 (-95.9%)   20000   \n",
       "libras                           0.8667/0.7944 (-8.3%)     360   \n",
       "low-res-spect                    0.8496/0.8604 (+1.3%)     531   \n",
       "lung-cancer                     0.3750/0.3125 (-16.7%)      32   \n",
       "lymphography                    0.7297/0.8378 (+14.8%)     148   \n",
       "magic                           0.8629/0.4118 (-52.3%)   19020   \n",
       "mammographic                     0.8083/0.7917 (-2.1%)     961   \n",
       "miniboone                       0.9250/0.7641 (-17.4%)  130064   \n",
       "molec-biol-promoter             0.7692/0.6226 (-19.1%)     106   \n",
       "molec-biol-splice                0.8482/0.8031 (-5.3%)    3190   \n",
       "monks-1                         0.6551/0.7523 (+14.8%)     556   \n",
       "monks-2                          0.6343/0.6644 (+4.7%)     601   \n",
       "monks-3                         0.7454/0.5417 (-27.3%)     554   \n",
       "mushroom                         1.0000/0.9953 (-0.5%)    8124   \n",
       "musk-1                          0.8655/0.7773 (-10.2%)     476   \n",
       "musk-2                          0.9945/0.1540 (-84.5%)    6598   \n",
       "nursery                         0.9988/0.3120 (-68.8%)   12960   \n",
       "oocytes_merluccius_nucleus_4d    0.8196/0.7730 (-5.7%)    1022   \n",
       "oocytes_merluccius_states_2f     0.9490/0.8943 (-5.8%)    1022   \n",
       "oocytes_trisopterus_nucleus_2f  0.8728/0.7654 (-12.3%)     912   \n",
       "oocytes_trisopterus_states_5b   0.9430/0.4079 (-56.7%)     912   \n",
       "optical                          0.9666/0.9688 (+0.2%)    5620   \n",
       "ozone                           0.9732/0.0284 (-97.1%)    2536   \n",
       "page-blocks                     0.9708/0.0599 (-93.8%)    5473   \n",
       "parkinsons                      0.9184/0.7526 (-18.1%)     195   \n",
       "pendigits                        0.9714/0.9777 (+0.6%)   10992   \n",
       "pima                             0.7656/0.7344 (-4.1%)     768   \n",
       "pittsburg-bridges-MATERIAL       0.8462/0.8679 (+2.6%)     106   \n",
       "pittsburg-bridges-REL-L         0.7692/0.5686 (-26.1%)     103   \n",
       "pittsburg-bridges-SPAN          0.5217/0.7391 (+41.7%)      92   \n",
       "pittsburg-bridges-T-OR-D         0.8800/0.8627 (-2.0%)     102   \n",
       "pittsburg-bridges-TYPE          0.6538/0.5577 (-14.7%)     105   \n",
       "planning                        0.6667/0.4615 (-30.8%)     182   \n",
       "plant-margin                     0.8125/0.7950 (-2.2%)    1600   \n",
       "plant-shape                     0.6350/0.5175 (-18.5%)    1600   \n",
       "plant-texture                    0.7900/0.7972 (+0.9%)    1599   \n",
       "post-operative                   0.7273/0.7111 (-2.2%)      90   \n",
       "primary-tumor                   0.5000/0.4121 (-17.6%)     330   \n",
       "ringnorm                         0.9843/0.9762 (-0.8%)    7400   \n",
       "seeds                           0.8654/0.5048 (-41.7%)     210   \n",
       "semeion                          0.9296/0.9296 (+0.0%)    1593   \n",
       "soybean                         0.8723/0.3590 (-58.8%)     683   \n",
       "spambase                        0.9461/0.3939 (-58.4%)    4601   \n",
       "spect                           0.6183/0.5484 (-11.3%)     265   \n",
       "spectf                          0.6043/0.9198 (+52.2%)     267   \n",
       "statlog-australian-credit        0.6802/0.6667 (-2.0%)     690   \n",
       "statlog-german-credit            0.7280/0.7000 (-3.8%)    1000   \n",
       "statlog-heart                    0.8358/0.8593 (+2.8%)     270   \n",
       "statlog-image                   0.9757/0.8234 (-15.6%)    2310   \n",
       "statlog-landsat                 0.9075/0.2305 (-74.6%)    6435   \n",
       "statlog-shuttle                  0.9983/0.9989 (+0.1%)   58000   \n",
       "statlog-vehicle                  0.8294/0.7612 (-8.2%)     846   \n",
       "steel-plates                     0.7567/0.6887 (-9.0%)    1941   \n",
       "synthetic-control               0.9800/0.7033 (-28.2%)     600   \n",
       "teaching                        0.6053/0.3333 (-44.9%)     151   \n",
       "thyroid                          0.9770/0.9767 (-0.0%)    7200   \n",
       "tic-tac-toe                      0.9833/0.9353 (-4.9%)     958   \n",
       "titanic                         0.7909/0.5464 (-30.9%)    2201   \n",
       "trains                              nan/0.4000 (+nan%)      10   \n",
       "twonorm                          0.9778/0.9700 (-0.8%)    7400   \n",
       "vertebral-column-2clases        0.8701/0.7548 (-13.2%)     310   \n",
       "vertebral-column-3clases         0.8052/0.8452 (+5.0%)     310   \n",
       "wall-following                  0.9076/0.4637 (-48.9%)    5456   \n",
       "waveform                         0.8312/0.8464 (+1.8%)    5000   \n",
       "waveform-noise                   0.8328/0.8252 (-0.9%)    5000   \n",
       "wine                            0.9318/0.2809 (-69.9%)     178   \n",
       "wine-quality-red                 0.6250/0.5770 (-7.7%)    1599   \n",
       "wine-quality-white              0.6479/0.4924 (-24.0%)    4898   \n",
       "yeast                            0.6173/0.5768 (-6.6%)    1484   \n",
       "zoo                              1.0000/0.9400 (-6.0%)     101   \n",
       "\n",
       "                                       ResNet (old/new)  \\\n",
       "abalone                          0.6466/0.3463 (-46.4%)   \n",
       "acute-inflammation               1.0000/0.7000 (-30.0%)   \n",
       "acute-nephritis                  1.0000/0.5833 (-41.7%)   \n",
       "adult                             0.8484/0.8502 (+0.2%)   \n",
       "annealing                       0.2600/0.7400 (+184.6%)   \n",
       "arrhythmia                       0.6460/0.4159 (-35.6%)   \n",
       "audiology-std                    0.8000/0.5200 (-35.0%)   \n",
       "balance-scale                    0.9167/0.4744 (-48.3%)   \n",
       "balloons                         1.0000/0.8750 (-12.5%)   \n",
       "bank                              0.8796/0.8801 (+0.1%)   \n",
       "blood                            0.8021/0.5963 (-25.7%)   \n",
       "breast-cancer                     0.7465/0.7203 (-3.5%)   \n",
       "breast-cancer-wisc               0.9714/0.4298 (-55.8%)   \n",
       "breast-cancer-wisc-diag          0.9507/0.7430 (-21.9%)   \n",
       "breast-cancer-wisc-prog          0.8163/0.6162 (-24.5%)   \n",
       "breast-tissue                    0.4231/0.6792 (+60.5%)   \n",
       "car                              0.9282/0.1493 (-83.9%)   \n",
       "cardiotocography-10clases        0.8173/0.2427 (-70.3%)   \n",
       "cardiotocography-3clases         0.9021/0.7893 (-12.5%)   \n",
       "chess-krvk                       0.8543/0.1647 (-80.7%)   \n",
       "chess-krvkp                      0.9912/0.5763 (-41.9%)   \n",
       "congressional-voting             0.5963/0.4286 (-28.1%)   \n",
       "conn-bench-sonar-mines-rocks     0.8077/0.6731 (-16.7%)   \n",
       "conn-bench-vowel-deterding       0.9935/0.8485 (-14.6%)   \n",
       "connect-4                        0.8716/0.2940 (-66.3%)   \n",
       "contrac                          0.5136/0.3465 (-32.5%)   \n",
       "credit-approval                  0.8430/0.7449 (-11.6%)   \n",
       "cylinder-bands                    0.7734/0.7305 (-5.6%)   \n",
       "dermatology                       0.9231/0.9016 (-2.3%)   \n",
       "echocardiogram                   0.8485/0.6615 (-22.0%)   \n",
       "ecoli                            0.8214/0.5833 (-29.0%)   \n",
       "energy-y1                        0.8177/0.1667 (-79.6%)   \n",
       "energy-y2                        0.8750/0.6849 (-21.7%)   \n",
       "fertility                         0.8400/0.8800 (+4.8%)   \n",
       "flags                             0.3750/0.4021 (+7.2%)   \n",
       "glass                            0.6415/0.4766 (-25.7%)   \n",
       "haberman-survival                0.6842/0.3660 (-46.5%)   \n",
       "hayes-roth                       0.7143/0.0357 (-95.0%)   \n",
       "heart-cleveland                   0.5658/0.6093 (+7.7%)   \n",
       "heart-hungarian                   0.8082/0.7347 (-9.1%)   \n",
       "heart-switzerland                0.3226/0.3607 (+11.8%)   \n",
       "heart-va                         0.2600/0.3400 (+30.8%)   \n",
       "hepatitis                         0.7692/0.7922 (+3.0%)   \n",
       "hill-valley                       0.5396/0.5776 (+7.0%)   \n",
       "horse-colic                      0.8088/0.6029 (-25.5%)   \n",
       "ilpd-indian-liver                 0.6712/0.6976 (+3.9%)   \n",
       "image-segmentation               0.8919/0.1881 (-78.9%)   \n",
       "ionosphere                       0.9545/0.7200 (-24.6%)   \n",
       "iris                             0.9730/0.7467 (-23.3%)   \n",
       "led-display                      0.7160/0.2080 (-70.9%)   \n",
       "lenses                            0.6667/0.6667 (-0.0%)   \n",
       "letter                           0.9762/0.3086 (-68.4%)   \n",
       "libras                           0.7111/0.5333 (-25.0%)   \n",
       "low-res-spect                    0.8647/0.3170 (-63.3%)   \n",
       "lung-cancer                     0.2500/0.5000 (+100.0%)   \n",
       "lymphography                      0.6757/0.7027 (+4.0%)   \n",
       "magic                            0.8723/0.3516 (-59.7%)   \n",
       "mammographic                     0.7833/0.4000 (-48.9%)   \n",
       "miniboone                        0.9254/0.7194 (-22.3%)   \n",
       "molec-biol-promoter              0.7692/0.6604 (-14.1%)   \n",
       "molec-biol-splice                0.8557/0.6075 (-29.0%)   \n",
       "monks-1                          0.7546/0.5208 (-31.0%)   \n",
       "monks-2                           0.6273/0.6713 (+7.0%)   \n",
       "monks-3                          0.5833/0.6806 (+16.7%)   \n",
       "mushroom                         1.0000/0.4820 (-51.8%)   \n",
       "musk-1                           0.8739/0.7311 (-16.3%)   \n",
       "musk-2                            0.9964/0.9536 (-4.3%)   \n",
       "nursery                          0.9994/0.5350 (-46.5%)   \n",
       "oocytes_merluccius_nucleus_4d    0.8000/0.6732 (-15.9%)   \n",
       "oocytes_merluccius_states_2f     0.9373/0.6849 (-26.9%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.7719/0.6162 (-20.2%)   \n",
       "oocytes_trisopterus_states_5b    0.8947/0.6754 (-24.5%)   \n",
       "optical                           0.9627/0.9627 (+0.0%)   \n",
       "ozone                             0.9669/0.9700 (+0.3%)   \n",
       "page-blocks                       0.9605/0.9364 (-2.5%)   \n",
       "parkinsons                        0.9184/0.8866 (-3.5%)   \n",
       "pendigits                         0.9708/0.9723 (+0.2%)   \n",
       "pima                             0.7135/0.3490 (-51.1%)   \n",
       "pittsburg-bridges-MATERIAL       0.9231/0.8302 (-10.1%)   \n",
       "pittsburg-bridges-REL-L          0.8462/0.6863 (-18.9%)   \n",
       "pittsburg-bridges-SPAN            0.5652/0.6087 (+7.7%)   \n",
       "pittsburg-bridges-T-OR-D          0.8800/0.8627 (-2.0%)   \n",
       "pittsburg-bridges-TYPE            0.6538/0.6538 (+0.0%)   \n",
       "planning                         0.7111/0.5385 (-24.3%)   \n",
       "plant-margin                     0.7975/0.4200 (-47.3%)   \n",
       "plant-shape                      0.5150/0.2575 (-50.0%)   \n",
       "plant-texture                    0.8000/0.5419 (-32.3%)   \n",
       "post-operative                   0.7273/0.5556 (-23.6%)   \n",
       "primary-tumor                    0.3902/0.2788 (-28.6%)   \n",
       "ringnorm                         0.9811/0.5503 (-43.9%)   \n",
       "seeds                            0.8654/0.6190 (-28.5%)   \n",
       "semeion                          0.9146/0.5452 (-40.4%)   \n",
       "soybean                          0.8670/0.3750 (-56.7%)   \n",
       "spambase                         0.9461/0.7070 (-25.3%)   \n",
       "spect                            0.6667/0.5484 (-17.7%)   \n",
       "spectf                           0.7005/0.9198 (+31.3%)   \n",
       "statlog-australian-credit         0.6395/0.6783 (+6.1%)   \n",
       "statlog-german-credit            0.7720/0.5840 (-24.4%)   \n",
       "statlog-heart                    0.8657/0.6889 (-20.4%)   \n",
       "statlog-image                    0.9584/0.3281 (-65.8%)   \n",
       "statlog-landsat                   0.9055/0.8620 (-4.8%)   \n",
       "statlog-shuttle                   0.9992/0.9970 (-0.2%)   \n",
       "statlog-vehicle                  0.7583/0.5910 (-22.1%)   \n",
       "steel-plates                     0.7629/0.4227 (-44.6%)   \n",
       "synthetic-control                0.9600/0.4433 (-53.8%)   \n",
       "teaching                         0.5526/0.3467 (-37.3%)   \n",
       "thyroid                           0.9799/0.9770 (-0.3%)   \n",
       "tic-tac-toe                      0.9623/0.8246 (-14.3%)   \n",
       "titanic                          0.7727/0.3227 (-58.2%)   \n",
       "trains                               nan/0.8000 (+nan%)   \n",
       "twonorm                          0.9735/0.4995 (-48.7%)   \n",
       "vertebral-column-2clases         0.8312/0.6452 (-22.4%)   \n",
       "vertebral-column-3clases          0.7532/0.7097 (-5.8%)   \n",
       "wall-following                   0.9223/0.5154 (-44.1%)   \n",
       "waveform                         0.8360/0.4664 (-44.2%)   \n",
       "waveform-noise                   0.8584/0.6572 (-23.4%)   \n",
       "wine                             0.9773/0.6517 (-33.3%)   \n",
       "wine-quality-red                 0.6150/0.4456 (-27.6%)   \n",
       "wine-quality-white               0.6307/0.4945 (-21.6%)   \n",
       "yeast                            0.5499/0.3976 (-27.7%)   \n",
       "zoo                               1.0000/0.9400 (-6.0%)   \n",
       "\n",
       "                                         SNN (old/new)  \\\n",
       "abalone                         0.6657/0.3166 (-52.4%)   \n",
       "acute-inflammation               1.0000/1.0000 (-0.0%)   \n",
       "acute-nephritis                  1.0000/1.0000 (-0.0%)   \n",
       "adult                            0.8476/0.8527 (+0.6%)   \n",
       "annealing                       0.7600/0.1300 (-82.9%)   \n",
       "arrhythmia                       0.6549/0.5973 (-8.8%)   \n",
       "audiology-std                   0.8000/0.6800 (-15.0%)   \n",
       "balance-scale                    0.9231/0.9679 (+4.9%)   \n",
       "balloons                        1.0000/0.3750 (-62.5%)   \n",
       "bank                             0.8903/0.8677 (-2.5%)   \n",
       "blood                            0.7701/0.7674 (-0.4%)   \n",
       "breast-cancer                   0.7183/0.5734 (-20.2%)   \n",
       "breast-cancer-wisc               0.9714/0.9484 (-2.4%)   \n",
       "breast-cancer-wisc-diag          0.9789/0.9859 (+0.7%)   \n",
       "breast-cancer-wisc-prog          0.6735/0.7071 (+5.0%)   \n",
       "breast-tissue                    0.7308/0.6981 (-4.5%)   \n",
       "car                             0.9838/0.7014 (-28.7%)   \n",
       "cardiotocography-10clases        0.8399/0.7846 (-6.6%)   \n",
       "cardiotocography-3clases         0.9153/0.9219 (+0.7%)   \n",
       "chess-krvk                      0.8805/0.1625 (-81.5%)   \n",
       "chess-krvkp                     0.9912/0.5225 (-47.3%)   \n",
       "congressional-voting            0.6147/0.3871 (-37.0%)   \n",
       "conn-bench-sonar-mines-rocks     0.7885/0.7692 (-2.4%)   \n",
       "conn-bench-vowel-deterding       0.9957/0.9978 (+0.2%)   \n",
       "connect-4                       0.8807/0.4176 (-52.6%)   \n",
       "contrac                          0.5190/0.5177 (-0.3%)   \n",
       "credit-approval                  0.8430/0.8841 (+4.9%)   \n",
       "cylinder-bands                   0.7266/0.7461 (+2.7%)   \n",
       "dermatology                      0.9231/0.9781 (+6.0%)   \n",
       "echocardiogram                  0.8182/0.7231 (-11.6%)   \n",
       "ecoli                            0.8929/0.8631 (-3.3%)   \n",
       "energy-y1                       0.9583/0.1771 (-81.5%)   \n",
       "energy-y2                        0.9063/0.9323 (+2.9%)   \n",
       "fertility                        0.9200/0.8800 (-4.3%)   \n",
       "flags                            0.4583/0.4742 (+3.5%)   \n",
       "glass                            0.7358/0.7103 (-3.5%)   \n",
       "haberman-survival                0.7368/0.6732 (-8.6%)   \n",
       "hayes-roth                      0.6786/0.0357 (-94.7%)   \n",
       "heart-cleveland                  0.6184/0.5828 (-5.8%)   \n",
       "heart-hungarian                  0.7945/0.7959 (+0.2%)   \n",
       "heart-switzerland               0.3548/0.0656 (-81.5%)   \n",
       "heart-va                        0.3600/0.2600 (-27.8%)   \n",
       "hepatitis                        0.7692/0.7922 (+3.0%)   \n",
       "hill-valley                      0.5248/0.5627 (+7.2%)   \n",
       "horse-colic                     0.8088/0.2794 (-65.5%)   \n",
       "ilpd-indian-liver                0.6986/0.6976 (-0.1%)   \n",
       "image-segmentation              0.9114/0.1724 (-81.1%)   \n",
       "ionosphere                       0.8864/0.8743 (-1.4%)   \n",
       "iris                             0.9730/0.8800 (-9.6%)   \n",
       "led-display                      0.7640/0.7180 (-6.0%)   \n",
       "lenses                          0.6667/0.7500 (+12.5%)   \n",
       "letter                          0.9726/0.0401 (-95.9%)   \n",
       "libras                           0.7889/0.7667 (-2.8%)   \n",
       "low-res-spect                    0.8571/0.8679 (+1.3%)   \n",
       "lung-cancer                     0.6250/0.5000 (-20.0%)   \n",
       "lymphography                    0.9189/0.7973 (-13.2%)   \n",
       "magic                           0.8692/0.6707 (-22.8%)   \n",
       "mammographic                     0.8250/0.8021 (-2.8%)   \n",
       "miniboone                       0.9307/0.7194 (-22.7%)   \n",
       "molec-biol-promoter              0.8462/0.8491 (+0.3%)   \n",
       "molec-biol-splice               0.9009/0.2408 (-73.3%)   \n",
       "monks-1                         0.7523/0.6458 (-14.2%)   \n",
       "monks-2                         0.5926/0.6667 (+12.5%)   \n",
       "monks-3                          0.6042/0.6435 (+6.5%)   \n",
       "mushroom                         1.0000/0.9983 (-0.2%)   \n",
       "musk-1                           0.8739/0.8613 (-1.4%)   \n",
       "musk-2                          0.9891/0.1540 (-84.4%)   \n",
       "nursery                          0.9978/0.9347 (-6.3%)   \n",
       "oocytes_merluccius_nucleus_4d    0.8235/0.8141 (-1.1%)   \n",
       "oocytes_merluccius_states_2f     0.9529/0.9119 (-4.3%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.7982/0.7632 (-4.4%)   \n",
       "oocytes_trisopterus_states_5b    0.9342/0.9057 (-3.1%)   \n",
       "optical                          0.9711/0.9638 (-0.7%)   \n",
       "ozone                            0.9700/0.9685 (-0.2%)   \n",
       "page-blocks                      0.9583/0.9437 (-1.5%)   \n",
       "parkinsons                      0.8980/0.7732 (-13.9%)   \n",
       "pendigits                        0.9706/0.9711 (+0.1%)   \n",
       "pima                             0.7552/0.7552 (+0.0%)   \n",
       "pittsburg-bridges-MATERIAL       0.8846/0.8302 (-6.2%)   \n",
       "pittsburg-bridges-REL-L         0.6923/0.5294 (-23.5%)   \n",
       "pittsburg-bridges-SPAN          0.6957/0.6087 (-12.5%)   \n",
       "pittsburg-bridges-T-OR-D         0.8400/0.8627 (+2.7%)   \n",
       "pittsburg-bridges-TYPE          0.6538/0.5192 (-20.6%)   \n",
       "planning                         0.6889/0.7143 (+3.7%)   \n",
       "plant-margin                     0.8125/0.7638 (-6.0%)   \n",
       "plant-shape                     0.7275/0.5637 (-22.5%)   \n",
       "plant-texture                    0.8125/0.7935 (-2.3%)   \n",
       "post-operative                  0.7273/0.2667 (-63.3%)   \n",
       "primary-tumor                   0.5244/0.4061 (-22.6%)   \n",
       "ringnorm                         0.9751/0.9832 (+0.8%)   \n",
       "seeds                            0.8846/0.9048 (+2.3%)   \n",
       "semeion                          0.9196/0.9209 (+0.1%)   \n",
       "soybean                         0.8511/0.0239 (-97.2%)   \n",
       "spambase                         0.9409/0.9243 (-1.8%)   \n",
       "spect                           0.6398/0.5645 (-11.8%)   \n",
       "spectf                          0.4973/0.9198 (+85.0%)   \n",
       "statlog-australian-credit        0.5988/0.6377 (+6.5%)   \n",
       "statlog-german-credit            0.7560/0.7720 (+2.1%)   \n",
       "statlog-heart                    0.9254/0.8593 (-7.1%)   \n",
       "statlog-image                    0.9549/0.9498 (-0.5%)   \n",
       "statlog-landsat                  0.9100/0.8780 (-3.5%)   \n",
       "statlog-shuttle                 0.9990/0.0009 (-99.9%)   \n",
       "statlog-vehicle                  0.8009/0.7329 (-8.5%)   \n",
       "steel-plates                    0.7835/0.7052 (-10.0%)   \n",
       "synthetic-control               0.9867/0.7833 (-20.6%)   \n",
       "teaching                        0.5000/0.3333 (-33.3%)   \n",
       "thyroid                          0.9816/0.9772 (-0.4%)   \n",
       "tic-tac-toe                      0.9665/0.9812 (+1.5%)   \n",
       "titanic                         0.7836/0.3300 (-57.9%)   \n",
       "trains                              nan/0.4000 (+nan%)   \n",
       "twonorm                         0.9805/0.4995 (-49.1%)   \n",
       "vertebral-column-2clases         0.8312/0.7806 (-6.1%)   \n",
       "vertebral-column-3clases         0.8312/0.8129 (-2.2%)   \n",
       "wall-following                  0.9098/0.1514 (-83.4%)   \n",
       "waveform                         0.8480/0.8304 (-2.1%)   \n",
       "waveform-noise                   0.8608/0.8336 (-3.2%)   \n",
       "wine                             0.9773/0.9888 (+1.2%)   \n",
       "wine-quality-red                0.6300/0.5081 (-19.3%)   \n",
       "wine-quality-white              0.6373/0.5517 (-13.4%)   \n",
       "yeast                           0.6307/0.4704 (-25.4%)   \n",
       "zoo                              0.9200/0.9800 (+6.5%)   \n",
       "\n",
       "                                                  WN (old/new)  \n",
       "abalone                                 0.6351/0.5465 (-14.0%)  \n",
       "acute-inflammation                       1.0000/1.0000 (-0.0%)  \n",
       "acute-nephritis                          1.0000/1.0000 (-0.0%)  \n",
       "adult                                    0.8453/0.8492 (+0.5%)  \n",
       "annealing                               0.6500/0.7700 (+18.5%)  \n",
       "arrhythmia                               0.6018/0.5796 (-3.7%)  \n",
       "audiology-std                            0.7200/0.7200 (-0.0%)  \n",
       "balance-scale                            0.9551/0.9295 (-2.7%)  \n",
       "balloons                        0.0000/0.7500 (+7499999900.0%)  \n",
       "bank                                     0.8850/0.8664 (-2.1%)  \n",
       "blood                                    0.7594/0.7086 (-6.7%)  \n",
       "breast-cancer                            0.6197/0.5734 (-7.5%)  \n",
       "breast-cancer-wisc                       0.9657/0.9570 (-0.9%)  \n",
       "breast-cancer-wisc-diag                  0.9718/0.9824 (+1.1%)  \n",
       "breast-cancer-wisc-prog                 0.8367/0.6768 (-19.1%)  \n",
       "breast-tissue                           0.5385/0.6038 (+12.1%)  \n",
       "car                                      0.9769/0.9826 (+0.6%)  \n",
       "cardiotocography-10clases                0.8606/0.7752 (-9.9%)  \n",
       "cardiotocography-3clases                0.8945/0.1383 (-84.5%)  \n",
       "chess-krvk                              0.7673/0.1625 (-78.8%)  \n",
       "chess-krvkp                              0.9912/0.9781 (-1.3%)  \n",
       "congressional-voting                    0.5872/0.4931 (-16.0%)  \n",
       "conn-bench-sonar-mines-rocks            0.8269/0.4615 (-44.2%)  \n",
       "conn-bench-vowel-deterding              0.9524/0.7273 (-23.6%)  \n",
       "connect-4                               0.8833/0.3747 (-57.6%)  \n",
       "contrac                                  0.4755/0.4878 (+2.6%)  \n",
       "credit-approval                         0.9070/0.7884 (-13.1%)  \n",
       "cylinder-bands                           0.7578/0.7305 (-3.6%)  \n",
       "dermatology                              0.9451/0.9727 (+2.9%)  \n",
       "echocardiogram                          0.7879/0.3231 (-59.0%)  \n",
       "ecoli                                    0.8452/0.8571 (+1.4%)  \n",
       "energy-y1                                0.9010/0.9219 (+2.3%)  \n",
       "energy-y2                                0.8906/0.8958 (+0.6%)  \n",
       "fertility                               0.6800/0.8800 (+29.4%)  \n",
       "flags                                   0.4167/0.5670 (+36.1%)  \n",
       "glass                                    0.6792/0.6449 (-5.1%)  \n",
       "haberman-survival                       0.7500/0.6601 (-12.0%)  \n",
       "hayes-roth                              0.5714/0.4643 (-18.7%)  \n",
       "heart-cleveland                          0.5658/0.5960 (+5.3%)  \n",
       "heart-hungarian                         0.7534/0.8299 (+10.2%)  \n",
       "heart-switzerland                       0.2581/0.3115 (+20.7%)  \n",
       "heart-va                                0.2200/0.3100 (+40.9%)  \n",
       "hepatitis                                0.8462/0.7922 (-6.4%)  \n",
       "hill-valley                             0.4934/0.5759 (+16.7%)  \n",
       "horse-colic                             0.7059/0.6029 (-14.6%)  \n",
       "ilpd-indian-liver                        0.6918/0.6392 (-7.6%)  \n",
       "image-segmentation                      0.8938/0.1429 (-84.0%)  \n",
       "ionosphere                               0.9318/0.9029 (-3.1%)  \n",
       "iris                                     1.0000/0.9333 (-6.7%)  \n",
       "led-display                              0.6920/0.6840 (-1.2%)  \n",
       "lenses                                  0.8333/0.1667 (-80.0%)  \n",
       "letter                                  0.9580/0.0401 (-95.8%)  \n",
       "libras                                   0.8000/0.7500 (-6.3%)  \n",
       "low-res-spect                            0.8872/0.8189 (-7.7%)  \n",
       "lung-cancer                             0.5000/0.5625 (+12.5%)  \n",
       "lymphography                             0.7568/0.7162 (-5.4%)  \n",
       "magic                                    0.8690/0.7857 (-9.6%)  \n",
       "mammographic                            0.8292/0.7000 (-15.6%)  \n",
       "miniboone                               0.9272/0.7506 (-19.0%)  \n",
       "molec-biol-promoter                     0.6923/0.6038 (-12.8%)  \n",
       "molec-biol-splice                        0.8494/0.8050 (-5.2%)  \n",
       "monks-1                                 0.5000/0.7292 (+45.8%)  \n",
       "monks-2                                  0.6644/0.6667 (+0.3%)  \n",
       "monks-3                                 0.5231/0.7407 (+41.6%)  \n",
       "mushroom                                0.9995/0.4828 (-51.7%)  \n",
       "musk-1                                   0.8992/0.8487 (-5.6%)  \n",
       "musk-2                                  0.9927/0.1540 (-84.5%)  \n",
       "nursery                                  0.9966/0.9861 (-1.1%)  \n",
       "oocytes_merluccius_nucleus_4d            0.8078/0.7808 (-3.3%)  \n",
       "oocytes_merluccius_states_2f             0.9020/0.8826 (-2.2%)  \n",
       "oocytes_trisopterus_nucleus_2f          0.7939/0.5899 (-25.7%)  \n",
       "oocytes_trisopterus_states_5b            0.9254/0.8947 (-3.3%)  \n",
       "optical                                  0.9638/0.9677 (+0.4%)  \n",
       "ozone                                    0.9748/0.9629 (-1.2%)  \n",
       "page-blocks                             0.9730/0.7876 (-19.0%)  \n",
       "parkinsons                               0.8163/0.7526 (-7.8%)  \n",
       "pendigits                                0.9620/0.9706 (+0.9%)  \n",
       "pima                                     0.6979/0.6693 (-4.1%)  \n",
       "pittsburg-bridges-MATERIAL               0.8077/0.7547 (-6.6%)  \n",
       "pittsburg-bridges-REL-L                  0.6538/0.6667 (+2.0%)  \n",
       "pittsburg-bridges-SPAN                   0.6522/0.6087 (-6.7%)  \n",
       "pittsburg-bridges-T-OR-D                 0.8800/0.8627 (-2.0%)  \n",
       "pittsburg-bridges-TYPE                  0.4615/0.6538 (+41.7%)  \n",
       "planning                                0.6444/0.7143 (+10.8%)  \n",
       "plant-margin                             0.8175/0.7725 (-5.5%)  \n",
       "plant-shape                             0.6575/0.0100 (-98.5%)  \n",
       "plant-texture                            0.8175/0.7960 (-2.6%)  \n",
       "post-operative                          0.5455/0.7111 (+30.4%)  \n",
       "primary-tumor                           0.5000/0.3879 (-22.4%)  \n",
       "ringnorm                                 0.9719/0.9762 (+0.4%)  \n",
       "seeds                                    0.8846/0.8762 (-1.0%)  \n",
       "semeion                                 0.9322/0.0992 (-89.4%)  \n",
       "soybean                                 0.8537/0.4149 (-51.4%)  \n",
       "spambase                                0.9504/0.8109 (-14.7%)  \n",
       "spect                                   0.6398/0.5484 (-14.3%)  \n",
       "spectf                                 0.4545/0.9198 (+102.4%)  \n",
       "statlog-australian-credit                0.6860/0.6783 (-1.1%)  \n",
       "statlog-german-credit                    0.7400/0.7000 (-5.4%)  \n",
       "statlog-heart                            0.8657/0.8667 (+0.1%)  \n",
       "statlog-image                            0.9515/0.9437 (-0.8%)  \n",
       "statlog-landsat                          0.8925/0.8810 (-1.3%)  \n",
       "statlog-shuttle                          0.9988/0.9988 (+0.0%)  \n",
       "statlog-vehicle                          0.8009/0.7565 (-5.5%)  \n",
       "steel-plates                            0.7856/0.6711 (-14.6%)  \n",
       "synthetic-control                       0.9867/0.4467 (-54.7%)  \n",
       "teaching                                 0.3158/0.3200 (+1.3%)  \n",
       "thyroid                                  0.9807/0.9749 (-0.6%)  \n",
       "tic-tac-toe                             0.9707/0.6534 (-32.7%)  \n",
       "titanic                                 0.7818/0.5236 (-33.0%)  \n",
       "trains                                  0.5000/0.6000 (+20.0%)  \n",
       "twonorm                                  0.9730/0.9743 (+0.1%)  \n",
       "vertebral-column-2clases                0.6623/0.8065 (+21.8%)  \n",
       "vertebral-column-3clases                0.7403/0.8452 (+14.2%)  \n",
       "wall-following                          0.9274/0.1514 (-83.7%)  \n",
       "waveform                                 0.8376/0.7972 (-4.8%)  \n",
       "waveform-noise                           0.8640/0.8424 (-2.5%)  \n",
       "wine                                     0.9773/0.9213 (-5.7%)  \n",
       "wine-quality-red                        0.5575/0.4693 (-15.8%)  \n",
       "wine-quality-white                       0.5482/0.4973 (-9.3%)  \n",
       "yeast                                    0.5876/0.5391 (-8.3%)  \n",
       "zoo                                     0.9600/0.8000 (-16.7%)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BN (old/new)</th>\n",
       "      <th>HW (old/new)</th>\n",
       "      <th>LN (old/new)</th>\n",
       "      <th>M</th>\n",
       "      <th>MS (old/new)</th>\n",
       "      <th>N</th>\n",
       "      <th>ResNet (old/new)</th>\n",
       "      <th>SNN (old/new)</th>\n",
       "      <th>WN (old/new)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>abalone</th>\n",
       "      <td>0.6303/0.5575 (-11.6%)</td>\n",
       "      <td>0.6427/0.5383 (-16.2%)</td>\n",
       "      <td>0.6178/0.6049 (-2.1%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.6284/0.6130 (-2.4%)</td>\n",
       "      <td>4177</td>\n",
       "      <td>0.6466/0.3463 (-46.4%)</td>\n",
       "      <td>0.6657/0.3166 (-52.4%)</td>\n",
       "      <td>0.6351/0.5465 (-14.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acute-inflammation</th>\n",
       "      <td>1.0000/0.9667 (-3.3%)</td>\n",
       "      <td>1.0000/0.6167 (-38.3%)</td>\n",
       "      <td>0.9000/1.0000 (+11.1%)</td>\n",
       "      <td>7</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>120</td>\n",
       "      <td>1.0000/0.7000 (-30.0%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acute-nephritis</th>\n",
       "      <td>1.0000/0.9333 (-6.7%)</td>\n",
       "      <td>1.0000/0.5833 (-41.7%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>7</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>120</td>\n",
       "      <td>1.0000/0.5833 (-41.7%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "      <td>1.0000/1.0000 (-0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>adult</th>\n",
       "      <td>0.8499/0.8522 (+0.3%)</td>\n",
       "      <td>0.8453/0.8524 (+0.8%)</td>\n",
       "      <td>0.8517/0.8522 (+0.1%)</td>\n",
       "      <td>15</td>\n",
       "      <td>0.8487/0.8511 (+0.3%)</td>\n",
       "      <td>48842</td>\n",
       "      <td>0.8484/0.8502 (+0.2%)</td>\n",
       "      <td>0.8476/0.8527 (+0.6%)</td>\n",
       "      <td>0.8453/0.8492 (+0.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>annealing</th>\n",
       "      <td>0.1200/0.0600 (-50.0%)</td>\n",
       "      <td>0.3600/0.6500 (+80.6%)</td>\n",
       "      <td>0.5000/0.4700 (-6.0%)</td>\n",
       "      <td>32</td>\n",
       "      <td>0.7300/0.7500 (+2.7%)</td>\n",
       "      <td>898</td>\n",
       "      <td>0.2600/0.7400 (+184.6%)</td>\n",
       "      <td>0.7600/0.1300 (-82.9%)</td>\n",
       "      <td>0.6500/0.7700 (+18.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>arrhythmia</th>\n",
       "      <td>0.5929/0.5044 (-14.9%)</td>\n",
       "      <td>0.6283/0.6062 (-3.5%)</td>\n",
       "      <td>0.5752/0.6106 (+6.2%)</td>\n",
       "      <td>263</td>\n",
       "      <td>0.6372/0.5929 (-6.9%)</td>\n",
       "      <td>452</td>\n",
       "      <td>0.6460/0.4159 (-35.6%)</td>\n",
       "      <td>0.6549/0.5973 (-8.8%)</td>\n",
       "      <td>0.6018/0.5796 (-3.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>audiology-std</th>\n",
       "      <td>0.6400/0.5200 (-18.8%)</td>\n",
       "      <td>0.7200/0.6400 (-11.1%)</td>\n",
       "      <td>0.8000/0.8000 (-0.0%)</td>\n",
       "      <td>60</td>\n",
       "      <td>0.6800/0.7200 (+5.9%)</td>\n",
       "      <td>196</td>\n",
       "      <td>0.8000/0.5200 (-35.0%)</td>\n",
       "      <td>0.8000/0.6800 (-15.0%)</td>\n",
       "      <td>0.7200/0.7200 (-0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balance-scale</th>\n",
       "      <td>0.9231/0.4712 (-49.0%)</td>\n",
       "      <td>0.9103/0.9679 (+6.3%)</td>\n",
       "      <td>0.9872/0.9359 (-5.2%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.9231/0.9615 (+4.2%)</td>\n",
       "      <td>625</td>\n",
       "      <td>0.9167/0.4744 (-48.3%)</td>\n",
       "      <td>0.9231/0.9679 (+4.9%)</td>\n",
       "      <td>0.9551/0.9295 (-2.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balloons</th>\n",
       "      <td>1.0000/0.3750 (-62.5%)</td>\n",
       "      <td>0.2500/0.6250 (+150.0%)</td>\n",
       "      <td>0.7500/0.5000 (-33.3%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.5000/0.5000 (-0.0%)</td>\n",
       "      <td>16</td>\n",
       "      <td>1.0000/0.8750 (-12.5%)</td>\n",
       "      <td>1.0000/0.3750 (-62.5%)</td>\n",
       "      <td>0.0000/0.7500 (+7499999900.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bank</th>\n",
       "      <td>0.8823/0.8850 (+0.3%)</td>\n",
       "      <td>0.8885/0.8221 (-7.5%)</td>\n",
       "      <td>0.8920/0.8735 (-2.1%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.8876/0.8854 (-0.2%)</td>\n",
       "      <td>4521</td>\n",
       "      <td>0.8796/0.8801 (+0.1%)</td>\n",
       "      <td>0.8903/0.8677 (-2.5%)</td>\n",
       "      <td>0.8850/0.8664 (-2.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>blood</th>\n",
       "      <td>0.7647/0.7567 (-1.0%)</td>\n",
       "      <td>0.7968/0.6765 (-15.1%)</td>\n",
       "      <td>0.7112/0.2380 (-66.5%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.7754/0.2380 (-69.3%)</td>\n",
       "      <td>748</td>\n",
       "      <td>0.8021/0.5963 (-25.7%)</td>\n",
       "      <td>0.7701/0.7674 (-0.4%)</td>\n",
       "      <td>0.7594/0.7086 (-6.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer</th>\n",
       "      <td>0.7324/0.2937 (-59.9%)</td>\n",
       "      <td>0.7465/0.6573 (-11.9%)</td>\n",
       "      <td>0.6620/0.6224 (-6.0%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.6901/0.6713 (-2.7%)</td>\n",
       "      <td>286</td>\n",
       "      <td>0.7465/0.7203 (-3.5%)</td>\n",
       "      <td>0.7183/0.5734 (-20.2%)</td>\n",
       "      <td>0.6197/0.5734 (-7.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer-wisc</th>\n",
       "      <td>0.9829/0.3438 (-65.0%)</td>\n",
       "      <td>0.9771/0.9284 (-5.0%)</td>\n",
       "      <td>0.9714/0.9599 (-1.2%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.9714/0.9599 (-1.2%)</td>\n",
       "      <td>699</td>\n",
       "      <td>0.9714/0.4298 (-55.8%)</td>\n",
       "      <td>0.9714/0.9484 (-2.4%)</td>\n",
       "      <td>0.9657/0.9570 (-0.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer-wisc-diag</th>\n",
       "      <td>0.9789/0.8873 (-9.4%)</td>\n",
       "      <td>0.9789/0.9754 (-0.4%)</td>\n",
       "      <td>0.9648/0.9683 (+0.4%)</td>\n",
       "      <td>31</td>\n",
       "      <td>0.9718/0.9472 (-2.5%)</td>\n",
       "      <td>569</td>\n",
       "      <td>0.9507/0.7430 (-21.9%)</td>\n",
       "      <td>0.9789/0.9859 (+0.7%)</td>\n",
       "      <td>0.9718/0.9824 (+1.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-cancer-wisc-prog</th>\n",
       "      <td>0.7755/0.5556 (-28.4%)</td>\n",
       "      <td>0.8367/0.6869 (-17.9%)</td>\n",
       "      <td>0.7959/0.5758 (-27.7%)</td>\n",
       "      <td>34</td>\n",
       "      <td>0.7347/0.6162 (-16.1%)</td>\n",
       "      <td>198</td>\n",
       "      <td>0.8163/0.6162 (-24.5%)</td>\n",
       "      <td>0.6735/0.7071 (+5.0%)</td>\n",
       "      <td>0.8367/0.6768 (-19.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>breast-tissue</th>\n",
       "      <td>0.4615/0.7358 (+59.4%)</td>\n",
       "      <td>0.6154/0.5849 (-5.0%)</td>\n",
       "      <td>0.5769/0.4528 (-21.5%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.4615/0.6792 (+47.2%)</td>\n",
       "      <td>106</td>\n",
       "      <td>0.4231/0.6792 (+60.5%)</td>\n",
       "      <td>0.7308/0.6981 (-4.5%)</td>\n",
       "      <td>0.5385/0.6038 (+12.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <td>0.9606/0.1146 (-88.1%)</td>\n",
       "      <td>0.9560/0.9664 (+1.1%)</td>\n",
       "      <td>0.9907/0.9826 (-0.8%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.9861/0.9213 (-6.6%)</td>\n",
       "      <td>1728</td>\n",
       "      <td>0.9282/0.1493 (-83.9%)</td>\n",
       "      <td>0.9838/0.7014 (-28.7%)</td>\n",
       "      <td>0.9769/0.9826 (+0.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cardiotocography-10clases</th>\n",
       "      <td>0.7910/0.0941 (-88.1%)</td>\n",
       "      <td>0.8456/0.7582 (-10.3%)</td>\n",
       "      <td>0.8362/0.8109 (-3.0%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.8418/0.5814 (-30.9%)</td>\n",
       "      <td>2126</td>\n",
       "      <td>0.8173/0.2427 (-70.3%)</td>\n",
       "      <td>0.8399/0.7846 (-6.6%)</td>\n",
       "      <td>0.8606/0.7752 (-9.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cardiotocography-3clases</th>\n",
       "      <td>0.9096/0.7789 (-14.4%)</td>\n",
       "      <td>0.9171/0.8965 (-2.2%)</td>\n",
       "      <td>0.9021/0.9040 (+0.2%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.8964/0.9106 (+1.6%)</td>\n",
       "      <td>2126</td>\n",
       "      <td>0.9021/0.7893 (-12.5%)</td>\n",
       "      <td>0.9153/0.9219 (+0.7%)</td>\n",
       "      <td>0.8945/0.1383 (-84.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chess-krvk</th>\n",
       "      <td>0.8781/0.0139 (-98.4%)</td>\n",
       "      <td>0.5255/0.1500 (-71.5%)</td>\n",
       "      <td>0.8938/0.1495 (-83.3%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.8606/0.1823 (-78.8%)</td>\n",
       "      <td>28056</td>\n",
       "      <td>0.8543/0.1647 (-80.7%)</td>\n",
       "      <td>0.8805/0.1625 (-81.5%)</td>\n",
       "      <td>0.7673/0.1625 (-78.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chess-krvkp</th>\n",
       "      <td>0.9862/0.5225 (-47.0%)</td>\n",
       "      <td>0.9900/0.9837 (-0.6%)</td>\n",
       "      <td>0.9875/0.9762 (-1.1%)</td>\n",
       "      <td>37</td>\n",
       "      <td>0.9900/0.5225 (-47.2%)</td>\n",
       "      <td>3196</td>\n",
       "      <td>0.9912/0.5763 (-41.9%)</td>\n",
       "      <td>0.9912/0.5225 (-47.3%)</td>\n",
       "      <td>0.9912/0.9781 (-1.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>congressional-voting</th>\n",
       "      <td>0.5872/0.3871 (-34.1%)</td>\n",
       "      <td>0.5872/0.5115 (-12.9%)</td>\n",
       "      <td>0.5780/0.5576 (-3.5%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.6055/0.5253 (-13.2%)</td>\n",
       "      <td>435</td>\n",
       "      <td>0.5963/0.4286 (-28.1%)</td>\n",
       "      <td>0.6147/0.3871 (-37.0%)</td>\n",
       "      <td>0.5872/0.4931 (-16.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>conn-bench-sonar-mines-rocks</th>\n",
       "      <td>0.7115/0.4712 (-33.8%)</td>\n",
       "      <td>0.8462/0.8750 (+3.4%)</td>\n",
       "      <td>0.6731/0.8077 (+20.0%)</td>\n",
       "      <td>61</td>\n",
       "      <td>0.8269/0.8365 (+1.2%)</td>\n",
       "      <td>208</td>\n",
       "      <td>0.8077/0.6731 (-16.7%)</td>\n",
       "      <td>0.7885/0.7692 (-2.4%)</td>\n",
       "      <td>0.8269/0.4615 (-44.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>conn-bench-vowel-deterding</th>\n",
       "      <td>0.9610/0.9004 (-6.3%)</td>\n",
       "      <td>0.9784/0.9870 (+0.9%)</td>\n",
       "      <td>0.9935/0.7468 (-24.8%)</td>\n",
       "      <td>12</td>\n",
       "      <td>0.9935/0.9978 (+0.4%)</td>\n",
       "      <td>990</td>\n",
       "      <td>0.9935/0.8485 (-14.6%)</td>\n",
       "      <td>0.9957/0.9978 (+0.2%)</td>\n",
       "      <td>0.9524/0.7273 (-23.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>connect-4</th>\n",
       "      <td>0.8729/0.2462 (-71.8%)</td>\n",
       "      <td>0.8599/0.3112 (-63.8%)</td>\n",
       "      <td>0.8856/0.2462 (-72.2%)</td>\n",
       "      <td>43</td>\n",
       "      <td>0.8831/0.3711 (-58.0%)</td>\n",
       "      <td>67557</td>\n",
       "      <td>0.8716/0.2940 (-66.3%)</td>\n",
       "      <td>0.8807/0.4176 (-52.6%)</td>\n",
       "      <td>0.8833/0.3747 (-57.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>contrac</th>\n",
       "      <td>0.4538/0.4348 (-4.2%)</td>\n",
       "      <td>0.5054/0.4321 (-14.5%)</td>\n",
       "      <td>0.4592/0.4864 (+5.9%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.5136/0.4946 (-3.7%)</td>\n",
       "      <td>1473</td>\n",
       "      <td>0.5136/0.3465 (-32.5%)</td>\n",
       "      <td>0.5190/0.5177 (-0.3%)</td>\n",
       "      <td>0.4755/0.4878 (+2.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>credit-approval</th>\n",
       "      <td>0.8721/0.7768 (-10.9%)</td>\n",
       "      <td>0.8547/0.8493 (-0.6%)</td>\n",
       "      <td>0.8547/0.8348 (-2.3%)</td>\n",
       "      <td>16</td>\n",
       "      <td>0.8430/0.8087 (-4.1%)</td>\n",
       "      <td>690</td>\n",
       "      <td>0.8430/0.7449 (-11.6%)</td>\n",
       "      <td>0.8430/0.8841 (+4.9%)</td>\n",
       "      <td>0.9070/0.7884 (-13.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinder-bands</th>\n",
       "      <td>0.7500/0.6172 (-17.7%)</td>\n",
       "      <td>0.7969/0.7656 (-3.9%)</td>\n",
       "      <td>0.7578/0.7461 (-1.5%)</td>\n",
       "      <td>36</td>\n",
       "      <td>0.7656/0.7031 (-8.2%)</td>\n",
       "      <td>512</td>\n",
       "      <td>0.7734/0.7305 (-5.6%)</td>\n",
       "      <td>0.7266/0.7461 (+2.7%)</td>\n",
       "      <td>0.7578/0.7305 (-3.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dermatology</th>\n",
       "      <td>0.9341/0.3716 (-60.2%)</td>\n",
       "      <td>0.9780/0.9563 (-2.2%)</td>\n",
       "      <td>0.9451/0.9563 (+1.2%)</td>\n",
       "      <td>35</td>\n",
       "      <td>0.9121/0.9672 (+6.0%)</td>\n",
       "      <td>366</td>\n",
       "      <td>0.9231/0.9016 (-2.3%)</td>\n",
       "      <td>0.9231/0.9781 (+6.0%)</td>\n",
       "      <td>0.9451/0.9727 (+2.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>echocardiogram</th>\n",
       "      <td>0.8485/0.6462 (-23.8%)</td>\n",
       "      <td>0.6061/0.3231 (-46.7%)</td>\n",
       "      <td>0.8182/0.7538 (-7.9%)</td>\n",
       "      <td>11</td>\n",
       "      <td>0.8485/0.6923 (-18.4%)</td>\n",
       "      <td>131</td>\n",
       "      <td>0.8485/0.6615 (-22.0%)</td>\n",
       "      <td>0.8182/0.7231 (-11.6%)</td>\n",
       "      <td>0.7879/0.3231 (-59.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ecoli</th>\n",
       "      <td>0.8214/0.7381 (-10.1%)</td>\n",
       "      <td>0.8690/0.8690 (+0.0%)</td>\n",
       "      <td>0.8571/0.8452 (-1.4%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8333/0.8333 (+0.0%)</td>\n",
       "      <td>336</td>\n",
       "      <td>0.8214/0.5833 (-29.0%)</td>\n",
       "      <td>0.8929/0.8631 (-3.3%)</td>\n",
       "      <td>0.8452/0.8571 (+1.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>energy-y1</th>\n",
       "      <td>0.8646/0.3516 (-59.3%)</td>\n",
       "      <td>0.8802/0.9349 (+6.2%)</td>\n",
       "      <td>0.9479/0.8021 (-15.4%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.9583/0.9089 (-5.2%)</td>\n",
       "      <td>768</td>\n",
       "      <td>0.8177/0.1667 (-79.6%)</td>\n",
       "      <td>0.9583/0.1771 (-81.5%)</td>\n",
       "      <td>0.9010/0.9219 (+2.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>energy-y2</th>\n",
       "      <td>0.8750/0.2448 (-72.0%)</td>\n",
       "      <td>0.9010/0.8438 (-6.4%)</td>\n",
       "      <td>0.8802/0.8880 (+0.9%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.8958/0.9089 (+1.5%)</td>\n",
       "      <td>768</td>\n",
       "      <td>0.8750/0.6849 (-21.7%)</td>\n",
       "      <td>0.9063/0.9323 (+2.9%)</td>\n",
       "      <td>0.8906/0.8958 (+0.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fertility</th>\n",
       "      <td>0.6800/0.8800 (+29.4%)</td>\n",
       "      <td>0.8800/0.8800 (-0.0%)</td>\n",
       "      <td>0.8800/0.8800 (-0.0%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.8800/0.1200 (-86.4%)</td>\n",
       "      <td>100</td>\n",
       "      <td>0.8400/0.8800 (+4.8%)</td>\n",
       "      <td>0.9200/0.8800 (-4.3%)</td>\n",
       "      <td>0.6800/0.8800 (+29.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flags</th>\n",
       "      <td>0.4167/0.4845 (+16.3%)</td>\n",
       "      <td>0.4375/0.4742 (+8.4%)</td>\n",
       "      <td>0.3542/0.1443 (-59.3%)</td>\n",
       "      <td>29</td>\n",
       "      <td>0.4583/0.5464 (+19.2%)</td>\n",
       "      <td>194</td>\n",
       "      <td>0.3750/0.4021 (+7.2%)</td>\n",
       "      <td>0.4583/0.4742 (+3.5%)</td>\n",
       "      <td>0.4167/0.5670 (+36.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>glass</th>\n",
       "      <td>0.5849/0.5888 (+0.7%)</td>\n",
       "      <td>0.6415/0.6075 (-5.3%)</td>\n",
       "      <td>0.6981/0.5421 (-22.4%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.6038/0.6449 (+6.8%)</td>\n",
       "      <td>214</td>\n",
       "      <td>0.6415/0.4766 (-25.7%)</td>\n",
       "      <td>0.7358/0.7103 (-3.5%)</td>\n",
       "      <td>0.6792/0.6449 (-5.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>haberman-survival</th>\n",
       "      <td>0.7368/0.6275 (-14.8%)</td>\n",
       "      <td>0.6447/0.2614 (-59.4%)</td>\n",
       "      <td>0.6842/0.6275 (-8.3%)</td>\n",
       "      <td>4</td>\n",
       "      <td>0.7237/0.6993 (-3.4%)</td>\n",
       "      <td>306</td>\n",
       "      <td>0.6842/0.3660 (-46.5%)</td>\n",
       "      <td>0.7368/0.6732 (-8.6%)</td>\n",
       "      <td>0.7500/0.6601 (-12.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hayes-roth</th>\n",
       "      <td>0.7500/0.0357 (-95.2%)</td>\n",
       "      <td>0.7857/0.2143 (-72.7%)</td>\n",
       "      <td>0.8929/0.4643 (-48.0%)</td>\n",
       "      <td>4</td>\n",
       "      <td>0.4643/0.0357 (-92.3%)</td>\n",
       "      <td>160</td>\n",
       "      <td>0.7143/0.0357 (-95.0%)</td>\n",
       "      <td>0.6786/0.0357 (-94.7%)</td>\n",
       "      <td>0.5714/0.4643 (-18.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-cleveland</th>\n",
       "      <td>0.5789/0.3510 (-39.4%)</td>\n",
       "      <td>0.6316/0.5497 (-13.0%)</td>\n",
       "      <td>0.5789/0.5364 (-7.3%)</td>\n",
       "      <td>14</td>\n",
       "      <td>0.6053/0.5232 (-13.6%)</td>\n",
       "      <td>303</td>\n",
       "      <td>0.5658/0.6093 (+7.7%)</td>\n",
       "      <td>0.6184/0.5828 (-5.8%)</td>\n",
       "      <td>0.5658/0.5960 (+5.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-hungarian</th>\n",
       "      <td>0.8493/0.7551 (-11.1%)</td>\n",
       "      <td>0.7945/0.7891 (-0.7%)</td>\n",
       "      <td>0.8493/0.7891 (-7.1%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.8356/0.8503 (+1.8%)</td>\n",
       "      <td>294</td>\n",
       "      <td>0.8082/0.7347 (-9.1%)</td>\n",
       "      <td>0.7945/0.7959 (+0.2%)</td>\n",
       "      <td>0.7534/0.8299 (+10.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-switzerland</th>\n",
       "      <td>0.3871/0.0656 (-83.1%)</td>\n",
       "      <td>0.5806/0.4098 (-29.4%)</td>\n",
       "      <td>0.5161/0.4754 (-7.9%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.3871/0.3607 (-6.8%)</td>\n",
       "      <td>123</td>\n",
       "      <td>0.3226/0.3607 (+11.8%)</td>\n",
       "      <td>0.3548/0.0656 (-81.5%)</td>\n",
       "      <td>0.2581/0.3115 (+20.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heart-va</th>\n",
       "      <td>0.2800/0.2800 (-0.0%)</td>\n",
       "      <td>0.4000/0.3100 (-22.5%)</td>\n",
       "      <td>0.2400/0.2900 (+20.8%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.2600/0.3000 (+15.4%)</td>\n",
       "      <td>200</td>\n",
       "      <td>0.2600/0.3400 (+30.8%)</td>\n",
       "      <td>0.3600/0.2600 (-27.8%)</td>\n",
       "      <td>0.2200/0.3100 (+40.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hepatitis</th>\n",
       "      <td>0.8718/0.7922 (-9.1%)</td>\n",
       "      <td>0.6667/0.7922 (+18.8%)</td>\n",
       "      <td>0.7436/0.7922 (+6.5%)</td>\n",
       "      <td>20</td>\n",
       "      <td>0.7692/0.7922 (+3.0%)</td>\n",
       "      <td>155</td>\n",
       "      <td>0.7692/0.7922 (+3.0%)</td>\n",
       "      <td>0.7692/0.7922 (+3.0%)</td>\n",
       "      <td>0.8462/0.7922 (-6.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hill-valley</th>\n",
       "      <td>0.5050/0.5693 (+12.7%)</td>\n",
       "      <td>0.5000/0.4934 (-1.3%)</td>\n",
       "      <td>0.5050/0.6287 (+24.5%)</td>\n",
       "      <td>101</td>\n",
       "      <td>0.5116/0.7030 (+37.4%)</td>\n",
       "      <td>1212</td>\n",
       "      <td>0.5396/0.5776 (+7.0%)</td>\n",
       "      <td>0.5248/0.5627 (+7.2%)</td>\n",
       "      <td>0.4934/0.5759 (+16.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horse-colic</th>\n",
       "      <td>0.8529/0.6029 (-29.3%)</td>\n",
       "      <td>0.7794/0.6029 (-22.6%)</td>\n",
       "      <td>0.7941/0.6324 (-20.4%)</td>\n",
       "      <td>26</td>\n",
       "      <td>0.8529/0.6029 (-29.3%)</td>\n",
       "      <td>368</td>\n",
       "      <td>0.8088/0.6029 (-25.5%)</td>\n",
       "      <td>0.8088/0.2794 (-65.5%)</td>\n",
       "      <td>0.7059/0.6029 (-14.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ilpd-indian-liver</th>\n",
       "      <td>0.5959/0.6529 (+9.6%)</td>\n",
       "      <td>0.6781/0.6667 (-1.7%)</td>\n",
       "      <td>0.6986/0.6942 (-0.6%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.6644/0.6701 (+0.9%)</td>\n",
       "      <td>583</td>\n",
       "      <td>0.6712/0.6976 (+3.9%)</td>\n",
       "      <td>0.6986/0.6976 (-0.1%)</td>\n",
       "      <td>0.6918/0.6392 (-7.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>image-segmentation</th>\n",
       "      <td>0.8481/0.1686 (-80.1%)</td>\n",
       "      <td>0.9024/0.2114 (-76.6%)</td>\n",
       "      <td>0.8838/0.4657 (-47.3%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.9090/0.2310 (-74.6%)</td>\n",
       "      <td>2310</td>\n",
       "      <td>0.8919/0.1881 (-78.9%)</td>\n",
       "      <td>0.9114/0.1724 (-81.1%)</td>\n",
       "      <td>0.8938/0.1429 (-84.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ionosphere</th>\n",
       "      <td>0.9432/0.7200 (-23.7%)</td>\n",
       "      <td>0.9432/0.6457 (-31.5%)</td>\n",
       "      <td>0.9432/0.6400 (-32.1%)</td>\n",
       "      <td>34</td>\n",
       "      <td>0.9091/0.9143 (+0.6%)</td>\n",
       "      <td>351</td>\n",
       "      <td>0.9545/0.7200 (-24.6%)</td>\n",
       "      <td>0.8864/0.8743 (-1.4%)</td>\n",
       "      <td>0.9318/0.9029 (-3.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>iris</th>\n",
       "      <td>0.9189/0.6267 (-31.8%)</td>\n",
       "      <td>0.8378/0.9467 (+13.0%)</td>\n",
       "      <td>0.9730/0.9333 (-4.1%)</td>\n",
       "      <td>5</td>\n",
       "      <td>0.9189/0.8933 (-2.8%)</td>\n",
       "      <td>150</td>\n",
       "      <td>0.9730/0.7467 (-23.3%)</td>\n",
       "      <td>0.9730/0.8800 (-9.6%)</td>\n",
       "      <td>1.0000/0.9333 (-6.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>led-display</th>\n",
       "      <td>0.6280/0.4980 (-20.7%)</td>\n",
       "      <td>0.7040/0.6080 (-13.6%)</td>\n",
       "      <td>0.6480/0.1100 (-83.0%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.7200/0.7000 (-2.8%)</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.7160/0.2080 (-70.9%)</td>\n",
       "      <td>0.7640/0.7180 (-6.0%)</td>\n",
       "      <td>0.6920/0.6840 (-1.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lenses</th>\n",
       "      <td>0.8333/0.1667 (-80.0%)</td>\n",
       "      <td>1.0000/0.6667 (-33.3%)</td>\n",
       "      <td>0.6667/0.1667 (-75.0%)</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0000/0.1667 (-83.3%)</td>\n",
       "      <td>24</td>\n",
       "      <td>0.6667/0.6667 (-0.0%)</td>\n",
       "      <td>0.6667/0.7500 (+12.5%)</td>\n",
       "      <td>0.8333/0.1667 (-80.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>letter</th>\n",
       "      <td>0.9796/0.0847 (-91.4%)</td>\n",
       "      <td>0.8984/0.8377 (-6.8%)</td>\n",
       "      <td>0.9742/0.0367 (-96.2%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.9712/0.0401 (-95.9%)</td>\n",
       "      <td>20000</td>\n",
       "      <td>0.9762/0.3086 (-68.4%)</td>\n",
       "      <td>0.9726/0.0401 (-95.9%)</td>\n",
       "      <td>0.9580/0.0401 (-95.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libras</th>\n",
       "      <td>0.7444/0.7500 (+0.8%)</td>\n",
       "      <td>0.8222/0.6056 (-26.3%)</td>\n",
       "      <td>0.8333/0.6667 (-20.0%)</td>\n",
       "      <td>91</td>\n",
       "      <td>0.8667/0.7944 (-8.3%)</td>\n",
       "      <td>360</td>\n",
       "      <td>0.7111/0.5333 (-25.0%)</td>\n",
       "      <td>0.7889/0.7667 (-2.8%)</td>\n",
       "      <td>0.8000/0.7500 (-6.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>low-res-spect</th>\n",
       "      <td>0.8571/0.6340 (-26.0%)</td>\n",
       "      <td>0.9023/0.5245 (-41.9%)</td>\n",
       "      <td>0.8947/0.8453 (-5.5%)</td>\n",
       "      <td>101</td>\n",
       "      <td>0.8496/0.8604 (+1.3%)</td>\n",
       "      <td>531</td>\n",
       "      <td>0.8647/0.3170 (-63.3%)</td>\n",
       "      <td>0.8571/0.8679 (+1.3%)</td>\n",
       "      <td>0.8872/0.8189 (-7.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lung-cancer</th>\n",
       "      <td>0.5000/0.4375 (-12.5%)</td>\n",
       "      <td>0.1250/0.4375 (+250.0%)</td>\n",
       "      <td>0.2500/0.5625 (+125.0%)</td>\n",
       "      <td>57</td>\n",
       "      <td>0.3750/0.3125 (-16.7%)</td>\n",
       "      <td>32</td>\n",
       "      <td>0.2500/0.5000 (+100.0%)</td>\n",
       "      <td>0.6250/0.5000 (-20.0%)</td>\n",
       "      <td>0.5000/0.5625 (+12.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lymphography</th>\n",
       "      <td>0.7568/0.7973 (+5.4%)</td>\n",
       "      <td>0.7297/0.5541 (-24.1%)</td>\n",
       "      <td>0.7838/0.7838 (-0.0%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.7297/0.8378 (+14.8%)</td>\n",
       "      <td>148</td>\n",
       "      <td>0.6757/0.7027 (+4.0%)</td>\n",
       "      <td>0.9189/0.7973 (-13.2%)</td>\n",
       "      <td>0.7568/0.7162 (-5.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>magic</th>\n",
       "      <td>0.8713/0.6485 (-25.6%)</td>\n",
       "      <td>0.8673/0.5792 (-33.2%)</td>\n",
       "      <td>0.8620/0.7728 (-10.4%)</td>\n",
       "      <td>11</td>\n",
       "      <td>0.8629/0.4118 (-52.3%)</td>\n",
       "      <td>19020</td>\n",
       "      <td>0.8723/0.3516 (-59.7%)</td>\n",
       "      <td>0.8692/0.6707 (-22.8%)</td>\n",
       "      <td>0.8690/0.7857 (-9.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mammographic</th>\n",
       "      <td>0.8167/0.6417 (-21.4%)</td>\n",
       "      <td>0.7917/0.7771 (-1.8%)</td>\n",
       "      <td>0.8208/0.7896 (-3.8%)</td>\n",
       "      <td>6</td>\n",
       "      <td>0.8083/0.7917 (-2.1%)</td>\n",
       "      <td>961</td>\n",
       "      <td>0.7833/0.4000 (-48.9%)</td>\n",
       "      <td>0.8250/0.8021 (-2.8%)</td>\n",
       "      <td>0.8292/0.7000 (-15.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>miniboone</th>\n",
       "      <td>0.9262/0.7194 (-22.3%)</td>\n",
       "      <td>0.9270/0.7355 (-20.7%)</td>\n",
       "      <td>0.9313/0.7450 (-20.0%)</td>\n",
       "      <td>51</td>\n",
       "      <td>0.9250/0.7641 (-17.4%)</td>\n",
       "      <td>130064</td>\n",
       "      <td>0.9254/0.7194 (-22.3%)</td>\n",
       "      <td>0.9307/0.7194 (-22.7%)</td>\n",
       "      <td>0.9272/0.7506 (-19.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>molec-biol-promoter</th>\n",
       "      <td>0.7692/0.6981 (-9.2%)</td>\n",
       "      <td>0.6923/0.8113 (+17.2%)</td>\n",
       "      <td>0.4615/0.7358 (+59.4%)</td>\n",
       "      <td>58</td>\n",
       "      <td>0.7692/0.6226 (-19.1%)</td>\n",
       "      <td>106</td>\n",
       "      <td>0.7692/0.6604 (-14.1%)</td>\n",
       "      <td>0.8462/0.8491 (+0.3%)</td>\n",
       "      <td>0.6923/0.6038 (-12.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>molec-biol-splice</th>\n",
       "      <td>0.8519/0.5273 (-38.1%)</td>\n",
       "      <td>0.8833/0.8401 (-4.9%)</td>\n",
       "      <td>0.8607/0.7197 (-16.4%)</td>\n",
       "      <td>61</td>\n",
       "      <td>0.8482/0.8031 (-5.3%)</td>\n",
       "      <td>3190</td>\n",
       "      <td>0.8557/0.6075 (-29.0%)</td>\n",
       "      <td>0.9009/0.2408 (-73.3%)</td>\n",
       "      <td>0.8494/0.8050 (-5.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monks-1</th>\n",
       "      <td>0.9074/0.5255 (-42.1%)</td>\n",
       "      <td>0.5833/0.6435 (+10.3%)</td>\n",
       "      <td>0.7014/0.6829 (-2.6%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.6551/0.7523 (+14.8%)</td>\n",
       "      <td>556</td>\n",
       "      <td>0.7546/0.5208 (-31.0%)</td>\n",
       "      <td>0.7523/0.6458 (-14.2%)</td>\n",
       "      <td>0.5000/0.7292 (+45.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monks-2</th>\n",
       "      <td>0.3287/0.6551 (+99.3%)</td>\n",
       "      <td>0.6389/0.6713 (+5.1%)</td>\n",
       "      <td>0.5162/0.6690 (+29.6%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.6343/0.6644 (+4.7%)</td>\n",
       "      <td>601</td>\n",
       "      <td>0.6273/0.6713 (+7.0%)</td>\n",
       "      <td>0.5926/0.6667 (+12.5%)</td>\n",
       "      <td>0.6644/0.6667 (+0.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monks-3</th>\n",
       "      <td>0.5278/0.5648 (+7.0%)</td>\n",
       "      <td>0.5880/0.5764 (-2.0%)</td>\n",
       "      <td>0.6991/0.5486 (-21.5%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.7454/0.5417 (-27.3%)</td>\n",
       "      <td>554</td>\n",
       "      <td>0.5833/0.6806 (+16.7%)</td>\n",
       "      <td>0.6042/0.6435 (+6.5%)</td>\n",
       "      <td>0.5231/0.7407 (+41.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mushroom</th>\n",
       "      <td>0.9990/0.4628 (-53.7%)</td>\n",
       "      <td>1.0000/0.4820 (-51.8%)</td>\n",
       "      <td>0.9995/0.9183 (-8.1%)</td>\n",
       "      <td>22</td>\n",
       "      <td>1.0000/0.9953 (-0.5%)</td>\n",
       "      <td>8124</td>\n",
       "      <td>1.0000/0.4820 (-51.8%)</td>\n",
       "      <td>1.0000/0.9983 (-0.2%)</td>\n",
       "      <td>0.9995/0.4828 (-51.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>musk-1</th>\n",
       "      <td>0.8235/0.7269 (-11.7%)</td>\n",
       "      <td>0.8992/0.6765 (-24.8%)</td>\n",
       "      <td>0.8992/0.8445 (-6.1%)</td>\n",
       "      <td>167</td>\n",
       "      <td>0.8655/0.7773 (-10.2%)</td>\n",
       "      <td>476</td>\n",
       "      <td>0.8739/0.7311 (-16.3%)</td>\n",
       "      <td>0.8739/0.8613 (-1.4%)</td>\n",
       "      <td>0.8992/0.8487 (-5.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>musk-2</th>\n",
       "      <td>0.9982/0.1540 (-84.6%)</td>\n",
       "      <td>0.9915/0.9900 (-0.2%)</td>\n",
       "      <td>0.9951/0.1540 (-84.5%)</td>\n",
       "      <td>167</td>\n",
       "      <td>0.9945/0.1540 (-84.5%)</td>\n",
       "      <td>6598</td>\n",
       "      <td>0.9964/0.9536 (-4.3%)</td>\n",
       "      <td>0.9891/0.1540 (-84.4%)</td>\n",
       "      <td>0.9927/0.1540 (-84.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nursery</th>\n",
       "      <td>0.9994/0.6267 (-37.3%)</td>\n",
       "      <td>1.0000/0.9647 (-3.5%)</td>\n",
       "      <td>0.9966/0.3120 (-68.7%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.9988/0.3120 (-68.8%)</td>\n",
       "      <td>12960</td>\n",
       "      <td>0.9994/0.5350 (-46.5%)</td>\n",
       "      <td>0.9978/0.9347 (-6.3%)</td>\n",
       "      <td>0.9966/0.9861 (-1.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_merluccius_nucleus_4d</th>\n",
       "      <td>0.8078/0.7065 (-12.5%)</td>\n",
       "      <td>0.7176/0.6712 (-6.5%)</td>\n",
       "      <td>0.7686/0.7847 (+2.1%)</td>\n",
       "      <td>42</td>\n",
       "      <td>0.8196/0.7730 (-5.7%)</td>\n",
       "      <td>1022</td>\n",
       "      <td>0.8000/0.6732 (-15.9%)</td>\n",
       "      <td>0.8235/0.8141 (-1.1%)</td>\n",
       "      <td>0.8078/0.7808 (-3.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_merluccius_states_2f</th>\n",
       "      <td>0.9333/0.6888 (-26.2%)</td>\n",
       "      <td>0.9490/0.9119 (-3.9%)</td>\n",
       "      <td>0.9412/0.9061 (-3.7%)</td>\n",
       "      <td>26</td>\n",
       "      <td>0.9490/0.8943 (-5.8%)</td>\n",
       "      <td>1022</td>\n",
       "      <td>0.9373/0.6849 (-26.9%)</td>\n",
       "      <td>0.9529/0.9119 (-4.3%)</td>\n",
       "      <td>0.9020/0.8826 (-2.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_trisopterus_nucleus_2f</th>\n",
       "      <td>0.7456/0.5811 (-22.1%)</td>\n",
       "      <td>0.8289/0.5789 (-30.2%)</td>\n",
       "      <td>0.8202/0.6031 (-26.5%)</td>\n",
       "      <td>26</td>\n",
       "      <td>0.8728/0.7654 (-12.3%)</td>\n",
       "      <td>912</td>\n",
       "      <td>0.7719/0.6162 (-20.2%)</td>\n",
       "      <td>0.7982/0.7632 (-4.4%)</td>\n",
       "      <td>0.7939/0.5899 (-25.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oocytes_trisopterus_states_5b</th>\n",
       "      <td>0.8947/0.7873 (-12.0%)</td>\n",
       "      <td>0.9342/0.8991 (-3.8%)</td>\n",
       "      <td>0.8991/0.9057 (+0.7%)</td>\n",
       "      <td>33</td>\n",
       "      <td>0.9430/0.4079 (-56.7%)</td>\n",
       "      <td>912</td>\n",
       "      <td>0.8947/0.6754 (-24.5%)</td>\n",
       "      <td>0.9342/0.9057 (-3.1%)</td>\n",
       "      <td>0.9254/0.8947 (-3.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>optical</th>\n",
       "      <td>0.9716/0.9644 (-0.7%)</td>\n",
       "      <td>0.9644/0.9738 (+1.0%)</td>\n",
       "      <td>0.9755/0.9699 (-0.6%)</td>\n",
       "      <td>63</td>\n",
       "      <td>0.9666/0.9688 (+0.2%)</td>\n",
       "      <td>5620</td>\n",
       "      <td>0.9627/0.9627 (+0.0%)</td>\n",
       "      <td>0.9711/0.9638 (-0.7%)</td>\n",
       "      <td>0.9638/0.9677 (+0.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ozone</th>\n",
       "      <td>0.9669/0.9685 (+0.2%)</td>\n",
       "      <td>0.9716/0.9685 (-0.3%)</td>\n",
       "      <td>0.9716/0.9669 (-0.5%)</td>\n",
       "      <td>73</td>\n",
       "      <td>0.9732/0.0284 (-97.1%)</td>\n",
       "      <td>2536</td>\n",
       "      <td>0.9669/0.9700 (+0.3%)</td>\n",
       "      <td>0.9700/0.9685 (-0.2%)</td>\n",
       "      <td>0.9748/0.9629 (-1.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>page-blocks</th>\n",
       "      <td>0.9613/0.9419 (-2.0%)</td>\n",
       "      <td>0.9656/0.9401 (-2.6%)</td>\n",
       "      <td>0.9708/0.9550 (-1.6%)</td>\n",
       "      <td>11</td>\n",
       "      <td>0.9708/0.0599 (-93.8%)</td>\n",
       "      <td>5473</td>\n",
       "      <td>0.9605/0.9364 (-2.5%)</td>\n",
       "      <td>0.9583/0.9437 (-1.5%)</td>\n",
       "      <td>0.9730/0.7876 (-19.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parkinsons</th>\n",
       "      <td>0.8571/0.7526 (-12.2%)</td>\n",
       "      <td>0.8367/0.8763 (+4.7%)</td>\n",
       "      <td>0.8571/0.9175 (+7.1%)</td>\n",
       "      <td>23</td>\n",
       "      <td>0.9184/0.7526 (-18.1%)</td>\n",
       "      <td>195</td>\n",
       "      <td>0.9184/0.8866 (-3.5%)</td>\n",
       "      <td>0.8980/0.7732 (-13.9%)</td>\n",
       "      <td>0.8163/0.7526 (-7.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pendigits</th>\n",
       "      <td>0.9734/0.9720 (-0.1%)</td>\n",
       "      <td>0.9671/0.9605 (-0.7%)</td>\n",
       "      <td>0.9657/0.9720 (+0.7%)</td>\n",
       "      <td>17</td>\n",
       "      <td>0.9714/0.9777 (+0.6%)</td>\n",
       "      <td>10992</td>\n",
       "      <td>0.9708/0.9723 (+0.2%)</td>\n",
       "      <td>0.9706/0.9711 (+0.1%)</td>\n",
       "      <td>0.9620/0.9706 (+0.9%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pima</th>\n",
       "      <td>0.7188/0.6719 (-6.5%)</td>\n",
       "      <td>0.7188/0.7135 (-0.7%)</td>\n",
       "      <td>0.6927/0.7474 (+7.9%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.7656/0.7344 (-4.1%)</td>\n",
       "      <td>768</td>\n",
       "      <td>0.7135/0.3490 (-51.1%)</td>\n",
       "      <td>0.7552/0.7552 (+0.0%)</td>\n",
       "      <td>0.6979/0.6693 (-4.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-MATERIAL</th>\n",
       "      <td>0.8846/0.8679 (-1.9%)</td>\n",
       "      <td>0.9231/0.8679 (-6.0%)</td>\n",
       "      <td>0.9231/0.7170 (-22.3%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8462/0.8679 (+2.6%)</td>\n",
       "      <td>106</td>\n",
       "      <td>0.9231/0.8302 (-10.1%)</td>\n",
       "      <td>0.8846/0.8302 (-6.2%)</td>\n",
       "      <td>0.8077/0.7547 (-6.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-REL-L</th>\n",
       "      <td>0.7692/0.5686 (-26.1%)</td>\n",
       "      <td>0.6923/0.4510 (-34.9%)</td>\n",
       "      <td>0.7308/0.1373 (-81.2%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.7692/0.5686 (-26.1%)</td>\n",
       "      <td>103</td>\n",
       "      <td>0.8462/0.6863 (-18.9%)</td>\n",
       "      <td>0.6923/0.5294 (-23.5%)</td>\n",
       "      <td>0.6538/0.6667 (+2.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-SPAN</th>\n",
       "      <td>0.5652/0.5000 (-11.5%)</td>\n",
       "      <td>0.5652/0.7609 (+34.6%)</td>\n",
       "      <td>0.6087/0.6304 (+3.6%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.5217/0.7391 (+41.7%)</td>\n",
       "      <td>92</td>\n",
       "      <td>0.5652/0.6087 (+7.7%)</td>\n",
       "      <td>0.6957/0.6087 (-12.5%)</td>\n",
       "      <td>0.6522/0.6087 (-6.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-T-OR-D</th>\n",
       "      <td>0.8800/0.7059 (-19.8%)</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>102</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "      <td>0.8400/0.8627 (+2.7%)</td>\n",
       "      <td>0.8800/0.8627 (-2.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pittsburg-bridges-TYPE</th>\n",
       "      <td>0.1154/0.5192 (+349.9%)</td>\n",
       "      <td>0.5385/0.5577 (+3.6%)</td>\n",
       "      <td>0.6538/0.5769 (-11.8%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.6538/0.5577 (-14.7%)</td>\n",
       "      <td>105</td>\n",
       "      <td>0.6538/0.6538 (+0.0%)</td>\n",
       "      <td>0.6538/0.5192 (-20.6%)</td>\n",
       "      <td>0.4615/0.6538 (+41.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>planning</th>\n",
       "      <td>0.6222/0.5165 (-17.0%)</td>\n",
       "      <td>0.6000/0.3956 (-34.1%)</td>\n",
       "      <td>0.6889/0.7143 (+3.7%)</td>\n",
       "      <td>13</td>\n",
       "      <td>0.6667/0.4615 (-30.8%)</td>\n",
       "      <td>182</td>\n",
       "      <td>0.7111/0.5385 (-24.3%)</td>\n",
       "      <td>0.6889/0.7143 (+3.7%)</td>\n",
       "      <td>0.6444/0.7143 (+10.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant-margin</th>\n",
       "      <td>0.7600/0.5375 (-29.3%)</td>\n",
       "      <td>0.8375/0.6663 (-20.4%)</td>\n",
       "      <td>0.8425/0.7688 (-8.8%)</td>\n",
       "      <td>65</td>\n",
       "      <td>0.8125/0.7950 (-2.2%)</td>\n",
       "      <td>1600</td>\n",
       "      <td>0.7975/0.4200 (-47.3%)</td>\n",
       "      <td>0.8125/0.7638 (-6.0%)</td>\n",
       "      <td>0.8175/0.7725 (-5.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant-shape</th>\n",
       "      <td>0.2850/0.2062 (-27.6%)</td>\n",
       "      <td>0.6325/0.3925 (-37.9%)</td>\n",
       "      <td>0.6775/0.6388 (-5.7%)</td>\n",
       "      <td>65</td>\n",
       "      <td>0.6350/0.5175 (-18.5%)</td>\n",
       "      <td>1600</td>\n",
       "      <td>0.5150/0.2575 (-50.0%)</td>\n",
       "      <td>0.7275/0.5637 (-22.5%)</td>\n",
       "      <td>0.6575/0.0100 (-98.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plant-texture</th>\n",
       "      <td>0.8200/0.7134 (-13.0%)</td>\n",
       "      <td>0.7900/0.7860 (-0.5%)</td>\n",
       "      <td>0.8350/0.0100 (-98.8%)</td>\n",
       "      <td>65</td>\n",
       "      <td>0.7900/0.7972 (+0.9%)</td>\n",
       "      <td>1599</td>\n",
       "      <td>0.8000/0.5419 (-32.3%)</td>\n",
       "      <td>0.8125/0.7935 (-2.3%)</td>\n",
       "      <td>0.8175/0.7960 (-2.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>post-operative</th>\n",
       "      <td>0.5909/0.6222 (+5.3%)</td>\n",
       "      <td>0.5909/0.6889 (+16.6%)</td>\n",
       "      <td>0.7727/0.0222 (-97.1%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.7273/0.7111 (-2.2%)</td>\n",
       "      <td>90</td>\n",
       "      <td>0.7273/0.5556 (-23.6%)</td>\n",
       "      <td>0.7273/0.2667 (-63.3%)</td>\n",
       "      <td>0.5455/0.7111 (+30.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>primary-tumor</th>\n",
       "      <td>0.5122/0.2727 (-46.8%)</td>\n",
       "      <td>0.4512/0.3939 (-12.7%)</td>\n",
       "      <td>0.4512/0.3455 (-23.4%)</td>\n",
       "      <td>18</td>\n",
       "      <td>0.5000/0.4121 (-17.6%)</td>\n",
       "      <td>330</td>\n",
       "      <td>0.3902/0.2788 (-28.6%)</td>\n",
       "      <td>0.5244/0.4061 (-22.6%)</td>\n",
       "      <td>0.5000/0.3879 (-22.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ringnorm</th>\n",
       "      <td>0.9843/0.5049 (-48.7%)</td>\n",
       "      <td>0.9692/0.9673 (-0.2%)</td>\n",
       "      <td>0.9827/0.5049 (-48.6%)</td>\n",
       "      <td>21</td>\n",
       "      <td>0.9843/0.9762 (-0.8%)</td>\n",
       "      <td>7400</td>\n",
       "      <td>0.9811/0.5503 (-43.9%)</td>\n",
       "      <td>0.9751/0.9832 (+0.8%)</td>\n",
       "      <td>0.9719/0.9762 (+0.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seeds</th>\n",
       "      <td>0.8654/0.6667 (-23.0%)</td>\n",
       "      <td>0.9423/0.8952 (-5.0%)</td>\n",
       "      <td>0.8846/0.8952 (+1.2%)</td>\n",
       "      <td>8</td>\n",
       "      <td>0.8654/0.5048 (-41.7%)</td>\n",
       "      <td>210</td>\n",
       "      <td>0.8654/0.6190 (-28.5%)</td>\n",
       "      <td>0.8846/0.9048 (+2.3%)</td>\n",
       "      <td>0.8846/0.8762 (-1.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>semeion</th>\n",
       "      <td>0.9372/0.0992 (-89.4%)</td>\n",
       "      <td>0.9447/0.9397 (-0.5%)</td>\n",
       "      <td>0.9447/0.0992 (-89.5%)</td>\n",
       "      <td>257</td>\n",
       "      <td>0.9296/0.9296 (+0.0%)</td>\n",
       "      <td>1593</td>\n",
       "      <td>0.9146/0.5452 (-40.4%)</td>\n",
       "      <td>0.9196/0.9209 (+0.1%)</td>\n",
       "      <td>0.9322/0.0992 (-89.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>soybean</th>\n",
       "      <td>0.8883/0.5053 (-43.1%)</td>\n",
       "      <td>0.8617/0.3856 (-55.2%)</td>\n",
       "      <td>0.8484/0.4468 (-47.3%)</td>\n",
       "      <td>36</td>\n",
       "      <td>0.8723/0.3590 (-58.8%)</td>\n",
       "      <td>683</td>\n",
       "      <td>0.8670/0.3750 (-56.7%)</td>\n",
       "      <td>0.8511/0.0239 (-97.2%)</td>\n",
       "      <td>0.8537/0.4149 (-51.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spambase</th>\n",
       "      <td>0.9426/0.8948 (-5.1%)</td>\n",
       "      <td>0.9435/0.9222 (-2.3%)</td>\n",
       "      <td>0.9513/0.9230 (-3.0%)</td>\n",
       "      <td>58</td>\n",
       "      <td>0.9461/0.3939 (-58.4%)</td>\n",
       "      <td>4601</td>\n",
       "      <td>0.9461/0.7070 (-25.3%)</td>\n",
       "      <td>0.9409/0.9243 (-1.8%)</td>\n",
       "      <td>0.9504/0.8109 (-14.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spect</th>\n",
       "      <td>0.6344/0.4624 (-27.1%)</td>\n",
       "      <td>0.6022/0.5484 (-8.9%)</td>\n",
       "      <td>0.6720/0.6237 (-7.2%)</td>\n",
       "      <td>23</td>\n",
       "      <td>0.6183/0.5484 (-11.3%)</td>\n",
       "      <td>265</td>\n",
       "      <td>0.6667/0.5484 (-17.7%)</td>\n",
       "      <td>0.6398/0.5645 (-11.8%)</td>\n",
       "      <td>0.6398/0.5484 (-14.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spectf</th>\n",
       "      <td>0.2299/0.6738 (+193.1%)</td>\n",
       "      <td>0.8930/0.8610 (-3.6%)</td>\n",
       "      <td>0.5561/0.9144 (+64.4%)</td>\n",
       "      <td>45</td>\n",
       "      <td>0.6043/0.9198 (+52.2%)</td>\n",
       "      <td>267</td>\n",
       "      <td>0.7005/0.9198 (+31.3%)</td>\n",
       "      <td>0.4973/0.9198 (+85.0%)</td>\n",
       "      <td>0.4545/0.9198 (+102.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-australian-credit</th>\n",
       "      <td>0.6802/0.6725 (-1.1%)</td>\n",
       "      <td>0.6802/0.6232 (-8.4%)</td>\n",
       "      <td>0.6279/0.6783 (+8.0%)</td>\n",
       "      <td>15</td>\n",
       "      <td>0.6802/0.6667 (-2.0%)</td>\n",
       "      <td>690</td>\n",
       "      <td>0.6395/0.6783 (+6.1%)</td>\n",
       "      <td>0.5988/0.6377 (+6.5%)</td>\n",
       "      <td>0.6860/0.6783 (-1.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-german-credit</th>\n",
       "      <td>0.7520/0.6980 (-7.2%)</td>\n",
       "      <td>0.7760/0.7120 (-8.2%)</td>\n",
       "      <td>0.7400/0.7420 (+0.3%)</td>\n",
       "      <td>25</td>\n",
       "      <td>0.7280/0.7000 (-3.8%)</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.7720/0.5840 (-24.4%)</td>\n",
       "      <td>0.7560/0.7720 (+2.1%)</td>\n",
       "      <td>0.7400/0.7000 (-5.4%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-heart</th>\n",
       "      <td>0.7910/0.4444 (-43.8%)</td>\n",
       "      <td>0.7761/0.4444 (-42.7%)</td>\n",
       "      <td>0.7910/0.8444 (+6.8%)</td>\n",
       "      <td>14</td>\n",
       "      <td>0.8358/0.8593 (+2.8%)</td>\n",
       "      <td>270</td>\n",
       "      <td>0.8657/0.6889 (-20.4%)</td>\n",
       "      <td>0.9254/0.8593 (-7.1%)</td>\n",
       "      <td>0.8657/0.8667 (+0.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-image</th>\n",
       "      <td>0.9671/0.1446 (-85.0%)</td>\n",
       "      <td>0.9584/0.9195 (-4.1%)</td>\n",
       "      <td>0.9757/0.1429 (-85.4%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.9757/0.8234 (-15.6%)</td>\n",
       "      <td>2310</td>\n",
       "      <td>0.9584/0.3281 (-65.8%)</td>\n",
       "      <td>0.9549/0.9498 (-0.5%)</td>\n",
       "      <td>0.9515/0.9437 (-0.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-landsat</th>\n",
       "      <td>0.9040/0.8755 (-3.2%)</td>\n",
       "      <td>0.9110/0.8920 (-2.1%)</td>\n",
       "      <td>0.9040/0.8945 (-1.1%)</td>\n",
       "      <td>37</td>\n",
       "      <td>0.9075/0.2305 (-74.6%)</td>\n",
       "      <td>6435</td>\n",
       "      <td>0.9055/0.8620 (-4.8%)</td>\n",
       "      <td>0.9100/0.8780 (-3.5%)</td>\n",
       "      <td>0.8925/0.8810 (-1.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-shuttle</th>\n",
       "      <td>0.9988/0.9961 (-0.3%)</td>\n",
       "      <td>0.9977/0.9977 (+0.0%)</td>\n",
       "      <td>0.9987/0.9991 (+0.0%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.9983/0.9989 (+0.1%)</td>\n",
       "      <td>58000</td>\n",
       "      <td>0.9992/0.9970 (-0.2%)</td>\n",
       "      <td>0.9990/0.0009 (-99.9%)</td>\n",
       "      <td>0.9988/0.9988 (+0.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>statlog-vehicle</th>\n",
       "      <td>0.7583/0.6809 (-10.2%)</td>\n",
       "      <td>0.7962/0.6879 (-13.6%)</td>\n",
       "      <td>0.7915/0.7636 (-3.5%)</td>\n",
       "      <td>19</td>\n",
       "      <td>0.8294/0.7612 (-8.2%)</td>\n",
       "      <td>846</td>\n",
       "      <td>0.7583/0.5910 (-22.1%)</td>\n",
       "      <td>0.8009/0.7329 (-8.5%)</td>\n",
       "      <td>0.8009/0.7565 (-5.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>steel-plates</th>\n",
       "      <td>0.7031/0.6124 (-12.9%)</td>\n",
       "      <td>0.7608/0.7082 (-6.9%)</td>\n",
       "      <td>0.7588/0.3474 (-54.2%)</td>\n",
       "      <td>28</td>\n",
       "      <td>0.7567/0.6887 (-9.0%)</td>\n",
       "      <td>1941</td>\n",
       "      <td>0.7629/0.4227 (-44.6%)</td>\n",
       "      <td>0.7835/0.7052 (-10.0%)</td>\n",
       "      <td>0.7856/0.6711 (-14.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>synthetic-control</th>\n",
       "      <td>0.9733/0.2133 (-78.1%)</td>\n",
       "      <td>0.9867/0.1667 (-83.1%)</td>\n",
       "      <td>0.9733/0.2067 (-78.8%)</td>\n",
       "      <td>61</td>\n",
       "      <td>0.9800/0.7033 (-28.2%)</td>\n",
       "      <td>600</td>\n",
       "      <td>0.9600/0.4433 (-53.8%)</td>\n",
       "      <td>0.9867/0.7833 (-20.6%)</td>\n",
       "      <td>0.9867/0.4467 (-54.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>teaching</th>\n",
       "      <td>0.5000/0.3467 (-30.7%)</td>\n",
       "      <td>0.5263/0.3867 (-26.5%)</td>\n",
       "      <td>0.6316/0.4400 (-30.3%)</td>\n",
       "      <td>6</td>\n",
       "      <td>0.6053/0.3333 (-44.9%)</td>\n",
       "      <td>151</td>\n",
       "      <td>0.5526/0.3467 (-37.3%)</td>\n",
       "      <td>0.5000/0.3333 (-33.3%)</td>\n",
       "      <td>0.3158/0.3200 (+1.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thyroid</th>\n",
       "      <td>0.9778/0.9787 (+0.1%)</td>\n",
       "      <td>0.9708/0.9770 (+0.6%)</td>\n",
       "      <td>0.9752/0.9761 (+0.1%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.9770/0.9767 (-0.0%)</td>\n",
       "      <td>7200</td>\n",
       "      <td>0.9799/0.9770 (-0.3%)</td>\n",
       "      <td>0.9816/0.9772 (-0.4%)</td>\n",
       "      <td>0.9807/0.9749 (-0.6%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tic-tac-toe</th>\n",
       "      <td>0.9833/0.7724 (-21.4%)</td>\n",
       "      <td>0.9749/0.9457 (-3.0%)</td>\n",
       "      <td>0.9791/0.9645 (-1.5%)</td>\n",
       "      <td>10</td>\n",
       "      <td>0.9833/0.9353 (-4.9%)</td>\n",
       "      <td>958</td>\n",
       "      <td>0.9623/0.8246 (-14.3%)</td>\n",
       "      <td>0.9665/0.9812 (+1.5%)</td>\n",
       "      <td>0.9707/0.6534 (-32.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>titanic</th>\n",
       "      <td>0.7800/0.7491 (-4.0%)</td>\n",
       "      <td>0.7927/0.3227 (-59.3%)</td>\n",
       "      <td>0.7891/0.3227 (-59.1%)</td>\n",
       "      <td>4</td>\n",
       "      <td>0.7909/0.5464 (-30.9%)</td>\n",
       "      <td>2201</td>\n",
       "      <td>0.7727/0.3227 (-58.2%)</td>\n",
       "      <td>0.7836/0.3300 (-57.9%)</td>\n",
       "      <td>0.7818/0.5236 (-33.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trains</th>\n",
       "      <td>0.5000/0.4000 (-20.0%)</td>\n",
       "      <td>nan/0.4000 (+nan%)</td>\n",
       "      <td>1.0000/0.6000 (-40.0%)</td>\n",
       "      <td>30</td>\n",
       "      <td>nan/0.4000 (+nan%)</td>\n",
       "      <td>10</td>\n",
       "      <td>nan/0.8000 (+nan%)</td>\n",
       "      <td>nan/0.4000 (+nan%)</td>\n",
       "      <td>0.5000/0.6000 (+20.0%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>twonorm</th>\n",
       "      <td>0.9757/0.9127 (-6.5%)</td>\n",
       "      <td>0.9708/0.9738 (+0.3%)</td>\n",
       "      <td>0.9724/0.9732 (+0.1%)</td>\n",
       "      <td>21</td>\n",
       "      <td>0.9778/0.9700 (-0.8%)</td>\n",
       "      <td>7400</td>\n",
       "      <td>0.9735/0.4995 (-48.7%)</td>\n",
       "      <td>0.9805/0.4995 (-49.1%)</td>\n",
       "      <td>0.9730/0.9743 (+0.1%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vertebral-column-2clases</th>\n",
       "      <td>0.8312/0.3226 (-61.2%)</td>\n",
       "      <td>0.8571/0.6581 (-23.2%)</td>\n",
       "      <td>0.8442/0.8000 (-5.2%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.8701/0.7548 (-13.2%)</td>\n",
       "      <td>310</td>\n",
       "      <td>0.8312/0.6452 (-22.4%)</td>\n",
       "      <td>0.8312/0.7806 (-6.1%)</td>\n",
       "      <td>0.6623/0.8065 (+21.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vertebral-column-3clases</th>\n",
       "      <td>0.7792/0.7935 (+1.8%)</td>\n",
       "      <td>0.7922/0.8194 (+3.4%)</td>\n",
       "      <td>0.8312/0.8065 (-3.0%)</td>\n",
       "      <td>7</td>\n",
       "      <td>0.8052/0.8452 (+5.0%)</td>\n",
       "      <td>310</td>\n",
       "      <td>0.7532/0.7097 (-5.8%)</td>\n",
       "      <td>0.8312/0.8129 (-2.2%)</td>\n",
       "      <td>0.7403/0.8452 (+14.2%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wall-following</th>\n",
       "      <td>0.9333/0.1514 (-83.8%)</td>\n",
       "      <td>0.9230/0.8904 (-3.5%)</td>\n",
       "      <td>0.9128/0.0601 (-93.4%)</td>\n",
       "      <td>25</td>\n",
       "      <td>0.9076/0.4637 (-48.9%)</td>\n",
       "      <td>5456</td>\n",
       "      <td>0.9223/0.5154 (-44.1%)</td>\n",
       "      <td>0.9098/0.1514 (-83.4%)</td>\n",
       "      <td>0.9274/0.1514 (-83.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>waveform</th>\n",
       "      <td>0.8360/0.7064 (-15.5%)</td>\n",
       "      <td>0.8320/0.8380 (+0.7%)</td>\n",
       "      <td>0.8448/0.8592 (+1.7%)</td>\n",
       "      <td>22</td>\n",
       "      <td>0.8312/0.8464 (+1.8%)</td>\n",
       "      <td>5000</td>\n",
       "      <td>0.8360/0.4664 (-44.2%)</td>\n",
       "      <td>0.8480/0.8304 (-2.1%)</td>\n",
       "      <td>0.8376/0.7972 (-4.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>waveform-noise</th>\n",
       "      <td>0.8480/0.6956 (-18.0%)</td>\n",
       "      <td>0.8696/0.8368 (-3.8%)</td>\n",
       "      <td>0.8504/0.8480 (-0.3%)</td>\n",
       "      <td>41</td>\n",
       "      <td>0.8328/0.8252 (-0.9%)</td>\n",
       "      <td>5000</td>\n",
       "      <td>0.8584/0.6572 (-23.4%)</td>\n",
       "      <td>0.8608/0.8336 (-3.2%)</td>\n",
       "      <td>0.8640/0.8424 (-2.5%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wine</th>\n",
       "      <td>0.9773/0.5056 (-48.3%)</td>\n",
       "      <td>0.9091/0.9663 (+6.3%)</td>\n",
       "      <td>0.9773/0.5393 (-44.8%)</td>\n",
       "      <td>14</td>\n",
       "      <td>0.9318/0.2809 (-69.9%)</td>\n",
       "      <td>178</td>\n",
       "      <td>0.9773/0.6517 (-33.3%)</td>\n",
       "      <td>0.9773/0.9888 (+1.2%)</td>\n",
       "      <td>0.9773/0.9213 (-5.7%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wine-quality-red</th>\n",
       "      <td>0.5450/0.4631 (-15.0%)</td>\n",
       "      <td>0.5625/0.5557 (-1.2%)</td>\n",
       "      <td>0.6100/0.5820 (-4.6%)</td>\n",
       "      <td>12</td>\n",
       "      <td>0.6250/0.5770 (-7.7%)</td>\n",
       "      <td>1599</td>\n",
       "      <td>0.6150/0.4456 (-27.6%)</td>\n",
       "      <td>0.6300/0.5081 (-19.3%)</td>\n",
       "      <td>0.5575/0.4693 (-15.8%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wine-quality-white</th>\n",
       "      <td>0.5335/0.4684 (-12.2%)</td>\n",
       "      <td>0.5564/0.4565 (-18.0%)</td>\n",
       "      <td>0.6544/0.5892 (-10.0%)</td>\n",
       "      <td>12</td>\n",
       "      <td>0.6479/0.4924 (-24.0%)</td>\n",
       "      <td>4898</td>\n",
       "      <td>0.6307/0.4945 (-21.6%)</td>\n",
       "      <td>0.6373/0.5517 (-13.4%)</td>\n",
       "      <td>0.5482/0.4973 (-9.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yeast</th>\n",
       "      <td>0.4906/0.0984 (-79.9%)</td>\n",
       "      <td>0.6065/0.4987 (-17.8%)</td>\n",
       "      <td>0.6092/0.5620 (-7.7%)</td>\n",
       "      <td>9</td>\n",
       "      <td>0.6173/0.5768 (-6.6%)</td>\n",
       "      <td>1484</td>\n",
       "      <td>0.5499/0.3976 (-27.7%)</td>\n",
       "      <td>0.6307/0.4704 (-25.4%)</td>\n",
       "      <td>0.5876/0.5391 (-8.3%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zoo</th>\n",
       "      <td>0.7200/0.8600 (+19.4%)</td>\n",
       "      <td>0.8800/0.2000 (-77.3%)</td>\n",
       "      <td>0.9600/0.8600 (-10.4%)</td>\n",
       "      <td>17</td>\n",
       "      <td>1.0000/0.9400 (-6.0%)</td>\n",
       "      <td>101</td>\n",
       "      <td>1.0000/0.9400 (-6.0%)</td>\n",
       "      <td>0.9200/0.9800 (+6.5%)</td>\n",
       "      <td>0.9600/0.8000 (-16.7%)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           BN (old/new)  \\\n",
       "abalone                          0.6303/0.5575 (-11.6%)   \n",
       "acute-inflammation                1.0000/0.9667 (-3.3%)   \n",
       "acute-nephritis                   1.0000/0.9333 (-6.7%)   \n",
       "adult                             0.8499/0.8522 (+0.3%)   \n",
       "annealing                        0.1200/0.0600 (-50.0%)   \n",
       "arrhythmia                       0.5929/0.5044 (-14.9%)   \n",
       "audiology-std                    0.6400/0.5200 (-18.8%)   \n",
       "balance-scale                    0.9231/0.4712 (-49.0%)   \n",
       "balloons                         1.0000/0.3750 (-62.5%)   \n",
       "bank                              0.8823/0.8850 (+0.3%)   \n",
       "blood                             0.7647/0.7567 (-1.0%)   \n",
       "breast-cancer                    0.7324/0.2937 (-59.9%)   \n",
       "breast-cancer-wisc               0.9829/0.3438 (-65.0%)   \n",
       "breast-cancer-wisc-diag           0.9789/0.8873 (-9.4%)   \n",
       "breast-cancer-wisc-prog          0.7755/0.5556 (-28.4%)   \n",
       "breast-tissue                    0.4615/0.7358 (+59.4%)   \n",
       "car                              0.9606/0.1146 (-88.1%)   \n",
       "cardiotocography-10clases        0.7910/0.0941 (-88.1%)   \n",
       "cardiotocography-3clases         0.9096/0.7789 (-14.4%)   \n",
       "chess-krvk                       0.8781/0.0139 (-98.4%)   \n",
       "chess-krvkp                      0.9862/0.5225 (-47.0%)   \n",
       "congressional-voting             0.5872/0.3871 (-34.1%)   \n",
       "conn-bench-sonar-mines-rocks     0.7115/0.4712 (-33.8%)   \n",
       "conn-bench-vowel-deterding        0.9610/0.9004 (-6.3%)   \n",
       "connect-4                        0.8729/0.2462 (-71.8%)   \n",
       "contrac                           0.4538/0.4348 (-4.2%)   \n",
       "credit-approval                  0.8721/0.7768 (-10.9%)   \n",
       "cylinder-bands                   0.7500/0.6172 (-17.7%)   \n",
       "dermatology                      0.9341/0.3716 (-60.2%)   \n",
       "echocardiogram                   0.8485/0.6462 (-23.8%)   \n",
       "ecoli                            0.8214/0.7381 (-10.1%)   \n",
       "energy-y1                        0.8646/0.3516 (-59.3%)   \n",
       "energy-y2                        0.8750/0.2448 (-72.0%)   \n",
       "fertility                        0.6800/0.8800 (+29.4%)   \n",
       "flags                            0.4167/0.4845 (+16.3%)   \n",
       "glass                             0.5849/0.5888 (+0.7%)   \n",
       "haberman-survival                0.7368/0.6275 (-14.8%)   \n",
       "hayes-roth                       0.7500/0.0357 (-95.2%)   \n",
       "heart-cleveland                  0.5789/0.3510 (-39.4%)   \n",
       "heart-hungarian                  0.8493/0.7551 (-11.1%)   \n",
       "heart-switzerland                0.3871/0.0656 (-83.1%)   \n",
       "heart-va                          0.2800/0.2800 (-0.0%)   \n",
       "hepatitis                         0.8718/0.7922 (-9.1%)   \n",
       "hill-valley                      0.5050/0.5693 (+12.7%)   \n",
       "horse-colic                      0.8529/0.6029 (-29.3%)   \n",
       "ilpd-indian-liver                 0.5959/0.6529 (+9.6%)   \n",
       "image-segmentation               0.8481/0.1686 (-80.1%)   \n",
       "ionosphere                       0.9432/0.7200 (-23.7%)   \n",
       "iris                             0.9189/0.6267 (-31.8%)   \n",
       "led-display                      0.6280/0.4980 (-20.7%)   \n",
       "lenses                           0.8333/0.1667 (-80.0%)   \n",
       "letter                           0.9796/0.0847 (-91.4%)   \n",
       "libras                            0.7444/0.7500 (+0.8%)   \n",
       "low-res-spect                    0.8571/0.6340 (-26.0%)   \n",
       "lung-cancer                      0.5000/0.4375 (-12.5%)   \n",
       "lymphography                      0.7568/0.7973 (+5.4%)   \n",
       "magic                            0.8713/0.6485 (-25.6%)   \n",
       "mammographic                     0.8167/0.6417 (-21.4%)   \n",
       "miniboone                        0.9262/0.7194 (-22.3%)   \n",
       "molec-biol-promoter               0.7692/0.6981 (-9.2%)   \n",
       "molec-biol-splice                0.8519/0.5273 (-38.1%)   \n",
       "monks-1                          0.9074/0.5255 (-42.1%)   \n",
       "monks-2                          0.3287/0.6551 (+99.3%)   \n",
       "monks-3                           0.5278/0.5648 (+7.0%)   \n",
       "mushroom                         0.9990/0.4628 (-53.7%)   \n",
       "musk-1                           0.8235/0.7269 (-11.7%)   \n",
       "musk-2                           0.9982/0.1540 (-84.6%)   \n",
       "nursery                          0.9994/0.6267 (-37.3%)   \n",
       "oocytes_merluccius_nucleus_4d    0.8078/0.7065 (-12.5%)   \n",
       "oocytes_merluccius_states_2f     0.9333/0.6888 (-26.2%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.7456/0.5811 (-22.1%)   \n",
       "oocytes_trisopterus_states_5b    0.8947/0.7873 (-12.0%)   \n",
       "optical                           0.9716/0.9644 (-0.7%)   \n",
       "ozone                             0.9669/0.9685 (+0.2%)   \n",
       "page-blocks                       0.9613/0.9419 (-2.0%)   \n",
       "parkinsons                       0.8571/0.7526 (-12.2%)   \n",
       "pendigits                         0.9734/0.9720 (-0.1%)   \n",
       "pima                              0.7188/0.6719 (-6.5%)   \n",
       "pittsburg-bridges-MATERIAL        0.8846/0.8679 (-1.9%)   \n",
       "pittsburg-bridges-REL-L          0.7692/0.5686 (-26.1%)   \n",
       "pittsburg-bridges-SPAN           0.5652/0.5000 (-11.5%)   \n",
       "pittsburg-bridges-T-OR-D         0.8800/0.7059 (-19.8%)   \n",
       "pittsburg-bridges-TYPE          0.1154/0.5192 (+349.9%)   \n",
       "planning                         0.6222/0.5165 (-17.0%)   \n",
       "plant-margin                     0.7600/0.5375 (-29.3%)   \n",
       "plant-shape                      0.2850/0.2062 (-27.6%)   \n",
       "plant-texture                    0.8200/0.7134 (-13.0%)   \n",
       "post-operative                    0.5909/0.6222 (+5.3%)   \n",
       "primary-tumor                    0.5122/0.2727 (-46.8%)   \n",
       "ringnorm                         0.9843/0.5049 (-48.7%)   \n",
       "seeds                            0.8654/0.6667 (-23.0%)   \n",
       "semeion                          0.9372/0.0992 (-89.4%)   \n",
       "soybean                          0.8883/0.5053 (-43.1%)   \n",
       "spambase                          0.9426/0.8948 (-5.1%)   \n",
       "spect                            0.6344/0.4624 (-27.1%)   \n",
       "spectf                          0.2299/0.6738 (+193.1%)   \n",
       "statlog-australian-credit         0.6802/0.6725 (-1.1%)   \n",
       "statlog-german-credit             0.7520/0.6980 (-7.2%)   \n",
       "statlog-heart                    0.7910/0.4444 (-43.8%)   \n",
       "statlog-image                    0.9671/0.1446 (-85.0%)   \n",
       "statlog-landsat                   0.9040/0.8755 (-3.2%)   \n",
       "statlog-shuttle                   0.9988/0.9961 (-0.3%)   \n",
       "statlog-vehicle                  0.7583/0.6809 (-10.2%)   \n",
       "steel-plates                     0.7031/0.6124 (-12.9%)   \n",
       "synthetic-control                0.9733/0.2133 (-78.1%)   \n",
       "teaching                         0.5000/0.3467 (-30.7%)   \n",
       "thyroid                           0.9778/0.9787 (+0.1%)   \n",
       "tic-tac-toe                      0.9833/0.7724 (-21.4%)   \n",
       "titanic                           0.7800/0.7491 (-4.0%)   \n",
       "trains                           0.5000/0.4000 (-20.0%)   \n",
       "twonorm                           0.9757/0.9127 (-6.5%)   \n",
       "vertebral-column-2clases         0.8312/0.3226 (-61.2%)   \n",
       "vertebral-column-3clases          0.7792/0.7935 (+1.8%)   \n",
       "wall-following                   0.9333/0.1514 (-83.8%)   \n",
       "waveform                         0.8360/0.7064 (-15.5%)   \n",
       "waveform-noise                   0.8480/0.6956 (-18.0%)   \n",
       "wine                             0.9773/0.5056 (-48.3%)   \n",
       "wine-quality-red                 0.5450/0.4631 (-15.0%)   \n",
       "wine-quality-white               0.5335/0.4684 (-12.2%)   \n",
       "yeast                            0.4906/0.0984 (-79.9%)   \n",
       "zoo                              0.7200/0.8600 (+19.4%)   \n",
       "\n",
       "                                           HW (old/new)  \\\n",
       "abalone                          0.6427/0.5383 (-16.2%)   \n",
       "acute-inflammation               1.0000/0.6167 (-38.3%)   \n",
       "acute-nephritis                  1.0000/0.5833 (-41.7%)   \n",
       "adult                             0.8453/0.8524 (+0.8%)   \n",
       "annealing                        0.3600/0.6500 (+80.6%)   \n",
       "arrhythmia                        0.6283/0.6062 (-3.5%)   \n",
       "audiology-std                    0.7200/0.6400 (-11.1%)   \n",
       "balance-scale                     0.9103/0.9679 (+6.3%)   \n",
       "balloons                        0.2500/0.6250 (+150.0%)   \n",
       "bank                              0.8885/0.8221 (-7.5%)   \n",
       "blood                            0.7968/0.6765 (-15.1%)   \n",
       "breast-cancer                    0.7465/0.6573 (-11.9%)   \n",
       "breast-cancer-wisc                0.9771/0.9284 (-5.0%)   \n",
       "breast-cancer-wisc-diag           0.9789/0.9754 (-0.4%)   \n",
       "breast-cancer-wisc-prog          0.8367/0.6869 (-17.9%)   \n",
       "breast-tissue                     0.6154/0.5849 (-5.0%)   \n",
       "car                               0.9560/0.9664 (+1.1%)   \n",
       "cardiotocography-10clases        0.8456/0.7582 (-10.3%)   \n",
       "cardiotocography-3clases          0.9171/0.8965 (-2.2%)   \n",
       "chess-krvk                       0.5255/0.1500 (-71.5%)   \n",
       "chess-krvkp                       0.9900/0.9837 (-0.6%)   \n",
       "congressional-voting             0.5872/0.5115 (-12.9%)   \n",
       "conn-bench-sonar-mines-rocks      0.8462/0.8750 (+3.4%)   \n",
       "conn-bench-vowel-deterding        0.9784/0.9870 (+0.9%)   \n",
       "connect-4                        0.8599/0.3112 (-63.8%)   \n",
       "contrac                          0.5054/0.4321 (-14.5%)   \n",
       "credit-approval                   0.8547/0.8493 (-0.6%)   \n",
       "cylinder-bands                    0.7969/0.7656 (-3.9%)   \n",
       "dermatology                       0.9780/0.9563 (-2.2%)   \n",
       "echocardiogram                   0.6061/0.3231 (-46.7%)   \n",
       "ecoli                             0.8690/0.8690 (+0.0%)   \n",
       "energy-y1                         0.8802/0.9349 (+6.2%)   \n",
       "energy-y2                         0.9010/0.8438 (-6.4%)   \n",
       "fertility                         0.8800/0.8800 (-0.0%)   \n",
       "flags                             0.4375/0.4742 (+8.4%)   \n",
       "glass                             0.6415/0.6075 (-5.3%)   \n",
       "haberman-survival                0.6447/0.2614 (-59.4%)   \n",
       "hayes-roth                       0.7857/0.2143 (-72.7%)   \n",
       "heart-cleveland                  0.6316/0.5497 (-13.0%)   \n",
       "heart-hungarian                   0.7945/0.7891 (-0.7%)   \n",
       "heart-switzerland                0.5806/0.4098 (-29.4%)   \n",
       "heart-va                         0.4000/0.3100 (-22.5%)   \n",
       "hepatitis                        0.6667/0.7922 (+18.8%)   \n",
       "hill-valley                       0.5000/0.4934 (-1.3%)   \n",
       "horse-colic                      0.7794/0.6029 (-22.6%)   \n",
       "ilpd-indian-liver                 0.6781/0.6667 (-1.7%)   \n",
       "image-segmentation               0.9024/0.2114 (-76.6%)   \n",
       "ionosphere                       0.9432/0.6457 (-31.5%)   \n",
       "iris                             0.8378/0.9467 (+13.0%)   \n",
       "led-display                      0.7040/0.6080 (-13.6%)   \n",
       "lenses                           1.0000/0.6667 (-33.3%)   \n",
       "letter                            0.8984/0.8377 (-6.8%)   \n",
       "libras                           0.8222/0.6056 (-26.3%)   \n",
       "low-res-spect                    0.9023/0.5245 (-41.9%)   \n",
       "lung-cancer                     0.1250/0.4375 (+250.0%)   \n",
       "lymphography                     0.7297/0.5541 (-24.1%)   \n",
       "magic                            0.8673/0.5792 (-33.2%)   \n",
       "mammographic                      0.7917/0.7771 (-1.8%)   \n",
       "miniboone                        0.9270/0.7355 (-20.7%)   \n",
       "molec-biol-promoter              0.6923/0.8113 (+17.2%)   \n",
       "molec-biol-splice                 0.8833/0.8401 (-4.9%)   \n",
       "monks-1                          0.5833/0.6435 (+10.3%)   \n",
       "monks-2                           0.6389/0.6713 (+5.1%)   \n",
       "monks-3                           0.5880/0.5764 (-2.0%)   \n",
       "mushroom                         1.0000/0.4820 (-51.8%)   \n",
       "musk-1                           0.8992/0.6765 (-24.8%)   \n",
       "musk-2                            0.9915/0.9900 (-0.2%)   \n",
       "nursery                           1.0000/0.9647 (-3.5%)   \n",
       "oocytes_merluccius_nucleus_4d     0.7176/0.6712 (-6.5%)   \n",
       "oocytes_merluccius_states_2f      0.9490/0.9119 (-3.9%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.8289/0.5789 (-30.2%)   \n",
       "oocytes_trisopterus_states_5b     0.9342/0.8991 (-3.8%)   \n",
       "optical                           0.9644/0.9738 (+1.0%)   \n",
       "ozone                             0.9716/0.9685 (-0.3%)   \n",
       "page-blocks                       0.9656/0.9401 (-2.6%)   \n",
       "parkinsons                        0.8367/0.8763 (+4.7%)   \n",
       "pendigits                         0.9671/0.9605 (-0.7%)   \n",
       "pima                              0.7188/0.7135 (-0.7%)   \n",
       "pittsburg-bridges-MATERIAL        0.9231/0.8679 (-6.0%)   \n",
       "pittsburg-bridges-REL-L          0.6923/0.4510 (-34.9%)   \n",
       "pittsburg-bridges-SPAN           0.5652/0.7609 (+34.6%)   \n",
       "pittsburg-bridges-T-OR-D          0.8800/0.8627 (-2.0%)   \n",
       "pittsburg-bridges-TYPE            0.5385/0.5577 (+3.6%)   \n",
       "planning                         0.6000/0.3956 (-34.1%)   \n",
       "plant-margin                     0.8375/0.6663 (-20.4%)   \n",
       "plant-shape                      0.6325/0.3925 (-37.9%)   \n",
       "plant-texture                     0.7900/0.7860 (-0.5%)   \n",
       "post-operative                   0.5909/0.6889 (+16.6%)   \n",
       "primary-tumor                    0.4512/0.3939 (-12.7%)   \n",
       "ringnorm                          0.9692/0.9673 (-0.2%)   \n",
       "seeds                             0.9423/0.8952 (-5.0%)   \n",
       "semeion                           0.9447/0.9397 (-0.5%)   \n",
       "soybean                          0.8617/0.3856 (-55.2%)   \n",
       "spambase                          0.9435/0.9222 (-2.3%)   \n",
       "spect                             0.6022/0.5484 (-8.9%)   \n",
       "spectf                            0.8930/0.8610 (-3.6%)   \n",
       "statlog-australian-credit         0.6802/0.6232 (-8.4%)   \n",
       "statlog-german-credit             0.7760/0.7120 (-8.2%)   \n",
       "statlog-heart                    0.7761/0.4444 (-42.7%)   \n",
       "statlog-image                     0.9584/0.9195 (-4.1%)   \n",
       "statlog-landsat                   0.9110/0.8920 (-2.1%)   \n",
       "statlog-shuttle                   0.9977/0.9977 (+0.0%)   \n",
       "statlog-vehicle                  0.7962/0.6879 (-13.6%)   \n",
       "steel-plates                      0.7608/0.7082 (-6.9%)   \n",
       "synthetic-control                0.9867/0.1667 (-83.1%)   \n",
       "teaching                         0.5263/0.3867 (-26.5%)   \n",
       "thyroid                           0.9708/0.9770 (+0.6%)   \n",
       "tic-tac-toe                       0.9749/0.9457 (-3.0%)   \n",
       "titanic                          0.7927/0.3227 (-59.3%)   \n",
       "trains                               nan/0.4000 (+nan%)   \n",
       "twonorm                           0.9708/0.9738 (+0.3%)   \n",
       "vertebral-column-2clases         0.8571/0.6581 (-23.2%)   \n",
       "vertebral-column-3clases          0.7922/0.8194 (+3.4%)   \n",
       "wall-following                    0.9230/0.8904 (-3.5%)   \n",
       "waveform                          0.8320/0.8380 (+0.7%)   \n",
       "waveform-noise                    0.8696/0.8368 (-3.8%)   \n",
       "wine                              0.9091/0.9663 (+6.3%)   \n",
       "wine-quality-red                  0.5625/0.5557 (-1.2%)   \n",
       "wine-quality-white               0.5564/0.4565 (-18.0%)   \n",
       "yeast                            0.6065/0.4987 (-17.8%)   \n",
       "zoo                              0.8800/0.2000 (-77.3%)   \n",
       "\n",
       "                                           LN (old/new)    M  \\\n",
       "abalone                           0.6178/0.6049 (-2.1%)    9   \n",
       "acute-inflammation               0.9000/1.0000 (+11.1%)    7   \n",
       "acute-nephritis                   1.0000/1.0000 (-0.0%)    7   \n",
       "adult                             0.8517/0.8522 (+0.1%)   15   \n",
       "annealing                         0.5000/0.4700 (-6.0%)   32   \n",
       "arrhythmia                        0.5752/0.6106 (+6.2%)  263   \n",
       "audiology-std                     0.8000/0.8000 (-0.0%)   60   \n",
       "balance-scale                     0.9872/0.9359 (-5.2%)    5   \n",
       "balloons                         0.7500/0.5000 (-33.3%)    5   \n",
       "bank                              0.8920/0.8735 (-2.1%)   17   \n",
       "blood                            0.7112/0.2380 (-66.5%)    5   \n",
       "breast-cancer                     0.6620/0.6224 (-6.0%)   10   \n",
       "breast-cancer-wisc                0.9714/0.9599 (-1.2%)   10   \n",
       "breast-cancer-wisc-diag           0.9648/0.9683 (+0.4%)   31   \n",
       "breast-cancer-wisc-prog          0.7959/0.5758 (-27.7%)   34   \n",
       "breast-tissue                    0.5769/0.4528 (-21.5%)   10   \n",
       "car                               0.9907/0.9826 (-0.8%)    7   \n",
       "cardiotocography-10clases         0.8362/0.8109 (-3.0%)   22   \n",
       "cardiotocography-3clases          0.9021/0.9040 (+0.2%)   22   \n",
       "chess-krvk                       0.8938/0.1495 (-83.3%)    7   \n",
       "chess-krvkp                       0.9875/0.9762 (-1.1%)   37   \n",
       "congressional-voting              0.5780/0.5576 (-3.5%)   17   \n",
       "conn-bench-sonar-mines-rocks     0.6731/0.8077 (+20.0%)   61   \n",
       "conn-bench-vowel-deterding       0.9935/0.7468 (-24.8%)   12   \n",
       "connect-4                        0.8856/0.2462 (-72.2%)   43   \n",
       "contrac                           0.4592/0.4864 (+5.9%)   10   \n",
       "credit-approval                   0.8547/0.8348 (-2.3%)   16   \n",
       "cylinder-bands                    0.7578/0.7461 (-1.5%)   36   \n",
       "dermatology                       0.9451/0.9563 (+1.2%)   35   \n",
       "echocardiogram                    0.8182/0.7538 (-7.9%)   11   \n",
       "ecoli                             0.8571/0.8452 (-1.4%)    8   \n",
       "energy-y1                        0.9479/0.8021 (-15.4%)    9   \n",
       "energy-y2                         0.8802/0.8880 (+0.9%)    9   \n",
       "fertility                         0.8800/0.8800 (-0.0%)   10   \n",
       "flags                            0.3542/0.1443 (-59.3%)   29   \n",
       "glass                            0.6981/0.5421 (-22.4%)   10   \n",
       "haberman-survival                 0.6842/0.6275 (-8.3%)    4   \n",
       "hayes-roth                       0.8929/0.4643 (-48.0%)    4   \n",
       "heart-cleveland                   0.5789/0.5364 (-7.3%)   14   \n",
       "heart-hungarian                   0.8493/0.7891 (-7.1%)   13   \n",
       "heart-switzerland                 0.5161/0.4754 (-7.9%)   13   \n",
       "heart-va                         0.2400/0.2900 (+20.8%)   13   \n",
       "hepatitis                         0.7436/0.7922 (+6.5%)   20   \n",
       "hill-valley                      0.5050/0.6287 (+24.5%)  101   \n",
       "horse-colic                      0.7941/0.6324 (-20.4%)   26   \n",
       "ilpd-indian-liver                 0.6986/0.6942 (-0.6%)   10   \n",
       "image-segmentation               0.8838/0.4657 (-47.3%)   19   \n",
       "ionosphere                       0.9432/0.6400 (-32.1%)   34   \n",
       "iris                              0.9730/0.9333 (-4.1%)    5   \n",
       "led-display                      0.6480/0.1100 (-83.0%)    8   \n",
       "lenses                           0.6667/0.1667 (-75.0%)    5   \n",
       "letter                           0.9742/0.0367 (-96.2%)   17   \n",
       "libras                           0.8333/0.6667 (-20.0%)   91   \n",
       "low-res-spect                     0.8947/0.8453 (-5.5%)  101   \n",
       "lung-cancer                     0.2500/0.5625 (+125.0%)   57   \n",
       "lymphography                      0.7838/0.7838 (-0.0%)   19   \n",
       "magic                            0.8620/0.7728 (-10.4%)   11   \n",
       "mammographic                      0.8208/0.7896 (-3.8%)    6   \n",
       "miniboone                        0.9313/0.7450 (-20.0%)   51   \n",
       "molec-biol-promoter              0.4615/0.7358 (+59.4%)   58   \n",
       "molec-biol-splice                0.8607/0.7197 (-16.4%)   61   \n",
       "monks-1                           0.7014/0.6829 (-2.6%)    7   \n",
       "monks-2                          0.5162/0.6690 (+29.6%)    7   \n",
       "monks-3                          0.6991/0.5486 (-21.5%)    7   \n",
       "mushroom                          0.9995/0.9183 (-8.1%)   22   \n",
       "musk-1                            0.8992/0.8445 (-6.1%)  167   \n",
       "musk-2                           0.9951/0.1540 (-84.5%)  167   \n",
       "nursery                          0.9966/0.3120 (-68.7%)    9   \n",
       "oocytes_merluccius_nucleus_4d     0.7686/0.7847 (+2.1%)   42   \n",
       "oocytes_merluccius_states_2f      0.9412/0.9061 (-3.7%)   26   \n",
       "oocytes_trisopterus_nucleus_2f   0.8202/0.6031 (-26.5%)   26   \n",
       "oocytes_trisopterus_states_5b     0.8991/0.9057 (+0.7%)   33   \n",
       "optical                           0.9755/0.9699 (-0.6%)   63   \n",
       "ozone                             0.9716/0.9669 (-0.5%)   73   \n",
       "page-blocks                       0.9708/0.9550 (-1.6%)   11   \n",
       "parkinsons                        0.8571/0.9175 (+7.1%)   23   \n",
       "pendigits                         0.9657/0.9720 (+0.7%)   17   \n",
       "pima                              0.6927/0.7474 (+7.9%)    9   \n",
       "pittsburg-bridges-MATERIAL       0.9231/0.7170 (-22.3%)    8   \n",
       "pittsburg-bridges-REL-L          0.7308/0.1373 (-81.2%)    8   \n",
       "pittsburg-bridges-SPAN            0.6087/0.6304 (+3.6%)    8   \n",
       "pittsburg-bridges-T-OR-D          0.8800/0.8627 (-2.0%)    8   \n",
       "pittsburg-bridges-TYPE           0.6538/0.5769 (-11.8%)    8   \n",
       "planning                          0.6889/0.7143 (+3.7%)   13   \n",
       "plant-margin                      0.8425/0.7688 (-8.8%)   65   \n",
       "plant-shape                       0.6775/0.6388 (-5.7%)   65   \n",
       "plant-texture                    0.8350/0.0100 (-98.8%)   65   \n",
       "post-operative                   0.7727/0.0222 (-97.1%)    9   \n",
       "primary-tumor                    0.4512/0.3455 (-23.4%)   18   \n",
       "ringnorm                         0.9827/0.5049 (-48.6%)   21   \n",
       "seeds                             0.8846/0.8952 (+1.2%)    8   \n",
       "semeion                          0.9447/0.0992 (-89.5%)  257   \n",
       "soybean                          0.8484/0.4468 (-47.3%)   36   \n",
       "spambase                          0.9513/0.9230 (-3.0%)   58   \n",
       "spect                             0.6720/0.6237 (-7.2%)   23   \n",
       "spectf                           0.5561/0.9144 (+64.4%)   45   \n",
       "statlog-australian-credit         0.6279/0.6783 (+8.0%)   15   \n",
       "statlog-german-credit             0.7400/0.7420 (+0.3%)   25   \n",
       "statlog-heart                     0.7910/0.8444 (+6.8%)   14   \n",
       "statlog-image                    0.9757/0.1429 (-85.4%)   19   \n",
       "statlog-landsat                   0.9040/0.8945 (-1.1%)   37   \n",
       "statlog-shuttle                   0.9987/0.9991 (+0.0%)   10   \n",
       "statlog-vehicle                   0.7915/0.7636 (-3.5%)   19   \n",
       "steel-plates                     0.7588/0.3474 (-54.2%)   28   \n",
       "synthetic-control                0.9733/0.2067 (-78.8%)   61   \n",
       "teaching                         0.6316/0.4400 (-30.3%)    6   \n",
       "thyroid                           0.9752/0.9761 (+0.1%)   22   \n",
       "tic-tac-toe                       0.9791/0.9645 (-1.5%)   10   \n",
       "titanic                          0.7891/0.3227 (-59.1%)    4   \n",
       "trains                           1.0000/0.6000 (-40.0%)   30   \n",
       "twonorm                           0.9724/0.9732 (+0.1%)   21   \n",
       "vertebral-column-2clases          0.8442/0.8000 (-5.2%)    7   \n",
       "vertebral-column-3clases          0.8312/0.8065 (-3.0%)    7   \n",
       "wall-following                   0.9128/0.0601 (-93.4%)   25   \n",
       "waveform                          0.8448/0.8592 (+1.7%)   22   \n",
       "waveform-noise                    0.8504/0.8480 (-0.3%)   41   \n",
       "wine                             0.9773/0.5393 (-44.8%)   14   \n",
       "wine-quality-red                  0.6100/0.5820 (-4.6%)   12   \n",
       "wine-quality-white               0.6544/0.5892 (-10.0%)   12   \n",
       "yeast                             0.6092/0.5620 (-7.7%)    9   \n",
       "zoo                              0.9600/0.8600 (-10.4%)   17   \n",
       "\n",
       "                                          MS (old/new)       N  \\\n",
       "abalone                          0.6284/0.6130 (-2.4%)    4177   \n",
       "acute-inflammation               1.0000/1.0000 (-0.0%)     120   \n",
       "acute-nephritis                  1.0000/1.0000 (-0.0%)     120   \n",
       "adult                            0.8487/0.8511 (+0.3%)   48842   \n",
       "annealing                        0.7300/0.7500 (+2.7%)     898   \n",
       "arrhythmia                       0.6372/0.5929 (-6.9%)     452   \n",
       "audiology-std                    0.6800/0.7200 (+5.9%)     196   \n",
       "balance-scale                    0.9231/0.9615 (+4.2%)     625   \n",
       "balloons                         0.5000/0.5000 (-0.0%)      16   \n",
       "bank                             0.8876/0.8854 (-0.2%)    4521   \n",
       "blood                           0.7754/0.2380 (-69.3%)     748   \n",
       "breast-cancer                    0.6901/0.6713 (-2.7%)     286   \n",
       "breast-cancer-wisc               0.9714/0.9599 (-1.2%)     699   \n",
       "breast-cancer-wisc-diag          0.9718/0.9472 (-2.5%)     569   \n",
       "breast-cancer-wisc-prog         0.7347/0.6162 (-16.1%)     198   \n",
       "breast-tissue                   0.4615/0.6792 (+47.2%)     106   \n",
       "car                              0.9861/0.9213 (-6.6%)    1728   \n",
       "cardiotocography-10clases       0.8418/0.5814 (-30.9%)    2126   \n",
       "cardiotocography-3clases         0.8964/0.9106 (+1.6%)    2126   \n",
       "chess-krvk                      0.8606/0.1823 (-78.8%)   28056   \n",
       "chess-krvkp                     0.9900/0.5225 (-47.2%)    3196   \n",
       "congressional-voting            0.6055/0.5253 (-13.2%)     435   \n",
       "conn-bench-sonar-mines-rocks     0.8269/0.8365 (+1.2%)     208   \n",
       "conn-bench-vowel-deterding       0.9935/0.9978 (+0.4%)     990   \n",
       "connect-4                       0.8831/0.3711 (-58.0%)   67557   \n",
       "contrac                          0.5136/0.4946 (-3.7%)    1473   \n",
       "credit-approval                  0.8430/0.8087 (-4.1%)     690   \n",
       "cylinder-bands                   0.7656/0.7031 (-8.2%)     512   \n",
       "dermatology                      0.9121/0.9672 (+6.0%)     366   \n",
       "echocardiogram                  0.8485/0.6923 (-18.4%)     131   \n",
       "ecoli                            0.8333/0.8333 (+0.0%)     336   \n",
       "energy-y1                        0.9583/0.9089 (-5.2%)     768   \n",
       "energy-y2                        0.8958/0.9089 (+1.5%)     768   \n",
       "fertility                       0.8800/0.1200 (-86.4%)     100   \n",
       "flags                           0.4583/0.5464 (+19.2%)     194   \n",
       "glass                            0.6038/0.6449 (+6.8%)     214   \n",
       "haberman-survival                0.7237/0.6993 (-3.4%)     306   \n",
       "hayes-roth                      0.4643/0.0357 (-92.3%)     160   \n",
       "heart-cleveland                 0.6053/0.5232 (-13.6%)     303   \n",
       "heart-hungarian                  0.8356/0.8503 (+1.8%)     294   \n",
       "heart-switzerland                0.3871/0.3607 (-6.8%)     123   \n",
       "heart-va                        0.2600/0.3000 (+15.4%)     200   \n",
       "hepatitis                        0.7692/0.7922 (+3.0%)     155   \n",
       "hill-valley                     0.5116/0.7030 (+37.4%)    1212   \n",
       "horse-colic                     0.8529/0.6029 (-29.3%)     368   \n",
       "ilpd-indian-liver                0.6644/0.6701 (+0.9%)     583   \n",
       "image-segmentation              0.9090/0.2310 (-74.6%)    2310   \n",
       "ionosphere                       0.9091/0.9143 (+0.6%)     351   \n",
       "iris                             0.9189/0.8933 (-2.8%)     150   \n",
       "led-display                      0.7200/0.7000 (-2.8%)    1000   \n",
       "lenses                          1.0000/0.1667 (-83.3%)      24   \n",
       "letter                          0.9712/0.0401 (-95.9%)   20000   \n",
       "libras                           0.8667/0.7944 (-8.3%)     360   \n",
       "low-res-spect                    0.8496/0.8604 (+1.3%)     531   \n",
       "lung-cancer                     0.3750/0.3125 (-16.7%)      32   \n",
       "lymphography                    0.7297/0.8378 (+14.8%)     148   \n",
       "magic                           0.8629/0.4118 (-52.3%)   19020   \n",
       "mammographic                     0.8083/0.7917 (-2.1%)     961   \n",
       "miniboone                       0.9250/0.7641 (-17.4%)  130064   \n",
       "molec-biol-promoter             0.7692/0.6226 (-19.1%)     106   \n",
       "molec-biol-splice                0.8482/0.8031 (-5.3%)    3190   \n",
       "monks-1                         0.6551/0.7523 (+14.8%)     556   \n",
       "monks-2                          0.6343/0.6644 (+4.7%)     601   \n",
       "monks-3                         0.7454/0.5417 (-27.3%)     554   \n",
       "mushroom                         1.0000/0.9953 (-0.5%)    8124   \n",
       "musk-1                          0.8655/0.7773 (-10.2%)     476   \n",
       "musk-2                          0.9945/0.1540 (-84.5%)    6598   \n",
       "nursery                         0.9988/0.3120 (-68.8%)   12960   \n",
       "oocytes_merluccius_nucleus_4d    0.8196/0.7730 (-5.7%)    1022   \n",
       "oocytes_merluccius_states_2f     0.9490/0.8943 (-5.8%)    1022   \n",
       "oocytes_trisopterus_nucleus_2f  0.8728/0.7654 (-12.3%)     912   \n",
       "oocytes_trisopterus_states_5b   0.9430/0.4079 (-56.7%)     912   \n",
       "optical                          0.9666/0.9688 (+0.2%)    5620   \n",
       "ozone                           0.9732/0.0284 (-97.1%)    2536   \n",
       "page-blocks                     0.9708/0.0599 (-93.8%)    5473   \n",
       "parkinsons                      0.9184/0.7526 (-18.1%)     195   \n",
       "pendigits                        0.9714/0.9777 (+0.6%)   10992   \n",
       "pima                             0.7656/0.7344 (-4.1%)     768   \n",
       "pittsburg-bridges-MATERIAL       0.8462/0.8679 (+2.6%)     106   \n",
       "pittsburg-bridges-REL-L         0.7692/0.5686 (-26.1%)     103   \n",
       "pittsburg-bridges-SPAN          0.5217/0.7391 (+41.7%)      92   \n",
       "pittsburg-bridges-T-OR-D         0.8800/0.8627 (-2.0%)     102   \n",
       "pittsburg-bridges-TYPE          0.6538/0.5577 (-14.7%)     105   \n",
       "planning                        0.6667/0.4615 (-30.8%)     182   \n",
       "plant-margin                     0.8125/0.7950 (-2.2%)    1600   \n",
       "plant-shape                     0.6350/0.5175 (-18.5%)    1600   \n",
       "plant-texture                    0.7900/0.7972 (+0.9%)    1599   \n",
       "post-operative                   0.7273/0.7111 (-2.2%)      90   \n",
       "primary-tumor                   0.5000/0.4121 (-17.6%)     330   \n",
       "ringnorm                         0.9843/0.9762 (-0.8%)    7400   \n",
       "seeds                           0.8654/0.5048 (-41.7%)     210   \n",
       "semeion                          0.9296/0.9296 (+0.0%)    1593   \n",
       "soybean                         0.8723/0.3590 (-58.8%)     683   \n",
       "spambase                        0.9461/0.3939 (-58.4%)    4601   \n",
       "spect                           0.6183/0.5484 (-11.3%)     265   \n",
       "spectf                          0.6043/0.9198 (+52.2%)     267   \n",
       "statlog-australian-credit        0.6802/0.6667 (-2.0%)     690   \n",
       "statlog-german-credit            0.7280/0.7000 (-3.8%)    1000   \n",
       "statlog-heart                    0.8358/0.8593 (+2.8%)     270   \n",
       "statlog-image                   0.9757/0.8234 (-15.6%)    2310   \n",
       "statlog-landsat                 0.9075/0.2305 (-74.6%)    6435   \n",
       "statlog-shuttle                  0.9983/0.9989 (+0.1%)   58000   \n",
       "statlog-vehicle                  0.8294/0.7612 (-8.2%)     846   \n",
       "steel-plates                     0.7567/0.6887 (-9.0%)    1941   \n",
       "synthetic-control               0.9800/0.7033 (-28.2%)     600   \n",
       "teaching                        0.6053/0.3333 (-44.9%)     151   \n",
       "thyroid                          0.9770/0.9767 (-0.0%)    7200   \n",
       "tic-tac-toe                      0.9833/0.9353 (-4.9%)     958   \n",
       "titanic                         0.7909/0.5464 (-30.9%)    2201   \n",
       "trains                              nan/0.4000 (+nan%)      10   \n",
       "twonorm                          0.9778/0.9700 (-0.8%)    7400   \n",
       "vertebral-column-2clases        0.8701/0.7548 (-13.2%)     310   \n",
       "vertebral-column-3clases         0.8052/0.8452 (+5.0%)     310   \n",
       "wall-following                  0.9076/0.4637 (-48.9%)    5456   \n",
       "waveform                         0.8312/0.8464 (+1.8%)    5000   \n",
       "waveform-noise                   0.8328/0.8252 (-0.9%)    5000   \n",
       "wine                            0.9318/0.2809 (-69.9%)     178   \n",
       "wine-quality-red                 0.6250/0.5770 (-7.7%)    1599   \n",
       "wine-quality-white              0.6479/0.4924 (-24.0%)    4898   \n",
       "yeast                            0.6173/0.5768 (-6.6%)    1484   \n",
       "zoo                              1.0000/0.9400 (-6.0%)     101   \n",
       "\n",
       "                                       ResNet (old/new)  \\\n",
       "abalone                          0.6466/0.3463 (-46.4%)   \n",
       "acute-inflammation               1.0000/0.7000 (-30.0%)   \n",
       "acute-nephritis                  1.0000/0.5833 (-41.7%)   \n",
       "adult                             0.8484/0.8502 (+0.2%)   \n",
       "annealing                       0.2600/0.7400 (+184.6%)   \n",
       "arrhythmia                       0.6460/0.4159 (-35.6%)   \n",
       "audiology-std                    0.8000/0.5200 (-35.0%)   \n",
       "balance-scale                    0.9167/0.4744 (-48.3%)   \n",
       "balloons                         1.0000/0.8750 (-12.5%)   \n",
       "bank                              0.8796/0.8801 (+0.1%)   \n",
       "blood                            0.8021/0.5963 (-25.7%)   \n",
       "breast-cancer                     0.7465/0.7203 (-3.5%)   \n",
       "breast-cancer-wisc               0.9714/0.4298 (-55.8%)   \n",
       "breast-cancer-wisc-diag          0.9507/0.7430 (-21.9%)   \n",
       "breast-cancer-wisc-prog          0.8163/0.6162 (-24.5%)   \n",
       "breast-tissue                    0.4231/0.6792 (+60.5%)   \n",
       "car                              0.9282/0.1493 (-83.9%)   \n",
       "cardiotocography-10clases        0.8173/0.2427 (-70.3%)   \n",
       "cardiotocography-3clases         0.9021/0.7893 (-12.5%)   \n",
       "chess-krvk                       0.8543/0.1647 (-80.7%)   \n",
       "chess-krvkp                      0.9912/0.5763 (-41.9%)   \n",
       "congressional-voting             0.5963/0.4286 (-28.1%)   \n",
       "conn-bench-sonar-mines-rocks     0.8077/0.6731 (-16.7%)   \n",
       "conn-bench-vowel-deterding       0.9935/0.8485 (-14.6%)   \n",
       "connect-4                        0.8716/0.2940 (-66.3%)   \n",
       "contrac                          0.5136/0.3465 (-32.5%)   \n",
       "credit-approval                  0.8430/0.7449 (-11.6%)   \n",
       "cylinder-bands                    0.7734/0.7305 (-5.6%)   \n",
       "dermatology                       0.9231/0.9016 (-2.3%)   \n",
       "echocardiogram                   0.8485/0.6615 (-22.0%)   \n",
       "ecoli                            0.8214/0.5833 (-29.0%)   \n",
       "energy-y1                        0.8177/0.1667 (-79.6%)   \n",
       "energy-y2                        0.8750/0.6849 (-21.7%)   \n",
       "fertility                         0.8400/0.8800 (+4.8%)   \n",
       "flags                             0.3750/0.4021 (+7.2%)   \n",
       "glass                            0.6415/0.4766 (-25.7%)   \n",
       "haberman-survival                0.6842/0.3660 (-46.5%)   \n",
       "hayes-roth                       0.7143/0.0357 (-95.0%)   \n",
       "heart-cleveland                   0.5658/0.6093 (+7.7%)   \n",
       "heart-hungarian                   0.8082/0.7347 (-9.1%)   \n",
       "heart-switzerland                0.3226/0.3607 (+11.8%)   \n",
       "heart-va                         0.2600/0.3400 (+30.8%)   \n",
       "hepatitis                         0.7692/0.7922 (+3.0%)   \n",
       "hill-valley                       0.5396/0.5776 (+7.0%)   \n",
       "horse-colic                      0.8088/0.6029 (-25.5%)   \n",
       "ilpd-indian-liver                 0.6712/0.6976 (+3.9%)   \n",
       "image-segmentation               0.8919/0.1881 (-78.9%)   \n",
       "ionosphere                       0.9545/0.7200 (-24.6%)   \n",
       "iris                             0.9730/0.7467 (-23.3%)   \n",
       "led-display                      0.7160/0.2080 (-70.9%)   \n",
       "lenses                            0.6667/0.6667 (-0.0%)   \n",
       "letter                           0.9762/0.3086 (-68.4%)   \n",
       "libras                           0.7111/0.5333 (-25.0%)   \n",
       "low-res-spect                    0.8647/0.3170 (-63.3%)   \n",
       "lung-cancer                     0.2500/0.5000 (+100.0%)   \n",
       "lymphography                      0.6757/0.7027 (+4.0%)   \n",
       "magic                            0.8723/0.3516 (-59.7%)   \n",
       "mammographic                     0.7833/0.4000 (-48.9%)   \n",
       "miniboone                        0.9254/0.7194 (-22.3%)   \n",
       "molec-biol-promoter              0.7692/0.6604 (-14.1%)   \n",
       "molec-biol-splice                0.8557/0.6075 (-29.0%)   \n",
       "monks-1                          0.7546/0.5208 (-31.0%)   \n",
       "monks-2                           0.6273/0.6713 (+7.0%)   \n",
       "monks-3                          0.5833/0.6806 (+16.7%)   \n",
       "mushroom                         1.0000/0.4820 (-51.8%)   \n",
       "musk-1                           0.8739/0.7311 (-16.3%)   \n",
       "musk-2                            0.9964/0.9536 (-4.3%)   \n",
       "nursery                          0.9994/0.5350 (-46.5%)   \n",
       "oocytes_merluccius_nucleus_4d    0.8000/0.6732 (-15.9%)   \n",
       "oocytes_merluccius_states_2f     0.9373/0.6849 (-26.9%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.7719/0.6162 (-20.2%)   \n",
       "oocytes_trisopterus_states_5b    0.8947/0.6754 (-24.5%)   \n",
       "optical                           0.9627/0.9627 (+0.0%)   \n",
       "ozone                             0.9669/0.9700 (+0.3%)   \n",
       "page-blocks                       0.9605/0.9364 (-2.5%)   \n",
       "parkinsons                        0.9184/0.8866 (-3.5%)   \n",
       "pendigits                         0.9708/0.9723 (+0.2%)   \n",
       "pima                             0.7135/0.3490 (-51.1%)   \n",
       "pittsburg-bridges-MATERIAL       0.9231/0.8302 (-10.1%)   \n",
       "pittsburg-bridges-REL-L          0.8462/0.6863 (-18.9%)   \n",
       "pittsburg-bridges-SPAN            0.5652/0.6087 (+7.7%)   \n",
       "pittsburg-bridges-T-OR-D          0.8800/0.8627 (-2.0%)   \n",
       "pittsburg-bridges-TYPE            0.6538/0.6538 (+0.0%)   \n",
       "planning                         0.7111/0.5385 (-24.3%)   \n",
       "plant-margin                     0.7975/0.4200 (-47.3%)   \n",
       "plant-shape                      0.5150/0.2575 (-50.0%)   \n",
       "plant-texture                    0.8000/0.5419 (-32.3%)   \n",
       "post-operative                   0.7273/0.5556 (-23.6%)   \n",
       "primary-tumor                    0.3902/0.2788 (-28.6%)   \n",
       "ringnorm                         0.9811/0.5503 (-43.9%)   \n",
       "seeds                            0.8654/0.6190 (-28.5%)   \n",
       "semeion                          0.9146/0.5452 (-40.4%)   \n",
       "soybean                          0.8670/0.3750 (-56.7%)   \n",
       "spambase                         0.9461/0.7070 (-25.3%)   \n",
       "spect                            0.6667/0.5484 (-17.7%)   \n",
       "spectf                           0.7005/0.9198 (+31.3%)   \n",
       "statlog-australian-credit         0.6395/0.6783 (+6.1%)   \n",
       "statlog-german-credit            0.7720/0.5840 (-24.4%)   \n",
       "statlog-heart                    0.8657/0.6889 (-20.4%)   \n",
       "statlog-image                    0.9584/0.3281 (-65.8%)   \n",
       "statlog-landsat                   0.9055/0.8620 (-4.8%)   \n",
       "statlog-shuttle                   0.9992/0.9970 (-0.2%)   \n",
       "statlog-vehicle                  0.7583/0.5910 (-22.1%)   \n",
       "steel-plates                     0.7629/0.4227 (-44.6%)   \n",
       "synthetic-control                0.9600/0.4433 (-53.8%)   \n",
       "teaching                         0.5526/0.3467 (-37.3%)   \n",
       "thyroid                           0.9799/0.9770 (-0.3%)   \n",
       "tic-tac-toe                      0.9623/0.8246 (-14.3%)   \n",
       "titanic                          0.7727/0.3227 (-58.2%)   \n",
       "trains                               nan/0.8000 (+nan%)   \n",
       "twonorm                          0.9735/0.4995 (-48.7%)   \n",
       "vertebral-column-2clases         0.8312/0.6452 (-22.4%)   \n",
       "vertebral-column-3clases          0.7532/0.7097 (-5.8%)   \n",
       "wall-following                   0.9223/0.5154 (-44.1%)   \n",
       "waveform                         0.8360/0.4664 (-44.2%)   \n",
       "waveform-noise                   0.8584/0.6572 (-23.4%)   \n",
       "wine                             0.9773/0.6517 (-33.3%)   \n",
       "wine-quality-red                 0.6150/0.4456 (-27.6%)   \n",
       "wine-quality-white               0.6307/0.4945 (-21.6%)   \n",
       "yeast                            0.5499/0.3976 (-27.7%)   \n",
       "zoo                               1.0000/0.9400 (-6.0%)   \n",
       "\n",
       "                                         SNN (old/new)  \\\n",
       "abalone                         0.6657/0.3166 (-52.4%)   \n",
       "acute-inflammation               1.0000/1.0000 (-0.0%)   \n",
       "acute-nephritis                  1.0000/1.0000 (-0.0%)   \n",
       "adult                            0.8476/0.8527 (+0.6%)   \n",
       "annealing                       0.7600/0.1300 (-82.9%)   \n",
       "arrhythmia                       0.6549/0.5973 (-8.8%)   \n",
       "audiology-std                   0.8000/0.6800 (-15.0%)   \n",
       "balance-scale                    0.9231/0.9679 (+4.9%)   \n",
       "balloons                        1.0000/0.3750 (-62.5%)   \n",
       "bank                             0.8903/0.8677 (-2.5%)   \n",
       "blood                            0.7701/0.7674 (-0.4%)   \n",
       "breast-cancer                   0.7183/0.5734 (-20.2%)   \n",
       "breast-cancer-wisc               0.9714/0.9484 (-2.4%)   \n",
       "breast-cancer-wisc-diag          0.9789/0.9859 (+0.7%)   \n",
       "breast-cancer-wisc-prog          0.6735/0.7071 (+5.0%)   \n",
       "breast-tissue                    0.7308/0.6981 (-4.5%)   \n",
       "car                             0.9838/0.7014 (-28.7%)   \n",
       "cardiotocography-10clases        0.8399/0.7846 (-6.6%)   \n",
       "cardiotocography-3clases         0.9153/0.9219 (+0.7%)   \n",
       "chess-krvk                      0.8805/0.1625 (-81.5%)   \n",
       "chess-krvkp                     0.9912/0.5225 (-47.3%)   \n",
       "congressional-voting            0.6147/0.3871 (-37.0%)   \n",
       "conn-bench-sonar-mines-rocks     0.7885/0.7692 (-2.4%)   \n",
       "conn-bench-vowel-deterding       0.9957/0.9978 (+0.2%)   \n",
       "connect-4                       0.8807/0.4176 (-52.6%)   \n",
       "contrac                          0.5190/0.5177 (-0.3%)   \n",
       "credit-approval                  0.8430/0.8841 (+4.9%)   \n",
       "cylinder-bands                   0.7266/0.7461 (+2.7%)   \n",
       "dermatology                      0.9231/0.9781 (+6.0%)   \n",
       "echocardiogram                  0.8182/0.7231 (-11.6%)   \n",
       "ecoli                            0.8929/0.8631 (-3.3%)   \n",
       "energy-y1                       0.9583/0.1771 (-81.5%)   \n",
       "energy-y2                        0.9063/0.9323 (+2.9%)   \n",
       "fertility                        0.9200/0.8800 (-4.3%)   \n",
       "flags                            0.4583/0.4742 (+3.5%)   \n",
       "glass                            0.7358/0.7103 (-3.5%)   \n",
       "haberman-survival                0.7368/0.6732 (-8.6%)   \n",
       "hayes-roth                      0.6786/0.0357 (-94.7%)   \n",
       "heart-cleveland                  0.6184/0.5828 (-5.8%)   \n",
       "heart-hungarian                  0.7945/0.7959 (+0.2%)   \n",
       "heart-switzerland               0.3548/0.0656 (-81.5%)   \n",
       "heart-va                        0.3600/0.2600 (-27.8%)   \n",
       "hepatitis                        0.7692/0.7922 (+3.0%)   \n",
       "hill-valley                      0.5248/0.5627 (+7.2%)   \n",
       "horse-colic                     0.8088/0.2794 (-65.5%)   \n",
       "ilpd-indian-liver                0.6986/0.6976 (-0.1%)   \n",
       "image-segmentation              0.9114/0.1724 (-81.1%)   \n",
       "ionosphere                       0.8864/0.8743 (-1.4%)   \n",
       "iris                             0.9730/0.8800 (-9.6%)   \n",
       "led-display                      0.7640/0.7180 (-6.0%)   \n",
       "lenses                          0.6667/0.7500 (+12.5%)   \n",
       "letter                          0.9726/0.0401 (-95.9%)   \n",
       "libras                           0.7889/0.7667 (-2.8%)   \n",
       "low-res-spect                    0.8571/0.8679 (+1.3%)   \n",
       "lung-cancer                     0.6250/0.5000 (-20.0%)   \n",
       "lymphography                    0.9189/0.7973 (-13.2%)   \n",
       "magic                           0.8692/0.6707 (-22.8%)   \n",
       "mammographic                     0.8250/0.8021 (-2.8%)   \n",
       "miniboone                       0.9307/0.7194 (-22.7%)   \n",
       "molec-biol-promoter              0.8462/0.8491 (+0.3%)   \n",
       "molec-biol-splice               0.9009/0.2408 (-73.3%)   \n",
       "monks-1                         0.7523/0.6458 (-14.2%)   \n",
       "monks-2                         0.5926/0.6667 (+12.5%)   \n",
       "monks-3                          0.6042/0.6435 (+6.5%)   \n",
       "mushroom                         1.0000/0.9983 (-0.2%)   \n",
       "musk-1                           0.8739/0.8613 (-1.4%)   \n",
       "musk-2                          0.9891/0.1540 (-84.4%)   \n",
       "nursery                          0.9978/0.9347 (-6.3%)   \n",
       "oocytes_merluccius_nucleus_4d    0.8235/0.8141 (-1.1%)   \n",
       "oocytes_merluccius_states_2f     0.9529/0.9119 (-4.3%)   \n",
       "oocytes_trisopterus_nucleus_2f   0.7982/0.7632 (-4.4%)   \n",
       "oocytes_trisopterus_states_5b    0.9342/0.9057 (-3.1%)   \n",
       "optical                          0.9711/0.9638 (-0.7%)   \n",
       "ozone                            0.9700/0.9685 (-0.2%)   \n",
       "page-blocks                      0.9583/0.9437 (-1.5%)   \n",
       "parkinsons                      0.8980/0.7732 (-13.9%)   \n",
       "pendigits                        0.9706/0.9711 (+0.1%)   \n",
       "pima                             0.7552/0.7552 (+0.0%)   \n",
       "pittsburg-bridges-MATERIAL       0.8846/0.8302 (-6.2%)   \n",
       "pittsburg-bridges-REL-L         0.6923/0.5294 (-23.5%)   \n",
       "pittsburg-bridges-SPAN          0.6957/0.6087 (-12.5%)   \n",
       "pittsburg-bridges-T-OR-D         0.8400/0.8627 (+2.7%)   \n",
       "pittsburg-bridges-TYPE          0.6538/0.5192 (-20.6%)   \n",
       "planning                         0.6889/0.7143 (+3.7%)   \n",
       "plant-margin                     0.8125/0.7638 (-6.0%)   \n",
       "plant-shape                     0.7275/0.5637 (-22.5%)   \n",
       "plant-texture                    0.8125/0.7935 (-2.3%)   \n",
       "post-operative                  0.7273/0.2667 (-63.3%)   \n",
       "primary-tumor                   0.5244/0.4061 (-22.6%)   \n",
       "ringnorm                         0.9751/0.9832 (+0.8%)   \n",
       "seeds                            0.8846/0.9048 (+2.3%)   \n",
       "semeion                          0.9196/0.9209 (+0.1%)   \n",
       "soybean                         0.8511/0.0239 (-97.2%)   \n",
       "spambase                         0.9409/0.9243 (-1.8%)   \n",
       "spect                           0.6398/0.5645 (-11.8%)   \n",
       "spectf                          0.4973/0.9198 (+85.0%)   \n",
       "statlog-australian-credit        0.5988/0.6377 (+6.5%)   \n",
       "statlog-german-credit            0.7560/0.7720 (+2.1%)   \n",
       "statlog-heart                    0.9254/0.8593 (-7.1%)   \n",
       "statlog-image                    0.9549/0.9498 (-0.5%)   \n",
       "statlog-landsat                  0.9100/0.8780 (-3.5%)   \n",
       "statlog-shuttle                 0.9990/0.0009 (-99.9%)   \n",
       "statlog-vehicle                  0.8009/0.7329 (-8.5%)   \n",
       "steel-plates                    0.7835/0.7052 (-10.0%)   \n",
       "synthetic-control               0.9867/0.7833 (-20.6%)   \n",
       "teaching                        0.5000/0.3333 (-33.3%)   \n",
       "thyroid                          0.9816/0.9772 (-0.4%)   \n",
       "tic-tac-toe                      0.9665/0.9812 (+1.5%)   \n",
       "titanic                         0.7836/0.3300 (-57.9%)   \n",
       "trains                              nan/0.4000 (+nan%)   \n",
       "twonorm                         0.9805/0.4995 (-49.1%)   \n",
       "vertebral-column-2clases         0.8312/0.7806 (-6.1%)   \n",
       "vertebral-column-3clases         0.8312/0.8129 (-2.2%)   \n",
       "wall-following                  0.9098/0.1514 (-83.4%)   \n",
       "waveform                         0.8480/0.8304 (-2.1%)   \n",
       "waveform-noise                   0.8608/0.8336 (-3.2%)   \n",
       "wine                             0.9773/0.9888 (+1.2%)   \n",
       "wine-quality-red                0.6300/0.5081 (-19.3%)   \n",
       "wine-quality-white              0.6373/0.5517 (-13.4%)   \n",
       "yeast                           0.6307/0.4704 (-25.4%)   \n",
       "zoo                              0.9200/0.9800 (+6.5%)   \n",
       "\n",
       "                                                  WN (old/new)  \n",
       "abalone                                 0.6351/0.5465 (-14.0%)  \n",
       "acute-inflammation                       1.0000/1.0000 (-0.0%)  \n",
       "acute-nephritis                          1.0000/1.0000 (-0.0%)  \n",
       "adult                                    0.8453/0.8492 (+0.5%)  \n",
       "annealing                               0.6500/0.7700 (+18.5%)  \n",
       "arrhythmia                               0.6018/0.5796 (-3.7%)  \n",
       "audiology-std                            0.7200/0.7200 (-0.0%)  \n",
       "balance-scale                            0.9551/0.9295 (-2.7%)  \n",
       "balloons                        0.0000/0.7500 (+7499999900.0%)  \n",
       "bank                                     0.8850/0.8664 (-2.1%)  \n",
       "blood                                    0.7594/0.7086 (-6.7%)  \n",
       "breast-cancer                            0.6197/0.5734 (-7.5%)  \n",
       "breast-cancer-wisc                       0.9657/0.9570 (-0.9%)  \n",
       "breast-cancer-wisc-diag                  0.9718/0.9824 (+1.1%)  \n",
       "breast-cancer-wisc-prog                 0.8367/0.6768 (-19.1%)  \n",
       "breast-tissue                           0.5385/0.6038 (+12.1%)  \n",
       "car                                      0.9769/0.9826 (+0.6%)  \n",
       "cardiotocography-10clases                0.8606/0.7752 (-9.9%)  \n",
       "cardiotocography-3clases                0.8945/0.1383 (-84.5%)  \n",
       "chess-krvk                              0.7673/0.1625 (-78.8%)  \n",
       "chess-krvkp                              0.9912/0.9781 (-1.3%)  \n",
       "congressional-voting                    0.5872/0.4931 (-16.0%)  \n",
       "conn-bench-sonar-mines-rocks            0.8269/0.4615 (-44.2%)  \n",
       "conn-bench-vowel-deterding              0.9524/0.7273 (-23.6%)  \n",
       "connect-4                               0.8833/0.3747 (-57.6%)  \n",
       "contrac                                  0.4755/0.4878 (+2.6%)  \n",
       "credit-approval                         0.9070/0.7884 (-13.1%)  \n",
       "cylinder-bands                           0.7578/0.7305 (-3.6%)  \n",
       "dermatology                              0.9451/0.9727 (+2.9%)  \n",
       "echocardiogram                          0.7879/0.3231 (-59.0%)  \n",
       "ecoli                                    0.8452/0.8571 (+1.4%)  \n",
       "energy-y1                                0.9010/0.9219 (+2.3%)  \n",
       "energy-y2                                0.8906/0.8958 (+0.6%)  \n",
       "fertility                               0.6800/0.8800 (+29.4%)  \n",
       "flags                                   0.4167/0.5670 (+36.1%)  \n",
       "glass                                    0.6792/0.6449 (-5.1%)  \n",
       "haberman-survival                       0.7500/0.6601 (-12.0%)  \n",
       "hayes-roth                              0.5714/0.4643 (-18.7%)  \n",
       "heart-cleveland                          0.5658/0.5960 (+5.3%)  \n",
       "heart-hungarian                         0.7534/0.8299 (+10.2%)  \n",
       "heart-switzerland                       0.2581/0.3115 (+20.7%)  \n",
       "heart-va                                0.2200/0.3100 (+40.9%)  \n",
       "hepatitis                                0.8462/0.7922 (-6.4%)  \n",
       "hill-valley                             0.4934/0.5759 (+16.7%)  \n",
       "horse-colic                             0.7059/0.6029 (-14.6%)  \n",
       "ilpd-indian-liver                        0.6918/0.6392 (-7.6%)  \n",
       "image-segmentation                      0.8938/0.1429 (-84.0%)  \n",
       "ionosphere                               0.9318/0.9029 (-3.1%)  \n",
       "iris                                     1.0000/0.9333 (-6.7%)  \n",
       "led-display                              0.6920/0.6840 (-1.2%)  \n",
       "lenses                                  0.8333/0.1667 (-80.0%)  \n",
       "letter                                  0.9580/0.0401 (-95.8%)  \n",
       "libras                                   0.8000/0.7500 (-6.3%)  \n",
       "low-res-spect                            0.8872/0.8189 (-7.7%)  \n",
       "lung-cancer                             0.5000/0.5625 (+12.5%)  \n",
       "lymphography                             0.7568/0.7162 (-5.4%)  \n",
       "magic                                    0.8690/0.7857 (-9.6%)  \n",
       "mammographic                            0.8292/0.7000 (-15.6%)  \n",
       "miniboone                               0.9272/0.7506 (-19.0%)  \n",
       "molec-biol-promoter                     0.6923/0.6038 (-12.8%)  \n",
       "molec-biol-splice                        0.8494/0.8050 (-5.2%)  \n",
       "monks-1                                 0.5000/0.7292 (+45.8%)  \n",
       "monks-2                                  0.6644/0.6667 (+0.3%)  \n",
       "monks-3                                 0.5231/0.7407 (+41.6%)  \n",
       "mushroom                                0.9995/0.4828 (-51.7%)  \n",
       "musk-1                                   0.8992/0.8487 (-5.6%)  \n",
       "musk-2                                  0.9927/0.1540 (-84.5%)  \n",
       "nursery                                  0.9966/0.9861 (-1.1%)  \n",
       "oocytes_merluccius_nucleus_4d            0.8078/0.7808 (-3.3%)  \n",
       "oocytes_merluccius_states_2f             0.9020/0.8826 (-2.2%)  \n",
       "oocytes_trisopterus_nucleus_2f          0.7939/0.5899 (-25.7%)  \n",
       "oocytes_trisopterus_states_5b            0.9254/0.8947 (-3.3%)  \n",
       "optical                                  0.9638/0.9677 (+0.4%)  \n",
       "ozone                                    0.9748/0.9629 (-1.2%)  \n",
       "page-blocks                             0.9730/0.7876 (-19.0%)  \n",
       "parkinsons                               0.8163/0.7526 (-7.8%)  \n",
       "pendigits                                0.9620/0.9706 (+0.9%)  \n",
       "pima                                     0.6979/0.6693 (-4.1%)  \n",
       "pittsburg-bridges-MATERIAL               0.8077/0.7547 (-6.6%)  \n",
       "pittsburg-bridges-REL-L                  0.6538/0.6667 (+2.0%)  \n",
       "pittsburg-bridges-SPAN                   0.6522/0.6087 (-6.7%)  \n",
       "pittsburg-bridges-T-OR-D                 0.8800/0.8627 (-2.0%)  \n",
       "pittsburg-bridges-TYPE                  0.4615/0.6538 (+41.7%)  \n",
       "planning                                0.6444/0.7143 (+10.8%)  \n",
       "plant-margin                             0.8175/0.7725 (-5.5%)  \n",
       "plant-shape                             0.6575/0.0100 (-98.5%)  \n",
       "plant-texture                            0.8175/0.7960 (-2.6%)  \n",
       "post-operative                          0.5455/0.7111 (+30.4%)  \n",
       "primary-tumor                           0.5000/0.3879 (-22.4%)  \n",
       "ringnorm                                 0.9719/0.9762 (+0.4%)  \n",
       "seeds                                    0.8846/0.8762 (-1.0%)  \n",
       "semeion                                 0.9322/0.0992 (-89.4%)  \n",
       "soybean                                 0.8537/0.4149 (-51.4%)  \n",
       "spambase                                0.9504/0.8109 (-14.7%)  \n",
       "spect                                   0.6398/0.5484 (-14.3%)  \n",
       "spectf                                 0.4545/0.9198 (+102.4%)  \n",
       "statlog-australian-credit                0.6860/0.6783 (-1.1%)  \n",
       "statlog-german-credit                    0.7400/0.7000 (-5.4%)  \n",
       "statlog-heart                            0.8657/0.8667 (+0.1%)  \n",
       "statlog-image                            0.9515/0.9437 (-0.8%)  \n",
       "statlog-landsat                          0.8925/0.8810 (-1.3%)  \n",
       "statlog-shuttle                          0.9988/0.9988 (+0.0%)  \n",
       "statlog-vehicle                          0.8009/0.7565 (-5.5%)  \n",
       "steel-plates                            0.7856/0.6711 (-14.6%)  \n",
       "synthetic-control                       0.9867/0.4467 (-54.7%)  \n",
       "teaching                                 0.3158/0.3200 (+1.3%)  \n",
       "thyroid                                  0.9807/0.9749 (-0.6%)  \n",
       "tic-tac-toe                             0.9707/0.6534 (-32.7%)  \n",
       "titanic                                 0.7818/0.5236 (-33.0%)  \n",
       "trains                                  0.5000/0.6000 (+20.0%)  \n",
       "twonorm                                  0.9730/0.9743 (+0.1%)  \n",
       "vertebral-column-2clases                0.6623/0.8065 (+21.8%)  \n",
       "vertebral-column-3clases                0.7403/0.8452 (+14.2%)  \n",
       "wall-following                          0.9274/0.1514 (-83.7%)  \n",
       "waveform                                 0.8376/0.7972 (-4.8%)  \n",
       "waveform-noise                           0.8640/0.8424 (-2.5%)  \n",
       "wine                                     0.9773/0.9213 (-5.7%)  \n",
       "wine-quality-red                        0.5575/0.4693 (-15.8%)  \n",
       "wine-quality-white                       0.5482/0.4973 (-9.3%)  \n",
       "yeast                                    0.5876/0.5391 (-8.3%)  \n",
       "zoo                                     0.9600/0.8000 (-16.7%)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n",
    "    display(pd.DataFrame.from_dict(overlap).transpose())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: we classify \"big\" and \"small\" datasets as having more or less than 1,000 elements, as in the paper. The paper is unclear on which bucket datasets with exactly 1,000 elements are in, but we should be able to reconstruct it using the fact that there are 75 small and 46 large datasets.\n",
    "\n",
    "We find that datasets with 1,000 examples are considered LARGE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "46"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([expected[e]['N'] >= 1000 for e in expected])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot implement tables S9 and S10 because they involve other algorithms, so we instead reproduce the left side of table 1 for overall (present in the paper) and split between small and large datasets (which is somewhat similar to table S9 and S10)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_rank_table(ranks, should_sub=True):\n",
    "    rank_list = [\n",
    "        (name, np.mean(rank)) for name, rank in ranks.items()\n",
    "    ]\n",
    "    rank_list.sort(key=lambda x: x[1])\n",
    "    subtr = 0\n",
    "    if should_sub:\n",
    "        subtr = len(ranks) / 2.0 - 0.5\n",
    "    rank_list = [\n",
    "        (name, avg_rank - subtr, scipy.stats.wilcoxon(ranks[rank_list[0][0]], ranks[name]).pvalue)\n",
    "        for name, avg_rank in rank_list\n",
    "    ]\n",
    "    rank_list = [\n",
    "        (name, f'{avg_rank:.3f}{\"*\" if pvalue < 0.05 else \"\"}', f'{pvalue:.1e}')\n",
    "        for name, avg_rank, pvalue in rank_list\n",
    "    ]\n",
    "    return pd.DataFrame(rank_list, columns=['algorithm', 'avg. rank', 'p-value']).set_index('algorithm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the combined table 1 like in the paper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table 1, original data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>0.013</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.680*</td>\n",
       "      <td>4.7e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>1.027*</td>\n",
       "      <td>8.3e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>1.080*</td>\n",
       "      <td>2.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>1.093*</td>\n",
       "      <td>3.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>1.133*</td>\n",
       "      <td>9.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>1.973*</td>\n",
       "      <td>1.4e-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           0.013      nan\n",
       "MS           0.680*  4.7e-02\n",
       "HW           1.027*  8.3e-03\n",
       "LN           1.080*  2.9e-03\n",
       "ResNet       1.093*  3.9e-03\n",
       "WN           1.133*  9.1e-04\n",
       "BN           1.973*  1.4e-06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table 1, new data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>-0.027</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>0.173</td>\n",
       "      <td>5.4e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.667</td>\n",
       "      <td>5.5e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>0.733*</td>\n",
       "      <td>1.6e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>1.213*</td>\n",
       "      <td>9.3e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>1.560*</td>\n",
       "      <td>1.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>2.680*</td>\n",
       "      <td>5.0e-11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN          -0.027      nan\n",
       "WN            0.173  5.4e-01\n",
       "MS            0.667  5.5e-02\n",
       "LN           0.733*  1.6e-02\n",
       "HW           1.213*  9.3e-04\n",
       "ResNet       1.560*  1.1e-04\n",
       "BN           2.680*  5.0e-11"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table 1, original data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>0.013</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.680*</td>\n",
       "      <td>4.7e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>1.027*</td>\n",
       "      <td>8.3e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>1.080*</td>\n",
       "      <td>2.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>1.093*</td>\n",
       "      <td>3.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>1.133*</td>\n",
       "      <td>9.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>1.973*</td>\n",
       "      <td>1.4e-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           0.013      nan\n",
       "MS           0.680*  4.7e-02\n",
       "HW           1.027*  8.3e-03\n",
       "LN           1.080*  2.9e-03\n",
       "ResNet       1.093*  3.9e-03\n",
       "WN           1.133*  9.1e-04\n",
       "BN           1.973*  1.4e-06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table 1, new data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>-0.027</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>0.173</td>\n",
       "      <td>5.4e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.667</td>\n",
       "      <td>5.5e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>0.733*</td>\n",
       "      <td>1.6e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>1.213*</td>\n",
       "      <td>9.3e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>1.560*</td>\n",
       "      <td>1.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>2.680*</td>\n",
       "      <td>5.0e-11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN          -0.027      nan\n",
       "WN            0.173  5.4e-01\n",
       "MS            0.667  5.5e-02\n",
       "LN           0.733*  1.6e-02\n",
       "HW           1.213*  9.3e-04\n",
       "ResNet       1.560*  1.1e-04\n",
       "BN           2.680*  5.0e-11"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.warnings.filterwarnings('ignore')\n",
    "\n",
    "print('Table 1, original data')\n",
    "display(gen_rank_table({**old_ranks[True], **old_ranks[False]}))\n",
    "print('Table 1, new data')\n",
    "display(gen_rank_table({**new_ranks[True], **new_ranks[False]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table S9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Small datasets, doesn't include other ml algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S9, original data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>3.013</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>3.680*</td>\n",
       "      <td>4.7e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>4.027*</td>\n",
       "      <td>8.3e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>4.080*</td>\n",
       "      <td>2.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>4.093*</td>\n",
       "      <td>3.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>4.133*</td>\n",
       "      <td>9.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>4.973*</td>\n",
       "      <td>1.4e-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           3.013      nan\n",
       "MS           3.680*  4.7e-02\n",
       "HW           4.027*  8.3e-03\n",
       "LN           4.080*  2.9e-03\n",
       "ResNet       4.093*  3.9e-03\n",
       "WN           4.133*  9.1e-04\n",
       "BN           4.973*  1.4e-06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S9, new data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>2.973</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>3.173</td>\n",
       "      <td>5.4e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>3.667</td>\n",
       "      <td>5.5e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>3.733*</td>\n",
       "      <td>1.6e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>4.213*</td>\n",
       "      <td>9.3e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>4.560*</td>\n",
       "      <td>1.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>5.680*</td>\n",
       "      <td>5.0e-11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           2.973      nan\n",
       "WN            3.173  5.4e-01\n",
       "MS            3.667  5.5e-02\n",
       "LN           3.733*  1.6e-02\n",
       "HW           4.213*  9.3e-04\n",
       "ResNet       4.560*  1.1e-04\n",
       "BN           5.680*  5.0e-11"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S9, original data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>3.013</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>3.680*</td>\n",
       "      <td>4.7e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>4.027*</td>\n",
       "      <td>8.3e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>4.080*</td>\n",
       "      <td>2.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>4.093*</td>\n",
       "      <td>3.9e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>4.133*</td>\n",
       "      <td>9.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>4.973*</td>\n",
       "      <td>1.4e-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           3.013      nan\n",
       "MS           3.680*  4.7e-02\n",
       "HW           4.027*  8.3e-03\n",
       "LN           4.080*  2.9e-03\n",
       "ResNet       4.093*  3.9e-03\n",
       "WN           4.133*  9.1e-04\n",
       "BN           4.973*  1.4e-06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S9, new data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>2.973</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>3.173</td>\n",
       "      <td>5.4e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>3.667</td>\n",
       "      <td>5.5e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>3.733*</td>\n",
       "      <td>1.6e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>4.213*</td>\n",
       "      <td>9.3e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>4.560*</td>\n",
       "      <td>1.1e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>5.680*</td>\n",
       "      <td>5.0e-11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           2.973      nan\n",
       "WN            3.173  5.4e-01\n",
       "MS            3.667  5.5e-02\n",
       "LN           3.733*  1.6e-02\n",
       "HW           4.213*  9.3e-04\n",
       "ResNet       4.560*  1.1e-04\n",
       "BN           5.680*  5.0e-11"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('Table S9, original data')\n",
    "display(gen_rank_table(old_ranks[False], False))\n",
    "print('Table S9, new data')\n",
    "display(gen_rank_table(new_ranks[False], False))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table S10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Large datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S10, original data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>-0.022</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>0.696</td>\n",
       "      <td>1.3e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.891*</td>\n",
       "      <td>3.3e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>1.130*</td>\n",
       "      <td>3.2e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>1.283*</td>\n",
       "      <td>4.4e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>1.348*</td>\n",
       "      <td>2.6e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>1.674*</td>\n",
       "      <td>1.9e-03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN          -0.022      nan\n",
       "LN            0.696  1.3e-01\n",
       "MS           0.891*  3.3e-02\n",
       "ResNet       1.130*  3.2e-03\n",
       "WN           1.283*  4.4e-03\n",
       "HW           1.348*  2.6e-03\n",
       "BN           1.674*  1.9e-03"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S10, new data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>0.348</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.413</td>\n",
       "      <td>9.6e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>0.543</td>\n",
       "      <td>7.2e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>0.630</td>\n",
       "      <td>4.4e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>0.717</td>\n",
       "      <td>4.8e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>2.065*</td>\n",
       "      <td>3.2e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>2.283*</td>\n",
       "      <td>2.8e-05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           0.348      nan\n",
       "MS            0.413  9.6e-01\n",
       "LN            0.543  7.2e-01\n",
       "HW            0.630  4.4e-01\n",
       "WN            0.717  4.8e-01\n",
       "ResNet       2.065*  3.2e-04\n",
       "BN           2.283*  2.8e-05"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S10, original data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>-0.022</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>0.696</td>\n",
       "      <td>1.3e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.891*</td>\n",
       "      <td>3.3e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>1.130*</td>\n",
       "      <td>3.2e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>1.283*</td>\n",
       "      <td>4.4e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>1.348*</td>\n",
       "      <td>2.6e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>1.674*</td>\n",
       "      <td>1.9e-03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN          -0.022      nan\n",
       "LN            0.696  1.3e-01\n",
       "MS           0.891*  3.3e-02\n",
       "ResNet       1.130*  3.2e-03\n",
       "WN           1.283*  4.4e-03\n",
       "HW           1.348*  2.6e-03\n",
       "BN           1.674*  1.9e-03"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table S10, new data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg. rank</th>\n",
       "      <th>p-value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SNN</th>\n",
       "      <td>0.348</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MS</th>\n",
       "      <td>0.413</td>\n",
       "      <td>9.6e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LN</th>\n",
       "      <td>0.543</td>\n",
       "      <td>7.2e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HW</th>\n",
       "      <td>0.630</td>\n",
       "      <td>4.4e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>0.717</td>\n",
       "      <td>4.8e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ResNet</th>\n",
       "      <td>2.065*</td>\n",
       "      <td>3.2e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BN</th>\n",
       "      <td>2.283*</td>\n",
       "      <td>2.8e-05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          avg. rank  p-value\n",
       "algorithm                   \n",
       "SNN           0.348      nan\n",
       "MS            0.413  9.6e-01\n",
       "LN            0.543  7.2e-01\n",
       "HW            0.630  4.4e-01\n",
       "WN            0.717  4.8e-01\n",
       "ResNet       2.065*  3.2e-04\n",
       "BN           2.283*  2.8e-05"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('Table S10, original data')\n",
    "display(gen_rank_table(old_ranks[True]))\n",
    "print('Table S10, new data')\n",
    "display(gen_rank_table(new_ranks[True]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Tox21"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Tox21 dataset evaluates different aspects of toxicity of various compounds. There are a total of 12 different labels, though not all compounds are labeled for all of them. A typical approach is to generate training and validation sets for each of the labels and train separate models for each. This is what I did, and perhaps not what the paper did (this might explain some of the differences we can see).\n",
    "\n",
    "According to the paper:\n",
    "> We used the validation sets of the challenge winners for hyperparameter selection (see Supplementary Section S4) and the challenge test set for performance comparison. We repeated the whole evaluation procedure 5 times to obtain error bars.\n",
    "\n",
    "Computation took a very long time for me, so I didn't have time to run the whole evaluation procedure 5 times. Further, it seems like there were some issues with my reproduction, as I did not get results that were as good as in the paper on the final test set, which is why I used the validation set instead in my final table. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we do a full grid search for each label and each network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def process_label(label, depth, force_cached=False):\n",
    "    X, y = load_tox21(label, 'train', cvfold=-1)\n",
    "    Xtrain, ytrain = load_tox21(label, 'train')\n",
    "    Xval, yval = load_tox21(label, 'val')\n",
    "    \n",
    "    folder = f'tox21_label_{label}'\n",
    "    \n",
    "    print(f'{uci_folder_to_name(folder)}: {len(y)} training samples')\n",
    "    Xtest, ytest = load_tox21(label, 'test')\n",
    "\n",
    "    try:\n",
    "        os.makedirs(f'pickled/uci/{uci_folder_to_name(folder)}')\n",
    "    except Exception:\n",
    "        pass\n",
    "        \n",
    "    perf = {}\n",
    "    for variant in tqdm(random.sample(list(TOX21_HYPER), len(TOX21_HYPER)), folder):\n",
    "        if not force_cached:\n",
    "            cur_name = folder + '::' + variant\n",
    "            bucket = int(hashlib.md5(cur_name.encode('utf8')).hexdigest(), 16) % N_BUCKETS\n",
    "            if bucket != BUCKET:\n",
    "                continue\n",
    "        perf[(variant, depth)] = get_performance(X, y, Xtest, ytest, TOX21_HYPER[variant], folder, variant, force_cached, metric='auc', force_depth=depth)\n",
    "    return perf\n",
    "\n",
    "SHOW_TQDM = True\n",
    "all_tox_perfs = {}\n",
    "for label in tqdm(range(12), 'Labels'):\n",
    "    for depth in tqdm(random.sample(TOX21_HYPER['SNN']['layers'], len(TOX21_HYPER['SNN']['layers'])), 'Depths'):\n",
    "        print(depth)\n",
    "        with torch.cuda.device(1 - CUDA_DEVICE):\n",
    "            all_tox_perfs[(label, depth)] = process_label(label, depth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "final_tox = defaultdict(list)\n",
    "for _, perfs in all_tox_perfs.items():\n",
    "    for k, v in perfs.items():\n",
    "        try:\n",
    "            final_tox[k].append(np.max(v[-2]))\n",
    "        except Exception:\n",
    "            pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "tox_df_dict = {'method': [], 2: [], 3: [], 4: [], 6: [], 8: [], 16: [], 32: []}\n",
    "methods = ['SNN', 'BatchNorm', 'WeightNorm', 'LayerNorm', 'Highway', 'MSRAinit', 'ResNet']\n",
    "\n",
    "for m in methods:\n",
    "    tox_df_dict['method'].append(m)\n",
    "    for k in tox_df_dict.keys():\n",
    "        if k == 'method':\n",
    "            continue\n",
    "        vals = [v * 100 for v in final_tox[(m, k)]]\n",
    "        if m != 'ResNet' or k != 32:\n",
    "            tox_df_dict[k].append(f'{np.mean(vals):.1f} ±{np.std(vals):.1f}')\n",
    "        else:\n",
    "            tox_df_dict[k].append('na')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to display the results. Let's first show the original table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col1 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col2 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col3 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col4 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col5 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col6 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col0 {\n",
       "            font-weight:  bold;\n",
       "        }</style>  \n",
       "<table id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"blank level0\" ></th> \n",
       "        <th class=\"col_heading level0 col0\" >2</th> \n",
       "        <th class=\"col_heading level0 col1\" >3</th> \n",
       "        <th class=\"col_heading level0 col2\" >4</th> \n",
       "        <th class=\"col_heading level0 col3\" >6</th> \n",
       "        <th class=\"col_heading level0 col4\" >8</th> \n",
       "        <th class=\"col_heading level0 col5\" >16</th> \n",
       "        <th class=\"col_heading level0 col6\" >32</th> \n",
       "    </tr>    <tr> \n",
       "        <th class=\"index_name level0\" >method</th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row0\" class=\"row_heading level0 row0\" >SNN</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col0\" class=\"data row0 col0\" >83.7 ± 0.3</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col1\" class=\"data row0 col1\" >84.4 ± 0.5</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col2\" class=\"data row0 col2\" >84.2 ± 0.4</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col3\" class=\"data row0 col3\" >83.9 ± 0.5</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col4\" class=\"data row0 col4\" >84.5 ± 0.2</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col5\" class=\"data row0 col5\" >83.5 ± 0.5</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row0_col6\" class=\"data row0 col6\" >82.5 ± 0.7</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row1\" class=\"row_heading level0 row1\" >Batchnorm</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col0\" class=\"data row1 col0\" >80.0 ± 0.5</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col1\" class=\"data row1 col1\" >79.8 ± 1.6</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col2\" class=\"data row1 col2\" >77.2 ± 1.1</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col3\" class=\"data row1 col3\" >77.0 ± 1.7</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col4\" class=\"data row1 col4\" >75.0 ± 0.9</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col5\" class=\"data row1 col5\" >73.7 ± 2.0</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row1_col6\" class=\"data row1 col6\" >76.0 ± 1.1</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row2\" class=\"row_heading level0 row2\" >WeightNorm</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col0\" class=\"data row2 col0\" >83.7 ± 0.8</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col1\" class=\"data row2 col1\" >82.9 ± 0.8</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col2\" class=\"data row2 col2\" >82.2 ± 0.9</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col3\" class=\"data row2 col3\" >82.5 ± 0.6</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col4\" class=\"data row2 col4\" >81.9 ± 1.2</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col5\" class=\"data row2 col5\" >78.1 ± 1.3</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row2_col6\" class=\"data row2 col6\" >56.6 ± 2.6</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row3\" class=\"row_heading level0 row3\" >LayerNorm</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col0\" class=\"data row3 col0\" >84.3 ± 0.3</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col1\" class=\"data row3 col1\" >84.3 ± 0.5</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col2\" class=\"data row3 col2\" >84.0 ± 0.2</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col3\" class=\"data row3 col3\" >82.5 ± 0.8</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col4\" class=\"data row3 col4\" >80.9 ± 1.8</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col5\" class=\"data row3 col5\" >78.7 ± 2.3</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row3_col6\" class=\"data row3 col6\" >78.8 ± 0.8</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row4\" class=\"row_heading level0 row4\" >Highway</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col0\" class=\"data row4 col0\" >83.3 ± 0.9</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col1\" class=\"data row4 col1\" >83.0 ± 0.5</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col2\" class=\"data row4 col2\" >82.6 ± 0.9</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col3\" class=\"data row4 col3\" >82.4 ± 0.8</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col4\" class=\"data row4 col4\" >80.3 ± 1.4</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col5\" class=\"data row4 col5\" >80.3 ± 2.4</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row4_col6\" class=\"data row4 col6\" >79.6 ± 0.8</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row5\" class=\"row_heading level0 row5\" >MSRAinit</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col0\" class=\"data row5 col0\" >82.7 ± 0.4</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col1\" class=\"data row5 col1\" >81.6 ± 0.9</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col2\" class=\"data row5 col2\" >81.1 ± 1.7</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col3\" class=\"data row5 col3\" >80.6 ± 0.6</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col4\" class=\"data row5 col4\" >80.9 ± 1.1</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col5\" class=\"data row5 col5\" >80.2 ± 1.1</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row5_col6\" class=\"data row5 col6\" >80.4 ± 1.9</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407level0_row6\" class=\"row_heading level0 row6\" >ResNet</th> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col0\" class=\"data row6 col0\" >82.2 ± 1.1</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col1\" class=\"data row6 col1\" >80.0 ± 2.0</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col2\" class=\"data row6 col2\" >80.5 ± 1.2</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col3\" class=\"data row6 col3\" >81.2 ± 0.7</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col4\" class=\"data row6 col4\" >81.8 ± 0.6</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col5\" class=\"data row6 col5\" >81.2 ± 0.6</td> \n",
       "        <td id=\"T_e562e1c6_242e_11e8_b307_24050f9e7407row6_col6\" class=\"data row6 col6\" >na</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2da3f4aeb8>"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def highlight_max(s):\n",
    "    '''\n",
    "    highlight the maximum in a Series yellow.\n",
    "    '''\n",
    "    is_max = s == max(e for e in s if e != 'na')\n",
    "    return ['font-weight: bold' if v else '' for v in is_max]\n",
    "\n",
    "df = pd.read_csv('tox21.csv').set_index('method')\n",
    "df.style.apply(highlight_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned, my reproduction is not very good in this case, and I didn't iterate on it very much simply due to lack of computational time. \n",
    "\n",
    "Below is the table I get using the *validation* set for each category (not the test set). As we can see this is not as good as the original paper and the error bars are much larger (presumably in part because I did not repeat the experiments 5 times). We also notice that although SNNs do dominate for sufficiently large networks, as claimed in the paper, it takes longer for them to become the best type of network, with MSRAinit dominating up to 6 layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_38d51324_242f_11e8_b307_24050f9e7407row0_col4 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_38d51324_242f_11e8_b307_24050f9e7407row0_col5 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_38d51324_242f_11e8_b307_24050f9e7407row0_col6 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_38d51324_242f_11e8_b307_24050f9e7407row5_col0 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_38d51324_242f_11e8_b307_24050f9e7407row5_col1 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_38d51324_242f_11e8_b307_24050f9e7407row5_col2 {\n",
       "            font-weight:  bold;\n",
       "        }    #T_38d51324_242f_11e8_b307_24050f9e7407row5_col3 {\n",
       "            font-weight:  bold;\n",
       "        }</style>  \n",
       "<table id=\"T_38d51324_242f_11e8_b307_24050f9e7407\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"blank level0\" ></th> \n",
       "        <th class=\"col_heading level0 col0\" >2</th> \n",
       "        <th class=\"col_heading level0 col1\" >3</th> \n",
       "        <th class=\"col_heading level0 col2\" >4</th> \n",
       "        <th class=\"col_heading level0 col3\" >6</th> \n",
       "        <th class=\"col_heading level0 col4\" >8</th> \n",
       "        <th class=\"col_heading level0 col5\" >16</th> \n",
       "        <th class=\"col_heading level0 col6\" >32</th> \n",
       "    </tr>    <tr> \n",
       "        <th class=\"index_name level0\" >method</th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "        <th class=\"blank\" ></th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row0\" class=\"row_heading level0 row0\" >SNN</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col0\" class=\"data row0 col0\" >75.6 ±5.3</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col1\" class=\"data row0 col1\" >75.9 ±5.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col2\" class=\"data row0 col2\" >76.1 ±4.9</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col3\" class=\"data row0 col3\" >77.0 ±5.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col4\" class=\"data row0 col4\" >78.9 ±4.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col5\" class=\"data row0 col5\" >81.4 ±3.8</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row0_col6\" class=\"data row0 col6\" >81.9 ±3.8</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row1\" class=\"row_heading level0 row1\" >BatchNorm</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col0\" class=\"data row1 col0\" >76.4 ±5.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col1\" class=\"data row1 col1\" >76.4 ±4.8</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col2\" class=\"data row1 col2\" >75.6 ±5.5</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col3\" class=\"data row1 col3\" >75.6 ±5.6</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col4\" class=\"data row1 col4\" >75.7 ±5.3</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col5\" class=\"data row1 col5\" >73.0 ±5.8</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row1_col6\" class=\"data row1 col6\" >69.1 ±9.3</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row2\" class=\"row_heading level0 row2\" >WeightNorm</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col0\" class=\"data row2 col0\" >76.1 ±5.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col1\" class=\"data row2 col1\" >76.3 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col2\" class=\"data row2 col2\" >76.7 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col3\" class=\"data row2 col3\" >76.8 ±4.5</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col4\" class=\"data row2 col4\" >76.9 ±4.6</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col5\" class=\"data row2 col5\" >76.6 ±4.4</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row2_col6\" class=\"data row2 col6\" >76.0 ±4.7</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row3\" class=\"row_heading level0 row3\" >LayerNorm</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col0\" class=\"data row3 col0\" >76.2 ±4.8</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col1\" class=\"data row3 col1\" >76.1 ±5.2</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col2\" class=\"data row3 col2\" >75.3 ±5.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col3\" class=\"data row3 col3\" >74.6 ±4.8</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col4\" class=\"data row3 col4\" >75.7 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col5\" class=\"data row3 col5\" >74.1 ±5.1</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row3_col6\" class=\"data row3 col6\" >73.4 ±4.8</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row4\" class=\"row_heading level0 row4\" >Highway</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col0\" class=\"data row4 col0\" >77.5 ±5.1</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col1\" class=\"data row4 col1\" >78.8 ±5.1</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col2\" class=\"data row4 col2\" >78.9 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col3\" class=\"data row4 col3\" >76.3 ±4.6</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col4\" class=\"data row4 col4\" >76.4 ±4.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col5\" class=\"data row4 col5\" >72.6 ±8.4</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row4_col6\" class=\"data row4 col6\" >52.0 ±6.8</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row5\" class=\"row_heading level0 row5\" >MSRAinit</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col0\" class=\"data row5 col0\" >81.5 ±4.1</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col1\" class=\"data row5 col1\" >82.9 ±3.9</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col2\" class=\"data row5 col2\" >81.3 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col3\" class=\"data row5 col3\" >79.1 ±5.0</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col4\" class=\"data row5 col4\" >77.4 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col5\" class=\"data row5 col5\" >76.8 ±4.5</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row5_col6\" class=\"data row5 col6\" >75.2 ±4.3</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_38d51324_242f_11e8_b307_24050f9e7407level0_row6\" class=\"row_heading level0 row6\" >ResNet</th> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col0\" class=\"data row6 col0\" >77.5 ±4.6</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col1\" class=\"data row6 col1\" >77.9 ±4.6</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col2\" class=\"data row6 col2\" >77.8 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col3\" class=\"data row6 col3\" >78.0 ±4.6</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col4\" class=\"data row6 col4\" >77.7 ±4.7</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col5\" class=\"data row6 col5\" >77.9 ±4.8</td> \n",
       "        <td id=\"T_38d51324_242f_11e8_b307_24050f9e7407row6_col6\" class=\"data row6 col6\" >na</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2da3f3beb8>"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame.from_dict(tox_df_dict).set_index('method')\n",
    "df.style.apply(highlight_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}