examples/cfrl_adult.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counterfactual with Reinforcement Learning (CFRL) on Adult Census"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is described in [Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning](https://arxiv.org/abs/2106.02597) and can generate counterfactual instances for any black-box model. The usual optimization procedure is transformed into a learnable process allowing to generate batches of counterfactual instances in a single forward pass even for high dimensional data. The training pipeline is model-agnostic and relies only on prediction feedback by querying the black-box model. Furthermore, the method allows target and feature conditioning. \n",
    "\n",
    "**We exemplify the use case for the TensorFlow backend. This means that all models: the autoencoder, the actor and the critic are TensorFlow models. Our implementation supports PyTorch backend as well.**\n",
    "\n",
    "CFRL uses [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/abs/1509.02971) by interleaving a state-action function approximator called critic, with a learning an approximator called actor to predict the optimal action. The method assumes that the critic is differentiable with respect to the action argument, thus allowing to optimize the actor's parameters efficiently through gradient-based methods.\n",
    "\n",
    "The DDPG algorithm requires two separate networks, an actor $\\mu$ and a critic $Q$. Given the encoded representation of the input instance $z = enc(x)$, the model prediction $y_M$, the target prediction\n",
    "$y_T$ and the conditioning vector $c$, the actor outputs the counterfactual’s latent representation $z_{CF} = \\mu(z, y_M, y_T, c)$. The decoder then projects the embedding $z_{CF}$ back to the original input space,\n",
    "followed by optional post-processing.\n",
    "\n",
    "The training step consists of simultaneously optimizing the actor and critic networks. The critic regresses on the reward $R$ determined by the model prediction, while the actor maximizes the critic’s output for the given instance through $L_{max}$. The actor also minimizes two objectives to encourage the generation of sparse, in-distribution counterfactuals. The sparsity loss $L_{sparsity}$ operates on the decoded counterfactual $x_{CF}$ and combines the $L_1$ loss over the standardized numerical features and the $L_0$ loss over the categorical ones. The consistency loss $L_{consist}$ aims to encode the counterfactual $x_{CF}$ back to the same latent representation where it was decoded from and helps to produce in-distribution counterfactual instances. Formally, the actor's loss can be written as:\n",
    "$L_{actor} = L_{max} + \\lambda_{1}L_{sparsity} + \\lambda_{2}L_{consistency}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from copy import deepcopy\n",
    "from typing import List, Tuple, Dict, Callable\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow.keras as keras\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from alibi.explainers import CounterfactualRLTabular, CounterfactualRL\n",
    "from alibi.datasets import fetch_adult\n",
    "from alibi.models.tensorflow.autoencoder import HeAE\n",
    "from alibi.models.tensorflow.actor_critic import Actor, Critic\n",
    "from alibi.models.tensorflow.cfrl_models import ADULTEncoder, ADULTDecoder\n",
    "from alibi.explainers.cfrl_base import Callback\n",
    "from alibi.explainers.backends.cfrl_tabular import get_he_preprocessor, get_statistics, \\\n",
    "    get_conditional_vector, apply_category_mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Adult Census Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch adult dataset\n",
    "adult = fetch_adult()\n",
    "\n",
    "# Separate columns in numerical and categorical.\n",
    "categorical_names = [adult.feature_names[i] for i in adult.category_map.keys()]\n",
    "categorical_ids = list(adult.category_map.keys())\n",
    "\n",
    "numerical_names = [name for i, name in enumerate(adult.feature_names) if i not in adult.category_map.keys()]\n",
    "numerical_ids = [i for i in range(len(adult.feature_names)) if i not in adult.category_map.keys()]\n",
    "\n",
    "# split data into train and test\n",
    "X, Y = adult.data, adult.target\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train black-box classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define numerical standard scaler.\n",
    "num_transf = StandardScaler()\n",
    "\n",
    "# Define categorical one-hot encoder.\n",
    "cat_transf = OneHotEncoder(\n",
    "    categories=[range(len(x)) for x in adult.category_map.values()],\n",
    "    handle_unknown=\"ignore\"\n",
    ")\n",
    "\n",
    "# Define column transformer\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        (\"cat\", cat_transf, categorical_ids),\n",
    "        (\"num\", num_transf, numerical_ids),\n",
    "    ],\n",
    "    sparse_threshold=0\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit preprocessor.\n",
    "preprocessor.fit(X_train)\n",
    "\n",
    "# Preprocess train and test dataset.\n",
    "X_train_ohe = preprocessor.transform(X_train)\n",
    "X_test_ohe = preprocessor.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(max_depth=15, min_samples_split=10, n_estimators=50)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select one of the below classifiers.\n",
    "# clf = XGBClassifier(min_child_weight=0.5, max_depth=3, gamma=0.2)\n",
    "# clf = LogisticRegression(C=10)\n",
    "# clf = DecisionTreeClassifier(max_depth=10, min_samples_split=5)\n",
    "clf = RandomForestClassifier(max_depth=15, min_samples_split=10, n_estimators=50)\n",
    "\n",
    "# Fit the classifier.\n",
    "clf.fit(X_train_ohe, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the predictor (black-box)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've trained the classifier, we can define the black-box model. Note that the output of the black-box is a distribution which can be either a soft-label distribution (probabilities/logits for each class) or a hard-label distribution (one-hot encoding). Internally, CFRL takes the `argmax`. Moreover the output **DOES NOT HAVE TO BE DIFFERENTIABLE**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define prediction function.\n",
    "predictor = lambda x: clf.predict_proba(preprocessor.transform(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.862\n"
     ]
    }
   ],
   "source": [
    "# Compute accuracy.\n",
    "acc = accuracy_score(y_true=Y_test, y_pred=predictor(X_test).argmax(axis=1))\n",
    "print(\"Accuracy: %.3f\" % acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define and train autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of directly modelling the perturbation vector in the potentially high-dimensional input space, we first train an autoencoder. The weights of the encoder are frozen and the actor applies the\n",
    "counterfactual perturbations in the latent space of the encoder. The pre-trained decoder maps the counterfactual embedding back to the input feature space. \n",
    "\n",
    "The autoencoder follows a standard design. The model is composed from two submodules, the encoder and the decoder. The forward pass consists of passing the input to the encoder, obtain the input embedding and pass the embedding through the decoder.\n",
    "\n",
    "```python\n",
    "class HeAE(keras.Model):\n",
    "    def __init__(self, encoder: keras.Model, decoder: keras.Model, **kwargs) -> None:\n",
    "        super().__init__(**kwargs)\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "\n",
    "    def call(self, x: tf.Tensor, **kwargs):\n",
    "        z = self.encoder(x)\n",
    "        x_hat = self.decoder(z)\n",
    "        return x_hat\n",
    "```\n",
    "\n",
    "The heterogeneous variant used in this example uses an additional type checking to ensure that the output of the decoder is a list of tensors.\n",
    "\n",
    "Heterogeneous dataset require special treatment. In this work we modeled the numerical features by normal distributions with constant standard deviation and categorical features by categorical distributions. Due to the choice of feature modeling, some numerical features can end up having different types than the original numerical features. For example, a feature like `Age` having the type of `int` can become a `float` due to the autoencoder reconstruction (e.g., `Age=26 -> Age=26.3`). This behavior can be undesirable. Thus we performed casting when process the output of the autoencoder (decoder component)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define attribute types, required for datatype conversion.\n",
    "feature_types = {\"Age\": int, \"Capital Gain\": int, \"Capital Loss\": int, \"Hours per week\": int}\n",
    "\n",
    "# Define data preprocessor and inverse preprocessor. The invers preprocessor include datatype conversions.\n",
    "heae_preprocessor, heae_inv_preprocessor = get_he_preprocessor(X=X_train,\n",
    "                                                               feature_names=adult.feature_names,\n",
    "                                                               category_map=adult.category_map,\n",
    "                                                               feature_types=feature_types)\n",
    "\n",
    "# Define trainset\n",
    "trainset_input = heae_preprocessor(X_train).astype(np.float32)\n",
    "trainset_outputs = {\n",
    "    \"output_1\": X_train_ohe[:, :len(numerical_ids)]\n",
    "}\n",
    "\n",
    "for i, cat_id in enumerate(categorical_ids):\n",
    "    trainset_outputs.update({\n",
    "        f\"output_{i+2}\": X_train[:, cat_id]\n",
    "    })\n",
    "    \n",
    "trainset = tf.data.Dataset.from_tensor_slices((trainset_input, trainset_outputs))\n",
    "trainset = trainset.shuffle(1024).batch(128, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "203/203 [==============================] - 3s 5ms/step - loss: 1.1433 - output_1_loss: 0.0294 - output_2_loss: 1.3754 - output_3_loss: 1.3523 - output_4_loss: 0.8716 - output_5_loss: 1.7635 - output_6_loss: 1.2288 - output_7_loss: 0.8067 - output_8_loss: 0.4318 - output_9_loss: 1.0806 - output_2_sparse_categorical_accuracy: 0.5897 - output_3_sparse_categorical_accuracy: 0.5329 - output_4_sparse_categorical_accuracy: 0.7434 - output_5_sparse_categorical_accuracy: 0.3613 - output_6_sparse_categorical_accuracy: 0.5369 - output_7_sparse_categorical_accuracy: 0.7659 - output_8_sparse_categorical_accuracy: 0.7948 - output_9_sparse_categorical_accuracy: 0.7540\n",
      "Epoch 2/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.2457 - output_1_loss: 0.0115 - output_2_loss: 0.2465 - output_3_loss: 0.3056 - output_4_loss: 0.1590 - output_5_loss: 0.4081 - output_6_loss: 0.2346 - output_7_loss: 0.1856 - output_8_loss: 0.0526 - output_9_loss: 0.2815 - output_2_sparse_categorical_accuracy: 0.9348 - output_3_sparse_categorical_accuracy: 0.9145 - output_4_sparse_categorical_accuracy: 0.9599 - output_5_sparse_categorical_accuracy: 0.8919 - output_6_sparse_categorical_accuracy: 0.9386 - output_7_sparse_categorical_accuracy: 0.9457 - output_8_sparse_categorical_accuracy: 0.9926 - output_9_sparse_categorical_accuracy: 0.9150\n",
      "Epoch 3/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.1157 - output_1_loss: 0.0080 - output_2_loss: 0.0972 - output_3_loss: 0.1292 - output_4_loss: 0.0829 - output_5_loss: 0.1507 - output_6_loss: 0.0912 - output_7_loss: 0.1000 - output_8_loss: 0.0246 - output_9_loss: 0.1861 - output_2_sparse_categorical_accuracy: 0.9795 - output_3_sparse_categorical_accuracy: 0.9685 - output_4_sparse_categorical_accuracy: 0.9804 - output_5_sparse_categorical_accuracy: 0.9637 - output_6_sparse_categorical_accuracy: 0.9800 - output_7_sparse_categorical_accuracy: 0.9710 - output_8_sparse_categorical_accuracy: 0.9951 - output_9_sparse_categorical_accuracy: 0.9432\n",
      "Epoch 4/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0728 - output_1_loss: 0.0060 - output_2_loss: 0.0533 - output_3_loss: 0.0784 - output_4_loss: 0.0445 - output_5_loss: 0.0890 - output_6_loss: 0.0554 - output_7_loss: 0.0643 - output_8_loss: 0.0157 - output_9_loss: 0.1340 - output_2_sparse_categorical_accuracy: 0.9892 - output_3_sparse_categorical_accuracy: 0.9833 - output_4_sparse_categorical_accuracy: 0.9896 - output_5_sparse_categorical_accuracy: 0.9786 - output_6_sparse_categorical_accuracy: 0.9882 - output_7_sparse_categorical_accuracy: 0.9824 - output_8_sparse_categorical_accuracy: 0.9971 - output_9_sparse_categorical_accuracy: 0.9572\n",
      "Epoch 5/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0530 - output_1_loss: 0.0048 - output_2_loss: 0.0376 - output_3_loss: 0.0578 - output_4_loss: 0.0302 - output_5_loss: 0.0643 - output_6_loss: 0.0397 - output_7_loss: 0.0456 - output_8_loss: 0.0103 - output_9_loss: 0.1003 - output_2_sparse_categorical_accuracy: 0.9922 - output_3_sparse_categorical_accuracy: 0.9872 - output_4_sparse_categorical_accuracy: 0.9932 - output_5_sparse_categorical_accuracy: 0.9843 - output_6_sparse_categorical_accuracy: 0.9913 - output_7_sparse_categorical_accuracy: 0.9873 - output_8_sparse_categorical_accuracy: 0.9986 - output_9_sparse_categorical_accuracy: 0.9686\n",
      "Epoch 6/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0417 - output_1_loss: 0.0040 - output_2_loss: 0.0298 - output_3_loss: 0.0464 - output_4_loss: 0.0226 - output_5_loss: 0.0512 - output_6_loss: 0.0314 - output_7_loss: 0.0358 - output_8_loss: 0.0082 - output_9_loss: 0.0763 - output_2_sparse_categorical_accuracy: 0.9942 - output_3_sparse_categorical_accuracy: 0.9894 - output_4_sparse_categorical_accuracy: 0.9952 - output_5_sparse_categorical_accuracy: 0.9866 - output_6_sparse_categorical_accuracy: 0.9935 - output_7_sparse_categorical_accuracy: 0.9907 - output_8_sparse_categorical_accuracy: 0.9987 - output_9_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 7/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0335 - output_1_loss: 0.0034 - output_2_loss: 0.0232 - output_3_loss: 0.0374 - output_4_loss: 0.0173 - output_5_loss: 0.0423 - output_6_loss: 0.0258 - output_7_loss: 0.0279 - output_8_loss: 0.0062 - output_9_loss: 0.0606 - output_2_sparse_categorical_accuracy: 0.9949 - output_3_sparse_categorical_accuracy: 0.9921 - output_4_sparse_categorical_accuracy: 0.9963 - output_5_sparse_categorical_accuracy: 0.9895 - output_6_sparse_categorical_accuracy: 0.9945 - output_7_sparse_categorical_accuracy: 0.9931 - output_8_sparse_categorical_accuracy: 0.9991 - output_9_sparse_categorical_accuracy: 0.9826\n",
      "Epoch 8/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0284 - output_1_loss: 0.0030 - output_2_loss: 0.0191 - output_3_loss: 0.0313 - output_4_loss: 0.0145 - output_5_loss: 0.0364 - output_6_loss: 0.0227 - output_7_loss: 0.0229 - output_8_loss: 0.0054 - output_9_loss: 0.0510 - output_2_sparse_categorical_accuracy: 0.9967 - output_3_sparse_categorical_accuracy: 0.9928 - output_4_sparse_categorical_accuracy: 0.9975 - output_5_sparse_categorical_accuracy: 0.9908 - output_6_sparse_categorical_accuracy: 0.9950 - output_7_sparse_categorical_accuracy: 0.9946 - output_8_sparse_categorical_accuracy: 0.9992 - output_9_sparse_categorical_accuracy: 0.9860\n",
      "Epoch 9/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0243 - output_1_loss: 0.0027 - output_2_loss: 0.0157 - output_3_loss: 0.0274 - output_4_loss: 0.0125 - output_5_loss: 0.0300 - output_6_loss: 0.0191 - output_7_loss: 0.0197 - output_8_loss: 0.0046 - output_9_loss: 0.0435 - output_2_sparse_categorical_accuracy: 0.9969 - output_3_sparse_categorical_accuracy: 0.9941 - output_4_sparse_categorical_accuracy: 0.9975 - output_5_sparse_categorical_accuracy: 0.9926 - output_6_sparse_categorical_accuracy: 0.9959 - output_7_sparse_categorical_accuracy: 0.9950 - output_8_sparse_categorical_accuracy: 0.9992 - output_9_sparse_categorical_accuracy: 0.9885\n",
      "Epoch 10/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0209 - output_1_loss: 0.0025 - output_2_loss: 0.0137 - output_3_loss: 0.0225 - output_4_loss: 0.0106 - output_5_loss: 0.0253 - output_6_loss: 0.0169 - output_7_loss: 0.0172 - output_8_loss: 0.0042 - output_9_loss: 0.0370 - output_2_sparse_categorical_accuracy: 0.9977 - output_3_sparse_categorical_accuracy: 0.9954 - output_4_sparse_categorical_accuracy: 0.9979 - output_5_sparse_categorical_accuracy: 0.9941 - output_6_sparse_categorical_accuracy: 0.9960 - output_7_sparse_categorical_accuracy: 0.9955 - output_8_sparse_categorical_accuracy: 0.9989 - output_9_sparse_categorical_accuracy: 0.9907\n",
      "Epoch 11/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0183 - output_1_loss: 0.0022 - output_2_loss: 0.0119 - output_3_loss: 0.0204 - output_4_loss: 0.0089 - output_5_loss: 0.0225 - output_6_loss: 0.0146 - output_7_loss: 0.0146 - output_8_loss: 0.0036 - output_9_loss: 0.0325 - output_2_sparse_categorical_accuracy: 0.9979 - output_3_sparse_categorical_accuracy: 0.9958 - output_4_sparse_categorical_accuracy: 0.9983 - output_5_sparse_categorical_accuracy: 0.9942 - output_6_sparse_categorical_accuracy: 0.9967 - output_7_sparse_categorical_accuracy: 0.9963 - output_8_sparse_categorical_accuracy: 0.9992 - output_9_sparse_categorical_accuracy: 0.9919\n",
      "Epoch 12/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0160 - output_1_loss: 0.0020 - output_2_loss: 0.0100 - output_3_loss: 0.0179 - output_4_loss: 0.0081 - output_5_loss: 0.0195 - output_6_loss: 0.0122 - output_7_loss: 0.0129 - output_8_loss: 0.0029 - output_9_loss: 0.0288 - output_2_sparse_categorical_accuracy: 0.9984 - output_3_sparse_categorical_accuracy: 0.9963 - output_4_sparse_categorical_accuracy: 0.9985 - output_5_sparse_categorical_accuracy: 0.9955 - output_6_sparse_categorical_accuracy: 0.9973 - output_7_sparse_categorical_accuracy: 0.9972 - output_8_sparse_categorical_accuracy: 0.9997 - output_9_sparse_categorical_accuracy: 0.9923\n",
      "Epoch 13/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0140 - output_1_loss: 0.0018 - output_2_loss: 0.0085 - output_3_loss: 0.0158 - output_4_loss: 0.0066 - output_5_loss: 0.0168 - output_6_loss: 0.0112 - output_7_loss: 0.0111 - output_8_loss: 0.0029 - output_9_loss: 0.0247 - output_2_sparse_categorical_accuracy: 0.9988 - output_3_sparse_categorical_accuracy: 0.9971 - output_4_sparse_categorical_accuracy: 0.9992 - output_5_sparse_categorical_accuracy: 0.9963 - output_6_sparse_categorical_accuracy: 0.9979 - output_7_sparse_categorical_accuracy: 0.9975 - output_8_sparse_categorical_accuracy: 0.9994 - output_9_sparse_categorical_accuracy: 0.9935\n",
      "Epoch 14/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0125 - output_1_loss: 0.0017 - output_2_loss: 0.0075 - output_3_loss: 0.0139 - output_4_loss: 0.0061 - output_5_loss: 0.0144 - output_6_loss: 0.0100 - output_7_loss: 0.0100 - output_8_loss: 0.0023 - output_9_loss: 0.0226 - output_2_sparse_categorical_accuracy: 0.9990 - output_3_sparse_categorical_accuracy: 0.9976 - output_4_sparse_categorical_accuracy: 0.9991 - output_5_sparse_categorical_accuracy: 0.9966 - output_6_sparse_categorical_accuracy: 0.9983 - output_7_sparse_categorical_accuracy: 0.9978 - output_8_sparse_categorical_accuracy: 0.9998 - output_9_sparse_categorical_accuracy: 0.9939\n",
      "Epoch 15/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0110 - output_1_loss: 0.0015 - output_2_loss: 0.0068 - output_3_loss: 0.0123 - output_4_loss: 0.0053 - output_5_loss: 0.0127 - output_6_loss: 0.0083 - output_7_loss: 0.0091 - output_8_loss: 0.0018 - output_9_loss: 0.0190 - output_2_sparse_categorical_accuracy: 0.9991 - output_3_sparse_categorical_accuracy: 0.9976 - output_4_sparse_categorical_accuracy: 0.9992 - output_5_sparse_categorical_accuracy: 0.9975 - output_6_sparse_categorical_accuracy: 0.9987 - output_7_sparse_categorical_accuracy: 0.9979 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9952\n",
      "Epoch 16/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0097 - output_1_loss: 0.0014 - output_2_loss: 0.0058 - output_3_loss: 0.0105 - output_4_loss: 0.0049 - output_5_loss: 0.0108 - output_6_loss: 0.0071 - output_7_loss: 0.0078 - output_8_loss: 0.0017 - output_9_loss: 0.0175 - output_2_sparse_categorical_accuracy: 0.9994 - output_3_sparse_categorical_accuracy: 0.9981 - output_4_sparse_categorical_accuracy: 0.9993 - output_5_sparse_categorical_accuracy: 0.9980 - output_6_sparse_categorical_accuracy: 0.9989 - output_7_sparse_categorical_accuracy: 0.9981 - output_8_sparse_categorical_accuracy: 0.9998 - output_9_sparse_categorical_accuracy: 0.9962\n",
      "Epoch 17/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0088 - output_1_loss: 0.0013 - output_2_loss: 0.0052 - output_3_loss: 0.0095 - output_4_loss: 0.0041 - output_5_loss: 0.0097 - output_6_loss: 0.0066 - output_7_loss: 0.0071 - output_8_loss: 0.0017 - output_9_loss: 0.0158 - output_2_sparse_categorical_accuracy: 0.9996 - output_3_sparse_categorical_accuracy: 0.9982 - output_4_sparse_categorical_accuracy: 0.9995 - output_5_sparse_categorical_accuracy: 0.9983 - output_6_sparse_categorical_accuracy: 0.9990 - output_7_sparse_categorical_accuracy: 0.9986 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9964\n",
      "Epoch 18/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0079 - output_1_loss: 0.0012 - output_2_loss: 0.0048 - output_3_loss: 0.0088 - output_4_loss: 0.0038 - output_5_loss: 0.0086 - output_6_loss: 0.0056 - output_7_loss: 0.0063 - output_8_loss: 0.0015 - output_9_loss: 0.0141 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9984 - output_4_sparse_categorical_accuracy: 0.9997 - output_5_sparse_categorical_accuracy: 0.9988 - output_6_sparse_categorical_accuracy: 0.9991 - output_7_sparse_categorical_accuracy: 0.9988 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9970\n",
      "Epoch 19/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0077 - output_1_loss: 0.0012 - output_2_loss: 0.0045 - output_3_loss: 0.0090 - output_4_loss: 0.0039 - output_5_loss: 0.0085 - output_6_loss: 0.0053 - output_7_loss: 0.0057 - output_8_loss: 0.0022 - output_9_loss: 0.0129 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9984 - output_4_sparse_categorical_accuracy: 0.9997 - output_5_sparse_categorical_accuracy: 0.9986 - output_6_sparse_categorical_accuracy: 0.9990 - output_7_sparse_categorical_accuracy: 0.9989 - output_8_sparse_categorical_accuracy: 0.9993 - output_9_sparse_categorical_accuracy: 0.9972\n",
      "Epoch 20/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0070 - output_1_loss: 0.0011 - output_2_loss: 0.0040 - output_3_loss: 0.0076 - output_4_loss: 0.0036 - output_5_loss: 0.0074 - output_6_loss: 0.0045 - output_7_loss: 0.0070 - output_8_loss: 0.0012 - output_9_loss: 0.0118 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9984 - output_4_sparse_categorical_accuracy: 0.9997 - output_5_sparse_categorical_accuracy: 0.9989 - output_6_sparse_categorical_accuracy: 0.9996 - output_7_sparse_categorical_accuracy: 0.9984 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9980\n",
      "Epoch 21/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0058 - output_1_loss: 9.9263e-04 - output_2_loss: 0.0033 - output_3_loss: 0.0063 - output_4_loss: 0.0027 - output_5_loss: 0.0061 - output_6_loss: 0.0038 - output_7_loss: 0.0047 - output_8_loss: 9.7739e-04 - output_9_loss: 0.0103 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9987 - output_4_sparse_categorical_accuracy: 0.9998 - output_5_sparse_categorical_accuracy: 0.9992 - output_6_sparse_categorical_accuracy: 0.9997 - output_7_sparse_categorical_accuracy: 0.9990 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9984\n",
      "Epoch 22/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0052 - output_1_loss: 9.5724e-04 - output_2_loss: 0.0029 - output_3_loss: 0.0057 - output_4_loss: 0.0025 - output_5_loss: 0.0056 - output_6_loss: 0.0032 - output_7_loss: 0.0041 - output_8_loss: 0.0010 - output_9_loss: 0.0089 - output_2_sparse_categorical_accuracy: 0.9998 - output_3_sparse_categorical_accuracy: 0.9987 - output_4_sparse_categorical_accuracy: 0.9998 - output_5_sparse_categorical_accuracy: 0.9992 - output_6_sparse_categorical_accuracy: 0.9998 - output_7_sparse_categorical_accuracy: 0.9994 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9987\n",
      "Epoch 23/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0051 - output_1_loss: 9.3460e-04 - output_2_loss: 0.0029 - output_3_loss: 0.0051 - output_4_loss: 0.0023 - output_5_loss: 0.0061 - output_6_loss: 0.0033 - output_7_loss: 0.0039 - output_8_loss: 8.0690e-04 - output_9_loss: 0.0087 - output_2_sparse_categorical_accuracy: 0.9998 - output_3_sparse_categorical_accuracy: 0.9993 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9988 - output_6_sparse_categorical_accuracy: 0.9997 - output_7_sparse_categorical_accuracy: 0.9991 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9987\n",
      "Epoch 24/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0047 - output_1_loss: 8.8066e-04 - output_2_loss: 0.0030 - output_3_loss: 0.0047 - output_4_loss: 0.0027 - output_5_loss: 0.0047 - output_6_loss: 0.0033 - output_7_loss: 0.0036 - output_8_loss: 8.2909e-04 - output_9_loss: 0.0079 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9991 - output_4_sparse_categorical_accuracy: 0.9997 - output_5_sparse_categorical_accuracy: 0.9995 - output_6_sparse_categorical_accuracy: 0.9998 - output_7_sparse_categorical_accuracy: 0.9994 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9989\n",
      "Epoch 25/50\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0043 - output_1_loss: 8.4130e-04 - output_2_loss: 0.0025 - output_3_loss: 0.0046 - output_4_loss: 0.0021 - output_5_loss: 0.0042 - output_6_loss: 0.0025 - output_7_loss: 0.0039 - output_8_loss: 6.7827e-04 - output_9_loss: 0.0071 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9990 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9995 - output_6_sparse_categorical_accuracy: 0.9997 - output_7_sparse_categorical_accuracy: 0.9991 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9989\n",
      "Epoch 26/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0037 - output_1_loss: 7.7661e-04 - output_2_loss: 0.0020 - output_3_loss: 0.0037 - output_4_loss: 0.0017 - output_5_loss: 0.0039 - output_6_loss: 0.0023 - output_7_loss: 0.0029 - output_8_loss: 7.6482e-04 - output_9_loss: 0.0062 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9995 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9996 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 0.9995 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9991\n",
      "Epoch 27/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0037 - output_1_loss: 7.5118e-04 - output_2_loss: 0.0022 - output_3_loss: 0.0043 - output_4_loss: 0.0015 - output_5_loss: 0.0039 - output_6_loss: 0.0024 - output_7_loss: 0.0027 - output_8_loss: 6.4810e-04 - output_9_loss: 0.0059 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9993 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9994 - output_6_sparse_categorical_accuracy: 0.9995 - output_7_sparse_categorical_accuracy: 0.9997 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9992\n",
      "Epoch 28/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0034 - output_1_loss: 7.1342e-04 - output_2_loss: 0.0019 - output_3_loss: 0.0036 - output_4_loss: 0.0016 - output_5_loss: 0.0039 - output_6_loss: 0.0020 - output_7_loss: 0.0024 - output_8_loss: 5.6566e-04 - output_9_loss: 0.0054 - output_2_sparse_categorical_accuracy: 0.9998 - output_3_sparse_categorical_accuracy: 0.9997 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9995 - output_6_sparse_categorical_accuracy: 0.9998 - output_7_sparse_categorical_accuracy: 0.9995 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9991\n",
      "Epoch 29/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0029 - output_1_loss: 6.4941e-04 - output_2_loss: 0.0018 - output_3_loss: 0.0028 - output_4_loss: 0.0014 - output_5_loss: 0.0026 - output_6_loss: 0.0016 - output_7_loss: 0.0022 - output_8_loss: 5.7546e-04 - output_9_loss: 0.0048 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9997 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9992\n",
      "Epoch 30/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0027 - output_1_loss: 6.2401e-04 - output_2_loss: 0.0015 - output_3_loss: 0.0025 - output_4_loss: 0.0013 - output_5_loss: 0.0027 - output_6_loss: 0.0015 - output_7_loss: 0.0019 - output_8_loss: 6.3529e-04 - output_9_loss: 0.0043 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9997 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9998 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 0.9999 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9994\n",
      "Epoch 31/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0026 - output_1_loss: 6.1968e-04 - output_2_loss: 0.0015 - output_3_loss: 0.0024 - output_4_loss: 0.0012 - output_5_loss: 0.0025 - output_6_loss: 0.0015 - output_7_loss: 0.0022 - output_8_loss: 5.2991e-04 - output_9_loss: 0.0043 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9995 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9994\n",
      "Epoch 32/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0030 - output_1_loss: 6.5021e-04 - output_2_loss: 0.0019 - output_3_loss: 0.0026 - output_4_loss: 0.0014 - output_5_loss: 0.0031 - output_6_loss: 0.0017 - output_7_loss: 0.0027 - output_8_loss: 5.7290e-04 - output_9_loss: 0.0050 - output_2_sparse_categorical_accuracy: 0.9998 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9995 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 0.9996 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9992\n",
      "Epoch 33/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0027 - output_1_loss: 6.5071e-04 - output_2_loss: 0.0018 - output_3_loss: 0.0026 - output_4_loss: 0.0013 - output_5_loss: 0.0024 - output_6_loss: 0.0018 - output_7_loss: 0.0020 - output_8_loss: 3.8678e-04 - output_9_loss: 0.0043 - output_2_sparse_categorical_accuracy: 0.9997 - output_3_sparse_categorical_accuracy: 0.9997 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9999 - output_6_sparse_categorical_accuracy: 0.9997 - output_7_sparse_categorical_accuracy: 0.9998 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9993\n",
      "Epoch 34/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0021 - output_1_loss: 5.5956e-04 - output_2_loss: 0.0012 - output_3_loss: 0.0017 - output_4_loss: 8.6624e-04 - output_5_loss: 0.0017 - output_6_loss: 0.0013 - output_7_loss: 0.0016 - output_8_loss: 3.5614e-04 - output_9_loss: 0.0037 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9999 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 0.9999 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 0.9998 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9995\n",
      "Epoch 35/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0019 - output_1_loss: 5.1479e-04 - output_2_loss: 0.0010 - output_3_loss: 0.0018 - output_4_loss: 9.6103e-04 - output_5_loss: 0.0016 - output_6_loss: 0.0010 - output_7_loss: 0.0014 - output_8_loss: 3.0842e-04 - output_9_loss: 0.0029 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9998 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9999 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9998\n",
      "Epoch 36/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0020 - output_1_loss: 5.3038e-04 - output_2_loss: 0.0012 - output_3_loss: 0.0019 - output_4_loss: 9.7982e-04 - output_5_loss: 0.0018 - output_6_loss: 0.0012 - output_7_loss: 0.0015 - output_8_loss: 3.8625e-04 - output_9_loss: 0.0031 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9999 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 0.9998 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9997\n",
      "Epoch 37/50\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0018 - output_1_loss: 4.9262e-04 - output_2_loss: 0.0010 - output_3_loss: 0.0016 - output_4_loss: 8.3019e-04 - output_5_loss: 0.0017 - output_6_loss: 0.0010 - output_7_loss: 0.0012 - output_8_loss: 2.9626e-04 - output_9_loss: 0.0025 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 0.9999 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9999 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9999\n",
      "Epoch 38/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0018 - output_1_loss: 4.9916e-04 - output_2_loss: 9.8594e-04 - output_3_loss: 0.0019 - output_4_loss: 8.7578e-04 - output_5_loss: 0.0014 - output_6_loss: 9.3181e-04 - output_7_loss: 0.0016 - output_8_loss: 3.1634e-04 - output_9_loss: 0.0027 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9997 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9997\n",
      "Epoch 39/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0018 - output_1_loss: 5.1158e-04 - output_2_loss: 0.0012 - output_3_loss: 0.0015 - output_4_loss: 8.4457e-04 - output_5_loss: 0.0016 - output_6_loss: 0.0011 - output_7_loss: 0.0012 - output_8_loss: 4.6680e-04 - output_9_loss: 0.0028 - output_2_sparse_categorical_accuracy: 0.9998 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9999 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 1.0000 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9996\n",
      "Epoch 40/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0019 - output_1_loss: 5.1722e-04 - output_2_loss: 0.0012 - output_3_loss: 0.0020 - output_4_loss: 0.0011 - output_5_loss: 0.0020 - output_6_loss: 0.0012 - output_7_loss: 0.0011 - output_8_loss: 2.2310e-04 - output_9_loss: 0.0025 - output_2_sparse_categorical_accuracy: 0.9998 - output_3_sparse_categorical_accuracy: 0.9998 - output_4_sparse_categorical_accuracy: 0.9998 - output_5_sparse_categorical_accuracy: 0.9997 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 1.0000 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9998\n",
      "Epoch 41/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0014 - output_1_loss: 4.4617e-04 - output_2_loss: 0.0010 - output_3_loss: 0.0013 - output_4_loss: 5.6881e-04 - output_5_loss: 0.0011 - output_6_loss: 6.9290e-04 - output_7_loss: 8.2740e-04 - output_8_loss: 1.9064e-04 - output_9_loss: 0.0020 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9999 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 1.0000 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9999\n",
      "Epoch 42/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0013 - output_1_loss: 4.1261e-04 - output_2_loss: 5.3990e-04 - output_3_loss: 9.4065e-04 - output_4_loss: 6.7139e-04 - output_5_loss: 0.0010 - output_6_loss: 7.6685e-04 - output_7_loss: 8.5373e-04 - output_8_loss: 1.3263e-04 - output_9_loss: 0.0020 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 1.0000 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 1.0000 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9998\n",
      "Epoch 43/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0011 - output_1_loss: 3.5602e-04 - output_2_loss: 5.6818e-04 - output_3_loss: 0.0011 - output_4_loss: 5.1920e-04 - output_5_loss: 7.7666e-04 - output_6_loss: 6.1909e-04 - output_7_loss: 6.1377e-04 - output_8_loss: 1.4408e-04 - output_9_loss: 0.0014 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9999 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 1.0000 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9999\n",
      "Epoch 44/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0011 - output_1_loss: 3.7045e-04 - output_2_loss: 6.0620e-04 - output_3_loss: 8.7592e-04 - output_4_loss: 4.1271e-04 - output_5_loss: 8.4469e-04 - output_6_loss: 5.5344e-04 - output_7_loss: 0.0011 - output_8_loss: 1.3968e-04 - output_9_loss: 0.0014 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 1.0000 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9998 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 1.0000\n",
      "Epoch 45/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0015 - output_1_loss: 4.3320e-04 - output_2_loss: 7.4801e-04 - output_3_loss: 0.0014 - output_4_loss: 0.0012 - output_5_loss: 0.0013 - output_6_loss: 9.3725e-04 - output_7_loss: 9.5972e-04 - output_8_loss: 2.5844e-04 - output_9_loss: 0.0020 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9999 - output_4_sparse_categorical_accuracy: 0.9996 - output_5_sparse_categorical_accuracy: 0.9997 - output_6_sparse_categorical_accuracy: 0.9998 - output_7_sparse_categorical_accuracy: 0.9999 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9998\n",
      "Epoch 46/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0016 - output_1_loss: 5.1220e-04 - output_2_loss: 0.0011 - output_3_loss: 0.0012 - output_4_loss: 7.4547e-04 - output_5_loss: 0.0011 - output_6_loss: 0.0010 - output_7_loss: 9.8855e-04 - output_8_loss: 3.3233e-04 - output_9_loss: 0.0023 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 1.0000 - output_4_sparse_categorical_accuracy: 0.9998 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 0.9999 - output_7_sparse_categorical_accuracy: 0.9999 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9998\n",
      "Epoch 47/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0012 - output_1_loss: 3.9966e-04 - output_2_loss: 5.8024e-04 - output_3_loss: 9.3850e-04 - output_4_loss: 5.1903e-04 - output_5_loss: 0.0011 - output_6_loss: 7.0110e-04 - output_7_loss: 7.8000e-04 - output_8_loss: 4.4502e-04 - output_9_loss: 0.0016 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 1.0000 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9999 - output_8_sparse_categorical_accuracy: 0.9999 - output_9_sparse_categorical_accuracy: 0.9998\n",
      "Epoch 48/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 9.4926e-04 - output_1_loss: 3.5535e-04 - output_2_loss: 5.0522e-04 - output_3_loss: 8.1496e-04 - output_4_loss: 2.8557e-04 - output_5_loss: 6.6599e-04 - output_6_loss: 4.2018e-04 - output_7_loss: 5.8909e-04 - output_8_loss: 1.5880e-04 - output_9_loss: 0.0013 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9999 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 1.0000 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9999\n",
      "Epoch 49/50\n",
      "203/203 [==============================] - 1s 6ms/step - loss: 0.0015 - output_1_loss: 4.0701e-04 - output_2_loss: 8.3141e-04 - output_3_loss: 0.0021 - output_4_loss: 8.7160e-04 - output_5_loss: 0.0016 - output_6_loss: 0.0011 - output_7_loss: 9.3522e-04 - output_8_loss: 1.9702e-04 - output_9_loss: 0.0015 - output_2_sparse_categorical_accuracy: 0.9999 - output_3_sparse_categorical_accuracy: 0.9996 - output_4_sparse_categorical_accuracy: 0.9999 - output_5_sparse_categorical_accuracy: 0.9997 - output_6_sparse_categorical_accuracy: 0.9998 - output_7_sparse_categorical_accuracy: 0.9998 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9998    \n",
      "Epoch 50/50\n",
      "203/203 [==============================] - 1s 5ms/step - loss: 0.0011 - output_1_loss: 3.6270e-04 - output_2_loss: 5.5289e-04 - output_3_loss: 0.0012 - output_4_loss: 2.9493e-04 - output_5_loss: 7.6597e-04 - output_6_loss: 4.7658e-04 - output_7_loss: 0.0010 - output_8_loss: 1.2022e-04 - output_9_loss: 0.0013 - output_2_sparse_categorical_accuracy: 1.0000 - output_3_sparse_categorical_accuracy: 0.9999 - output_4_sparse_categorical_accuracy: 1.0000 - output_5_sparse_categorical_accuracy: 1.0000 - output_6_sparse_categorical_accuracy: 1.0000 - output_7_sparse_categorical_accuracy: 0.9995 - output_8_sparse_categorical_accuracy: 1.0000 - output_9_sparse_categorical_accuracy: 0.9999\n",
      "INFO:tensorflow:Assets written to: tensorflow/ADULT_autoencoder/assets\n"
     ]
    }
   ],
   "source": [
    "# Define autoencoder path and create dir if it doesn't exist.\n",
    "heae_path = os.path.join(\"tensorflow\", \"ADULT_autoencoder\")\n",
    "if not os.path.exists(heae_path):\n",
    "    os.makedirs(heae_path)\n",
    "\n",
    "# Define constants.\n",
    "EPOCHS = 50              # epochs to train the autoencoder\n",
    "HIDDEN_DIM = 128         # hidden dimension of the autoencoder\n",
    "LATENT_DIM = 15          # define latent dimension\n",
    "\n",
    "# Define output dimensions.\n",
    "OUTPUT_DIMS = [len(numerical_ids)]\n",
    "OUTPUT_DIMS += [len(adult.category_map[cat_id]) for cat_id in categorical_ids]\n",
    "\n",
    "# Define the heterogeneous auto-encoder.\n",
    "heae = HeAE(encoder=ADULTEncoder(hidden_dim=HIDDEN_DIM, latent_dim=LATENT_DIM),\n",
    "            decoder=ADULTDecoder(hidden_dim=HIDDEN_DIM, output_dims=OUTPUT_DIMS))\n",
    "\n",
    "# Define loss functions.\n",
    "he_loss = [keras.losses.MeanSquaredError()]\n",
    "he_loss_weights = [1.]\n",
    "\n",
    "# Add categorical losses.\n",
    "for i in range(len(categorical_names)):\n",
    "    he_loss.append(keras.losses.SparseCategoricalCrossentropy(from_logits=True))\n",
    "    he_loss_weights.append(1./len(categorical_names))\n",
    "\n",
    "# Define metrics.\n",
    "metrics = {}\n",
    "for i, cat_name in enumerate(categorical_names):\n",
    "    metrics.update({f\"output_{i+2}\": keras.metrics.SparseCategoricalAccuracy()})\n",
    "    \n",
    "# Compile model.\n",
    "heae.compile(optimizer=keras.optimizers.Adam(learning_rate=1e-3),\n",
    "             loss=he_loss,\n",
    "             loss_weights=he_loss_weights,\n",
    "             metrics=metrics)\n",
    "\n",
    "if len(os.listdir(heae_path)) == 0:\n",
    "    # Fit and save autoencoder.\n",
    "    heae.fit(trainset, epochs=EPOCHS)\n",
    "    heae.save(heae_path, save_format=\"tf\")\n",
    "else:\n",
    "    # Load the model.\n",
    "    heae = keras.models.load_model(heae_path, compile=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counterfactual with Reinforcement Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define constants\n",
    "COEFF_SPARSITY = 0.5               # sparisty coefficient\n",
    "COEFF_CONSISTENCY = 0.5            # consisteny coefficient -> No consistency.\n",
    "TRAIN_STEPS = 20000                # number of training steps -> consider increasing the number of steps\n",
    "BATCH_SIZE = 100                   # batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define dataset specific attributes and constraints\n",
    "\n",
    "A desirable property of a method for generating counterfactuals is to allow feature conditioning. Real-world datasets usually include immutable features such as *gender* or *race*, which should remain unchanged throughout the counterfactual search procedure. Similarly, a numerical feature such as `Age` should only increase for a counterfactual to be actionable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define immutable features.\n",
    "immutable_features = ['Marital Status', 'Relationship', 'Race', 'Sex']\n",
    "\n",
    "# Define ranges. This means that the `Age` feature can not decrease.\n",
    "ranges = {'Age': [0.0, 1.0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define and fit the explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainer = CounterfactualRLTabular(predictor=predictor,\n",
    "                                    encoder=heae.encoder,\n",
    "                                    decoder=heae.decoder,\n",
    "                                    latent_dim=LATENT_DIM,\n",
    "                                    encoder_preprocessor=heae_preprocessor,\n",
    "                                    decoder_inv_preprocessor=heae_inv_preprocessor,\n",
    "                                    coeff_sparsity=COEFF_SPARSITY,\n",
    "                                    coeff_consistency=COEFF_CONSISTENCY,\n",
    "                                    category_map=adult.category_map,\n",
    "                                    feature_names=adult.feature_names,\n",
    "                                    ranges=ranges,\n",
    "                                    immutable_features=immutable_features,\n",
    "                                    train_steps=TRAIN_STEPS,\n",
    "                                    batch_size=BATCH_SIZE,\n",
    "                                    backend=\"tensorflow\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20000/20000 [11:22<00:00, 29.29it/s]\n"
     ]
    }
   ],
   "source": [
    "# Fit the explainer.\n",
    "explainer = explainer.fit(X=X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select some positive examples.\n",
    "X_positive = X_test[np.argmax(predictor(X_test), axis=1) == 1]\n",
    "\n",
    "X = X_positive[:1000]\n",
    "Y_t = np.array([0])\n",
    "C = [{\"Age\": [0, 20], \"Workclass\": [\"State-gov\", \"?\", \"Local-gov\"]}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:00<00:00, 36.32it/s]\n"
     ]
    }
   ],
   "source": [
    "# Generate counterfactual instances.\n",
    "explanation = explainer.explain(X, Y_t, C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concat labels to the original instances.\n",
    "orig = np.concatenate(\n",
    "    [explanation.data['orig']['X'], explanation.data['orig']['class']],\n",
    "    axis=1\n",
    ")\n",
    "\n",
    "# Concat labels to the counterfactual instances.\n",
    "cf = np.concatenate(\n",
    "    [explanation.data['cf']['X'], explanation.data['cf']['class']],\n",
    "    axis=1\n",
    ")\n",
    "\n",
    "# Define new feature names and category map by including the label.\n",
    "feature_names = adult.feature_names + [\"Label\"]\n",
    "category_map = deepcopy(adult.category_map)\n",
    "category_map.update({feature_names.index(\"Label\"): adult.target_names})\n",
    "\n",
    "# Replace label encodings with strings.\n",
    "orig_pd = pd.DataFrame(\n",
    "    apply_category_mapping(orig, category_map),\n",
    "    columns=feature_names\n",
    ")\n",
    "\n",
    "cf_pd = pd.DataFrame(\n",
    "    apply_category_mapping(cf, category_map),\n",
    "    columns=feature_names\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Workclass</th>\n",
       "      <th>Education</th>\n",
       "      <th>Marital Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital Gain</th>\n",
       "      <th>Capital Loss</th>\n",
       "      <th>Hours per week</th>\n",
       "      <th>Country</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>Private</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Blue-Collar</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>7298</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>35</td>\n",
       "      <td>Private</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>White-Collar</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>7688</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Masters</td>\n",
       "      <td>Married</td>\n",
       "      <td>Professional</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>5178</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44</td>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>44</td>\n",
       "      <td>Federal-gov</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Admin</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Age     Workclass         Education Marital Status    Occupation  \\\n",
       "0  60       Private  High School grad        Married   Blue-Collar   \n",
       "1  35       Private  High School grad        Married  White-Collar   \n",
       "2  39     State-gov           Masters        Married  Professional   \n",
       "3  44  Self-emp-inc  High School grad        Married         Sales   \n",
       "4  44   Federal-gov  High School grad        Married         Admin   \n",
       "\n",
       "  Relationship   Race     Sex Capital Gain Capital Loss Hours per week  \\\n",
       "0      Husband  White    Male         7298            0             40   \n",
       "1      Husband  White    Male         7688            0             50   \n",
       "2         Wife  White  Female         5178            0             38   \n",
       "3      Husband  White    Male            0            0             50   \n",
       "4      Husband  White    Male            0            0             40   \n",
       "\n",
       "         Country Label  \n",
       "0  United-States  >50K  \n",
       "1  United-States  >50K  \n",
       "2  United-States  >50K  \n",
       "3  United-States  >50K  \n",
       "4  United-States  >50K  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orig_pd.head(n=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Workclass</th>\n",
       "      <th>Education</th>\n",
       "      <th>Marital Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital Gain</th>\n",
       "      <th>Capital Loss</th>\n",
       "      <th>Hours per week</th>\n",
       "      <th>Country</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>Private</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Blue-Collar</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>983</td>\n",
       "      <td>86</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Service</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>1027</td>\n",
       "      <td>89</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Service</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>860</td>\n",
       "      <td>94</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44</td>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Service</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>1080</td>\n",
       "      <td>85</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>44</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>Admin</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>988</td>\n",
       "      <td>89</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Age     Workclass         Education Marital Status   Occupation  \\\n",
       "0  60       Private  High School grad        Married  Blue-Collar   \n",
       "1  38       Private  High School grad        Married      Service   \n",
       "2  40     State-gov  High School grad        Married      Service   \n",
       "3  44  Self-emp-inc  High School grad        Married      Service   \n",
       "4  44     State-gov  High School grad        Married        Admin   \n",
       "\n",
       "  Relationship   Race     Sex Capital Gain Capital Loss Hours per week  \\\n",
       "0      Husband  White    Male          983           86             40   \n",
       "1      Husband  White    Male         1027           89             40   \n",
       "2         Wife  White  Female          860           94             40   \n",
       "3      Husband  White    Male         1080           85             40   \n",
       "4      Husband  White    Male          988           89             40   \n",
       "\n",
       "         Country  Label  \n",
       "0  United-States  <=50K  \n",
       "1  United-States  <=50K  \n",
       "2  United-States  <=50K  \n",
       "3  United-States  <=50K  \n",
       "4  United-States  <=50K  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_pd.head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Diversity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "13it [00:00, 31.34it/s]\n"
     ]
    }
   ],
   "source": [
    "# Generate counterfactual instances.\n",
    "X = X_positive[2].reshape(1, -1)\n",
    "explanation = explainer.explain(X=X, Y_t=Y_t, C=C, diversity=True, num_samples=100, batch_size=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concat label column.\n",
    "orig = np.concatenate(\n",
    "    [explanation.data['orig']['X'], explanation.data['orig']['class']],\n",
    "    axis=1\n",
    ")\n",
    "\n",
    "cf = np.concatenate(\n",
    "    [explanation.data['cf']['X'], explanation.data['cf']['class']],\n",
    "    axis=1\n",
    ")\n",
    "\n",
    "# Transfrom label encodings to string.\n",
    "orig_pd = pd.DataFrame(\n",
    "    apply_category_mapping(orig, category_map),\n",
    "    columns=feature_names,\n",
    ")\n",
    "\n",
    "cf_pd = pd.DataFrame(\n",
    "    apply_category_mapping(cf, category_map),\n",
    "    columns=feature_names,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Workclass</th>\n",
       "      <th>Education</th>\n",
       "      <th>Marital Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital Gain</th>\n",
       "      <th>Capital Loss</th>\n",
       "      <th>Hours per week</th>\n",
       "      <th>Country</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Masters</td>\n",
       "      <td>Married</td>\n",
       "      <td>Professional</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>5178</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Age  Workclass Education Marital Status    Occupation Relationship   Race  \\\n",
       "0  39  State-gov   Masters        Married  Professional         Wife  White   \n",
       "\n",
       "      Sex Capital Gain Capital Loss Hours per week        Country Label  \n",
       "0  Female         5178            0             38  United-States  >50K  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orig_pd.head(n=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Workclass</th>\n",
       "      <th>Education</th>\n",
       "      <th>Marital Status</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Relationship</th>\n",
       "      <th>Race</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Capital Gain</th>\n",
       "      <th>Capital Loss</th>\n",
       "      <th>Hours per week</th>\n",
       "      <th>Country</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>38</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Dropout</td>\n",
       "      <td>Married</td>\n",
       "      <td>?</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>916</td>\n",
       "      <td>88</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Dropout</td>\n",
       "      <td>Married</td>\n",
       "      <td>?</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>4656</td>\n",
       "      <td>90</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Dropout</td>\n",
       "      <td>Married</td>\n",
       "      <td>Admin</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>1014</td>\n",
       "      <td>89</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>38</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Dropout</td>\n",
       "      <td>Married</td>\n",
       "      <td>Other</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>737</td>\n",
       "      <td>92</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>High School grad</td>\n",
       "      <td>Married</td>\n",
       "      <td>?</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>1031</td>\n",
       "      <td>87</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Age  Workclass         Education Marital Status Occupation Relationship  \\\n",
       "0  38  State-gov           Dropout        Married          ?         Wife   \n",
       "1  38  State-gov           Dropout        Married          ?         Wife   \n",
       "2  38  State-gov           Dropout        Married      Admin         Wife   \n",
       "3  38  State-gov           Dropout        Married      Other         Wife   \n",
       "4  38  State-gov  High School grad        Married          ?         Wife   \n",
       "\n",
       "    Race     Sex Capital Gain Capital Loss Hours per week        Country  \\\n",
       "0  White  Female          916           88             40  United-States   \n",
       "1  White  Female         4656           90             40  United-States   \n",
       "2  White  Female         1014           89             40  United-States   \n",
       "3  White  Female          737           92             40  United-States   \n",
       "4  White  Female         1031           87             40  United-States   \n",
       "\n",
       "   Label  \n",
       "0  <=50K  \n",
       "1  <=50K  \n",
       "2  <=50K  \n",
       "3  <=50K  \n",
       "4  <=50K  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_pd.head(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logging is clearly important when dealing with deep learning models. Thus, we provide an interface to write custom callbacks for logging purposes after each training step which we defined [here](../api/alibi.explainers.cfrl_base.rst#alibi.explainers.cfrl_base.Callback). In the following cells we provide some example to log in **Weights and Biases**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logging reward callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RewardCallback(Callback):\n",
    "    def __call__(self,\n",
    "                 step: int, \n",
    "                 update: int, \n",
    "                 model: CounterfactualRL,\n",
    "                 sample: Dict[str, np.ndarray],\n",
    "                 losses: Dict[str, float]):\n",
    "        \n",
    "        if (step + update) % 100 != 0:\n",
    "            return\n",
    "        \n",
    "        # get the counterfactual and target\n",
    "        Y_t = sample[\"Y_t\"]\n",
    "        X_cf = model.params[\"decoder_inv_preprocessor\"](sample[\"X_cf\"])\n",
    "        \n",
    "        # get prediction label\n",
    "        Y_m_cf = predictor(X_cf)\n",
    "        \n",
    "        # compute reward\n",
    "        reward = np.mean(model.params[\"reward_func\"](Y_m_cf, Y_t))\n",
    "        wandb.log({\"reward\": reward})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logging losses callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LossCallback(Callback):\n",
    "    def __call__(self,\n",
    "                 step: int, \n",
    "                 update: int, \n",
    "                 model: CounterfactualRL,\n",
    "                 sample: Dict[str, np.ndarray],\n",
    "                 losses: Dict[str, float]):\n",
    "        # Log training losses.\n",
    "        if (step + update) % 100 == 0:\n",
    "            wandb.log(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logging tables callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TablesCallback(Callback):\n",
    "    def __call__(self,\n",
    "                 step: int, \n",
    "                 update: int, \n",
    "                 model: CounterfactualRL,\n",
    "                 sample: Dict[str, np.ndarray],\n",
    "                 losses: Dict[str, float]):\n",
    "        # Log every 1000 steps\n",
    "        if step % 1000 != 0:\n",
    "            return\n",
    "        \n",
    "        # Define number of samples to be displayed.\n",
    "        NUM_SAMPLES = 5\n",
    "        \n",
    "        X = heae_inv_preprocessor(sample[\"X\"][:NUM_SAMPLES])        # input instance\n",
    "        X_cf = heae_inv_preprocessor(sample[\"X_cf\"][:NUM_SAMPLES])  # counterfactual\n",
    "        \n",
    "        Y_m = np.argmax(sample[\"Y_m\"][:NUM_SAMPLES], axis=1).astype(int).reshape(-1, 1) # input labels\n",
    "        Y_t = np.argmax(sample[\"Y_t\"][:NUM_SAMPLES], axis=1).astype(int).reshape(-1, 1) # target labels\n",
    "        Y_m_cf = np.argmax(predictor(X_cf), axis=1).astype(int).reshape(-1, 1)          # counterfactual labels\n",
    "        \n",
    "        # Define feature names and category map for input.\n",
    "        feature_names = adult.feature_names + [\"Label\"]\n",
    "        category_map = deepcopy(adult.category_map)\n",
    "        category_map.update({feature_names.index(\"Label\"): adult.target_names})\n",
    "        \n",
    "        # Construct input array.\n",
    "        inputs = np.concatenate([X, Y_m], axis=1)\n",
    "        inputs = pd.DataFrame(apply_category_mapping(inputs, category_map),\n",
    "                              columns=feature_names)\n",
    "        \n",
    "        # Define feature names and category map for counterfactual output.\n",
    "        feature_names += [\"Target\"]\n",
    "        category_map.update({feature_names.index(\"Target\"): adult.target_names})\n",
    "        \n",
    "        # Construct output array.\n",
    "        outputs = np.concatenate([X_cf, Y_m_cf, Y_t], axis=1)\n",
    "        outputs = pd.DataFrame(apply_category_mapping(outputs, category_map),\n",
    "                               columns=feature_names)\n",
    "        \n",
    "        # Log table.\n",
    "        wandb.log({\n",
    "            \"Input\": wandb.Table(dataframe=inputs),\n",
    "            \"Output\": wandb.Table(dataframe=outputs)\n",
    "        })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having defined the callbacks, we can define a new explainer that will include logging.\n",
    "\n",
    "```python\n",
    "import wandb\n",
    "\n",
    "# Initialize wandb.\n",
    "wandb_project = \"Adult Census Counterfactual with Reinforcement Learning\"\n",
    "wandb.init(project=wandb_project)\n",
    "\n",
    "# Define explainer as before and include callbacks.\n",
    "explainer = CounterfactualRLTabular(...,\n",
    "                                    callbacks=[LossCallback(), RewardCallback(), TablesCallback()])\n",
    "\n",
    "# Fit the explainers.\n",
    "explainer = explainer.fit(X=X_train)\n",
    "\n",
    "# Close wandb.\n",
    "wandb.finish()\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}