doc/source/examples/cd_text_amazon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Drift detection on Amazon reviews\n",
    "\n",
    "\n",
    "## Methods\n",
    "\n",
    "We illustrate drift detection on text data using the following detectors:\n",
    "\n",
    "- [Maximum Mean Discrepancy (MMD) detector](https://docs.seldon.io/projects/alibi-detect/en/stable/cd/methods/mmddrift.html) using [pre-trained transformers](https://huggingface.co/transformers/) to flag drift in the embedding space.\n",
    "\n",
    "- [Classifier drift detector](https://docs.seldon.io/projects/alibi-detect/en/stable/cd/methods/classifierdrift.html) to detect drift in the input space.\n",
    "\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The *Amazon* dataset contains product reviews with a star rating. We will test whether drift can be detected if the ratings start to drift. For more information, check the [WILDS documentation page](https://wilds.stanford.edu/datasets/#amazon).\n",
    "\n",
    "\n",
    "## Dependencies\n",
    "\n",
    "Besides `alibi-detect`, this example notebook also uses the *Amazon* dataset through the [WILDS](https://wilds.stanford.edu/datasets/) package. WILDS is a curated collection of benchmark datasets that represent distribution shifts faced in the wild and can be installed via `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install wilds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Throughout the notebook we use detectors with both `PyTorch` and `TensorFlow` backends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "def set_seed(seed: int) -> None:\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "seed = 1234\n",
    "set_seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and prepare data\n",
    "\n",
    "We first load the dataset and create reference data, data which should not be rejected under the null of the test (H0) and data which should exhibit drift (H1). The drift is introduced later by specifying a specific star rating for the test instances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "AMAZON_PATH = './data/amazon' # path to save data\n",
    "DOWNLOAD = False  # set to True for first run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "The following cell will download the Amazon dataset (if DOWNLOAD=True). The download size is ~7GB and size on disk is ~7GB.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "from sklearn.model_selection import train_test_split\n",
    "from torch.utils.data import DataLoader, Subset\n",
    "from wilds import get_dataset\n",
    "from wilds.common.data_loaders import get_train_loader\n",
    "\n",
    "ds = get_dataset(dataset='amazon', root_dir=AMAZON_PATH, download=DOWNLOAD)\n",
    "ds_tr = ds.get_subset('train')\n",
    "idx_ref, idx_h0 = train_test_split(np.arange(len(ds_tr)), train_size=.5, random_state=seed, shuffle=True)\n",
    "ds_ref = Subset(ds_tr, idx_ref)\n",
    "ds_h0 = Subset(ds_tr, idx_h0)\n",
    "ds_h1 = ds.get_subset('test')\n",
    "dl = partial(DataLoader, shuffle=True, batch_size=100, collate_fn=ds.collate, num_workers=2)\n",
    "dl_ref, dl_h0, dl_h1 = dl(ds_ref), dl(ds_h0), dl(ds_h1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detect drift\n",
    "\n",
    "### MMD detector on transformer embeddings\n",
    "\n",
    "First we embed instances using a pretrained transformer model and detect data drift using the [MMD detector](https://docs.seldon.io/projects/alibi-detect/en/stable/cd/methods/mmddrift.html) on the embeddings.\n",
    "\n",
    "Helper functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "\n",
    "def update_flat_list(x: List[list]):\n",
    "    return [item for sublist in x for item in sublist]\n",
    "\n",
    "\n",
    "def accumulate_sample(dataloader: DataLoader, sample_size: int, stars: int = None):\n",
    "    \"\"\" Create batches of data from dataloaders. \"\"\"\n",
    "    batch_count, stars_count = 0, 0\n",
    "    x_out, y_out, meta_out = [], [], []\n",
    "    for x, y, meta in dataloader:\n",
    "        y, meta = y.numpy(), meta.numpy()\n",
    "        if isinstance(stars, int):\n",
    "            idx_stars = np.where(y == stars)[0]\n",
    "            y, meta = y[idx_stars], meta[idx_stars]\n",
    "            x = tuple([x[idx] for idx in idx_stars])\n",
    "        n_batch = y.shape[0]\n",
    "        idx = min(sample_size - batch_count, n_batch)\n",
    "        batch_count += n_batch\n",
    "        x_out += [x[:idx]]\n",
    "        y_out += [y[:idx]]\n",
    "        meta_out += [meta[:idx]]\n",
    "        if batch_count >= sample_size:\n",
    "            break\n",
    "    x_out = update_flat_list(x_out)\n",
    "    y_out = np.concatenate(y_out, axis=0)\n",
    "    meta_out = np.concatenate(meta_out, axis=0)\n",
    "    return x_out, y_out, meta_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the transformer embedding preprocessing step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from alibi_detect.cd import MMDDrift\n",
    "from alibi_detect.cd.pytorch import preprocess_drift\n",
    "from alibi_detect.models.pytorch import TransformerEmbedding\n",
    "from functools import partial\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "emb_type = 'hidden_state'  # pooler_output, last_hidden_state or hidden_state\n",
    "# layers to extract hidden states from for the embedding used in drift detection\n",
    "# only relevant for emb_type = 'hidden_state'\n",
    "n_layers = 8\n",
    "layers = [-_ for _ in range(1, n_layers + 1)]\n",
    "max_len = 100  # max length for the tokenizer\n",
    "\n",
    "model_name = 'bert-base-cased'  # a model supported by the transformers library\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "embedding = TransformerEmbedding(model_name, emb_type, layers).to(device).eval()\n",
    "preprocess_fn = partial(preprocess_drift, model=embedding, tokenizer=tokenizer, max_len=max_len, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function which will for a specified number of iterations (`n_sample`):\n",
    "- Configure the `MMDDrift` detector with a new reference data sample\n",
    "- Detect drift on the H0 and H1 splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = ['No!', 'Yes!']\n",
    "\n",
    "\n",
    "def print_preds(preds: dict, preds_name: str) -> None:\n",
    "    print(preds_name)\n",
    "    print('Drift? {}'.format(labels[preds['data']['is_drift']]))\n",
    "    print(f'p-value: {preds[\"data\"][\"p_val\"]:.3f}')\n",
    "    print('') \n",
    "    \n",
    "\n",
    "def make_predictions(ref_size: int, test_size: int, n_sample: int, stars_h1: int = 4) -> None:\n",
    "    \"\"\" Create drift MMD detector, init, sample data and make predictions. \"\"\"\n",
    "    for _ in range(n_sample):\n",
    "        # sample data\n",
    "        x_ref, y_ref, meta_ref = accumulate_sample(dl_ref, ref_size)\n",
    "        x_h0, y_h0, meta_h0 = accumulate_sample(dl_h0, test_size)\n",
    "        x_h1, y_h1, meta_h1 = accumulate_sample(dl_h1, test_size, stars=stars_h1)\n",
    "        # init and run detector\n",
    "        dd = MMDDrift(x_ref, backend='pytorch', p_val=.05, preprocess_fn=preprocess_fn, n_permutations=1000)\n",
    "        preds_h0 = dd.predict(x_h0)\n",
    "        preds_h1 = dd.predict(x_h1)\n",
    "        print_preds(preds_h0, 'H0')\n",
    "        print_preds(preds_h1, 'H1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:alibi_detect.cd.utils:Input shape could not be inferred. If alibi_detect.models.tensorflow.embedding.TransformerEmbedding is used as preprocessing step, a saved detector cannot be reinitialized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H0\n",
      "Drift? No!\n",
      "p-value: 0.205\n",
      "\n",
      "H1\n",
      "Drift? Yes!\n",
      "p-value: 0.000\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:alibi_detect.cd.utils:Input shape could not be inferred. If alibi_detect.models.tensorflow.embedding.TransformerEmbedding is used as preprocessing step, a saved detector cannot be reinitialized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H0\n",
      "Drift? No!\n",
      "p-value: 0.898\n",
      "\n",
      "H1\n",
      "Drift? Yes!\n",
      "p-value: 0.000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "make_predictions(ref_size=1000, test_size=1000, n_sample=2, stars_h1=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classifier drift detector\n",
    "\n",
    "Now we will use the [ClassifierDrift detector](https://docs.seldon.io/projects/alibi-detect/en/stable/cd/methods/classifierdrift.html) which uses a binary classification model to try and distinguish the reference from the test (H0 or H1) data. Drift is then detected on the difference between the prediction distributions on out-of-fold reference vs. test instances using a Kolmogorov-Smirnov 2 sample test on the prediction probabilities or via a binomial test on the binarized predictions. We use a pretrained transformer model but freeze its weights and only train the head which consists of 2 dense layers with a leaky ReLU non-linearity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from transformers import DistilBertModel\n",
    "\n",
    "model_name = 'distilbert-base-uncased'\n",
    "\n",
    "class Classifier(nn.Module):\n",
    "    def __init__(self) -> None:\n",
    "        super().__init__()\n",
    "        self.lm = DistilBertModel.from_pretrained(model_name)\n",
    "        for param in self.lm.parameters():  # freeze language model weights\n",
    "            param.requires_grad = False\n",
    "        self.head = nn.Sequential(nn.Linear(768, 512), nn.LeakyReLU(.1), nn.Linear(512, 2))\n",
    "    \n",
    "    def forward(self, tokens) -> torch.Tensor:\n",
    "        h = self.lm(**tokens).last_hidden_state\n",
    "        h = nn.MaxPool1d(kernel_size=100)(h.permute(0, 2, 1)).squeeze(-1)\n",
    "        return self.head(h)\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = Classifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from alibi_detect.cd import ClassifierDrift\n",
    "from alibi_detect.utils.prediction import tokenize_transformer\n",
    "\n",
    "\n",
    "def make_predictions(model, backend: str, ref_size: int, test_size: int, n_sample: int, stars_h1: int = 4) -> None:\n",
    "    \"\"\" Create drift Classifier detector, init, sample data and make predictions. \"\"\"\n",
    "    \n",
    "    # batch_fn tokenizes each batch of instances of the reference and test set during training\n",
    "    b = 'pt' if backend == 'pytorch' else 'tf'\n",
    "    batch_fn = partial(tokenize_transformer, tokenizer=tokenizer, max_len=max_len, backend=b)\n",
    "    \n",
    "    for _ in range(n_sample):\n",
    "        # sample data\n",
    "        x_ref, y_ref, meta_ref = accumulate_sample(dl_ref, ref_size)\n",
    "        x_h0, y_h0, meta_h0 = accumulate_sample(dl_h0, test_size)\n",
    "        x_h1, y_h1, meta_h1 = accumulate_sample(dl_h1, test_size, stars=stars_h1)\n",
    "        # init and run detector\n",
    "        # since our classifier returns logits, we set preds_type to 'logits'\n",
    "        # n_folds determines the number of folds used for cross-validation, this makes sure all \n",
    "        #   test data is used but only out-of-fold predictions taken into account for the drift detection\n",
    "        #   alternatively we can set train_size to a fraction between 0 and 1 and not apply cross-validation\n",
    "        # epochs specifies how many epochs the classifier will be trained for each sample or fold\n",
    "        # preprocess_batch_fn is applied to each batch of instances and translates the text into tokens\n",
    "        dd = ClassifierDrift(x_ref, model, backend=backend, p_val=.05, preds_type='logits', \n",
    "                             n_folds=3, epochs=2, preprocess_batch_fn=batch_fn, train_size=None)\n",
    "        preds_h0 = dd.predict(x_h0)\n",
    "        preds_h1 = dd.predict(x_h1)\n",
    "        print_preds(preds_h0, 'H0')\n",
    "        print_preds(preds_h1, 'H1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H0\n",
      "Drift? No!\n",
      "p-value: 0.644\n",
      "\n",
      "H1\n",
      "Drift? Yes!\n",
      "p-value: 0.006\n",
      "\n",
      "H0\n",
      "Drift? No!\n",
      "p-value: 0.697\n",
      "\n",
      "H1\n",
      "Drift? Yes!\n",
      "p-value: 0.003\n",
      "\n"
     ]
    }
   ],
   "source": [
    "make_predictions(model, 'pytorch', ref_size=1000, test_size=1000, n_sample=2, stars_h1=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow drift detector\n",
    "\n",
    "We can do the same using TensorFlow instead of PyTorch as backend. We first define the classifier again and then simply run the detector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Dense, LeakyReLU, MaxPool1D\n",
    "from transformers import TFDistilBertModel\n",
    "\n",
    "class ClassifierTF(tf.keras.Model):\n",
    "    def __init__(self) -> None:\n",
    "        super(ClassifierTF, self).__init__()\n",
    "        self.lm = TFDistilBertModel.from_pretrained(model_name)\n",
    "        self.lm.trainable = False  # freeze language model weights\n",
    "        self.head = tf.keras.Sequential([Dense(512), LeakyReLU(alpha=.1), Dense(2)])\n",
    "    \n",
    "    def call(self, tokens) -> tf.Tensor:\n",
    "        h = self.lm(**tokens).last_hidden_state\n",
    "        h = tf.squeeze(MaxPool1D(pool_size=100)(h), axis=1)\n",
    "        return self.head(h)\n",
    "    \n",
    "    @classmethod\n",
    "    def from_config(cls, config):  # not needed for sequential/functional API models\n",
    "        return cls(**config)\n",
    "\n",
    "model = ClassifierTF()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H0\n",
      "Drift? No!\n",
      "p-value: 0.100\n",
      "\n",
      "H1\n",
      "Drift? Yes!\n",
      "p-value: 0.000\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n",
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'activation_13', 'vocab_layer_norm']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H0\n",
      "Drift? No!\n",
      "p-value: 0.589\n",
      "\n",
      "H1\n",
      "Drift? Yes!\n",
      "p-value: 0.002\n",
      "\n"
     ]
    }
   ],
   "source": [
    "make_predictions(model, 'tensorflow', ref_size=1000, test_size=1000, n_sample=2, stars_h1=4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}