notebooks/cybert/cybert_example_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cyBERT: a flexible log parser based on the BERT language model\n",
    "\n",
    "## Authors\n",
    " - Rachel Allen, PhD (NVIDIA)\n",
    " - Bartley Richardson, PhD (NVIDIA)\n",
    " \n",
    "## Table of Contents\n",
    "* Introduction\n",
    "* Generating Labeled Logs\n",
    "* Tokenization\n",
    "* Data Loading\n",
    "* Fine-tuning pretrained BERT\n",
    "* Model Evaluation\n",
    "\n",
    "## Introduction\n",
    "\n",
    "One of the most arduous tasks of any security operation (and equally as time consuming for a data scientist) is ETL and parsing. This notebook illustrates how to train a BERT language model using a toy dataset of just 1000 previously parsed windows event logs as a labeled data. We will fine-tune a pretrained BERT model using the PyTorch interface from the [HuggingFace](https://github.com/huggingface) library with a classification layer for Named Entity Recognition. The HuggingFace library is th emost widely accepted interface for working with BERT. It includes pre-built modifications of BERT for specific tasks. In our case we are using `BertForTokenClassification`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "from pytorch_transformers import BertTokenizer, BertModel, BertForTokenClassification, AdamW\n",
    "from torch.optim import Adam\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "import torch.nn.functional as F\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from seqeval.metrics import classification_report,accuracy_score,f1_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tqdm import tqdm,trange\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Labeled Logs\n",
    "\n",
    "To train our model we begin with a dataframe containing parsed logs and additional `raw` column containing the whole raw log as a string. We will use the column names as our labels. Note that this original version pre-processing is on CPU, future versions will be end-to-end GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "logs_df = pd.read_csv('./data/winevt_sample.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time                                                                              1525108499000\n",
       "eventcode                                                                                  4624\n",
       "uuid                                                       fc2f8767-9917-40ce-b1b2-19ac4ec80e76\n",
       "computername                                                                 laptop-70.wong.com\n",
       "eventtype                                                                                     0\n",
       "keywords                                                                          Audit Success\n",
       "logname                                                                                Security\n",
       "message                                                  An account was successfully logged on.\n",
       "opcode                                                                                     Info\n",
       "recordnumber                                                                         1920306769\n",
       "sourcename                                                 Microsoft Windows security auditing.\n",
       "taskcategory                                                                              Logon\n",
       "type                                                                                Information\n",
       "insert_time                                                              10/04/2018 12:42:12 AM\n",
       "network_information_source_network_address                                         198.49.63.55\n",
       "network_information_workstation_name                                        desktop-tonyfreeman\n",
       "new_logon_account_domain                                                           stephens.biz\n",
       "new_logon_account_name                                                     tonyfreeman@acme.com\n",
       "subject_account_domain                                                              elliott.net\n",
       "subject_account_name                                                                tonyfreeman\n",
       "raw                                           10/04/2018 12:42:12 AM LogName= Security Sourc...\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sample parsed log for training\n",
    "logs_df.loc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'02/28/2019 12:49:04 AM LogName= Security SourceName= Microsoft Windows security auditing. EventCode= 4624 EventType= 0 Type= Information ComputerName= lt-95.melton.com TaskCategory= Logon OpCode= Info RecordNumber= 474033423 Keywords= Audit Success Message= An account was successfully logged on.    Subject:   Account Name:  gonzalespeter   Account Domain:  taylor.com   New Logon:   Account Name:  gonzalespeter@acme.com   Account Domain:  blair.com    Network Information:   Workstation Name:  desktop-gonzalespeter   Network Address:  192.175.54.118'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sample raw log\n",
    "logs_df.raw[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = logs_df.columns.values.tolist()\n",
    "cols.remove('raw')\n",
    "\n",
    "def labeler(row):\n",
    "    raw_split = row['raw'].split()\n",
    "    label_list = ['other'] * len(raw_split) \n",
    "    for col in cols:\n",
    "        if str(row[col]) not in {'','-','None','NaN'}:\n",
    "            sublist = str(row[col]).split()\n",
    "            sublist_len=len(sublist)\n",
    "            match_count = 0\n",
    "            for ind in (i for i,el in enumerate(raw_split) if el==sublist[0]):\n",
    "                if match_count < 1:\n",
    "                    if raw_split[ind:ind+sublist_len]==sublist:\n",
    "                        if label_list[ind:ind+sublist_len] == ['other'] * sublist_len:\n",
    "                            label_list[ind:ind+sublist_len] = [col] * sublist_len\n",
    "                            match_count = 1\n",
    "    return label_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "logs_df['labels'] = logs_df.apply(lambda x : labeler(x), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['insert_time', 'insert_time', 'insert_time', 'other', 'logname', 'other', 'sourcename', 'sourcename', 'sourcename', 'sourcename', 'other', 'eventcode', 'other', 'eventtype', 'other', 'type', 'other', 'computername', 'other', 'taskcategory', 'other', 'opcode', 'other', 'recordnumber', 'other', 'keywords', 'keywords', 'other', 'message', 'message', 'message', 'message', 'message', 'message', 'other', 'other', 'other', 'subject_account_name', 'other', 'other', 'subject_account_domain', 'other', 'other', 'other', 'other', 'new_logon_account_name', 'other', 'other', 'new_logon_account_domain', 'other', 'other', 'other', 'other', 'network_information_workstation_name', 'other', 'other', 'network_information_source_network_address']\n"
     ]
    }
   ],
   "source": [
    "print(logs_df.labels[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "logs_df['logs'] = logs_df['raw'].apply(lambda x: x.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = logs_df['labels'].tolist()\n",
    "logs = logs_df['logs'].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tag/Label list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a set list of all labels(tags) from our dataset, add `X` for wordpiece tokens we will not have tags for and `[PAD]` for logs shorter than the length of the model's embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set of tags\n",
    "tag_values = list(set(x for l in labels for x in l))\n",
    "\n",
    "# add 'X' tag for wordpiece \n",
    "tag_values.append('X')\n",
    "tag_values.append('[PAD]')\n",
    "\n",
    "# Set a dict for mapping id to tag name\n",
    "tag2idx = {t: i for i, t in enumerate(tag_values)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wordpiece tokenization\n",
    "We are using the `bert-base-uncased` tokenizer from the pretrained BERT library from [HuggingFace](https://github.com/huggingface). This tokenizer splits our whitespace separated words further into in dictionary sub-word pieces. The model eventually uses the label from the first piece of a word as it's tag, so we do not care about the model's ability to predict labels for the sub-word pieces. For training, the tag used for these pieces is `X`. To learn more see the [BERT paper](https://arxiv.org/abs/1810.04805)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n",
    "\n",
    "tokenized_texts = []\n",
    "new_labels = []\n",
    "for sentence, tags in zip(logs,labels):\n",
    "    new_tags = []\n",
    "    new_text = []\n",
    "    for word, tag in zip(sentence,tags):\n",
    "        sub_words = tokenizer.wordpiece_tokenizer.tokenize(word.lower())\n",
    "        for count, sub_word in enumerate(sub_words):\n",
    "            if count > 0:\n",
    "                tag = 'X'\n",
    "            new_tags.append(tag)\n",
    "            new_text.append(sub_word)\n",
    "    tokenized_texts.append(new_text)\n",
    "    new_labels.append(new_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model inputs\n",
    "For training our models needs (1) wordpiece tokens as integers padded to the specific length of the model (2) corresponding tags as integers and (3) a binary attention mask that ignores padding. Here we have have used 256 for the model size for each log or log piece. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert string tokens into ints\n",
    "input_ids = [tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts]\n",
    "\n",
    "# pad with input_ids with zeros and labels with [PAD]\n",
    "def pad(l, content, width):\n",
    "    l.extend([content] * (width - len(l)))\n",
    "    return l\n",
    "\n",
    "input_ids = [pad(x, 0, 256) for x in input_ids]\n",
    "\n",
    "new_labels = [pad(x, '[PAD]', 256) for x in new_labels]\n",
    "\n",
    "\n",
    "# attention mask for model to ignore padding\n",
    "attention_masks = [[int(i>0) for i in ii] for ii in input_ids]\n",
    "\n",
    "# convert labels/tags to ints\n",
    "tags = [[tag2idx.get(l) for l in lab] for lab in new_labels]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and testing datasets\n",
    "We split the data into training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr_inputs, val_inputs, tr_tags, val_tags,tr_masks, val_masks = train_test_split(input_ids, tags, attention_masks, random_state=1234, test_size=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Move the datasets to the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr_inputs = torch.tensor(tr_inputs)\n",
    "val_inputs = torch.tensor(val_inputs)\n",
    "tr_tags = torch.tensor(tr_tags)\n",
    "val_tags = torch.tensor(val_tags)\n",
    "tr_masks = torch.tensor(tr_masks)\n",
    "val_masks = torch.tensor(val_masks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create dataloaders to make batches of data ready to feed into the model. Authors recommend batch size of 16 or 32. Here we use a batch size of 32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = TensorDataset(tr_inputs, tr_masks, tr_tags)\n",
    "train_sampler = RandomSampler(train_data)\n",
    "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=32)\n",
    "\n",
    "valid_data = TensorDataset(val_inputs, val_masks, val_tags)\n",
    "valid_sampler = SequentialSampler(valid_data)\n",
    "valid_dataloader = DataLoader(valid_data, sampler=valid_sampler, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-tuning pretrained BERT\n",
    "Download pretrained model from HuggingFace and move to GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = BertForTokenClassification.from_pretrained(\"bert-base-uncased\", num_labels=len(tag2idx))\n",
    "#model to gpu\n",
    "model.cuda();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine tune all parameter layers from the pretrained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "FULL_FINETUNING = True\n",
    "if FULL_FINETUNING:\n",
    "    #fine tune all layer parameters\n",
    "    param_optimizer = list(model.named_parameters())\n",
    "    no_decay = ['bias', 'gamma', 'beta']\n",
    "    optimizer_grouped_parameters = [\n",
    "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
    "         'weight_decay_rate': 0.01},\n",
    "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
    "         'weight_decay_rate': 0.0}\n",
    "    ]\n",
    "else:\n",
    "    # only fine tune classifier parameters\n",
    "    param_optimizer = list(model.classifier.named_parameters()) \n",
    "    optimizer_grouped_parameters = [{\"params\": [p for n, p in param_optimizer]}]\n",
    "optimizer = Adam(optimizer_grouped_parameters, lr=3e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're using an simple measure for accuracy. Total correct predictions over total number of labeled tokens accross logs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def flat_accuracy(preds, labels):\n",
    "    pred_flat = np.argmax(preds, axis=2).flatten()\n",
    "    labels_flat = labels.flatten()\n",
    "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch:   0%|          | 0/2 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loss: 1.1063271555407295\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch:  50%|█████     | 1/2 [00:15<00:15, 15.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.2913314178586006\n",
      "Validation Accuracy: 0.550140380859375\n",
      "F1-Score: 0.8453539528062924\n",
      "Train loss: 0.14663322655291394\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 100%|██████████| 2/2 [00:30<00:00, 15.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.026326983235776424\n",
      "Validation Accuracy: 0.58355712890625\n",
      "F1-Score: 0.9789872096058471\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# using 2 epochs to avoid overfitting- paper recommends 2 to 4 epochs\n",
    "\n",
    "\n",
    "epochs = 2\n",
    "max_grad_norm = 1.0\n",
    "\n",
    "for _ in trange(epochs, desc=\"Epoch\"):\n",
    "    # TRAIN loop\n",
    "    model.train()\n",
    "    tr_loss = 0\n",
    "    nb_tr_examples, nb_tr_steps = 0, 0\n",
    "    for step, batch in enumerate(train_dataloader):\n",
    "        # add batch to gpu\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_input_mask, b_labels = batch\n",
    "        # forward pass\n",
    "        loss, scores = model(b_input_ids, token_type_ids=None,\n",
    "                     attention_mask=b_input_mask, labels=b_labels)\n",
    "        # backward pass\n",
    "        loss.backward()\n",
    "        # track train loss\n",
    "        tr_loss += loss.item()\n",
    "        nb_tr_examples += b_input_ids.size(0)\n",
    "        nb_tr_steps += 1\n",
    "        # gradient clipping\n",
    "        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)\n",
    "        # update parameters\n",
    "        optimizer.step()\n",
    "        model.zero_grad()\n",
    "    # print train loss per epoch\n",
    "    print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n",
    "    # VALIDATION on validation set\n",
    "    model.eval()\n",
    "    eval_loss, eval_accuracy = 0, 0\n",
    "    nb_eval_steps, nb_eval_examples = 0, 0\n",
    "    predictions , true_labels = [], []\n",
    "    for batch in valid_dataloader:\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_input_mask, b_labels = batch\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            tmp_eval_loss, logits = model(b_input_ids, token_type_ids=None,\n",
    "                           attention_mask=b_input_mask, labels=b_labels)\n",
    "        logits = logits.detach().cpu().numpy()\n",
    "        label_ids = b_labels.to('cpu').numpy()\n",
    "        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n",
    "        true_labels.append(label_ids)\n",
    "        \n",
    "        tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
    "        \n",
    "        eval_loss += tmp_eval_loss.mean().item()\n",
    "        eval_accuracy += tmp_eval_accuracy\n",
    "        \n",
    "        nb_eval_examples += b_input_ids.size(0)\n",
    "        nb_eval_steps += 1\n",
    "    eval_loss = eval_loss/nb_eval_steps\n",
    "    print(\"Validation loss: {}\".format(eval_loss))\n",
    "    print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))\n",
    "    pred_tags = [tag_values[p_i] for p in predictions for p_i in p]\n",
    "    valid_tags = [tag_values[l_ii] for l in true_labels for l_i in l for l_ii in l_i]\n",
    "    print(\"F1-Score: {}\".format(f1_score(pred_tags, valid_tags)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "\n",
    "We want to look at our model's performance for individual fields and ignore the predictions for the `X` tag that we used for subword pieces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mapping index to name\n",
    "tag2name={tag2idx[key] : key for key in tag2idx.keys()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1 score: 0.996850\n",
      "Accuracy score: 0.998094\n",
      "                                            precision    recall  f1-score   support\n",
      "\n",
      "                                     other     1.0000    1.0000    1.0000      1696\n",
      "                    subject_account_domain     1.0000    0.9400    0.9691       100\n",
      "                                    opcode     1.0000    1.0000    1.0000       100\n",
      "                              recordnumber     1.0000    1.0000    1.0000       100\n",
      "                                 eventcode     1.0000    1.0000    1.0000       100\n",
      "                    new_logon_account_name     1.0000    1.0000    1.0000       100\n",
      "                              computername     1.0000    1.0000    1.0000       100\n",
      "                              taskcategory     1.0000    1.0000    1.0000       100\n",
      "network_information_source_network_address     1.0000    1.0000    1.0000       100\n",
      "                                   logname     0.9524    1.0000    0.9756       100\n",
      "                                   message     1.0000    1.0000    1.0000       100\n",
      "                               insert_time     1.0000    1.0000    1.0000       100\n",
      "      network_information_workstation_name     1.0000    0.9500    0.9744       100\n",
      "                                sourcename     1.0000    1.0000    1.0000       100\n",
      "                                  keywords     1.0000    1.0000    1.0000        96\n",
      "                  new_logon_account_domain     0.9615    1.0000    0.9804       100\n",
      "                      subject_account_name     0.9804    1.0000    0.9901       100\n",
      "                                      type     1.0000    1.0000    1.0000       100\n",
      "                                 eventtype     1.0000    1.0000    1.0000       100\n",
      "\n",
      "                                 micro avg     0.9968    0.9968    0.9968      3492\n",
      "                                 macro avg     0.9970    0.9968    0.9968      3492\n",
      "\n"
     ]
    }
   ],
   "source": [
    "eval_loss, eval_accuracy = 0, 0\n",
    "nb_eval_steps, nb_eval_examples = 0, 0\n",
    "y_true = []\n",
    "y_pred = []\n",
    "\n",
    "for step, batch in enumerate(valid_dataloader):\n",
    "    batch = tuple(t.to(device) for t in batch)\n",
    "    input_ids, input_mask, label_ids = batch\n",
    "        \n",
    "    with torch.no_grad():\n",
    "        outputs = model(input_ids, token_type_ids=None,\n",
    "        attention_mask=input_mask,)\n",
    "        \n",
    "        # For eval mode, the first result of outputs is logits\n",
    "        logits = outputs[0] \n",
    "        \n",
    "    # Get NER predict result\n",
    "    logits = torch.argmax(F.log_softmax(logits,dim=2),dim=2)\n",
    "    logits = logits.detach().cpu().numpy()\n",
    "    \n",
    "    # Get NER true result\n",
    "    label_ids = label_ids.to('cpu').numpy()\n",
    "    \n",
    "    # Only predict the groud truth, mask=0, will not calculate\n",
    "    input_mask = input_mask.to('cpu').numpy()\n",
    "    \n",
    "    # Compare the valuable predict result\n",
    "    for i,mask in enumerate(input_mask):\n",
    "        # ground truth \n",
    "        temp_1 = []\n",
    "        # Prediction\n",
    "        temp_2 = []\n",
    "        \n",
    "        for j, m in enumerate(mask):\n",
    "            # Mask=0 is PAD, do not compare\n",
    "            if m: # Exclude the X label\n",
    "                if tag2name[label_ids[i][j]] != \"X\" : \n",
    "                    temp_1.append(tag2name[label_ids[i][j]])\n",
    "                    temp_2.append(tag2name[logits[i][j]])\n",
    "            else:\n",
    "                break      \n",
    "        y_true.append(temp_1)\n",
    "        y_pred.append(temp_2)\n",
    "\n",
    "print(\"f1 score: %f\"%(f1_score(y_true, y_pred)))\n",
    "print(\"Accuracy score: %f\"%(accuracy_score(y_true, y_pred)))\n",
    "\n",
    "# Get acc , recall, F1 result report\n",
    "print(classification_report(y_true, y_pred,digits=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even using a small toy dataset, the model performs pretty well!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stay tuned-- All GPU pipeline coming soon!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}