jupyter-notebooks/notebooks/ner-drugs-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e4f4b62d",
   "metadata": {},
   "source": [
    "<h1>Processo de Treinamento do Ner-Drugs-Model</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07f620d0",
   "metadata": {},
   "source": [
    "Etapas no processo de treinamento/utilização do <b>Ner-Drugs-Model</b>:\n",
    "<ul>\n",
    "    <li><b>Dataset</b></li>\n",
    "    <li><b>Pré-processamento</b></li>\n",
    "    <li><b>Config File</b></li>\n",
    "    <li><b>Treinamento</b></li>\n",
    "    <li><b>Implementação</b></li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a09355",
   "metadata": {},
   "source": [
    "<h1>O que é o Ner-Drugs-Model?</h1><br>\n",
    "É um <b>spaCy model</b> (spaCy é uma biblioteca focada em processamento avançado de linguagem natural, ou NLP) treinado de maneira customizada para reconhecer palavras referentes a drogas na língua inglesa, como Oxycodone, Weed e Opium. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e9ae69",
   "metadata": {},
   "source": [
    "<h1>Dataset</h1><br>\n",
    "<p1>O <b>Dataset</b> utilizado no processo de treinamento e na avaliação do <b>Ner-Drugs-Model</b>, foi retirado do <a href=\"https://github.com/explosion/projects/tree/v3/tutorials/ner_drugs/assets\">github</a> dos próprios developers da <b>spaCy</b> , onde esse Dataset é utilizado em um projeto de demonstração, esse dados foram retirados de comentários no reddit.<br>\n",
    "    Dois arquivos foram baixados desse github, <b>drugs_train.jsonl</b>, utilizado para treinamento, e <b>drugs_eval.jsonl</b>, utilizado para avaliação.<br>\n",
    "    Ambos arquivos estão em formato JSONL e cada data point contém o seguinte campos importantes:<br>\n",
    "<ul>\n",
    "    <li><b>text</b>: Contendo a string ou comentário.</li>\n",
    "    <li><b>tokens</b>: Uma lista de dicionários com as palavras tokenizadas.</li>\n",
    "    <li><b>spans</b>: Uma lista de dicionários que marca as entidades encontradas na string.</li>\n",
    "</ul>\n",
    "    Em seguida um exemplo de como é estruturado o Dataset:</p1><br><br>\n",
    "<b><pre>{\n",
    "  \"text\": \"Idk if that Xanax or ur just an ass hole\",\n",
    "  \"tokens\": [\n",
    "    { \"text\": \"Idk\", \"start\": 0, \"end\": 3, \"id\": 0 },\n",
    "    { \"text\": \"if\", \"start\": 4, \"end\": 6, \"id\": 1 },\n",
    "    { \"text\": \"that\", \"start\": 7, \"end\": 11, \"id\": 2 },\n",
    "    { \"text\": \"Xanax\", \"start\": 12, \"end\": 17, \"id\": 3 },\n",
    "    { \"text\": \"or\", \"start\": 18, \"end\": 20, \"id\": 4 },\n",
    "    { \"text\": \"ur\", \"start\": 21, \"end\": 23, \"id\": 5 },\n",
    "    { \"text\": \"just\", \"start\": 24, \"end\": 28, \"id\": 6 },\n",
    "    { \"text\": \"an\", \"start\": 29, \"end\": 31, \"id\": 7 },\n",
    "    { \"text\": \"ass\", \"start\": 32, \"end\": 35, \"id\": 8 },\n",
    "    { \"text\": \"hole\", \"start\": 36, \"end\": 40, \"id\": 9 }\n",
    "  ],\n",
    "  \"spans\": [\n",
    "    {\n",
    "      \"start\": 12,\n",
    "      \"end\": 17,\n",
    "      \"token_start\": 3,\n",
    "      \"token_end\": 3,\n",
    "      \"label\": \"DRUG\"\n",
    "    }\n",
    "  ],\n",
    "  \"_input_hash\": -2128862848,\n",
    "  \"_task_hash\": -334208479,\n",
    "  \"answer\": \"accept\"\n",
    "}</pre></b><br><br>\n",
    "Os <b>Datasets</b> anteriormente mencionados são compostos por centenas de linhas com objetos JSONL seguindo a estrutura descrita acima.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b1897a",
   "metadata": {},
   "source": [
    "<h1>Pré-processamento</h1><br>\n",
    "O <b>Pré-processamento</b> foi feito para converter o formato do <b>Dataset</b> de JSONL para <b>.spacy</b>, que é o formato padrão utilizado pela biblioteca <b>spaCy</b> para treinar os modelos.<br>\n",
    "O Pré-processamento foi feito pelo seguinte código:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3daee62",
   "metadata": {},
   "outputs": [],
   "source": [
    "import typer\n",
    "import srsly\n",
    "from pathlib import Path\n",
    "from spacy.util import get_words_and_spaces\n",
    "from spacy.tokens import Doc, DocBin\n",
    "import spacy\n",
    "\n",
    "\n",
    "def main(\n",
    "    input_path: Path = typer.Argument(..., exists=True, dir_okay=False),\n",
    "    output_path: Path = typer.Argument(..., dir_okay=False),\n",
    "):\n",
    "    nlp = spacy.blank(\"en\")\n",
    "    doc_bin = DocBin(attrs=[\"ENT_IOB\", \"ENT_TYPE\"])\n",
    "    for eg in srsly.read_jsonl(input_path):\n",
    "        if eg[\"answer\"] != \"accept\":\n",
    "            continue\n",
    "        tokens = [token[\"text\"] for token in eg[\"tokens\"]]\n",
    "        words, spaces = get_words_and_spaces(tokens, eg[\"text\"])\n",
    "        doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "        doc.ents = [\n",
    "            doc.char_span(s[\"start\"], s[\"end\"], label=s[\"label\"])\n",
    "            for s in eg.get(\"spans\", [])\n",
    "        ]\n",
    "        doc_bin.add(doc)\n",
    "    doc_bin.to_disk(output_path)\n",
    "    print(f\"Processed {len(doc_bin)} documents: {output_path.name}\")\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    typer.run(main)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f15188",
   "metadata": {},
   "source": [
    "Com o <b>Pré-processamento</b> concluído, foram gerados dois arquivos, <b>drugs_eval.jsonl.spacy</b> e <b>drugs_train.jsonl.spacy</b>."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf5cf50",
   "metadata": {},
   "source": [
    "<h1>Config File</h1><br>\n",
    "Em seguida a próxima etapa é o download do arquivo <b>base_config.cfg</b>, que pode ser obtido nesse <a href=\"https://spacy.io/usage/training\">link</a>, esse arquivo base_config possui as informações necessárias básicas para se gerar um arquivo <b>config.ctg</b> definitivo, o qual será usado para treinar o modelo, mas antes o seguinte comando deve ser executado no terminal para gerar tal arquivo:<br><br>\n",
    "<b>spacy init fill-config configs/base_config.cfg configs/config.ctg</b><br><br>\n",
    "Consequentemente o arquivo <b>config.ctg</b> gerado é estruturado da seguinte forma: <br>\n",
    "<pre>\n",
    "    <b>\n",
    "        [paths]\n",
    "        train = \"./corpus/drugs_train.jsonl.spacy\"\n",
    "        dev = \"./corpus/drugs_eval.jsonl.spacy\"\n",
    "        vectors = null\n",
    "        init_tok2vec = null\n",
    "\n",
    "        [system]\n",
    "        gpu_allocator = null\n",
    "        seed = 0\n",
    "\n",
    "        [nlp]\n",
    "        lang = \"en\"\n",
    "        pipeline = [\"tok2vec\",\"ner\"]\n",
    "        batch_size = 1000\n",
    "        disabled = []\n",
    "        before_creation = null\n",
    "        after_creation = null\n",
    "        after_pipeline_creation = null\n",
    "        tokenizer = {\"@tokenizers\":\"spacy.Tokenizer.v1\"}\n",
    "\n",
    "        [components]\n",
    "\n",
    "        [components.ner]\n",
    "        factory = \"ner\"\n",
    "        incorrect_spans_key = null\n",
    "        moves = null\n",
    "        update_with_oracle_cut_size = 100\n",
    "\n",
    "        [components.ner.model]\n",
    "        @architectures = \"spacy.TransitionBasedParser.v2\"\n",
    "        state_type = \"ner\"\n",
    "        extra_state_tokens = false\n",
    "        hidden_width = 64\n",
    "        maxout_pieces = 2\n",
    "        use_upper = true\n",
    "        nO = null\n",
    "\n",
    "        [components.ner.model.tok2vec]\n",
    "        @architectures = \"spacy.Tok2VecListener.v1\"\n",
    "        width = ${components.tok2vec.model.encode.width}\n",
    "        upstream = \"*\"\n",
    "\n",
    "        [components.tok2vec]\n",
    "        factory = \"tok2vec\"\n",
    "\n",
    "        [components.tok2vec.model]\n",
    "        @architectures = \"spacy.Tok2Vec.v2\"\n",
    "\n",
    "        [components.tok2vec.model.embed]\n",
    "        @architectures = \"spacy.MultiHashEmbed.v2\"\n",
    "        width = ${components.tok2vec.model.encode.width}\n",
    "        attrs = [\"ORTH\",\"SHAPE\"]\n",
    "        rows = [5000,2500]\n",
    "        include_static_vectors = false\n",
    "\n",
    "        [components.tok2vec.model.encode]\n",
    "        @architectures = \"spacy.MaxoutWindowEncoder.v2\"\n",
    "        width = 96\n",
    "        depth = 4\n",
    "        window_size = 1\n",
    "        maxout_pieces = 3\n",
    "\n",
    "        [corpora]\n",
    "\n",
    "        [corpora.dev]\n",
    "        @readers = \"spacy.Corpus.v1\"\n",
    "        path = ${paths.dev}\n",
    "        max_length = 0\n",
    "        gold_preproc = false\n",
    "        limit = 0\n",
    "        augmenter = null\n",
    "\n",
    "        [corpora.train]\n",
    "        @readers = \"spacy.Corpus.v1\"\n",
    "        path = ${paths.train}\n",
    "        max_length = 0\n",
    "        gold_preproc = false\n",
    "        limit = 0\n",
    "        augmenter = null\n",
    "\n",
    "        [training]\n",
    "        dev_corpus = \"corpora.dev\"\n",
    "        train_corpus = \"corpora.train\"\n",
    "        seed = ${system.seed}\n",
    "        gpu_allocator = ${system.gpu_allocator}\n",
    "        dropout = 0.1\n",
    "        accumulate_gradient = 1\n",
    "        patience = 1600\n",
    "        max_epochs = 0\n",
    "        max_steps = 20000\n",
    "        eval_frequency = 200\n",
    "        frozen_components = []\n",
    "        annotating_components = []\n",
    "        before_to_disk = null\n",
    "\n",
    "        [training.batcher]\n",
    "        @batchers = \"spacy.batch_by_words.v1\"\n",
    "        discard_oversize = false\n",
    "        tolerance = 0.2\n",
    "        get_length = null\n",
    "\n",
    "        [training.batcher.size]\n",
    "        @schedules = \"compounding.v1\"\n",
    "        start = 100\n",
    "        stop = 1000\n",
    "        compound = 1.001\n",
    "        t = 0.0\n",
    "\n",
    "        [training.logger]\n",
    "        @loggers = \"spacy.ConsoleLogger.v1\"\n",
    "        progress_bar = false\n",
    "\n",
    "        [training.optimizer]\n",
    "        @optimizers = \"Adam.v1\"\n",
    "        beta1 = 0.9\n",
    "        beta2 = 0.999\n",
    "        L2_is_weight_decay = true\n",
    "        L2 = 0.01\n",
    "        grad_clip = 1.0\n",
    "        use_averages = false\n",
    "        eps = 0.00000001\n",
    "        learn_rate = 0.001\n",
    "\n",
    "        [training.score_weights]\n",
    "        ents_f = 1.0\n",
    "        ents_p = 0.0\n",
    "        ents_r = 0.0\n",
    "        ents_per_type = null\n",
    "\n",
    "        [pretraining]\n",
    "\n",
    "        [initialize]\n",
    "        vectors = ${paths.vectors}\n",
    "        init_tok2vec = ${paths.init_tok2vec}\n",
    "        vocab_data = null\n",
    "        lookups = null\n",
    "        before_init = null\n",
    "        after_init = null\n",
    "\n",
    "        [initialize.components]\n",
    "\n",
    "        [initialize.tokenizer]\n",
    "    \n",
    "</b></pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac38271",
   "metadata": {},
   "source": [
    "<h1>Treinamento</h1><br>\n",
    "Nessa etapa todos os preparativos já foram feitos, só nós resta treinar o modelo, utilizando o seguinte comando:<br><br>\n",
    "<b>spacy train ./configs/config.ctg --output ./output</b><br><br>\n",
    "Dessa maneira o modelo está prontamente treinado e armazenado como <b>model-best</b> na pasta <b>output</b>, definida no comando anterior, e pode ser utilizado por meio do método <b>spacy.load()</b>.<br>\n",
    "Só nós resta transformar tal modelo em uma <b>package</b> para que possa ser utilizado em qualquer python script, por meio do seguinte comando:<br><br>\n",
    "<b>spacy package ./output/model-best ./model --name ner-drugs-model</b><br><br>\n",
    "Agora o modelo pode ser facilmente importado."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ee87854",
   "metadata": {},
   "source": [
    "<h1>Implementação</h1><br>\n",
    "Agora na etapa final, uma pequena demonstração de como funciona, e é implementado, o Ner-Drugs-Model treinado."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b5ad6518",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weed DRUG\n",
      "oxycodone DRUG\n",
      "cocaine DRUG\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load(\"ner-drugs-model\")\n",
    "doc = nlp(\"Tell me more about weed, oxycodone and cocaine.\")\n",
    "for ent in doc.ents:\n",
    "    print(ent.text,ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74cca5b",
   "metadata": {},
   "source": [
    "Como pode ser visto acima o <b>Ner-Drugs-Model</b> reconheceu três <b>entidades</b> referentes a <b>label DRUG</b>.<br><br>\n",
    "Assim concluindo a demonstração e o processo de treinamento."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}