demo/spacy_annotator_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e676eccd-bb3b-45ab-b47e-3ecd935df5bc",
   "metadata": {},
   "source": [
    "## Example notebook\n",
    "In this notebook we show 3 different ways to you the spacy annotator:   \n",
    "\n",
    "    - annotation without spaCy model   \n",
    "    - annotations with spaCy model   \n",
    "    - annotations with spaCy model and entityRuler   \n",
    "    \n",
    "Enjoy :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "extensive-creativity",
   "metadata": {},
   "outputs": [],
   "source": [
    "# python -m pip install -e ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f2facdf-b736-43a9-b6db-925d29c5cc3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pending-joshua",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import spacy_annotator as spa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conditional-dating",
   "metadata": {},
   "source": [
    "## Import data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "underlying-contrary",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\n",
    "    \"text\": [\n",
    "        \"New york is lovely, Milan is nice, but london is amazing!\",\n",
    "        \"Stockholm is too cold. Ingrid Bergman says so.\"\n",
    "    ]})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decimal-phoenix",
   "metadata": {},
   "source": [
    "## Annotation _without_ spaCy model\n",
    "Basic implementation of the spacy annotator. The user input labels and entities manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "oriental-labor",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator = spa.Annotator(labels=[\"GPE\", \"PERSON\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "competent-hometown",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator.instructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "digital-making",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels = annotator.annotate(df=df, col_text=\"text\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dec3d71-96f9-45d4-baea-2094260d8d27",
   "metadata": {},
   "source": [
    "### Inspect output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "green-savage",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "structured-anaheim",
   "metadata": {},
   "source": [
    "## Annotation _with_ spaCy model\n",
    "Use the small, medium, large spaCy model or even transformers to label you data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "favorite-selection",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "desirable-evolution",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hybrid-christian",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator = spa.Annotator(labels=[\"GPE\", \"PERSON\"], model=nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "palestinian-korean",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels = annotator.annotate(df=df, col_text=\"text\", shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b2da8d-3a58-4d8a-b217-8fce7cb3caff",
   "metadata": {},
   "source": [
    "### Inspect output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "respective-founder",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affiliated-coordination",
   "metadata": {},
   "source": [
    "## Annotation _with_ spaCy model _and_ EntityRuler\n",
    "Use a combinations of spaCy models and entity ruler patters to label those entities that even a large model might miss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "brave-netherlands",
   "metadata": {},
   "outputs": [],
   "source": [
    "patterns = [\n",
    "    {\"label\": \"GPE\", \"pattern\": \"london\"}, # this one isn't picked up by \"ner\"\n",
    "    {\"label\": \"GPE\", \"pattern\": \"Stockholm\"},\n",
    "    {\"label\": \"PERSON\", \"pattern\": \"Humphrey Bogart\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "conditional-tolerance",
   "metadata": {},
   "outputs": [],
   "source": [
    "ruler = nlp.add_pipe(\"entity_ruler\", config={\"phrase_matcher_attr\": \"LOWER\"}, before=\"ner\")\n",
    "ruler.add_patterns(patterns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "frequent-deficit",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator = spa.Annotator(labels=[\"GPE\", \"PERSON\"], model=nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "brave-appearance",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_labels = annotator.annotate(df=df, col_text=\"text\", shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f48861-f12e-4dd0-bdae-6dc59d17e136",
   "metadata": {},
   "source": [
    "## Inspect output and save dataframe of annotations to .spacy format for training in Spacy3 pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "confirmed-announcement",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect labels\n",
    "df_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16559caf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to the current working directory with the default name 'annotations.spacy'\n",
    "spacy_annotations = annotator.to_spacy(df_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39948269",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to the current working directory with the specified name or path\n",
    "spacy_annotations = annotator.to_spacy(df_labels, \"spacy_labels.spacy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc59b18f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saves to a specified directory\n",
    "spacy_annotations = annotator.to_spacy(df_labels, \"C:\\pick_your_directory\\spacy_labels.spacy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "requested-devon",
   "metadata": {},
   "outputs": [],
   "source": [
    "#fin"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "fd9bfda413e31d4b70aff7aba05d93ec548db95e8f72d9c6d6b86034e46221e6"
  },
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit ('spacy-annotator': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}