docs/notebooks/getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "56249f9f-e9e8-482a-bbbe-516fa48a02a5",
   "metadata": {},
   "source": [
    "# NoOffense Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd900070-723b-453b-a7a3-d9a338911cec",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Installation\n",
    "\n",
    "NoOffense is a package that provides a smart and fast tool for detecting offensive language in Turkish text. It utilizes transformers and is largely based on the Hugging Face library, making it fully compatible with it.\n",
    "\n",
    "To install this package, simply run the following command in your Python environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d0e365f-3856-4684-9f02-6bb9a0095ae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/ertugrul-dmr/NoOffense.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "688b1b3f-b5cc-4104-b55b-f889be3b6b9b",
   "metadata": {},
   "source": [
    "This command will install the package directly from the GitHub repository.\n",
    "\n",
    "**PyTorch with CUDA**\n",
    "\n",
    "If you wish to use a GPU / CUDA, you must install PyTorch with the matching CUDA Version."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66919cab-d1dd-488f-9834-1df7c65ac4d8",
   "metadata": {},
   "source": [
    "## Getting Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bb26859-6238-477b-b780-f704a97a2a62",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from noofense import Predictor\n",
    "\n",
    "# Predictor applies some text cleaning&preps internally\n",
    "# In case of multiple trained models use list of model paths,\n",
    "# and NoOffense will yield ensemble results internally\n",
    "# you can use gpu support by setting device for faster inference\n",
    "\n",
    "clf = Predictor(model_path='path_to_your_model_weight_folder', device='cuda')\n",
    "predictions = clf.predict(['sen ne gerizekalısın ya', 'ne demek istedin anlamadım'], progress_bar=False)\n",
    "\n",
    "# Code above will return dict contains these keys:\n",
    "# ['probas']: probabilities of each class\n",
    "# ['predictions']: label encoding for each predicted class\n",
    "# ['predicted_labels']: str label mappings for each class, INSULT, Racist etc.\n",
    "\n",
    "print(predictions['predicted_labels'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644081e4-95d5-4f76-a341-4eae3bf5c56c",
   "metadata": {},
   "source": [
    "NoOffense also supports making predictions on Pandas dataframes:\n",
    "\n",
    "It expects a column named \"text\" to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4be0923e-f1b8-413a-8d43-95672991d02d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from nooffense import DFPredictor\n",
    "\n",
    "clf = DFPredictor(model_path='path_to_your_model_weight_folder', device='cpu')\n",
    "predicted_df = clf.predict_df(df_path=\"path_to_your_csv_file\", save_csv=False, progress_bar=True)\n",
    "# Code above will return a copy of original dataframe with predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a26121-c7b6-4409-aa79-601132d4f925",
   "metadata": {},
   "source": [
    "## Fill Masks\n",
    "\n",
    "NoOffense models are trained in a self-supervised fashion using a large Turkish offensive text corpus. These models are then fine-tuned for text classification tasks. However, as a byproduct of our Masked Language Model training, you can also use it as a fill-mask pipe!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d55cda25-1faa-43f8-bb12-0fce971dd39c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nooffense import MaskPredictor\n",
    "model = MaskPredictor(\"path_to_your_model_weight_folder\")\n",
    "model.mask_filler('sen tam bir [MASK]')\n",
    "# output:\n",
    "# [{'score': 0.26599952578544617,\n",
    "#   'token': 127526,\n",
    "#   'token_str': 'aptalsın',\n",
    "#   'sequence': 'sen tam bir aptalsın'},\n",
    "#  {'score': 0.10496608912944794,\n",
    "#   'token': 69461,\n",
    "#   'token_str': 'ibne',\n",
    "#   'sequence': 'sen tam bir ibne'},\n",
    "#  {'score': 0.10287725925445557,\n",
    "#   'token': 19111,\n",
    "#   'token_str': 'pislik',\n",
    "#   'sequence': 'sen tam bir pislik'},\n",
    "#  {'score': 0.05552872270345688,\n",
    "#   'token': 53442,\n",
    "#   'token_str': 'kaltak',\n",
    "#   'sequence': 'sen tam bir kaltak'}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7db7e83-250b-4268-8c39-7ffee023f2dc",
   "metadata": {},
   "source": [
    "As you can see, it fills the [MASK] token with pretty offensive content. To compare, let's use it with a casual NLP model. By the way, NoOffense also supports Hugging Face's model hub!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc037474-c2e2-4b7d-a126-45500ed374f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nooffense import MaskPredictor\n",
    "model = MaskPredictor(\"dbmdz/bert-base-turkish-128k-uncased\")\n",
    "model.mask_filler('sen tam bir [MASK]')\n",
    "# output:\n",
    "# [{'score': 0.1943783015012741,\n",
    "#   'token': 18,\n",
    "#   'token_str': '.',\n",
    "#   'sequence': 'sen tam bir.'},\n",
    "#  {'score': 0.09647665172815323,\n",
    "#   'token': 5,\n",
    "#   'token_str': '!',\n",
    "#   'sequence': 'sen tam bir!'},\n",
    "#  {'score': 0.030896930024027824,\n",
    "#   'token': 2289,\n",
    "#   'token_str': 'sen',\n",
    "#   'sequence': 'sen tam bir sen'},\n",
    "#  {'score': 0.024956168606877327,\n",
    "#   'token': 3191,\n",
    "#   'token_str': 'sey',\n",
    "#   'sequence': 'sen tam bir sey'}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09a663ed-be02-4a4b-a548-38163f4dad7c",
   "metadata": {},
   "source": [
    "As you can see, the casual NLP model produces much less offensive content than NoOffense."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d91b291-a20c-49e5-b8ea-035a1fc509e4",
   "metadata": {},
   "source": [
    "## Get Sentence Embeddings\n",
    "\n",
    "Sometimes, you need to encode your sentences in vector form to use them in different tasks than classification. With NoOffense, you can get sentence embeddings for offensive language inputs and use them for downstream tasks like sentence similarity, retrieval, and ranking, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8d26380-0265-41a6-a715-c248d54852bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nooffense import SentenceEncoder\n",
    "model = SentenceEncoder(\"path_to_your_model_weight_folder\")\n",
    "model.encode(['input_text_here'])\n",
    "\n",
    "# The code above will resuls mxn matrix where m is number of sentences given and n is dimension of encoder model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff8bdcb4-2bee-44c2-ab34-cfc4036d61ed",
   "metadata": {},
   "source": [
    "Speaking of sentence similarity, NoOffense library has method that helps you find most similar sentences using cosine similarity based on SentenceEncodings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "304259dd-b486-4c52-9a1f-dfe427b2baa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nooffense import SentenceEncoder\n",
    "model = SentenceEncoder(\"path_to_your_model_weight_folder\")\n",
    "model.find_most_similar(\"bir küfürlü içerik\", [\"text1\", \"text2\", ..., \"text_n\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90021cda-1228-4198-9e5c-2b31964ec8b9",
   "metadata": {},
   "source": [
    "The code above will result sorted dictionary starting with most similar text given from list of strings."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}