g3doc/tutorials/Fairness_Indicators_on_TF_Hub_Text_Embeddings.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aalPefrUUplk"
      },
      "source": [
        "# Fairness Indicators on TF-Hub Text Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/responsible_ai/fairness_indicators/tutorials/Fairness_Indicators_on_TF_Hub_Text_Embeddings\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_on_TF_Hub_Text_Embeddings.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_on_TF_Hub_Text_Embeddings.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/fairness-indicators/g3doc/tutorials/Fairness_Indicators_on_TF_Hub_Text_Embeddings.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://tfhub.dev/google/random-nnlm-en-dim128/1\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" /\u003eSee TF Hub model\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w0zsksbydmNp"
      },
      "source": [
        "In this tutorial, you will learn how to use [Fairness Indicators](https://github.com/tensorflow/fairness-indicators) to evaluate embeddings from [TF Hub](https://www.tensorflow.org/hub). This notebook uses the [Civil Comments dataset](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u33JXdluZ2lG"
      },
      "source": [
        "## Setup\n",
        "\n",
        "Install the required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BAUEkqYlzP3W"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U pip==20.2\n",
        "\n",
        "!pip install fairness-indicators \\\n",
        "  \"absl-py==0.12.0\" \\\n",
        "  \"pyarrow==10.0.0\" \\\n",
        "  \"apache-beam==2.47.0\" \\\n",
        "  \"avro-python3==1.9.1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6pe8c6L7kCW"
      },
      "source": [
        "Import other required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B8dlyTyiTe-9"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import tempfile\n",
        "import apache_beam as beam\n",
        "from datetime import datetime\n",
        "import tensorflow as tf\n",
        
        "import tensorflow_hub as hub\n",
        "import tensorflow_model_analysis as tfma\n",
        "from tensorflow_model_analysis.addons.fairness.view import widget_view\n",
        "from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators\n",
        "from fairness_indicators import example_model\n",
        "from fairness_indicators.tutorial_utils import util"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xz4PcI0hSVcq"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "In this notebook, you work with the [Civil Comments dataset](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification) which contains approximately 2 million public comments made public by the [Civil Comments platform](https://github.com/reaktivstudios/civil-comments) in 2017 for ongoing research. This effort was sponsored by Jigsaw, who have hosted competitions on Kaggle to help classify toxic comments as well as minimize unintended model bias.\n",
        "\n",
        "Each individual text comment in the dataset has a toxicity label, with the label being 1 if the comment is toxic and 0 if the comment is non-toxic. Within the data, a subset of comments are labeled with a variety of identity attributes, including categories for gender, sexual orientation, religion, and race or ethnicity."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ekzb7vVnPCc"
      },
      "source": [
        "### Prepare the data\n",
        "\n",
        "TensorFlow parses features from data using [`tf.io.FixedLenFeature`](https://www.tensorflow.org/api_docs/python/tf/io/FixedLenFeature) and [`tf.io.VarLenFeature`](https://www.tensorflow.org/api_docs/python/tf/io/VarLenFeature). Map out the input feature, output feature, and all other slicing features of interest."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n4_nXQDykX6W"
      },
      "outputs": [],
      "source": [
        "BASE_DIR = tempfile.gettempdir()\n",
        "\n",
        "# The input and output features of the classifier\n",
        "TEXT_FEATURE = 'comment_text'\n",
        "LABEL = 'toxicity'\n",
        "\n",
        "FEATURE_MAP = {\n",
        "    # input and output features\n",
        "    LABEL: tf.io.FixedLenFeature([], tf.float32),\n",
        "    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),\n",
        "\n",
        "    # slicing features\n",
        "    'sexual_orientation': tf.io.VarLenFeature(tf.string),\n",
        "    'gender': tf.io.VarLenFeature(tf.string),\n",
        "    'religion': tf.io.VarLenFeature(tf.string),\n",
        "    'race': tf.io.VarLenFeature(tf.string),\n",
        "    'disability': tf.io.VarLenFeature(tf.string)\n",
        "}\n",
        "\n",
        "IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeUtnaT49Doq"
      },
      "source": [
        "By default, the notebook downloads a preprocessed version of this dataset, but\n",
        "you may use the original dataset and re-run the processing steps if\n",
        "desired.\n",
        "\n",
        "In the original dataset, each comment is labeled with the percentage\n",
        "of raters who believed that a comment corresponds to a particular\n",
        "identity. For example, a comment might be labeled with the following:\n",
        "`{ male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8,\n",
        "homosexual_gay_or_lesbian: 1.0 }`.\n",
        "\n",
        "The processing step groups identity by category (gender,\n",
        "sexual_orientation, etc.) and removes identities with a score less\n",
        "than 0.5. So the example above would be converted to the following:\n",
        "of raters who believed that a comment corresponds to a particular\n",
        "identity. For example, the comment above would be labeled with the\n",
        "following:\n",
        "`{ gender: [female], sexual_orientation: [heterosexual,\n",
        "homosexual_gay_or_lesbian] }`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHxa31VX9eP2"
      },
      "source": [
        "Download the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NUmSmqYGS0n8"
      },
      "outputs": [],
      "source": [
        "download_original_data = False #@param {type:\"boolean\"}\n",
        "\n",
        "if download_original_data:\n",
        "  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',\n",
        "                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')\n",
        "  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',\n",
        "                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')\n",
        "\n",
        "  # The identity terms list will be grouped together by their categories\n",
        "  # (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,\n",
        "  # text column and label column will be kept after processing.\n",
        "  train_tf_file = util.convert_comments_data(train_tf_file)\n",
        "  validate_tf_file = util.convert_comments_data(validate_tf_file)\n",
        "\n",
        "else:\n",
        "  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',\n",
        "                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')\n",
        "  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',\n",
        "                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zz1NLR5Uu3oQ"
      },
      "source": [
        "## Create a TensorFlow Model Analysis Pipeline\n",
        "\n",
        "The Fairness Indicators library operates on [TensorFlow Model Analysis (TFMA) models](https://www.tensorflow.org/tfx/model_analysis/get_started). TFMA models wrap TensorFlow models with additional functionality to evaluate and visualize their results. The actual evaluation occurs inside of an [Apache Beam pipeline](https://beam.apache.org/documentation/programming-guide/).\n",
        "\n",
        "The steps you follow to create a TFMA pipeline are:\n",
        "1. Build a TensorFlow model\n",
        "2. Build a TFMA model on top of the TensorFlow model\n",
        "3. Run the model analysis in an orchestrator. The example model in this notebook uses Apache Beam as the orchestrator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7nSvu4IUCigW"
      },
      "outputs": [],
      "source": [
        "def embedding_fairness_result(embedding, identity_term='gender'):\n",
        "  \n",
        "  model_dir = os.path.join(BASE_DIR, 'train',\n",
        "                         datetime.now().strftime('%Y%m%d-%H%M%S'))\n",
        "\n",
        "  print(\"Training classifier for \" + embedding)\n",
        "  classifier = example_model.train_model(model_dir,\n",
        "                                         train_tf_file,\n",
        "                                         LABEL,\n",
        "                                         TEXT_FEATURE,\n",
        "                                         FEATURE_MAP,\n",
        "                                         embedding)\n",
        "\n",
        "  # Create a unique path to store the results for this embedding.\n",
        "  embedding_name = embedding.split('/')[-2]\n",
        "  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)\n",
        "\n",
        "  example_model.evaluate_model(classifier,\n",
        "                               validate_tf_file,\n",
        "                               eval_result_path,\n",
        "                               identity_term,\n",
        "                               LABEL,\n",
        "                               FEATURE_MAP)\n",
        "  return tfma.load_eval_result(output_path=eval_result_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jTPqije9Eg5b"
      },
      "source": [
        "## Run TFMA \u0026 Fairness Indicators"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8AvInTNt8Gyn"
      },
      "source": [
        "### Fairness Indicators Metrics\n",
        "\n",
        "Some of the metrics available with Fairness Indicators are:\n",
        "\n",
        "* [Negative Rate, False Negative Rate (FNR), and True Negative Rate (TNR)](https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_and_false_negative_rates)\n",
        "* [Positive Rate, False Positive Rate (FPR), and True Positive Rate (TPR)](https://en.wikipedia.org/wiki/False_positives_and_false_negatives#False_positive_and_false_negative_rates)\n",
        "* [Accuracy](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Accuracy)\n",
        "* [Precision and Recall](https://en.wikipedia.org/wiki/Precision_and_recall)\n",
        "* [Precision-Recall AUC](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC)\n",
        "* [ROC AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LGXCFtScblYt"
      },
      "source": [
        "### Text Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CI-1M5qXGjG"
      },
      "source": [
        "**[TF-Hub](https://www.tensorflow.org/hub)** provides several **text embeddings**. These embeddings will serve as the feature column for the different models. This tutorial uses the following embeddings:\n",
        "\n",
        "* [**random-nnlm-en-dim128**](https://tfhub.dev/google/random-nnlm-en-dim128/1): random text embeddings, this serves as a convenient baseline.\n",
        "* [**nnlm-en-dim128**](https://tfhub.dev/google/nnlm-en-dim128/1): a text embedding based on [A Neural Probabilistic Language Model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf). \n",
        "* [**universal-sentence-encoder**](https://tfhub.dev/google/universal-sentence-encoder/2): a text embedding based on [Universal Sentence Encoder](https://arxiv.org/pdf/1803.11175.pdf)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xxq97Qt7itVL"
      },
      "source": [
        "## Fairness Indicator Results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "27FX15awixuK"
      },
      "source": [
        "Compute fairness indicators with the `embedding_fairness_result` pipeline, and then render the results in the Fairness Indicator UI widget with `widget_view.render_fairness_indicator` for all the above embeddings.\n",
        "\n",
        "Note: You may need to run the `widget_view.render_fairness_indicator` cells twice for the visualization to be displayed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEUbZ93y8NCW"
      },
      "source": [
        "#### Random NNLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DkSuox-Pb6Pz"
      },
      "outputs": [],
      "source": [
        "eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "05xUesz6VpAe"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jmKe8Z1b8SBy"
      },
      "source": [
        "#### NNLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5b8HcTUBckj1"
      },
      "outputs": [],
      "source": [
        "eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n6hasLzFVrDN"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1I4xEDNq8T0X"
      },
      "source": [
        "#### Universal Sentence Encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GrdweWRkck8A"
      },
      "outputs": [],
      "source": [
        "eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JBABAkZMVtTK"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(eval_result=eval_result_use)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "402oTKbap77R"
      },
      "source": [
        "### Comparing Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UgnqwNjpqBuv"
      },
      "source": [
        "You can also use Fairness Indicators to compare embeddings directly. For example, compare the models generated from the NNLM and USE embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "49ECfYWUp7Kk"
      },
      "outputs": [],
      "source": [
        "widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Fairness Indicators on TF-Hub Text Embeddings",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}