Fine_tune_ALBERT_sentence_pair_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Fine_tune_ALBERT_sentence_pair_classification.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNlbkBhVp6bNo8ze7MJzkQI",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "6f797e5377fe4ab281003d537c15f279": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_9b46878367244033ac0d96f6f83766cd",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_b9b102512ad548ac96f5560eead35201",
              "IPY_MODEL_cb3cf3d7089347a092499762c552f785"
            ]
          }
        },
        "9b46878367244033ac0d96f6f83766cd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "b9b102512ad548ac96f5560eead35201": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_4ae80720349c409bba8dffa6ba429e9e",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 684,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 684,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_c74fcaee125642908ab8fa8905f0cf6f"
          }
        },
        "cb3cf3d7089347a092499762c552f785": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_6a229158e0db415b8561dc22a8a4ab47",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 684/684 [00:00&lt;00:00, 1.52kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_ef906f1d1e444404bd88e789e14e977a"
          }
        },
        "4ae80720349c409bba8dffa6ba429e9e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "c74fcaee125642908ab8fa8905f0cf6f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "6a229158e0db415b8561dc22a8a4ab47": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "ef906f1d1e444404bd88e789e14e977a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "80d383da347f41a6927d659dc5e1a855": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_26425569e06e4fc5b32405bcdba37dad",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_c6897db24c284263879bcc92e6d56f39",
              "IPY_MODEL_7468d3da8c8547d38e146b8adee8948e"
            ]
          }
        },
        "26425569e06e4fc5b32405bcdba37dad": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "c6897db24c284263879bcc92e6d56f39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_e90a00d220bc4546923517f13daf29be",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 760289,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 760289,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_13b7985246244fcb89926039ded8eb20"
          }
        },
        "7468d3da8c8547d38e146b8adee8948e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_3e108b6d60874974aaf0eea761e7da59",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 760k/760k [00:01&lt;00:00, 479kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_afe53d24858d446592d1825713664d9b"
          }
        },
        "e90a00d220bc4546923517f13daf29be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "13b7985246244fcb89926039ded8eb20": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "3e108b6d60874974aaf0eea761e7da59": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "afe53d24858d446592d1825713664d9b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "ac95062870cd4eb3afc93dfcc8ef9a6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_d79473c349744b6ba4d1532889a3dd3b",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_bac3fa1e8ff04a1680267702e35c89b1",
              "IPY_MODEL_c425744e85d44659b143679eccca5b0f"
            ]
          }
        },
        "d79473c349744b6ba4d1532889a3dd3b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "bac3fa1e8ff04a1680267702e35c89b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_a9c4247af57f46d485a4f509eafc79e3",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 47376696,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 47376696,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_6a4d58a9a2a84f09b3d341529de530ed"
          }
        },
        "c425744e85d44659b143679eccca5b0f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_c07487bb1e8c4415986d31752fa0d7f5",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 47.4M/47.4M [00:02&lt;00:00, 20.5MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_9a003cc981484793b31167d114362ea4"
          }
        },
        "a9c4247af57f46d485a4f509eafc79e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "6a4d58a9a2a84f09b3d341529de530ed": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "c07487bb1e8c4415986d31752fa0d7f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "9a003cc981484793b31167d114362ea4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TuLR0BMJhIub",
        "colab_type": "text"
      },
      "source": [
        "# Fine-tune ALBERT for sentence-pair classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SZIHhCYNhN-H",
        "colab_type": "text"
      },
      "source": [
        "## Introduction \n",
        "\n",
        "You will learn in this notebook how to fine-tune ALBERT and other BERT-based models for the **sentence-pair classification** task. This PyTorch implementation leverages the Hugging face *transformers* and *datasets* libraries to download pre-trained models, enable quick research experiments, access datasets and evaluation metrics.\n",
        "\n",
        "This task is part of the semantic textual similarity problem. You have two pair of sentences and you want to model the textual interaction between them.\n",
        "\n",
        "The dataset used in this notebook is Microsoft Research Paraphrase Corpus (MRPC) which is part of the GLUE benchmark : you have two sentences and you want to predict if one sentence is the paraphrase of the other one. The evaluation metrics are F1 and accuracy.\n",
        "\n",
        "You should be able to reach on the validation set **91.19** as F1 score (the score reported in the ALBERT paper is 90.9) and **87.5** as accuracy. The fine-tuning takes 35 seconds per epoch and the inference takes 2 seconds.\n",
        "\n",
        "The main features of this tutorial are : \n",
        "- End-to-end ML implementation (training, validation, prediction, evaluation)\n",
        "- Easy adaptability to your own datasets\n",
        "- Facilitation of quick experiments with other BERT-based models (BERT, ALBERT, ...)\n",
        "- Quick training with limited computational resources (mixed-precision, gradient accumulation, ...)\n",
        "- Multi-GPU execution\n",
        "- Threshold choice for the classification decision (not necessarily 0.5)\n",
        "- Freeze BERT layers and only update the classification layer weights or update all the weights\n",
        "- Reproducible results with seed settings\n",
        "\n",
        "#### Sections\n",
        "\n",
        "1. [Installation of libraries and imports](#section01)\n",
        "\n",
        "2. [Loading the dataset](#section02)\n",
        "\n",
        "3. [Classes and functions](#section03)\n",
        "\n",
        "4. [Parameters](#section04)\n",
        "\n",
        "5. [Training and validation](#section05)\n",
        "\n",
        "6. [Prediction](#section06)\n",
        "\n",
        "7. [Evaluation](#section07)\n",
        "\n",
        "8. [Experiments' ideas](#section08)\n",
        "\n",
        "9. [Limitations](#section09)\n",
        "\n",
        "10. [Future works](#section10)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YIv7rF4V6lyE",
        "colab_type": "text"
      },
      "source": [
        "## Installation of libraries and imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "42-lJ1u9IHT6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install datasets==1.0.1\n",
        "!pip install transformers==3.1.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KE_TpNaSZQ5n",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "e100b424-e5d9-45e5-ba3a-abe466edeb16"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "import copy\n",
        "import torch.optim as optim\n",
        "import random\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from torch.utils.data import DataLoader, Dataset\n",
        "from torch.cuda.amp import autocast, GradScaler\n",
        "from tqdm import tqdm\n",
        "from transformers import AutoTokenizer, AutoModel, AdamW, get_linear_schedule_with_warmup\n",
        "from datasets import load_dataset, load_metric\n",
        "\n",
        "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "PyTorch version 1.6.0+cu101 available.\n",
            "TensorFlow version 2.3.0 available.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aF8IWdEowPP-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "cba1400c-7447-48d9-d71c-3381e7315f70"
      },
      "source": [
        "# Check that we are using 100% of GPU memory footprint support libraries/code\n",
        "# from https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb\n",
        "!ln -sf /opt/bin/nvidia-smi /usr/bin/nvidia-smi\n",
        "!pip -q install gputil\n",
        "!pip -q install psutil\n",
        "!pip -q install humanize\n",
        "import psutil\n",
        "import humanize\n",
        "import os\n",
        "import GPUtil as GPU\n",
        "GPUs = GPU.getGPUs()\n",
        "# XXX: only one GPU on Colab and isn’t guaranteed\n",
        "gpu = GPUs[0]\n",
        "def printm():\n",
        " process = psutil.Process(os.getpid())\n",
        " print(\"Gen RAM Free: \" + humanize.naturalsize( psutil.virtual_memory().available ), \" | Proc size: \" + humanize.naturalsize( process.memory_info().rss))\n",
        " print(\"GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB\".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))\n",
        "printm()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  Building wheel for gputil (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Gen RAM Free: 12.6 GB  | Proc size: 525.1 MB\n",
            "GPU RAM Free: 15079MB | Used: 0MB | Util   0% | Total 15079MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yoGB7Arrwb4A",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "In case GPU utilisation (Util) is not at 0%, you can uncomment and run the following line to kill all processes to get the full GPU afterwards. Make sure to comment out the line again to not constantly crash the notebook on purpose."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CNdVkubWwdiD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# !kill -9 -1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lpKx43Iq6znw",
        "colab_type": "text"
      },
      "source": [
        "## Loading the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x0hfGEe2LB9s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load the MRPC dataset (train, validation and test)\n",
        "dataset = load_dataset('glue', 'mrpc')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mpobOzx-Lht7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "split = dataset['train'].train_test_split(test_size=0.1, seed=1)  # split the original training data for validation\n",
        "train = split['train']  # 90 % of the original training data\n",
        "val = split['test']   # 10 % of the original training data\n",
        "test = dataset['validation']  # the original validation data is used as test data because the test labels are not available with the datasets library\n",
        "\n",
        "# Transform data into pandas dataframes\n",
        "df_train = pd.DataFrame(train)\n",
        "df_val = pd.DataFrame(val)\n",
        "df_test = pd.DataFrame(test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RsjCBsJFF4Kn",
        "colab_type": "text"
      },
      "source": [
        "If you want to use your own dataset, you can upload it on the left of the notebook. \n",
        "\n",
        "For now, only csv files are handled, you need to upload three files : training data, validation data and test data (with or without labels)\n",
        "\n",
        "Here is a script to load data from csv files with the headers below : \n",
        "- sentence1\n",
        "- sentence2 \n",
        "- label"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OCA-bKfuEbjH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load your dataset from csv files\n",
        "\n",
        "# Some useful UNIX commands : \n",
        "# !pwd -> print working directory\n",
        "# !ls -> list directory contents of files and directories\n",
        "# %cd -> change the directory/folder of the terminal's shell\n",
        "\n",
        "\n",
        "# path_to_train_data = '/content/...'\n",
        "# path_to_val_data = '/content/...'\n",
        "# path_to_test_data = '/content/...'\n",
        "\n",
        "# delimiter = \";\" \n",
        "\n",
        "# df_train = pd.read_csv(path_to_train_data, delimiter=delimiter)\n",
        "# df_val = pd.read_csv(path_to_val_data, delimiter=delimiter)\n",
        "# df_test = pd.read_csv(path_to_test_data, delimiter=delimiter)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nNs2FWNJSLSq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "3157c71e-8b2d-4a86-9c3d-7433be8f7022"
      },
      "source": [
        "print(df_train.shape)\n",
        "print(df_val.shape)\n",
        "print(df_test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(3301, 4)\n",
            "(367, 4)\n",
            "(408, 4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "irj7itV0UCF_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "outputId": "c7aedad8-f1c1-4407-f9d0-9c2d0b4b96b6"
      },
      "source": [
        "df_train.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>idx</th>\n",
              "      <th>label</th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3349</td>\n",
              "      <td>1</td>\n",
              "      <td>The American troops , who also defend the mayo...</td>\n",
              "      <td>The American troops from 3-15 infantry , who a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1806</td>\n",
              "      <td>0</td>\n",
              "      <td>Last week , his lawyers asked Warner to grant ...</td>\n",
              "      <td>Last week , his lawyers asked Gov. Mark R. War...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2681</td>\n",
              "      <td>1</td>\n",
              "      <td>Their election increases the board from seven ...</td>\n",
              "      <td>Their appointments increase Berkshire 's board...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2001</td>\n",
              "      <td>1</td>\n",
              "      <td>Dolores Mahoy , 68 , of Colorado Springs , Col...</td>\n",
              "      <td>Dolores E. Mahoy , 68 , of Colorado Springs is...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2993</td>\n",
              "      <td>1</td>\n",
              "      <td>\" They were an inspirational couple , selfless...</td>\n",
              "      <td>He said : “ They were an inspirational couple ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    idx  ...                                          sentence2\n",
              "0  3349  ...  The American troops from 3-15 infantry , who a...\n",
              "1  1806  ...  Last week , his lawyers asked Gov. Mark R. War...\n",
              "2  2681  ...  Their appointments increase Berkshire 's board...\n",
              "3  2001  ...  Dolores E. Mahoy , 68 , of Colorado Springs is...\n",
              "4  2993  ...  He said : “ They were an inspirational couple ...\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TWJfh6DV7CB5",
        "colab_type": "text"
      },
      "source": [
        "## Classes and functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tc1GQh7yEm4C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CustomDataset(Dataset):\n",
        "\n",
        "    def __init__(self, data, maxlen, with_labels=True, bert_model='albert-base-v2'):\n",
        "\n",
        "        self.data = data  # pandas dataframe\n",
        "        #Initialize the tokenizer\n",
        "        self.tokenizer = AutoTokenizer.from_pretrained(bert_model)  \n",
        "\n",
        "        self.maxlen = maxlen\n",
        "        self.with_labels = with_labels \n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.data)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "\n",
        "        # Selecting sentence1 and sentence2 at the specified index in the data frame\n",
        "        sent1 = str(self.data.loc[index, 'sentence1'])\n",
        "        sent2 = str(self.data.loc[index, 'sentence2'])\n",
        "\n",
        "        # Tokenize the pair of sentences to get token ids, attention masks and token type ids\n",
        "        encoded_pair = self.tokenizer(sent1, sent2, \n",
        "                                      padding='max_length',  # Pad to max_length\n",
        "                                      truncation=True,  # Truncate to max_length\n",
        "                                      max_length=self.maxlen,  \n",
        "                                      return_tensors='pt')  # Return torch.Tensor objects\n",
        "        \n",
        "        token_ids = encoded_pair['input_ids'].squeeze(0)  # tensor of token ids\n",
        "        attn_masks = encoded_pair['attention_mask'].squeeze(0)  # binary tensor with \"0\" for padded values and \"1\" for the other values\n",
        "        token_type_ids = encoded_pair['token_type_ids'].squeeze(0)  # binary tensor with \"0\" for the 1st sentence tokens & \"1\" for the 2nd sentence tokens\n",
        "\n",
        "        if self.with_labels:  # True if the dataset has labels\n",
        "            label = self.data.loc[index, 'label']\n",
        "            return token_ids, attn_masks, token_type_ids, label  \n",
        "        else:\n",
        "            return token_ids, attn_masks, token_type_ids"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hm0lAXvTZChm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SentencePairClassifier(nn.Module):\n",
        "\n",
        "    def __init__(self, bert_model=\"albert-base-v2\", freeze_bert=False):\n",
        "        super(SentencePairClassifier, self).__init__()\n",
        "        #  Instantiating BERT-based model object\n",
        "        self.bert_layer = AutoModel.from_pretrained(bert_model)\n",
        "\n",
        "        #  Fix the hidden-state size of the encoder outputs (If you want to add other pre-trained models here, search for the encoder output size)\n",
        "        if bert_model == \"albert-base-v2\":  # 12M parameters\n",
        "            hidden_size = 768\n",
        "        elif bert_model == \"albert-large-v2\":  # 18M parameters\n",
        "            hidden_size = 1024\n",
        "        elif bert_model == \"albert-xlarge-v2\":  # 60M parameters\n",
        "            hidden_size = 2048\n",
        "        elif bert_model == \"albert-xxlarge-v2\":  # 235M parameters\n",
        "            hidden_size = 4096\n",
        "        elif bert_model == \"bert-base-uncased\": # 110M parameters\n",
        "            hidden_size = 768\n",
        "\n",
        "        # Freeze bert layers and only train the classification layer weights\n",
        "        if freeze_bert:\n",
        "            for p in self.bert_layer.parameters():\n",
        "                p.requires_grad = False\n",
        "\n",
        "        # Classification layer\n",
        "        self.cls_layer = nn.Linear(hidden_size, 1)\n",
        "\n",
        "        self.dropout = nn.Dropout(p=0.1)\n",
        "\n",
        "    @autocast()  # run in mixed precision\n",
        "    def forward(self, input_ids, attn_masks, token_type_ids):\n",
        "        '''\n",
        "        Inputs:\n",
        "            -input_ids : Tensor  containing token ids\n",
        "            -attn_masks : Tensor containing attention masks to be used to focus on non-padded values\n",
        "            -token_type_ids : Tensor containing token type ids to be used to identify sentence1 and sentence2\n",
        "        '''\n",
        "\n",
        "        # Feeding the inputs to the BERT-based model to obtain contextualized representations\n",
        "        cont_reps, pooler_output = self.bert_layer(input_ids, attn_masks, token_type_ids)\n",
        "\n",
        "        # Feeding to the classifier layer the last layer hidden-state of the [CLS] token further processed by a\n",
        "        # Linear Layer and a Tanh activation. The Linear layer weights were trained from the sentence order prediction (ALBERT) or next sentence prediction (BERT)\n",
        "        # objective during pre-training.\n",
        "        logits = self.cls_layer(self.dropout(pooler_output))\n",
        "\n",
        "        return logits"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5SrSNNYTjwe8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def set_seed(seed):\n",
        "    \"\"\" Set all seeds to make results reproducible \"\"\"\n",
        "    torch.manual_seed(seed)\n",
        "    torch.cuda.manual_seed_all(seed)\n",
        "    torch.backends.cudnn.deterministic = True\n",
        "    torch.backends.cudnn.benchmark = False\n",
        "    np.random.seed(seed)\n",
        "    random.seed(seed)\n",
        "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
        "    \n",
        "\n",
        "def evaluate_loss(net, device, criterion, dataloader):\n",
        "    net.eval()\n",
        "\n",
        "    mean_loss = 0\n",
        "    count = 0\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for it, (seq, attn_masks, token_type_ids, labels) in enumerate(tqdm(dataloader)):\n",
        "            seq, attn_masks, token_type_ids, labels = \\\n",
        "                seq.to(device), attn_masks.to(device), token_type_ids.to(device), labels.to(device)\n",
        "            logits = net(seq, attn_masks, token_type_ids)\n",
        "            mean_loss += criterion(logits.squeeze(-1), labels.float()).item()\n",
        "            count += 1\n",
        "\n",
        "    return mean_loss / count"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hl-rhuWsrg01",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "469f2b56-d92d-448e-d92c-b68ebe76f531"
      },
      "source": [
        "print(\"Creation of the models' folder...\")\n",
        "!mkdir models"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Creation of the models' folder...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZsL9VvmX0NLC",
        "colab_type": "text"
      },
      "source": [
        "Link for mixed precision training, gradient scaling and gradient accumulation  : https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples\n",
        "\n",
        "If you would like to learn more about Training Neural Nets on Larger Batches, I suggest reading this post of Thomas Wolf :\n",
        "https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I-o6KyaFkU5u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_bert(net, criterion, opti, lr, lr_scheduler, train_loader, val_loader, epochs, iters_to_accumulate):\n",
        "\n",
        "    best_loss = np.Inf\n",
        "    best_ep = 1\n",
        "    nb_iterations = len(train_loader)\n",
        "    print_every = nb_iterations // 5  # print the training loss 5 times per epoch\n",
        "    iters = []\n",
        "    train_losses = []\n",
        "    val_losses = []\n",
        "\n",
        "    scaler = GradScaler()\n",
        "\n",
        "    for ep in range(epochs):\n",
        "\n",
        "        net.train()\n",
        "        running_loss = 0.0\n",
        "        for it, (seq, attn_masks, token_type_ids, labels) in enumerate(tqdm(train_loader)):\n",
        "\n",
        "            # Converting to cuda tensors\n",
        "            seq, attn_masks, token_type_ids, labels = \\\n",
        "                seq.to(device), attn_masks.to(device), token_type_ids.to(device), labels.to(device)\n",
        "    \n",
        "            # Enables autocasting for the forward pass (model + loss)\n",
        "            with autocast():\n",
        "                # Obtaining the logits from the model\n",
        "                logits = net(seq, attn_masks, token_type_ids)\n",
        "\n",
        "                # Computing loss\n",
        "                loss = criterion(logits.squeeze(-1), labels.float())\n",
        "                loss = loss / iters_to_accumulate  # Normalize the loss because it is averaged\n",
        "\n",
        "            # Backpropagating the gradients\n",
        "            # Scales loss.  Calls backward() on scaled loss to create scaled gradients.\n",
        "            scaler.scale(loss).backward()\n",
        "\n",
        "            if (it + 1) % iters_to_accumulate == 0:\n",
        "                # Optimization step\n",
        "                # scaler.step() first unscales the gradients of the optimizer's assigned params.\n",
        "                # If these gradients do not contain infs or NaNs, opti.step() is then called,\n",
        "                # otherwise, opti.step() is skipped.\n",
        "                scaler.step(opti)\n",
        "                # Updates the scale for next iteration.\n",
        "                scaler.update()\n",
        "                # Adjust the learning rate based on the number of iterations.\n",
        "                lr_scheduler.step()\n",
        "                # Clear gradients\n",
        "                opti.zero_grad()\n",
        "\n",
        "\n",
        "            running_loss += loss.item()\n",
        "\n",
        "            if (it + 1) % print_every == 0:  # Print training loss information\n",
        "                print()\n",
        "                print(\"Iteration {}/{} of epoch {} complete. Loss : {} \"\n",
        "                      .format(it+1, nb_iterations, ep+1, running_loss / print_every))\n",
        "\n",
        "                running_loss = 0.0\n",
        "\n",
        "\n",
        "        val_loss = evaluate_loss(net, device, criterion, val_loader)  # Compute validation loss\n",
        "        print()\n",
        "        print(\"Epoch {} complete! Validation Loss : {}\".format(ep+1, val_loss))\n",
        "\n",
        "        if val_loss < best_loss:\n",
        "            print(\"Best validation loss improved from {} to {}\".format(best_loss, val_loss))\n",
        "            print()\n",
        "            net_copy = copy.deepcopy(net)  # save a copy of the model\n",
        "            best_loss = val_loss\n",
        "            best_ep = ep + 1\n",
        "\n",
        "    # Saving the model\n",
        "    path_to_model='models/{}_lr_{}_val_loss_{}_ep_{}.pt'.format(bert_model, lr, round(best_loss, 5), best_ep)\n",
        "    torch.save(net_copy.state_dict(), path_to_model)\n",
        "    print(\"The model has been saved in {}\".format(path_to_model))\n",
        "\n",
        "    del loss\n",
        "    torch.cuda.empty_cache()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WFy9kQ2-SvQ2",
        "colab_type": "text"
      },
      "source": [
        "## Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b6bzDp4FreS6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_model = \"albert-base-v2\"  # 'albert-base-v2', 'albert-large-v2', 'albert-xlarge-v2', 'albert-xxlarge-v2', 'bert-base-uncased', ...\n",
        "freeze_bert = False  # if True, freeze the encoder weights and only update the classification layer weights\n",
        "maxlen = 128  # maximum length of the tokenized input sentence pair : if greater than \"maxlen\", the input is truncated and else if smaller, the input is padded\n",
        "bs = 16  # batch size\n",
        "iters_to_accumulate = 2  # the gradient accumulation adds gradients over an effective batch of size : bs * iters_to_accumulate. If set to \"1\", you get the usual batch size\n",
        "lr = 2e-5  # learning rate\n",
        "epochs = 4  # number of training epochs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_abThXlSr6n",
        "colab_type": "text"
      },
      "source": [
        "## Training and validation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wwXCoj9_h1hY",
        "colab_type": "text"
      },
      "source": [
        "Link for the AdamW optimizer and the learning rate scheduler :\n",
        "https://huggingface.co/transformers/main_classes/optimizer_schedules.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VZWGPomoryxy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "6f797e5377fe4ab281003d537c15f279",
            "9b46878367244033ac0d96f6f83766cd",
            "b9b102512ad548ac96f5560eead35201",
            "cb3cf3d7089347a092499762c552f785",
            "4ae80720349c409bba8dffa6ba429e9e",
            "c74fcaee125642908ab8fa8905f0cf6f",
            "6a229158e0db415b8561dc22a8a4ab47",
            "ef906f1d1e444404bd88e789e14e977a",
            "80d383da347f41a6927d659dc5e1a855",
            "26425569e06e4fc5b32405bcdba37dad",
            "c6897db24c284263879bcc92e6d56f39",
            "7468d3da8c8547d38e146b8adee8948e",
            "e90a00d220bc4546923517f13daf29be",
            "13b7985246244fcb89926039ded8eb20",
            "3e108b6d60874974aaf0eea761e7da59",
            "afe53d24858d446592d1825713664d9b",
            "ac95062870cd4eb3afc93dfcc8ef9a6e",
            "d79473c349744b6ba4d1532889a3dd3b",
            "bac3fa1e8ff04a1680267702e35c89b1",
            "c425744e85d44659b143679eccca5b0f",
            "a9c4247af57f46d485a4f509eafc79e3",
            "6a4d58a9a2a84f09b3d341529de530ed",
            "c07487bb1e8c4415986d31752fa0d7f5",
            "9a003cc981484793b31167d114362ea4"
          ]
        },
        "outputId": "25d2f91d-ed8e-4d59-b6cf-cffaff9aaf4b"
      },
      "source": [
        "#  Set all seeds to make reproducible results\n",
        "set_seed(1)\n",
        "\n",
        "# Creating instances of training and validation set\n",
        "print(\"Reading training data...\")\n",
        "train_set = CustomDataset(df_train, maxlen, bert_model)\n",
        "print(\"Reading validation data...\")\n",
        "val_set = CustomDataset(df_val, maxlen, bert_model)\n",
        "# Creating instances of training and validation dataloaders\n",
        "train_loader = DataLoader(train_set, batch_size=bs, num_workers=5)\n",
        "val_loader = DataLoader(val_set, batch_size=bs, num_workers=5)\n",
        "\n",
        "\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "net = SentencePairClassifier(bert_model, freeze_bert=freeze_bert)\n",
        "\n",
        "if torch.cuda.device_count() > 1:  # if multiple GPUs\n",
        "    print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n",
        "    net = nn.DataParallel(net)\n",
        "\n",
        "net.to(device)\n",
        "\n",
        "criterion = nn.BCEWithLogitsLoss()\n",
        "opti = AdamW(net.parameters(), lr=lr, weight_decay=1e-2)\n",
        "num_warmup_steps = 0 # The number of steps for the warmup phase.\n",
        "num_training_steps = epochs * len(train_loader)  # The total number of training steps\n",
        "t_total = (len(train_loader) // iters_to_accumulate) * epochs  # Necessary to take into account Gradient accumulation\n",
        "lr_scheduler = get_linear_schedule_with_warmup(optimizer=opti, num_warmup_steps=num_warmup_steps, num_training_steps=t_total)\n",
        "\n",
        "train_bert(net, criterion, opti, lr, lr_scheduler, train_loader, val_loader, epochs, iters_to_accumulate)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading training data...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6f797e5377fe4ab281003d537c15f279",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=684.0, style=ProgressStyle(description_…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "80d383da347f41a6927d659dc5e1a855",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=760289.0, style=ProgressStyle(descripti…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Reading validation data...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ac95062870cd4eb3afc93dfcc8ef9a6e",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=47376696.0, style=ProgressStyle(descrip…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 20%|██        | 42/207 [00:07<00:27,  6.03it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 41/207 of epoch 1 complete. Loss : 0.3171079242374839 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 40%|████      | 83/207 [00:14<00:20,  6.00it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 82/207 of epoch 1 complete. Loss : 0.28335858190931923 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 60%|█████▉    | 124/207 [00:20<00:13,  5.97it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 123/207 of epoch 1 complete. Loss : 0.2626715171627882 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 80%|███████▉  | 165/207 [00:27<00:07,  5.90it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 164/207 of epoch 1 complete. Loss : 0.22988062079359844 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|█████████▉| 206/207 [00:34<00:00,  5.94it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 205/207 of epoch 1 complete. Loss : 0.22290468016048756 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 207/207 [00:34<00:00,  5.93it/s]\n",
            "100%|██████████| 23/23 [00:01<00:00, 12.30it/s]\n",
            "  0%|          | 0/207 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch 1 complete! Validation Loss : 0.35299917602020764\n",
            "Best validation loss improved from inf to 0.35299917602020764\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 20%|██        | 42/207 [00:07<00:28,  5.77it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 41/207 of epoch 2 complete. Loss : 0.19836873188614845 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 40%|████      | 83/207 [00:14<00:21,  5.78it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 82/207 of epoch 2 complete. Loss : 0.16286275281411847 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 60%|█████▉    | 124/207 [00:21<00:14,  5.76it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 123/207 of epoch 2 complete. Loss : 0.15275843041699108 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 80%|███████▉  | 165/207 [00:28<00:07,  5.74it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 164/207 of epoch 2 complete. Loss : 0.13304086556521857 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|█████████▉| 206/207 [00:35<00:00,  5.69it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 205/207 of epoch 2 complete. Loss : 0.13053428790554766 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 207/207 [00:36<00:00,  5.74it/s]\n",
            "100%|██████████| 23/23 [00:01<00:00, 11.73it/s]\n",
            "  0%|          | 0/207 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch 2 complete! Validation Loss : 0.43609277385732403\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 20%|██        | 42/207 [00:07<00:28,  5.69it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 41/207 of epoch 3 complete. Loss : 0.13481714503794182 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 40%|████      | 83/207 [00:14<00:21,  5.78it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 82/207 of epoch 3 complete. Loss : 0.08921206479028958 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 60%|█████▉    | 124/207 [00:21<00:14,  5.78it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 123/207 of epoch 3 complete. Loss : 0.08746417448288057 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 80%|███████▉  | 165/207 [00:28<00:07,  5.79it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 164/207 of epoch 3 complete. Loss : 0.0712746214348732 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|█████████▉| 206/207 [00:36<00:00,  5.81it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 205/207 of epoch 3 complete. Loss : 0.06408452383446984 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 207/207 [00:36<00:00,  5.71it/s]\n",
            "100%|██████████| 23/23 [00:01<00:00, 11.89it/s]\n",
            "  0%|          | 0/207 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch 3 complete! Validation Loss : 0.35006705865911814\n",
            "Best validation loss improved from 0.35299917602020764 to 0.35006705865911814\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 20%|██        | 42/207 [00:07<00:28,  5.73it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 41/207 of epoch 4 complete. Loss : 0.06930875144444587 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 40%|████      | 83/207 [00:14<00:21,  5.77it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 82/207 of epoch 4 complete. Loss : 0.04216044627856917 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 60%|█████▉    | 124/207 [00:21<00:14,  5.75it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 123/207 of epoch 4 complete. Loss : 0.038601048358875074 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            " 80%|███████▉  | 165/207 [00:28<00:07,  5.82it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 164/207 of epoch 4 complete. Loss : 0.041417054536684254 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|█████████▉| 206/207 [00:35<00:00,  5.78it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Iteration 205/207 of epoch 4 complete. Loss : 0.03673171890308944 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 207/207 [00:36<00:00,  5.75it/s]\n",
            "100%|██████████| 23/23 [00:01<00:00, 11.96it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch 4 complete! Validation Loss : 0.36528593172197754\n",
            "The model has been saved in models/albert-base-v2_lr_2e-05_val_loss_0.35007_ep_3.pt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gw2sOrIvCEZz",
        "colab_type": "text"
      },
      "source": [
        "You can download the model saved in the folder \"models\" by browsing the files on the left of the colab notebook"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_f4zThMtvqC1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# If you encounter a CUDA out of memory error: \n",
        "# - uncomment the kill command, run the \"kill\" command (and comment it)\n",
        "# - reduce the batch size\n",
        "# - then run all cells from the begining \n",
        "\n",
        "# If you get an ugly print of tqdm (all iterations are showed), follow the above first and last steps\n",
        "\n",
        "printm()\n",
        "# !kill -9 -1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDBtVu7JSbUK",
        "colab_type": "text"
      },
      "source": [
        "## Prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kAfbt0FjkCfM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "6ffdbfb5-fa8d-4329-9075-79c5152c952b"
      },
      "source": [
        "print(\"Creation of the results' folder...\")\n",
        "!mkdir results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Creation of the results' folder...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m3r8_npVf30D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_probs_from_logits(logits):\n",
        "    \"\"\"\n",
        "    Converts a tensor of logits into an array of probabilities by applying the sigmoid function\n",
        "    \"\"\"\n",
        "    probs = torch.sigmoid(logits.unsqueeze(-1))\n",
        "    return probs.detach().cpu().numpy()\n",
        "\n",
        "def test_prediction(net, device, dataloader, with_labels=True, result_file=\"results/output.txt\"):\n",
        "    \"\"\"\n",
        "    Predict the probabilities on a dataset with or without labels and print the result in a file\n",
        "    \"\"\"\n",
        "    net.eval()\n",
        "    w = open(result_file, 'w')\n",
        "    probs_all = []\n",
        "\n",
        "    with torch.no_grad():\n",
        "        if with_labels:\n",
        "            for seq, attn_masks, token_type_ids, _ in tqdm(dataloader):\n",
        "                seq, attn_masks, token_type_ids = seq.to(device), attn_masks.to(device), token_type_ids.to(device)\n",
        "                logits = net(seq, attn_masks, token_type_ids)\n",
        "                probs = get_probs_from_logits(logits.squeeze(-1)).squeeze(-1)\n",
        "                probs_all += probs.tolist()\n",
        "        else:\n",
        "            for seq, attn_masks, token_type_ids in tqdm(dataloader):\n",
        "                seq, attn_masks, token_type_ids = seq.to(device), attn_masks.to(device), token_type_ids.to(device)\n",
        "                logits = net(seq, attn_masks, token_type_ids)\n",
        "                probs = get_probs_from_logits(logits.squeeze(-1)).squeeze(-1)\n",
        "                probs_all += probs.tolist()\n",
        "\n",
        "    w.writelines(str(prob)+'\\n' for prob in probs_all)\n",
        "    w.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5-VJObbxwL6r",
        "colab_type": "text"
      },
      "source": [
        "I'm sharing below an ALBERT pre-trained model (45 Mo) so you can reproduce my results on the MRPC validation set (**91.19** as F1 score and **87.5** as accuracy). It's just in case but if all the code run as expected, you should get after the model training the correct model in the *models* folder\n",
        "\n",
        "You can download it and upload it (~ 3 minutes) in the *models* folder by browsing the files on the left of the colab notebook :\n",
        "\n",
        "https://drive.google.com/file/d/1AcRLGvALAH3BVSiDVjY_b8CggJgVfksp/view?usp=sharing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWoWiw6MlPm-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        },
        "outputId": "31072be1-20ed-43b4-a7c3-50a2d9fd78da"
      },
      "source": [
        "path_to_model = '/content/models/albert-base-v2_lr_2e-05_val_loss_0.35007_ep_3.pt'  \n",
        "# path_to_model = '/content/models/...'  # You can add here your trained model\n",
        "\n",
        "path_to_output_file = 'results/output.txt'\n",
        "\n",
        "print(\"Reading test data...\")\n",
        "test_set = CustomDataset(df_test, maxlen, bert_model)\n",
        "test_loader = DataLoader(test_set, batch_size=bs, num_workers=5)\n",
        "\n",
        "model = SentencePairClassifier(bert_model)\n",
        "if torch.cuda.device_count() > 1:  # if multiple GPUs\n",
        "    print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n",
        "    model = nn.DataParallel(model)\n",
        "\n",
        "print()\n",
        "print(\"Loading the weights of the model...\")\n",
        "model.load_state_dict(torch.load(path_to_model))\n",
        "model.to(device)\n",
        "\n",
        "print(\"Predicting on test data...\")\n",
        "test_prediction(net=model, device=device, dataloader=test_loader, with_labels=True,  # set the with_labels parameter to False if your want to get predictions on a dataset without labels\n",
        "                result_file=path_to_output_file)\n",
        "print()\n",
        "print(\"Predictions are available in : {}\".format(path_to_output_file))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading test data...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/26 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Loading the weights of the model...\n",
            "Predicting on test data...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 26/26 [00:02<00:00, 12.07it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Predictions are available in : results/output.txt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HCVAtClcC1qT",
        "colab_type": "text"
      },
      "source": [
        "You can download the predictions saved in the folder \"results\" by browsing the files on the left of the colab notebook"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ywxq1c8DSiV3",
        "colab_type": "text"
      },
      "source": [
        "## Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6JYwEPtrlBFX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "path_to_output_file = 'results/output.txt'  # path to the file with prediction probabilities\n",
        "\n",
        "labels_test = df_test['label']  # true labels\n",
        "\n",
        "probs_test = pd.read_csv(path_to_output_file, header=None)[0]  # prediction probabilities\n",
        "threshold = 0.5   # you can adjust this threshold for your own dataset\n",
        "preds_test=(probs_test>=threshold).astype('uint8') # predicted labels using the above fixed threshold\n",
        "\n",
        "metric = load_metric(\"glue\", \"mrpc\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5LGJRzdN7UOT",
        "colab_type": "text"
      },
      "source": [
        "Link for the threshold choice problem : https://machinelearningmastery.com/threshold-moving-for-imbalanced-classification/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G3ReQL2CUj3I",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "079e3357-e84e-41b8-8d14-64fae01180d7"
      },
      "source": [
        "# Compute the accuracy and F1 scores\n",
        "metric._compute(predictions=preds_test, references=labels_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 0.875, 'f1': 0.911917098445596}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ExhSydG0eeq",
        "colab_type": "text"
      },
      "source": [
        "## Experiments' ideas\n",
        "\n",
        "- Try other pre-trained models: https://huggingface.co/models\n",
        "- Try other optimizers and learning rate schedulers\n",
        "- Tune the hyperparameters : batch size, gradient accumulation parameter (iters_to_accumulate), number of epochs, learning rate\n",
        "- Change the *maxlen* parameter (max : 512). If you increase it, the training will take longer\n",
        "- Observe the influence of freezing the encoder weights and only updating the classifier weights\n",
        "- Use other metrics (Precision, Recall, ROC AUC, Precision-recall AUC, etc.) depending on the task and the dataset\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xtkcmIY9t6AF",
        "colab_type": "text"
      },
      "source": [
        "## Limitations\n",
        "\n",
        "- As said in the BERT github repository of Google Research (https://github.com/google-research/bert), \"Small sets like MRPC have a high variance in the Dev set accuracy, even when starting from the same re-training checkpoint.\"\n",
        "So I suggest taking that into account if you want to compare models on this dataset.\n",
        "- Distinct random seeds for models trained on GLUE datasets including MRPC can have a significant impact on results : for more details, you can read the paper *Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping* by Dodge et al. (https://arxiv.org/abs/2002.06305)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZp5xdVwvWVC",
        "colab_type": "text"
      },
      "source": [
        "## Future works\n",
        "\n",
        "- Adapt the code so other BERT-based models like RoBERTa and DistillBERT can also be fine-tuned for this task\n",
        "- Experiment feeding to the classification layer the last layer hidden states' average of all input tokens or other operations with multiple encoder layers instead of the pooler output\n",
        "\n"
      ]
    }
  ]
}