nlp_how_to_add_a_domain_specific_vocabulary_new_tokens_to_a_subword_tokenizer_already_trained_like_BERT_WordPiece.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "nlp_how_to_add_a_domain_specific_vocabulary_new_tokens_to_a_subword_tokenizer_already_trained_like_BERT_WordPiece.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "GaH0DSElUHyP",
        "luZPFwswULij",
        "DI_Q7C-kUT5k",
        "lC2Y9cHCUpzk"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "100fd06bbed24d28bc2171a429358d8f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_3e054786917b483797b115a61e8c8232",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_5cf18bcc6c8f441b9957d938d5a7e4df",
              "IPY_MODEL_2cc74411b685469785a3bd286653fe13"
            ]
          }
        },
        "3e054786917b483797b115a61e8c8232": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "5cf18bcc6c8f441b9957d938d5a7e4df": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_cb9cbf26853e4b1380194417bbbe68a5",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 433,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 433,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_2246b97547f7412ea238f19dd7fa7307"
          }
        },
        "2cc74411b685469785a3bd286653fe13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_d27b7661bc36430da5cb1c877a92a0b6",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 433/433 [00:14&lt;00:00, 28.9B/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_7e69803076d6458bb57c4b7ebd53a7f9"
          }
        },
        "cb9cbf26853e4b1380194417bbbe68a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "2246b97547f7412ea238f19dd7fa7307": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "d27b7661bc36430da5cb1c877a92a0b6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "7e69803076d6458bb57c4b7ebd53a7f9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "2ac599dbc3eb49b58d86d01b09fc7e15": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_a594cd6462ea4251b14eed20f89a615f",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_77d9a40ddfdf413e8e853f10c44167b9",
              "IPY_MODEL_8eb66f875c1447c2a425e767acded7ae"
            ]
          }
        },
        "a594cd6462ea4251b14eed20f89a615f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "77d9a40ddfdf413e8e853f10c44167b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_9f35d5cd96ef4cd6be78f95c0927804a",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 213450,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 213450,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_e61e48f043b54d7f87eeffc90fcbf612"
          }
        },
        "8eb66f875c1447c2a425e767acded7ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_7b4ee78dd68d4321bc5de73729996d2d",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 213k/213k [00:00&lt;00:00, 422kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_afd14a80292e487fa460ac7849ff64b5"
          }
        },
        "9f35d5cd96ef4cd6be78f95c0927804a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "e61e48f043b54d7f87eeffc90fcbf612": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "7b4ee78dd68d4321bc5de73729996d2d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "afd14a80292e487fa460ac7849ff64b5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "7272c6c137f641f28b0acd921f989696": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_73f1f0fb45234ae5a68779dafbe6cb0a",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_e4faad36f01a4327a546a58b6d1e190b",
              "IPY_MODEL_8a8f724fb2464eccbe6b2a74fe668113"
            ]
          }
        },
        "73f1f0fb45234ae5a68779dafbe6cb0a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "e4faad36f01a4327a546a58b6d1e190b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_f33842d1395e43e889a6a08086ec7f92",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 435797,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 435797,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_715471da5c024329b8bbc0b4ba498ed2"
          }
        },
        "8a8f724fb2464eccbe6b2a74fe668113": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_20958048dd6a426fa112ce8e16a176e0",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 436k/436k [00:01&lt;00:00, 235kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_fc0b1681bc4142218974328a0da24682"
          }
        },
        "f33842d1395e43e889a6a08086ec7f92": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "715471da5c024329b8bbc0b4ba498ed2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "20958048dd6a426fa112ce8e16a176e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "fc0b1681bc4142218974328a0da24682": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "2a0aa92ad92b4d0aa96145e3561977c4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_6cc27fc238a341dab14b92f2ce03fd7e",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_a6a72063e27548578c9f21c9e08a1543",
              "IPY_MODEL_0427fdc7cf6b46b5961a58eb43a4877d"
            ]
          }
        },
        "6cc27fc238a341dab14b92f2ce03fd7e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "a6a72063e27548578c9f21c9e08a1543": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_92f385d9519c4c019bb4db503facdd3b",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 29,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 29,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_b20c06c3d4b741c99fcef1b7d43c0ffd"
          }
        },
        "0427fdc7cf6b46b5961a58eb43a4877d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_6530d242138547a0a411aedff704ef9f",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 29.0/29.0 [00:00&lt;00:00, 40.3B/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_1a97b7e1ec094ba0a26dcb2b9822eb21"
          }
        },
        "92f385d9519c4c019bb4db503facdd3b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "b20c06c3d4b741c99fcef1b7d43c0ffd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "6530d242138547a0a411aedff704ef9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "1a97b7e1ec094ba0a26dcb2b9822eb21": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "8b596eba209d4f2aa59e13bbcf2f7eb6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_addb8bca20f04b7d9a8fbe8fcd358122",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_ec3a71b56e344ceb922c3afaead0670c",
              "IPY_MODEL_fe44027f5bc54320889b8ff2f74abd45"
            ]
          }
        },
        "addb8bca20f04b7d9a8fbe8fcd358122": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "ec3a71b56e344ceb922c3afaead0670c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_188c3a52b3ee4517b866f46f2c1f3afb",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 435779157,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 435779157,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_90c17258c2ce4df8bee77dab31b460a4"
          }
        },
        "fe44027f5bc54320889b8ff2f74abd45": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_3ca5fdfd8b5a466485f2d42785c53b4b",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 436M/436M [00:11&lt;00:00, 36.4MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_8caf62e6ec554ef4b69be8fb7e991383"
          }
        },
        "188c3a52b3ee4517b866f46f2c1f3afb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "90c17258c2ce4df8bee77dab31b460a4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "3ca5fdfd8b5a466485f2d42785c53b4b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "8caf62e6ec554ef4b69be8fb7e991383": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Ej5ImPsRvTN"
      },
      "source": [
        "# NLP | How to add a domain-specific vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C9pW-pZ0aGUp"
      },
      "source": [
        "* **Author**: [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/)\n",
        "* **Date**: April 05, 2021\n",
        "* **Blog post**: [NLP & domain specific | How to add a specialized vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece](https://medium.com/@pierre_guillou/nlp-how-to-add-a-domain-specific-vocabulary-new-tokens-to-a-subword-tokenizer-already-trained-33ab15613a41)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bNxQvuVqEzYQ"
      },
      "source": [
        "**Summary**: In some cases, it may be crucial to enrich the vocabulary of an already trained natural language model with vocabulary from a specialized domain (medicine, law, etc.) in order to perform new tasks (classification, NER, summary, translation, etc.). While the Hugging Face library allows you to easily add new tokens to the vocabulary of an existing tokenizer like BERT WordPiece, those tokens must be whole words, not subwords. This article explains why and how to obtain these new tokens from a specialized corpus."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GaH0DSElUHyP"
      },
      "source": [
        "## Install libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Voe1oG_wHTj",
        "outputId": "fc6d6f88-aba4-42bc-f04b-53ddf89fe989"
      },
      "source": [
        "!nvcc --version"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "nvcc: NVIDIA (R) Cuda compiler driver\n",
            "Copyright (c) 2005-2020 NVIDIA Corporation\n",
            "Built on Wed_Jul_22_19:09:09_PDT_2020\n",
            "Cuda compilation tools, release 11.0, V11.0.221\n",
            "Build cuda_11.0_bu.TC445_37.28845127_0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gi4ZoDXOR1mm",
        "outputId": "233e5a7e-1236-4795-b553-c34f00e9c4c3"
      },
      "source": [
        "# Install last Hugging Face libraries (datasets & transformers)\n",
        "!pip install datasets git+https://github.com/huggingface/transformers/\n",
        "# install spaCY\n",
        "!pip install -U pip setuptools wheel\n",
        "!pip install -U spacy[cuda110]\n",
        "!python -m spacy download en_core_web_sm\n",
        "# install scikit-learn\n",
        "!pip install -U scikit-learn\n",
        "# install matplotlib\n",
        "!pip install matplotlib\n",
        "# install wikipedia\n",
        "!pip install wikipedia"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+https://github.com/huggingface/transformers/\n",
            "  Cloning https://github.com/huggingface/transformers/ to /tmp/pip-req-build-0o5hb4jb\n",
            "  Running command git clone -q https://github.com/huggingface/transformers/ /tmp/pip-req-build-0o5hb4jb\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting datasets\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/54/90/43b396481a8298c6010afb93b3c1e71d4ba6f8c10797a7da8eb005e45081/datasets-1.5.0-py3-none-any.whl (192kB)\n",
            "\u001b[K     |████████████████████████████████| 194kB 7.3MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/08/cd/342e584ee544d044fb573ae697404ce22ede086c9e87ce5960772084cad0/sacremoses-0.0.44.tar.gz (862kB)\n",
            "\u001b[K     |████████████████████████████████| 870kB 11.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (4.41.1)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (3.8.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (2019.12.20)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (3.0.12)\n",
            "Collecting tokenizers<0.11,>=0.10.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/71/23/2ddc317b2121117bf34dd00f5b0de194158f2a44ee2bf5e47c7166878a97/tokenizers-0.10.1-cp37-cp37m-manylinux2010_x86_64.whl (3.2MB)\n",
            "\u001b[K     |████████████████████████████████| 3.2MB 24.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (1.19.5)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (20.9)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==4.5.0.dev0) (2.23.0)\n",
            "Collecting xxhash\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e7/27/1c0b37c53a7852f1c190ba5039404d27b3ae96a55f48203a74259f8213c9/xxhash-2.0.0-cp37-cp37m-manylinux2010_x86_64.whl (243kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 60.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets) (0.3.3)\n",
            "Collecting fsspec\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/0d/a6bfee0ddf47b254286b9bd574e6f50978c69897647ae15b14230711806e/fsspec-0.8.7-py3-none-any.whl (103kB)\n",
            "\u001b[K     |████████████████████████████████| 112kB 57.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets) (1.1.5)\n",
            "Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.7/dist-packages (from datasets) (3.0.0)\n",
            "Collecting huggingface-hub<0.1.0\n",
            "  Downloading https://files.pythonhosted.org/packages/af/07/bf95f398e6598202d878332280f36e589512174882536eb20d792532a57d/huggingface_hub-0.0.7-py3-none-any.whl\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets) (0.70.11.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.5.0.dev0) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.5.0.dev0) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==4.5.0.dev0) (1.0.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers==4.5.0.dev0) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers==4.5.0.dev0) (3.4.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==4.5.0.dev0) (2.4.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.5.0.dev0) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.5.0.dev0) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.5.0.dev0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==4.5.0.dev0) (2020.12.5)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets) (2.8.1)\n",
            "Building wheels for collected packages: transformers\n",
            "  Building wheel for transformers (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for transformers: filename=transformers-4.5.0.dev0-cp37-none-any.whl size=2059073 sha256=cf2f464fdcf57aeed458e61554358c3fd9006f8b1ff3d6e93ac0e445a9d12562\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-e_xbrs4d/wheels/61/69/33/974fccec4d0ab5feee9fe83bd93e680d269a805be9ede5ec60\n",
            "Successfully built transformers\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.44-cp37-none-any.whl size=886084 sha256=f0642b425ba2e9624a97c8191e2729bb40c0f798efe7cb758856d3896391c1f4\n",
            "  Stored in directory: /root/.cache/pip/wheels/3e/fb/c0/13ab4d63d537658f448366744654323077c4d90069b6512f3c\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: xxhash, fsspec, huggingface-hub, datasets, sacremoses, tokenizers, transformers\n",
            "Successfully installed datasets-1.5.0 fsspec-0.8.7 huggingface-hub-0.0.7 sacremoses-0.0.44 tokenizers-0.10.1 transformers-4.5.0.dev0 xxhash-2.0.0\n",
            "Collecting pip\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)\n",
            "\u001b[K     |████████████████████████████████| 1.5MB 8.4MB/s \n",
            "\u001b[?25hRequirement already up-to-date: setuptools in /usr/local/lib/python3.7/dist-packages (54.2.0)\n",
            "Requirement already up-to-date: wheel in /usr/local/lib/python3.7/dist-packages (0.36.2)\n",
            "Installing collected packages: pip\n",
            "  Found existing installation: pip 19.3.1\n",
            "    Uninstalling pip-19.3.1:\n",
            "      Successfully uninstalled pip-19.3.1\n",
            "Successfully installed pip-21.0.1\n",
            "Requirement already satisfied: spacy[cuda110] in /usr/local/lib/python3.7/dist-packages (2.2.4)\n",
            "Collecting spacy[cuda110]\n",
            "  Downloading spacy-3.0.5-cp37-cp37m-manylinux2014_x86_64.whl (12.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 12.8 MB 344 kB/s \n",
            "\u001b[?25hRequirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (2.0.5)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (0.4.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (54.2.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (1.19.5)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (2.11.3)\n",
            "Collecting spacy-legacy<3.1.0,>=3.0.0\n",
            "  Downloading spacy_legacy-3.0.2-py2.py3-none-any.whl (7.8 kB)\n",
            "Collecting pydantic<1.8.0,>=1.7.1\n",
            "  Downloading pydantic-1.7.3-cp37-cp37m-manylinux2014_x86_64.whl (9.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 9.1 MB 71.9 MB/s \n",
            "\u001b[?25hCollecting thinc<8.1.0,>=8.0.2\n",
            "  Downloading thinc-8.0.2-cp37-cp37m-manylinux2014_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 69.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (1.0.5)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (3.0.5)\n",
            "Requirement already satisfied: importlib-metadata>=0.20 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (3.8.1)\n",
            "Collecting catalogue<2.1.0,>=2.0.1\n",
            "  Downloading catalogue-2.0.1-py3-none-any.whl (9.6 kB)\n",
            "Collecting srsly<3.0.0,>=2.4.0\n",
            "  Downloading srsly-2.4.0-cp37-cp37m-manylinux2014_x86_64.whl (456 kB)\n",
            "\u001b[K     |████████████████████████████████| 456 kB 53.5 MB/s \n",
            "\u001b[?25hCollecting pathy>=0.3.5\n",
            "  Downloading pathy-0.4.0-py3-none-any.whl (36 kB)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (3.7.4.3)\n",
            "Collecting typer<0.4.0,>=0.3.0\n",
            "  Downloading typer-0.3.2-py3-none-any.whl (21 kB)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (0.8.2)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (4.41.1)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (20.9)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy[cuda110]) (2.23.0)\n",
            "Collecting cupy-cuda110<9.0.0,>=5.0.0b4\n",
            "  Downloading cupy_cuda110-8.6.0-cp37-cp37m-manylinux1_x86_64.whl (165.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 165.3 MB 47 kB/s \n",
            "\u001b[?25hRequirement already satisfied: fastrlock>=0.3 in /usr/local/lib/python3.7/dist-packages (from cupy-cuda110<9.0.0,>=5.0.0b4->spacy[cuda110]) (0.6)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20->spacy[cuda110]) (3.4.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy[cuda110]) (2.4.7)\n",
            "Collecting smart-open<4.0.0,>=2.2.0\n",
            "  Downloading smart_open-3.0.0.tar.gz (113 kB)\n",
            "\u001b[K     |████████████████████████████████| 113 kB 80.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda110]) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda110]) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda110]) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda110]) (2.10)\n",
            "Requirement already satisfied: click<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer<0.4.0,>=0.3.0->spacy[cuda110]) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy[cuda110]) (1.1.1)\n",
            "Building wheels for collected packages: smart-open\n",
            "  Building wheel for smart-open (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for smart-open: filename=smart_open-3.0.0-py3-none-any.whl size=107097 sha256=2a62b976232df07d033a2d93fed4ef67f17c651b653e6f442353fcedba1c5f82\n",
            "  Stored in directory: /root/.cache/pip/wheels/83/a6/12/bf3c1a667bde4251be5b7a3368b2d604c9af2105b5c1cb1870\n",
            "Successfully built smart-open\n",
            "Installing collected packages: catalogue, typer, srsly, smart-open, pydantic, thinc, spacy-legacy, pathy, spacy, cupy-cuda110\n",
            "  Attempting uninstall: catalogue\n",
            "    Found existing installation: catalogue 1.0.0\n",
            "    Uninstalling catalogue-1.0.0:\n",
            "      Successfully uninstalled catalogue-1.0.0\n",
            "  Attempting uninstall: srsly\n",
            "    Found existing installation: srsly 1.0.5\n",
            "    Uninstalling srsly-1.0.5:\n",
            "      Successfully uninstalled srsly-1.0.5\n",
            "  Attempting uninstall: smart-open\n",
            "    Found existing installation: smart-open 4.2.0\n",
            "    Uninstalling smart-open-4.2.0:\n",
            "      Successfully uninstalled smart-open-4.2.0\n",
            "  Attempting uninstall: thinc\n",
            "    Found existing installation: thinc 7.4.0\n",
            "    Uninstalling thinc-7.4.0:\n",
            "      Successfully uninstalled thinc-7.4.0\n",
            "  Attempting uninstall: spacy\n",
            "    Found existing installation: spacy 2.2.4\n",
            "    Uninstalling spacy-2.2.4:\n",
            "      Successfully uninstalled spacy-2.2.4\n",
            "Successfully installed catalogue-2.0.1 cupy-cuda110-8.6.0 pathy-0.4.0 pydantic-1.7.3 smart-open-3.0.0 spacy-3.0.5 spacy-legacy-3.0.2 srsly-2.4.0 thinc-8.0.2 typer-0.3.2\n",
            "2021-04-05 16:38:16.310887: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n",
            "Collecting en-core-web-sm==3.0.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0-py3-none-any.whl (13.7 MB)\n",
            "\u001b[K     |████████████████████████████████| 13.7 MB 105 kB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy<3.1.0,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from en-core-web-sm==3.0.0) (3.0.5)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.19.5)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.4.1)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.23.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (20.9)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.0.1)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.2)\n",
            "Requirement already satisfied: importlib-metadata>=0.20 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.8.1)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.0.5)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (8.0.2)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.8.2)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.4.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (4.41.1)\n",
            "Requirement already satisfied: typer<0.4.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.3.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.4.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.0.5)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (54.2.0)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.7.4.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.11.3)\n",
            "Requirement already satisfied: pydantic<1.8.0,>=1.7.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.7.3)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.5)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.4.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.4.7)\n",
            "Requirement already satisfied: smart-open<4.0.0,>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2020.12.5)\n",
            "Requirement already satisfied: click<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer<0.4.0,>=0.3.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.1.1)\n",
            "Installing collected packages: en-core-web-sm\n",
            "  Attempting uninstall: en-core-web-sm\n",
            "    Found existing installation: en-core-web-sm 2.2.5\n",
            "    Uninstalling en-core-web-sm-2.2.5:\n",
            "      Successfully uninstalled en-core-web-sm-2.2.5\n",
            "Successfully installed en-core-web-sm-3.0.0\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_sm')\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (0.22.2.post1)\n",
            "Collecting scikit-learn\n",
            "  Downloading scikit_learn-0.24.1-cp37-cp37m-manylinux2010_x86_64.whl (22.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 22.3 MB 65.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.4.1)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.19.5)\n",
            "Collecting threadpoolctl>=2.0.0\n",
            "  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.0.1)\n",
            "Installing collected packages: threadpoolctl, scikit-learn\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 0.22.2.post1\n",
            "    Uninstalling scikit-learn-0.22.2.post1:\n",
            "      Successfully uninstalled scikit-learn-0.22.2.post1\n",
            "Successfully installed scikit-learn-0.24.1 threadpoolctl-2.1.0\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (3.2.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.10.0)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.19.5)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.8.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.3.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.4.7)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib) (1.15.0)\n",
            "Collecting wikipedia\n",
            "  Downloading wikipedia-1.4.0.tar.gz (27 kB)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from wikipedia) (4.6.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from wikipedia) (2.23.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2020.12.5)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.0.4)\n",
            "Building wheels for collected packages: wikipedia\n",
            "  Building wheel for wikipedia (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wikipedia: filename=wikipedia-1.4.0-py3-none-any.whl size=11686 sha256=d7118db6a3b5ed78c9d5c9896f954890cd18779a4cc32007a2fe9dee981406da\n",
            "  Stored in directory: /root/.cache/pip/wheels/15/93/6d/5b2c68b8a64c7a7a04947b4ed6d89fb557dcc6bc27d1d7f3ba\n",
            "Successfully built wikipedia\n",
            "Installing collected packages: wikipedia\n",
            "Successfully installed wikipedia-1.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "luZPFwswULij"
      },
      "source": [
        "## Download a BERT model and its WordPiece tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0,
          "referenced_widgets": [
            "100fd06bbed24d28bc2171a429358d8f",
            "3e054786917b483797b115a61e8c8232",
            "5cf18bcc6c8f441b9957d938d5a7e4df",
            "2cc74411b685469785a3bd286653fe13",
            "cb9cbf26853e4b1380194417bbbe68a5",
            "2246b97547f7412ea238f19dd7fa7307",
            "d27b7661bc36430da5cb1c877a92a0b6",
            "7e69803076d6458bb57c4b7ebd53a7f9",
            "2ac599dbc3eb49b58d86d01b09fc7e15",
            "a594cd6462ea4251b14eed20f89a615f",
            "77d9a40ddfdf413e8e853f10c44167b9",
            "8eb66f875c1447c2a425e767acded7ae",
            "9f35d5cd96ef4cd6be78f95c0927804a",
            "e61e48f043b54d7f87eeffc90fcbf612",
            "7b4ee78dd68d4321bc5de73729996d2d",
            "afd14a80292e487fa460ac7849ff64b5",
            "7272c6c137f641f28b0acd921f989696",
            "73f1f0fb45234ae5a68779dafbe6cb0a",
            "e4faad36f01a4327a546a58b6d1e190b",
            "8a8f724fb2464eccbe6b2a74fe668113",
            "f33842d1395e43e889a6a08086ec7f92",
            "715471da5c024329b8bbc0b4ba498ed2",
            "20958048dd6a426fa112ce8e16a176e0",
            "fc0b1681bc4142218974328a0da24682",
            "2a0aa92ad92b4d0aa96145e3561977c4",
            "6cc27fc238a341dab14b92f2ce03fd7e",
            "a6a72063e27548578c9f21c9e08a1543",
            "0427fdc7cf6b46b5961a58eb43a4877d",
            "92f385d9519c4c019bb4db503facdd3b",
            "b20c06c3d4b741c99fcef1b7d43c0ffd",
            "6530d242138547a0a411aedff704ef9f",
            "1a97b7e1ec094ba0a26dcb2b9822eb21",
            "8b596eba209d4f2aa59e13bbcf2f7eb6",
            "addb8bca20f04b7d9a8fbe8fcd358122",
            "ec3a71b56e344ceb922c3afaead0670c",
            "fe44027f5bc54320889b8ff2f74abd45",
            "188c3a52b3ee4517b866f46f2c1f3afb",
            "90c17258c2ce4df8bee77dab31b460a4",
            "3ca5fdfd8b5a466485f2d42785c53b4b",
            "8caf62e6ec554ef4b69be8fb7e991383"
          ]
        },
        "id": "nnV5TNmUR2QL",
        "outputId": "1d2d8429-979f-47a2-f38c-c32d935fea2d"
      },
      "source": [
        "from transformers import AutoModelForMaskedLM, AutoTokenizer\n",
        "model_name = \"bert-base-cased\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_name)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "100fd06bbed24d28bc2171a429358d8f",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2ac599dbc3eb49b58d86d01b09fc7e15",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=213450.0, style=ProgressStyle(descripti…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7272c6c137f641f28b0acd921f989696",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=435797.0, style=ProgressStyle(descripti…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2a0aa92ad92b4d0aa96145e3561977c4",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=29.0, style=ProgressStyle(description_w…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8b596eba209d4f2aa59e13bbcf2f7eb6",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=435779157.0, style=ProgressStyle(descri…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DI_Q7C-kUT5k"
      },
      "source": [
        "## Tokenize a phrase about COVID"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T9u3-n92Sska"
      },
      "source": [
        "text = \"COVID-19 affects different people in different ways. Most infected people will develop mild to moderate illness and recover without hospitalization.\""
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j97tA5WeS2yV",
        "outputId": "0101711f-b115-4991-f7fd-88b7e1be0ecd"
      },
      "source": [
        "# tokenization of the text\n",
        "tokens = tokenizer.tokenize(text)\n",
        "print(tokens)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['CO', '##VI', '##D', '-', '19', 'affects', 'different', 'people', 'in', 'different', 'ways', '.', 'Most', 'infected', 'people', 'will', 'develop', 'mild', 'to', 'moderate', 'illness', 'and', 'recover', 'without', 'hospital', '##ization', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "LgH1SvInzW11",
        "outputId": "5cfa1623-cf3f-4767-d90b-34fb9c68b4ce"
      },
      "source": [
        "# back to text\n",
        "tokenizer.decode(tokenizer.encode(text), skip_special_tokens=True)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'COVID - 19 affects different people in different ways. Most infected people will develop mild to moderate illness and recover without hospitalization.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OMa--sCnJp4Z",
        "outputId": "71dc498c-f785-4e8e-c59f-c4cbd013738a"
      },
      "source": [
        "print(tokenizer.tokenize('COVID'))\n",
        "print(tokenizer.tokenize('hospitalization'))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['CO', '##VI', '##D']\n",
            "['hospital', '##ization']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FXwgWX9zpRHX"
      },
      "source": [
        "**We can notice that the BERT WordPiece tokenizer (from the bert-base-cased model) tokenize the words COVID and hospitalization with subwords because they do not exist as words in the tokenizer vocabulary.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JrETrcNdFt7i",
        "outputId": "092f3649-3231-4510-c6a8-6874481f7b9c"
      },
      "source": [
        "# Verify that the words COVID and hospitalization DO NOT belong to the tokenizer vocabulary\n",
        "vocab = [tok for tok,index in tokenizer.get_vocab().items()]\n",
        "\"COVID\" in vocab, \"hospitalization\" in vocab"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(False, False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lC2Y9cHCUpzk"
      },
      "source": [
        "## [ First test ] Add 2 new tokens (whole words) into the tokenizer vocab"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BR68tWtPTTen"
      },
      "source": [
        "new_tokens = ['COVID', 'hospitalization']"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y3vXXLqPSWBg",
        "outputId": "f234d9ad-1e4f-440a-8c55-63f0f109ca3b"
      },
      "source": [
        "print(\"[ BEFORE ] tokenizer vocab size:\", len(tokenizer)) \n",
        "added_tokens = tokenizer.add_tokens(new_tokens)\n",
        "\n",
        "print(\"[ AFTER ] tokenizer vocab size:\", len(tokenizer)) \n",
        "print()\n",
        "print('added_tokens:',added_tokens)\n",
        "print()\n",
        "\n",
        "# resize the embeddings matrix of the model \n",
        "model.resize_token_embeddings(len(tokenizer)) "
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ BEFORE ] tokenizer vocab size: 28996\n",
            "[ AFTER ] tokenizer vocab size: 28998\n",
            "\n",
            "added_tokens: 2\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Embedding(28998, 768)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oO8ruf7-LFQR",
        "outputId": "7e1610bd-3f1d-4f6e-c7e5-68a67444247b"
      },
      "source": [
        "# Verify that the words COVID and hospitalization DO belong to the tokenizer vocabulary\n",
        "vocab = [tok for tok,index in tokenizer.get_vocab().items()]\n",
        "\"COVID\" in vocab, \"hospitalization\" in vocab"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(True, True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oOz_GyDyIUTG"
      },
      "source": [
        "Let's call tokenizer_exBERT our tokenizer with the 2 new tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IzpHF5Fvzs5X"
      },
      "source": [
        "tokenizer_exBERT = tokenizer"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jh1C-86lTN-8",
        "outputId": "d8bd9916-4067-4cf1-f818-7f745cc7b553"
      },
      "source": [
        "# tokenization of the text\n",
        "tokens = tokenizer_exBERT.tokenize(text)\n",
        "print(tokens)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['COVID', '-', '19', 'affects', 'different', 'people', 'in', 'different', 'ways', '.', 'Most', 'infected', 'people', 'will', 'develop', 'mild', 'to', 'moderate', 'illness', 'and', 'recover', 'without', 'hospitalization', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "H9iJexuPTzAU",
        "outputId": "35393c66-c7e7-4396-cb23-61dd4d77748b"
      },
      "source": [
        "# back to text\n",
        "tokenizer_exBERT.decode(tokenizer_exBERT.encode(text), skip_special_tokens=True)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'COVID - 19 affects different people in different ways. Most infected people will develop mild to moderate illness and recover without hospitalization.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcBX4c8hIl29"
      },
      "source": [
        "**The tokenizer with the 2 new tokens succeeded in tokenizing the words COVID and hospitalization without subwords as they belong now to the vocabulary tokenizer.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nzWCYzijJA4H",
        "outputId": "02aaeaf1-4eb2-4872-df01-65cee2286893"
      },
      "source": [
        "# tokenization of the words COVID and hospitalization\n",
        "print(tokenizer_exBERT.tokenize('COVID'))\n",
        "print(tokenizer_exBERT.tokenize('hospitalization'))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['COVID']\n",
            "['hospitalization']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vaKONVnBVHav"
      },
      "source": [
        "## [ Second test ] Add more new tokens (subwords and words) into the tokenizer vocab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k58Ya5neFvxj"
      },
      "source": [
        "What if we want to detect the whole vocabulary of a specialized corpus (and not only 2 words) in order to add it to an existing corpus? \n",
        "\n",
        "Let's use a WordpIece tokenizer for this! (Why a WordPiece tokenizer? This is our first guess: since the BERT tokenizer is a WordPiece tokenizer, let's use a tokenizer of the same type)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VcgpRtetz-ZE"
      },
      "source": [
        "### 1) Import pages about COVID from English Wikipedia"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4XKt2He0v0kH",
        "outputId": "e0ad1101-5e0d-4cfa-fb51-8301a6b2fbbb"
      },
      "source": [
        "import wikipedia\n",
        "\n",
        "# let's choose 2 Wikipedia pages for our demonstration (we could have choosen an infinity)\n",
        "pages = [\"COVID-19\",\"COVID-19 pandemic\"]\n",
        "\n",
        "documents = list()\n",
        "for p in pages:\n",
        "  page = wikipedia.page(p)\n",
        "  documents.append(page.content)\n",
        "  print(page.title,page.url)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "COVID-19 https://en.wikipedia.org/wiki/COVID-19\n",
            "COVID-19 pandemic https://en.wikipedia.org/wiki/COVID-19_pandemic\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AP5hQgIV0Fz6"
      },
      "source": [
        "### 2) Train a WordPiece tokenizer on the imported Wikipedia pages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XTYi2eC30Mw7"
      },
      "source": [
        "Source: [All together: a BERT tokenizer from scratch](https://huggingface.co/docs/tokenizers/python/latest/pipeline.html#all-together-a-bert-tokenizer-from-scratch)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CIcwCvJmvBtp"
      },
      "source": [
        "# tokenzer WordPiece\n",
        "from tokenizers import Tokenizer\n",
        "from tokenizers.models import WordPiece\n",
        "\n",
        "bert_tokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\n",
        "\n",
        "# normalizer\n",
        "from tokenizers import normalizers\n",
        "from tokenizers.normalizers import Lowercase, NFD, StripAccents\n",
        "\n",
        "bert_tokenizer.normalizer = normalizers.Sequence([NFD()])\n",
        "\n",
        "# pre-tokenizer\n",
        "from tokenizers.pre_tokenizers import Whitespace\n",
        "\n",
        "bert_tokenizer.pre_tokenizer = Whitespace()\n",
        "\n",
        "# template\n",
        "from tokenizers.processors import TemplateProcessing\n",
        "\n",
        "bert_tokenizer.post_processor = TemplateProcessing(\n",
        "    single=\"[CLS] $A [SEP]\",\n",
        "    pair=\"[CLS] $A [SEP] $B:1 [SEP]:1\",\n",
        "    special_tokens=[\n",
        "        (\"[CLS]\", 1),\n",
        "        (\"[SEP]\", 2),\n",
        "    ],\n",
        ")\n",
        "\n",
        "# instantiate a trainer\n",
        "from tokenizers.trainers import WordPieceTrainer\n",
        "trainer = WordPieceTrainer(\n",
        "    vocab_size=30522, \n",
        "    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"]\n",
        "    )\n",
        "\n",
        "# Train \n",
        "files = documents\n",
        "bert_tokenizer.train_from_iterator(files, trainer)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NsRvZ32q0etr"
      },
      "source": [
        "### 3) Get the vocabulary that is not in the original BERT tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2yxColA0BQ5e"
      },
      "source": [
        "This step is not necessary, as the `tokenizer.add_tokens()` method will add new tokens only if they do not belong to the existing tokenizer vocabulary. However, it helps us to see what these new tokens are."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PNmp_5H7VVIf"
      },
      "source": [
        "old_vocab = [k for k,v in tokenizer.get_vocab().items()]\n",
        "new_vocab = [k for k,v in bert_tokenizer.get_vocab().items()]\n",
        "idx_old_vocab_list = list()\n",
        "same_tokens_list = list()\n",
        "different_tokens_list = list()\n",
        "\n",
        "for idx_new,w in enumerate(new_vocab): \n",
        "  try:\n",
        "    idx_old = old_vocab.index(w)\n",
        "  except:\n",
        "    idx_old = -1\n",
        "  if idx_old>=0:\n",
        "      idx_old_vocab_list.append(idx_old)\n",
        "      same_tokens_list.append((w,idx_new))\n",
        "  else:\n",
        "      different_tokens_list.append((w,idx_new))"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U7rIlrLcxK1u",
        "outputId": "28185ea2-807d-4c6e-ca78-b781145039b3"
      },
      "source": [
        "len(same_tokens_list),len(different_tokens_list),len(same_tokens_list)+len(different_tokens_list)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(4707, 3651, 8358)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q2VXO6LXSp4f"
      },
      "source": [
        "**We found 3651 tokens (subwords or words) that are not in the vocabulary of the original tokenizer.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CL7I-xL4yfJf",
        "outputId": "986d247f-4721-460c-9b74-b5c25e7300e7"
      },
      "source": [
        "# get list of new tokens\n",
        "new_tokens = [k for k,v in different_tokens_list]\n",
        "len(new_tokens), new_tokens[:10]"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3651,\n",
              " ['infusion',\n",
              "  'hotsp',\n",
              "  'pathogenic',\n",
              "  'Pf',\n",
              "  'exudation',\n",
              "  '##rolling',\n",
              "  '634',\n",
              "  '##ipp',\n",
              "  '##ffici',\n",
              "  'tripling'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehq09ep2mQKj"
      },
      "source": [
        "### 4) Add the new tokens (subwords and words) in the vocabulary of the original BERT tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1krFfLlsra1e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0c99656e-8c28-401d-cf03-d61fa18a7091"
      },
      "source": [
        "from transformers import AutoModelForMaskedLM, AutoTokenizer\n",
        "model_name = \"bert-base-cased\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_name)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g-sMnbkHrjx9",
        "outputId": "9354de98-6dc0-4d48-a2df-50e7705f8a34"
      },
      "source": [
        "print(\"[ BEFORE ] tokenizer vocab size:\", len(tokenizer)) \n",
        "added_tokens = tokenizer.add_tokens(new_tokens)\n",
        "\n",
        "print(\"[ AFTER ] tokenizer vocab size:\", len(tokenizer)) \n",
        "print()\n",
        "print('added_tokens:',added_tokens)\n",
        "print()\n",
        "\n",
        "# resize the embeddings matrix of the model \n",
        "model.resize_token_embeddings(len(tokenizer)) "
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ BEFORE ] tokenizer vocab size: 28996\n",
            "[ AFTER ] tokenizer vocab size: 32647\n",
            "\n",
            "added_tokens: 3651\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Embedding(32647, 768)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eKN_fGADOL-Q",
        "outputId": "f83963a4-fa24-4bc8-c345-dadae22c0fee"
      },
      "source": [
        "# Verify if  the words COVID and hospitalization belong or not to the tokenizer vocabulary\n",
        "vocab = [tok for tok,index in tokenizer.get_vocab().items()]\n",
        "\"COVID\" in vocab, \"hospitalization\" in vocab"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(False, False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lPLK_yTNWHlP"
      },
      "source": [
        "Let's call tokenizer_exBERT our tokenizer with the new tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "njAuBZSnWHlR"
      },
      "source": [
        "tokenizer_exBERT = tokenizer"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZrIwgDkGWHlS",
        "outputId": "9169cf0e-82dc-45e5-cd79-12b127ba5d2a"
      },
      "source": [
        "# tokenization of the text\n",
        "tokens = tokenizer_exBERT.tokenize(text)\n",
        "print(tokens)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['CO', '##VI', '##D', '-', '19', 'a', '##f', 'fec', 't', '##s', 'dif', 'fer', 'en', '##t', 'pe', 'o', 'ple', 'in', 'dif', 'fer', 'en', '##t', 'ways', '.', 'Mo', 'st', 'infec', 'te', '##d', 'pe', 'o', 'ple', 'will', 'd', 'ev', 'e', 'lop', 'mil', 'd', 'to', 'mod', 'e', 'ra', 'te', 'illness', 'and', 'rec', 'over', 'without', 'ho', 'sp', 'i', 'tal', 'i', '##zation', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "Fzc1vszBWHlU",
        "outputId": "cf54c6e8-61c3-4b01-868d-88c37a4c6113"
      },
      "source": [
        "# back to text\n",
        "tokenizer_exBERT.decode(tokenizer_exBERT.encode(text), skip_special_tokens=True)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'COVID - 19 af fec ts dif fer ent pe o ple in dif fer ent ways. Mo st infec ted pe o ple will d ev e lop mil d to mod e ra te illness and rec over without ho sp i tal ization.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eb7f6GZ8WHlV"
      },
      "source": [
        "**As the words COVID and hospitalization do not belong to the tokenizer vocabulary, they continue to be tokenized with subwords. That's right.**\n",
        "\n",
        "**However, only the word COVID is well tokenized: the word hospitalization is tokenized with subwords that do not start with ##. But except the first token, all other subword tokens should have started with ##!**\n",
        "\n",
        "**And we can see that many other words in the sentence are not well tokenized, too.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N0N541hTWHlW",
        "outputId": "d7ddd761-fbe9-433f-9be1-361ccf19006f"
      },
      "source": [
        "# tokenization of the words COVID and hospitalization\n",
        "print(tokenizer_exBERT.tokenize('COVID'))\n",
        "print(tokenizer_exBERT.tokenize('hospitalization'))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['CO', '##VI', '##D']\n",
            "['ho', 'sp', 'i', 'tal', 'i', '##zation']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpvHAok0mU0r"
      },
      "source": [
        "### 5) Add only the new tokens that do not start with ## in the vocabulary of the original BERT tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YN6TITaiCBBt"
      },
      "source": [
        "We know that a subword is not just a token that starts with ##, but let's see what happens if we remove all those subwords from the list of new tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yL_L7DxYmgP9",
        "outputId": "9ef81f7e-8a39-44d6-da65-1fd9912090a8"
      },
      "source": [
        "# get list of new tokens as whole words\n",
        "new_tokens = [tok for tok in new_tokens if tok.startswith(\"#\") == False]\n",
        "len(new_tokens), new_tokens[:10]"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2501,\n",
              " ['infusion',\n",
              "  'hotsp',\n",
              "  'pathogenic',\n",
              "  'Pf',\n",
              "  'exudation',\n",
              "  '634',\n",
              "  'tripling',\n",
              "  'produc',\n",
              "  'undernourishment',\n",
              "  'isolate'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2fDb-siInHha",
        "outputId": "01f57acc-7373-4926-8462-236248645e9e"
      },
      "source": [
        "from transformers import AutoModelForMaskedLM, AutoTokenizer\n",
        "model_name = \"bert-base-cased\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_name)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ko6CfvHRnHhc",
        "outputId": "8c61c8d4-4903-42a5-8b82-873d1c588b9e"
      },
      "source": [
        "print(\"[ BEFORE ] tokenizer vocab size:\", len(tokenizer)) \n",
        "added_tokens = tokenizer.add_tokens(new_tokens)\n",
        "\n",
        "print(\"[ AFTER ] tokenizer vocab size:\", len(tokenizer)) \n",
        "print()\n",
        "print('added_tokens:',added_tokens)\n",
        "print()\n",
        "\n",
        "# resize the embeddings matrix of the model \n",
        "model.resize_token_embeddings(len(tokenizer)) "
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ BEFORE ] tokenizer vocab size: 28996\n",
            "[ AFTER ] tokenizer vocab size: 31497\n",
            "\n",
            "added_tokens: 2501\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Embedding(31497, 768)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d95HnOr3nHhe"
      },
      "source": [
        "Let's call tokenizer_exBERT our tokenizer with the new tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Nj1S1WwnHhf"
      },
      "source": [
        "tokenizer_exBERT = tokenizer"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J00Kg8L0nHhg",
        "outputId": "0b4fb761-6183-4db9-9bd5-b9b9c1edfc18"
      },
      "source": [
        "# tokenization of the text\n",
        "tokens = tokenizer_exBERT.tokenize(text)\n",
        "print(tokens)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['CO', '##VI', '##D', '-', '19', 'a', '##f', 'fec', 't', '##s', 'dif', 'fer', 'en', '##t', 'pe', 'o', 'ple', 'in', 'dif', 'fer', 'en', '##t', 'ways', '.', 'Mo', 'st', 'infec', 'te', '##d', 'pe', 'o', 'ple', 'will', 'd', 'ev', 'e', 'lop', 'mil', 'd', 'to', 'mod', 'e', 'ra', 'te', 'illness', 'and', 'rec', 'over', 'without', 'ho', 'sp', 'i', 'tal', 'i', '##zation', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "2hPy_t1snHhh",
        "outputId": "edab46dd-ef11-46fc-b91a-b1b1f3f3f52f"
      },
      "source": [
        "# back to text\n",
        "tokenizer_exBERT.decode(tokenizer_exBERT.encode(text), skip_special_tokens=True)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'COVID - 19 af fec ts dif fer ent pe o ple in dif fer ent ways. Mo st infec ted pe o ple will d ev e lop mil d to mod e ra te illness and rec over without ho sp i tal ization.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXUmp1COnHhi"
      },
      "source": [
        "**The tokenizer continues to fail!**\n",
        "\n",
        "**It means that we must improve the new tokens list by taking out as well the subwords that begin a word (ie, they don't start by ##).**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E1Vn7QEPnHhj",
        "outputId": "ccf51db5-d0c5-4168-ced0-b09de7fc43a9"
      },
      "source": [
        "# tokenization of the words COVID and hospitalization\n",
        "print(tokenizer_exBERT.tokenize('COVID'))\n",
        "print(tokenizer_exBERT.tokenize('hospitalization'))"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['CO', '##VI', '##D']\n",
            "['ho', 'sp', 'i', 'tal', 'i', '##zation']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWzH7OfSDLUp"
      },
      "source": [
        "## [ Third test ] Add new tokens (only words, not subwords) into the tokenizer vocab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJO7bBUPyG2D"
      },
      "source": [
        "Let's add only the new tokens that are words, not subwords (that do not start with ## or do not are followed by a subword with ##) in the vocabulary of the original BERT tokenizer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uY4pga75DlTt"
      },
      "source": [
        "### 1) Let's use a word tokenizer (spaCY) to find the most frequent words of our corpus by using scikit-learn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CoXcomhe103D"
      },
      "source": [
        "**Yes but how?** Let's use a **words tokenizer like spaCY** to find the most frequent words of our corpus instead of a WordPiece tokenizer which generates subwords as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gx6_R5TCWUQP"
      },
      "source": [
        "**Observation**: here, the expression \"most frequent words\" means: the tokens present in most of the documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-pEZGsG4xSJj"
      },
      "source": [
        "import spacy\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5q6AxoWWxL9c"
      },
      "source": [
        "# initialize our tokenizer with the English spaCY one\n",
        "nlp = spacy.load(\"en_core_web_sm\", exclude=['morphologizer', 'parser', 'ner', 'attribute_ruler', 'lemmatizer'])"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vx0sA1jNRZH1"
      },
      "source": [
        "def spacy_tokenizer(document, nlp=nlp):\n",
        "    # tokenize the document with spaCY\n",
        "    doc = nlp(document)\n",
        "    # Remove stop words and punctuation symbols\n",
        "    tokens = [\n",
        "        token.text for token in doc if (\n",
        "        token.is_stop == False and \\\n",
        "        token.is_punct == False and \\\n",
        "        token.text.strip() != '' and \\\n",
        "        token.text.find(\"\\n\") == -1)]\n",
        "    return tokens\n",
        "\n",
        "def dfreq(idf, N):\n",
        "    return (1+N) / np.exp(idf - 1) - 1"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "efQustrf2eCd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "38fb1907-cb6f-4ac9-d3ca-19cdd6abd6af"
      },
      "source": [
        "%%time\n",
        "# https://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting\n",
        "tfidf_vectorizer = TfidfVectorizer(lowercase=False, tokenizer=spacy_tokenizer, \n",
        "                                   norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=False)\n",
        "# parse matrix of tfidf\n",
        "docs = documents\n",
        "length = len(docs)\n",
        "result = tfidf_vectorizer.fit_transform(docs)\n",
        "# print(result.shape)\n",
        "\n",
        "# idf\n",
        "idf = tfidf_vectorizer.idf_\n",
        "\n",
        "# sorted idf, tokens and docs frequencies\n",
        "idf_sorted_indexes = sorted(range(len(idf)), key=lambda k: idf[k])\n",
        "idf_sorted = idf[idf_sorted_indexes]\n",
        "tokens_by_df = np.array(tfidf_vectorizer.get_feature_names())[idf_sorted_indexes]\n",
        "dfreqs_sorted = dfreq(idf_sorted, length).astype(np.int32)\n",
        "tokens_dfreqs = {tok:dfreq for tok, dfreq in zip(tokens_by_df,dfreqs_sorted)}\n",
        "tokens_pct_list = [int(round(dfreq/length*100,2)) for token,dfreq in tokens_dfreqs.items()]"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 1.22 s, sys: 16.3 ms, total: 1.23 s\n",
            "Wall time: 1.25 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L7_K5rnqVR3s"
      },
      "source": [
        "# we have only 2 documents (that's why we range the intervale [1,101] with a step of 50)\n",
        "number_tokens_with_DF_above_pct = list()\n",
        "for pct in range(1,101,50):\n",
        "    index_max = len(np.array(tokens_pct_list)[np.array(tokens_pct_list)>=pct])\n",
        "    number_tokens_with_DF_above_pct.append(index_max)"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 111
        },
        "id": "-MunEm7_Rmnj",
        "outputId": "3618f2b2-be7c-429d-d0ea-86d7cb0f9f0a"
      },
      "source": [
        "# DF = Document Frequency\n",
        "\n",
        "# df_docfreqs = pd.DataFrame(number_tokens_with_DF_above_pct, columns=['number of tokens with DF above x%'])\n",
        "# df_docfreqs.index += 1 \n",
        "# df_docfreqs.transpose()\n",
        "\n",
        "# plt.plot(number_tokens_with_DF_above_pct)\n",
        "# plt.title(f'Document Frequency above of {pct}%')\n",
        "# plt.show()\n",
        "\n",
        "df_docfreqs = pd.DataFrame({'pct':list(range(1,101,50)),'number of tokens with DF above pct%':number_tokens_with_DF_above_pct})\n",
        "df_docfreqs.transpose()"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>pct</th>\n",
              "      <td>1</td>\n",
              "      <td>51</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>number of tokens with DF above pct%</th>\n",
              "      <td>4186</td>\n",
              "      <td>1058</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        0     1\n",
              "pct                                     1    51\n",
              "number of tokens with DF above pct%  4186  1058"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0yD7cdavZ4b9"
      },
      "source": [
        "**There are 4186 words which appear in one or two documents from our 2 documents list, and 1058 which are in the 2 documents.**\n",
        "\n",
        "**Let's consider that the 4186 words are all important and relevant to our COVID corpus.**\n",
        "\n",
        "**Observation**: within a corpus with more documents, we could have used another rule as for example: keeping only words which are at least in 10% of the documents list."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oW3KDhoQ7w2R"
      },
      "source": [
        "### Get the vocabulary that is not in the original BERT tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IMxPFT1XK_E"
      },
      "source": [
        "This step is not necessary, as the `tokenizer.add_tokens()` method will add new tokens only if they do not belong to the existing tokenizer vocabulary. However, it helps us to see what these new tokens are."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pYJR9G_97w2T"
      },
      "source": [
        "# list of new tokens\n",
        "pct = 1\n",
        "index_max = len(np.array(tokens_pct_list)[np.array(tokens_pct_list)>=pct])\n",
        "new_tokens = tokens_by_df[:index_max]\n",
        "# print(len(new_tokens))\n",
        "\n",
        "old_vocab = [k for k,v in tokenizer.get_vocab().items()]\n",
        "new_vocab = [token for token in new_tokens]\n",
        "idx_old_vocab_list = list()\n",
        "same_tokens_list = list()\n",
        "different_tokens_list = list()\n",
        "\n",
        "for idx_new,w in enumerate(new_vocab): \n",
        "  try:\n",
        "    idx_old = old_vocab.index(w)\n",
        "  except:\n",
        "    idx_old = -1\n",
        "  if idx_old>=0:\n",
        "      idx_old_vocab_list.append(idx_old)\n",
        "      same_tokens_list.append((w,idx_new))\n",
        "  else:\n",
        "      different_tokens_list.append((w,idx_new))"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZQwh9j6b7w2U",
        "outputId": "8b41253f-0410-40cb-e67c-183224e4cc1f"
      },
      "source": [
        "len(same_tokens_list),len(different_tokens_list),len(same_tokens_list)+len(different_tokens_list)"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3960, 226, 4186)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L69ZMFu47w2W"
      },
      "source": [
        "**We found 226 tokens (whole words) that are not in the vocabulary of the original tokenizer, and the words COVID and hospitalization belong to the new tokens list.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mYB3RcY67w2Y",
        "outputId": "a098ebc0-9cc4-4ddb-9c22-1f26bede4ae8"
      },
      "source": [
        "# get list of new tokens\n",
        "new_tokens = [k for k,v in different_tokens_list]\n",
        "print(len(new_tokens), new_tokens[:20])"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "226 ['0.002', '0.01', '0.1', '0.4', '0.5', '0.5–1', '1.4', '1.7', '2.2', '202012/01', '4.6', '50,000', '6,174', 'B.1.1.7', 'COVID-19', 'CoV-2', 'CoV.', 'P.1', 'U.S.', 'U07.1']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IBiR25A-ZksW",
        "outputId": "15cc67ee-dfbe-4e7d-b44c-6790623697c3"
      },
      "source": [
        "\"COVID\" in new_tokens, \"hospitalization\" in new_tokens"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(True, True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KILEkSFlB_53"
      },
      "source": [
        "### Add the new tokens (only whole words, not subwords!) in the vocabulary of the original BERT tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7yXmxO4sBcsM",
        "outputId": "5dbe7c65-ab59-44dd-8a34-b899df80c1c7"
      },
      "source": [
        "# import model and tokenizer\n",
        "from transformers import AutoModelForMaskedLM, AutoTokenizer\n",
        "model_name = \"bert-base-cased\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_name)"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cJJyqSeRBcsP",
        "outputId": "8f9ab271-f9e1-445d-b41e-30f823128631"
      },
      "source": [
        "print(\"[ BEFORE ] tokenizer vocab size:\", len(tokenizer)) \n",
        "added_tokens = tokenizer.add_tokens(new_tokens)\n",
        "\n",
        "print(\"[ AFTER ] tokenizer vocab size:\", len(tokenizer)) \n",
        "print()\n",
        "print('added_tokens:',added_tokens)\n",
        "print()\n",
        "\n",
        "# resize the embeddings matrix of the model \n",
        "model.resize_token_embeddings(len(tokenizer)) "
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ BEFORE ] tokenizer vocab size: 28996\n",
            "[ AFTER ] tokenizer vocab size: 29222\n",
            "\n",
            "added_tokens: 226\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Embedding(29222, 768)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bX4x_NCdBcsR"
      },
      "source": [
        "Let's call tokenizer_exBERT our tokenizer with the new tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QeTmSR92BcsS"
      },
      "source": [
        "tokenizer_exBERT = tokenizer"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8vmGm43nBcsT",
        "outputId": "67ea20fa-e763-4daf-ede9-af8da5f9dfe3"
      },
      "source": [
        "# tokenization of the text\n",
        "tokens = tokenizer_exBERT.tokenize(text)\n",
        "print(tokens)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['COVID-19', 'affects', 'different', 'people', 'in', 'different', 'ways', '.', 'Most', 'infected', 'people', 'will', 'develop', 'mild', 'to', 'moderate', 'illness', 'and', 'recover', 'without', 'hospitalization', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "HN9Qf_L3BcsU",
        "outputId": "c7487150-00e3-4f9f-c650-11f3d6f2141c"
      },
      "source": [
        "# back to text\n",
        "tokenizer_exBERT.decode(tokenizer_exBERT.encode(text), skip_special_tokens=True)"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'COVID-19 affects different people in different ways. Most infected people will develop mild to moderate illness and recover without hospitalization.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n4LXqD3rBcsZ"
      },
      "source": [
        "**The tokenizer with the new tokens (only whole words!) did succeed in tokenizing the words COVID and hospitalization correctly (and not only these ones: all of them!)**\n",
        "\n",
        "**It means that is fundamental to add new tokens that are only whole words to an existing subword tokenizer like WordPiece, and not subwords!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LYqykZLLBcsb",
        "outputId": "29c530df-807f-4b1e-e2bb-4cbcdec49ed4"
      },
      "source": [
        "# tokenization of the words COVID and hospitalization\n",
        "print(tokenizer_exBERT.tokenize('COVID'))\n",
        "print(tokenizer_exBERT.tokenize('hospitalization'))"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['COVID']\n",
            "['hospitalization']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hr7NGchV5X6I"
      },
      "source": [
        "## Let's check the impact of our enriched tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PA31nW1s8Fng"
      },
      "source": [
        "Let's use a text about COVID taken from a newspaper site (not from Wikipedia)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMIaazSd5j31"
      },
      "source": [
        "# source: https://edition.cnn.com/2021/04/05/health/us-coronavirus-monday/index.html\n",
        "text = 'Experts say Covid-19 vaccinations in the US are going extremely well -- but not enough people are protected yet and the country may be at the start of another surge. \\\n",
        "The US reported a record over the weekend with more than 4 million Covid-19 vaccine doses administered in 24 hours, according to the Centers for Disease Control and Prevention. \\\n",
        "And the country now averages more than 3 million doses daily, according to CDC data. \\\n",
        "But only about 18.5% of Americans are fully vaccinated, CDC data shows, and Covid-19 cases in the country have recently seen concerning increases. \\\n",
        "\"I do think we still have a few more rough weeks ahead,\" Dr. Celine Gounder, an infectious diseases specialist and epidemiologist, told CNN on Sunday. \\\n",
        "\"What we know from the past year of the pandemic is that we tend to trend about three to four weeks behind Europe in terms of our pandemic patterns.\"'"
      ],
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7xyUZXSn8PhV"
      },
      "source": [
        "Now, let's tokenize this text both with the original BERT tokenizer and its enriched version."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IesgAZnE8haq"
      },
      "source": [
        "from transformers import AutoModelForMaskedLM, AutoTokenizer\n",
        "model_name = \"bert-base-cased\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cSlyHJ-O7NuT",
        "outputId": "fe5a5d70-125b-474a-e3a7-082e40ea5e85"
      },
      "source": [
        "tokens = tokenizer.tokenize(text)\n",
        "print('number of tokens by the original BERT tokenizer:', len(tokens))\n",
        "\n",
        "tokens = tokenizer_exBERT.tokenize(text)\n",
        "print('number of tokens by the enriched tokenizer:', len(tokens))"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "number of tokens by the original BERT tokenizer: 203\n",
            "number of tokens by the enriched tokenizer: 193\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LxNQ6ROO8qeN"
      },
      "source": [
        "**As expected, we find that the enriched tokenizer needs less tokens (here, 5%) to tokenize the text on COVID than the original BERT tokenizer.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PHXgeeC-l-lJ"
      },
      "source": [
        "## To be continued..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OkTSGj-doRdb"
      },
      "source": [
        "Now that we have augmented our tokenizer vocabulary with words specific to our corpus, we need to fine-tune the natural language model it is associated with (here, the bert-base-cased model). Indeed, the addition of new words led to the increase of the matrix of embeddings of the model by the same number: **with each new word added, a new vector of embeddings with random values was added as well** thanks to the `model.resize_token_embeddings(len(tokenizer))` method. So we need to train (or fine-tune) our model on our body so that the model can learn the embeddings of these new words.\n",
        "\n",
        "Hugging Face provided a script and a notebook to fine tune a natural language model on a new corpus (*How to fine-tune a model on language modeling*: [script](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) | [github](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) | [colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb)). **We therefore have a ready-to-use code. However, it is possible that this code is not adapted to your situation** because if the number of new words (and therefore of new embeddings vectors) is high, it is possible that the training by this code leads to a Catastrophic Forgetting by modifying in a sensitive way the vectors of embeddings of the tokens of the initial vocabulary.\n",
        "\n",
        "**My advice**: do a Google search with this type of \"*fine-tune a pre-trained model for a specific domain*\" query. You will get all the interesting articles and documents on this topic. Good job to you!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "itMeujJv14E0"
      },
      "source": [
        "# END"
      ]
    }
  ]
}