Chinese_Poem_Search_based_on_GuwenBERT_and_ScaNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Chinese Poem Search based on GuwenBERT and ScaNN",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm",
      "authorship_tag": "ABX9TyNe3gK6hg26D/T98eF8JmSs",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "607348976b844e8ca2f15ffef04ed2ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_75134c7de0714e33a5b846506fc6a354",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_4e62cd9328e247e5b280314ac2f41551",
              "IPY_MODEL_e4f6b6b4aa3e4f59b98d88da2dd8878a",
              "IPY_MODEL_4e2f4fd19a8645269bb3c884ec184c2a"
            ]
          }
        },
        "75134c7de0714e33a5b846506fc6a354": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "4e62cd9328e247e5b280314ac2f41551": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_1a46711ed65542f5bd73791006b186d4",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "Dynasty tang: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_e014d81915174ba49b71d47b7ecb2d43"
          }
        },
        "e4f6b6b4aa3e4f59b98d88da2dd8878a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_302eeb232c3a440f90a27bea4785a8cc",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "",
            "max": 58,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 58,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_ae36f3a38a2d4ea499a26193729b3ca5"
          }
        },
        "4e2f4fd19a8645269bb3c884ec184c2a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_c9cb35cca13642b3872dd9861c3be312",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 58/58 [00:14&lt;00:00,  4.41it/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_fb5dfeeb6a8e4789ad0906ec983d8431"
          }
        },
        "1a46711ed65542f5bd73791006b186d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "e014d81915174ba49b71d47b7ecb2d43": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "302eeb232c3a440f90a27bea4785a8cc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "ae36f3a38a2d4ea499a26193729b3ca5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "c9cb35cca13642b3872dd9861c3be312": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "fb5dfeeb6a8e4789ad0906ec983d8431": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "b7b37039c2cd4ee9aeea7e8795be5539": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_54b979080ce94d0886fbff43fdbdc8da",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_d504c5569b564713b041af2f165a9a77",
              "IPY_MODEL_a940111be892426a91ad8763b2a735fd",
              "IPY_MODEL_47a08cd829a9406cb1175e04d9e95487"
            ]
          }
        },
        "54b979080ce94d0886fbff43fdbdc8da": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "d504c5569b564713b041af2f165a9a77": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_16309ab3881b4dbeb8b1b1e4fc64f64b",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "Dynasty song: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_047d003c14914c808d71eabd2d38acbb"
          }
        },
        "a940111be892426a91ad8763b2a735fd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_675987f4064d4327bab2d80435efaf87",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "",
            "max": 255,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 255,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_19196c75894648388b52b7e73f0023b9"
          }
        },
        "47a08cd829a9406cb1175e04d9e95487": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_2386fb9b857b4cf1b846d8b24142b049",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 255/255 [01:05&lt;00:00,  3.59it/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_77284cf0cc8a43aca9dd14f4851901d1"
          }
        },
        "16309ab3881b4dbeb8b1b1e4fc64f64b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "047d003c14914c808d71eabd2d38acbb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "675987f4064d4327bab2d80435efaf87": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "19196c75894648388b52b7e73f0023b9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "2386fb9b857b4cf1b846d8b24142b049": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "77284cf0cc8a43aca9dd14f4851901d1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "bd3dd2645e1847c581c18d95d0a2b0d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_f18e98dc55dd4577aaa57c3d8f21544a",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_cebc2381b49e41779fe1b56fdea42a6d",
              "IPY_MODEL_6528516ac48a44b19651733593437e51",
              "IPY_MODEL_f6aca07378d541ebb953da340df492c0"
            ]
          }
        },
        "f18e98dc55dd4577aaa57c3d8f21544a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "cebc2381b49e41779fe1b56fdea42a6d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_a80a14ae80694804b2966089674db98b",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "break up into sentences: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_b3f30be984754a6988f34c9d88b51477"
          }
        },
        "6528516ac48a44b19651733593437e51": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_02c26c9b513d4eb4bfcac7137512c8b0",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "",
            "max": 311860,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 311860,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_7f09e900e3444f23bca8424ab368d4a6"
          }
        },
        "f6aca07378d541ebb953da340df492c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_038a001e97534a8d8973556dc71d08cf",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 311860/311860 [00:55&lt;00:00, 8709.26it/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_25c3a76a4a1b401d84a103373636ced4"
          }
        },
        "a80a14ae80694804b2966089674db98b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "b3f30be984754a6988f34c9d88b51477": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "02c26c9b513d4eb4bfcac7137512c8b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "7f09e900e3444f23bca8424ab368d4a6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "038a001e97534a8d8973556dc71d08cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "25c3a76a4a1b401d84a103373636ced4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "d7e095275dff4309a79bef080fcfe3e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_ff070ece72ba4315895d16dc5848cbb8",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_a03c308c29f84bae8d80433023bc5745",
              "IPY_MODEL_c41512860f014e7dbb9ecd2bbaa0e145",
              "IPY_MODEL_80836c9367504f08a50a76f920383c17"
            ]
          }
        },
        "ff070ece72ba4315895d16dc5848cbb8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "a03c308c29f84bae8d80433023bc5745": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_83997112934345169c14eae5bed613d5",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "Downloading: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_ad179e3f9bdb4a1aac0497ec598d5221"
          }
        },
        "c41512860f014e7dbb9ecd2bbaa0e145": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_09965f59b5d4401b9d4c8694c8b99da6",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 519,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 519,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_24ec8e1691f84189b1750a826312971c"
          }
        },
        "80836c9367504f08a50a76f920383c17": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_d1637f7d80494a4ca354ec58e7731fa8",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 519/519 [00:00&lt;00:00, 19.1kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_ca34e05a674b4e828f0459d541d6df42"
          }
        },
        "83997112934345169c14eae5bed613d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "ad179e3f9bdb4a1aac0497ec598d5221": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "09965f59b5d4401b9d4c8694c8b99da6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "24ec8e1691f84189b1750a826312971c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "d1637f7d80494a4ca354ec58e7731fa8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "ca34e05a674b4e828f0459d541d6df42": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "766f659cfb5d4e8682f045ce65469845": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_c9bb67f1a3d747d886ea3453ed514a91",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_a5d1b0755c7f4726a0ec9e19f5a3528d",
              "IPY_MODEL_5807417077f74541ba85f30388a8da53",
              "IPY_MODEL_f3b5cb6c582e4023829e7cc45446ea4e"
            ]
          }
        },
        "c9bb67f1a3d747d886ea3453ed514a91": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "a5d1b0755c7f4726a0ec9e19f5a3528d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_7f2e06cc5e524164bd9f3ffd3ec9fa47",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "Downloading: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_a43e2247711240118d55d66e7f43bb26"
          }
        },
        "5807417077f74541ba85f30388a8da53": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_95526d25a2fe4e57b072034eb33c6a16",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 93479,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 93479,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_cb87bbc8ac8e45aba8a7ede3694b5999"
          }
        },
        "f3b5cb6c582e4023829e7cc45446ea4e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_144843fbeff24104b867dd85672e8d0a",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 93.5k/93.5k [00:00&lt;00:00, 1.50MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_8e9e9f18b9c94d75ae8b084b79a0d573"
          }
        },
        "7f2e06cc5e524164bd9f3ffd3ec9fa47": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "a43e2247711240118d55d66e7f43bb26": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "95526d25a2fe4e57b072034eb33c6a16": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "cb87bbc8ac8e45aba8a7ede3694b5999": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "144843fbeff24104b867dd85672e8d0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "8e9e9f18b9c94d75ae8b084b79a0d573": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "ab62bb42114348a29a4bd892b2f96460": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_62dd82a5caa94bec928dbd9c45f78f2e",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_8da2456942d0469ead0b080866028eff",
              "IPY_MODEL_2af326464adc4780833e882298d65654",
              "IPY_MODEL_3a8395b78e2b49a2b7bdc0cbc6c6448d"
            ]
          }
        },
        "62dd82a5caa94bec928dbd9c45f78f2e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "8da2456942d0469ead0b080866028eff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_afcbbf74944b45259bdd7771fa6992f4",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "Downloading: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_2e1b79a637244be4a33ceabf5b873fd8"
          }
        },
        "2af326464adc4780833e882298d65654": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_c489d75c37b044a2bdde71f3a0634bf8",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 418326249,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 418326249,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_9314230a17bf4413b8c46af688e23296"
          }
        },
        "3a8395b78e2b49a2b7bdc0cbc6c6448d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_5a99cdae60ca4706bb8528fd1f5ff36f",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 418M/418M [00:07&lt;00:00, 58.4MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_8efab33f22b8486598723f08a3f5328c"
          }
        },
        "afcbbf74944b45259bdd7771fa6992f4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "2e1b79a637244be4a33ceabf5b873fd8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "c489d75c37b044a2bdde71f3a0634bf8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "9314230a17bf4413b8c46af688e23296": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "5a99cdae60ca4706bb8528fd1f5ff36f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "8efab33f22b8486598723f08a3f5328c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "9c70f870e9904902b80e82164ddc86bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_cc355189d55a4c6287bb2833170d0fe4",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_3cf09109ef4048cd896701e51c9c75b2",
              "IPY_MODEL_c4ac33d857374095b107ab988cb007ad",
              "IPY_MODEL_99152cb0a847437e8e40369484f9b3b9"
            ]
          }
        },
        "cc355189d55a4c6287bb2833170d0fe4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "3cf09109ef4048cd896701e51c9c75b2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_5eb551a593da41698c9a2eb5c4bb9f35",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": "Fetch embeddings: 100%",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_b84f72f590b5431780eb396b30e03c05"
          }
        },
        "c4ac33d857374095b107ab988cb007ad": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_6ce259fc74844c7798847c3c78476881",
            "_dom_classes": [],
            "description": "",
            "_model_name": "FloatProgressModel",
            "bar_style": "",
            "max": 1340301,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 1340301,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_8c9163e362074af094d511794eb04865"
          }
        },
        "99152cb0a847437e8e40369484f9b3b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_6c406df96ed64ef3a6a2d2208b190f62",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 1340301/1340301 [24:17&lt;00:00, 703.27it/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_9222b79cab814a43992137215de2acfc"
          }
        },
        "5eb551a593da41698c9a2eb5c4bb9f35": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "b84f72f590b5431780eb396b30e03c05": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "6ce259fc74844c7798847c3c78476881": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "8c9163e362074af094d511794eb04865": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "6c406df96ed64ef3a6a2d2208b190f62": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "9222b79cab814a43992137215de2acfc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hululuzhu/chinese-poem-search/blob/main/Chinese_Poem_Search_based_on_GuwenBERT_and_ScaNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLqw917vqOLw"
      },
      "source": [
        "# This is an end-to-end colab to search for most relevant Chinese tang/song dynasty poems given your string query.\n",
        "\n",
        "Sample\n",
        "\n",
        "```\n",
        "    输入 你看那长江的水从天上来\n",
        "      1: 李白 [鼓吹曲辞将进酒]\n",
        "      君不见黄河之水天上来，奔流到海不复回．．．\n",
        "      2: 马之纯 [新亭其二]\n",
        "      新亭见说在山头，看见江河衮衮流．．．\n",
        "      3: 释善果 [偈其五]\n",
        "      苏州有，常州有，吸尽西江只一口．．．\n",
        "\n",
        "\n",
        "    输入：忆长安\n",
        "      1: 宋祁 [农阁]\n",
        "      ...看云记巫峡，望日省长安。...\n",
        "      2: 徐凝 [寄白司马]\n",
        "      ...争遣江州白司马，五年风景忆长安。\n",
        "      3: 崔涂 [春晚怀进士韦澹]\n",
        "      ...二年春怅望，不似在长安。\n",
        "```\n",
        "\n",
        "Some notes:\n",
        "\n",
        "*   Based on [guwenBERT](https://huggingface.co/ethanyt/guwenbert-base) (an ancient chinese pre-trained [RoBERTa](https://arxiv.org/abs/1907.11692) language model), [HF Transformers](https://github.com/huggingface/transformers) (model inference), and [Google sanNN](https://github.com/google-research/google-research/tree/master/scann) (approximate nearest neighbor search)\n",
        "*   We fetch chinese poems from this [chinese-poetry github project](https://github.com/chinese-poetry/chinese-poetry), and divide to sentence pieces\n",
        "*   Converted to simplified Chinese input using [chinese-converter package](https://github.com/zachary822/chinese-converter), please skip if you prefer traditional Chinese\n",
        "*   Use the last layer hidden output as embedding to balance quality and memory constraints, [literature](http://jalammar.github.io/illustrated-bert/) recommends last 4 layers but cannot afford memory\n",
        "*   Note the colab runs successfully with high RAM GPU colab instance (paid class, Tesla P100 GPU with 16G GPU ram, 2 core CPU with 24G ram). If you encourter with OOM issue, reduce the `SAMPLE_SIZE` value will help, or consider to upgrade to paid colab class ($9.99 per month with awesome GPU!!).\n",
        "*   Colab takes about 1 hour to fully load/transform/initialize, after that, the inference should take <100ms per call.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKi5pzXVpuuk"
      },
      "source": [
        "## Verify GPU is used\n",
        "\n",
        "\n",
        "\n",
        "*   Embedding lookup is pretty slow for CPU\n",
        "*   HuggingFace AutoModel does not work with TPU easily\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g2nriByN5eRr",
        "outputId": "b9dc03c9-4a8a-4074-d240-afd1de9a4a39"
      },
      "source": [
        "!nvidia-smi -L"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GPU 0: Tesla P100-PCIE-16GB (UUID: GPU-68fe9d9b-6ec5-1df8-3117-cd0e19d71493)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xPxIdWfop8oB"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c214ftqjxu5C",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b5baf508-d733-4d4d-9ff6-060e42aef1ce"
      },
      "source": [
        "import json\n",
        "import urllib.request\n",
        "!pip install -q \"tqdm>=4.36.1\" > /tmp/na\n",
        "from tqdm.notebook import tqdm\n",
        "!pip install chinese-converter > /tmp/na\n",
        "import chinese_converter\n",
        "import pickle\n",
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import gc\n",
        "import sys\n",
        "import re\n",
        "!pip install -q transformers\n",
        "from transformers import AutoTokenizer, AutoModel\n",
        "import torch\n",
        "!pip install -q scann\n",
        "import scann\n",
        "from google.colab import drive\n",
        "import tensorflow as tf\n",
        "from tensorboard.plugins import projector\n",
        "%load_ext tensorboard"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The tensorboard extension is already loaded. To reload it, use:\n",
            "  %reload_ext tensorboard\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BfgHSN1bqfHS"
      },
      "source": [
        "## Fetch Data from github"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ApFU9yJKx-ws"
      },
      "source": [
        "# https://github.com/chinese-poetry/chinese-poetry\n",
        "POEM_CONTENT = {\n",
        "    'tang': {\n",
        "        'total': 58,\n",
        "        'pattern': \"https://raw.githubusercontent.com/chinese-poetry/chinese-poetry/master/json/poet.tang.{0}.json\"\n",
        "    },\n",
        "    'song': {\n",
        "        'total': 255,\n",
        "        'pattern': \"https://raw.githubusercontent.com/chinese-poetry/chinese-poetry/master/json/poet.song.{0}.json\"\n",
        "    }\n",
        "}\n",
        "\n",
        "def get_poems(is_test=True, verbose=True):\n",
        "  df_list = []\n",
        "  for dynasty in POEM_CONTENT:\n",
        "    size = 3 if is_test else POEM_CONTENT[dynasty]['total']\n",
        "    pbar = tqdm(total=size, desc=\"Dynasty \" + dynasty)\n",
        "    for i in range(size):\n",
        "      url = POEM_CONTENT[dynasty]['pattern'].format(i * 1000)\n",
        "      if verbose:\n",
        "        print(f\"download {url} now\")\n",
        "      df_list.append(pd.read_json(url))\n",
        "      pbar.update(1)\n",
        "  return pd.concat(df_list)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81,
          "referenced_widgets": [
            "607348976b844e8ca2f15ffef04ed2ea",
            "75134c7de0714e33a5b846506fc6a354",
            "4e62cd9328e247e5b280314ac2f41551",
            "e4f6b6b4aa3e4f59b98d88da2dd8878a",
            "4e2f4fd19a8645269bb3c884ec184c2a",
            "1a46711ed65542f5bd73791006b186d4",
            "e014d81915174ba49b71d47b7ecb2d43",
            "302eeb232c3a440f90a27bea4785a8cc",
            "ae36f3a38a2d4ea499a26193729b3ca5",
            "c9cb35cca13642b3872dd9861c3be312",
            "fb5dfeeb6a8e4789ad0906ec983d8431",
            "b7b37039c2cd4ee9aeea7e8795be5539",
            "54b979080ce94d0886fbff43fdbdc8da",
            "d504c5569b564713b041af2f165a9a77",
            "a940111be892426a91ad8763b2a735fd",
            "47a08cd829a9406cb1175e04d9e95487",
            "16309ab3881b4dbeb8b1b1e4fc64f64b",
            "047d003c14914c808d71eabd2d38acbb",
            "675987f4064d4327bab2d80435efaf87",
            "19196c75894648388b52b7e73f0023b9",
            "2386fb9b857b4cf1b846d8b24142b049",
            "77284cf0cc8a43aca9dd14f4851901d1"
          ]
        },
        "id": "pQf1tWhZyAWg",
        "outputId": "cff8c091-ee1a-4529-b855-6d14d0919839"
      },
      "source": [
        "df = get_poems(is_test=False, verbose=False)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "607348976b844e8ca2f15ffef04ed2ea",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Dynasty tang:   0%|          | 0/58 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b7b37039c2cd4ee9aeea7e8795be5539",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Dynasty song:   0%|          | 0/255 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDIGpbSqqm6x"
      },
      "source": [
        "## Transform to simplified Chinese and cleaning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_i0amuvyFfD"
      },
      "source": [
        "df['concat_paragraphs'] = [''.join(map(str, l)) for l in df['paragraphs']]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W05fzSr7ydMm"
      },
      "source": [
        "def convert_schinese(tchinese):\n",
        "  return chinese_converter.to_simplified(tchinese)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oOq80VfRyfOm"
      },
      "source": [
        "df['s_content'] = df.apply(lambda row: convert_schinese(''.join(row.concat_paragraphs)), axis=1)\n",
        "df['s_title'] = df.apply(lambda row: convert_schinese(''.join(row.title)), axis=1)\n",
        "df['s_author'] = df.apply(lambda row: convert_schinese(''.join(row.author)), axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ywGpoUSXyQSf"
      },
      "source": [
        "my_df = df[['s_content', 's_title', 's_author']].copy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "bd3dd2645e1847c581c18d95d0a2b0d5",
            "f18e98dc55dd4577aaa57c3d8f21544a",
            "cebc2381b49e41779fe1b56fdea42a6d",
            "6528516ac48a44b19651733593437e51",
            "f6aca07378d541ebb953da340df492c0",
            "a80a14ae80694804b2966089674db98b",
            "b3f30be984754a6988f34c9d88b51477",
            "02c26c9b513d4eb4bfcac7137512c8b0",
            "7f09e900e3444f23bca8424ab368d4a6",
            "038a001e97534a8d8973556dc71d08cf",
            "25c3a76a4a1b401d84a103373636ced4"
          ]
        },
        "id": "Y8t0cZQqer_j",
        "outputId": "c65474e8-ad2a-47e4-ab37-8a0b46a0b79f"
      },
      "source": [
        "SPLIT_STR = '。|！|？|\\t|\\n|\\r'\n",
        "converted_sents = []\n",
        "pbar = tqdm(total=len(df), desc=\"break up into sentences\")\n",
        "for idx, row in df.iterrows():\n",
        "  res = re.split(SPLIT_STR, row['s_content'])\n",
        "  for s in res:\n",
        "    if s.strip() != \"\":\n",
        "      converted_sents.append({\n",
        "          \"s_content\": row['s_content'],\n",
        "          'piece': s,\n",
        "          's_author': row['s_author'],\n",
        "          's_title': row['s_title'],\n",
        "      })\n",
        "  pbar.update(1)\n",
        "\n",
        "sent_pd = pd.DataFrame(converted_sents)\n",
        "sent_pd['piece_len'] = sent_pd.piece.str.len()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bd3dd2645e1847c581c18d95d0a2b0d5",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "break up into sentences:   0%|          | 0/311860 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "98C7o3PXiMC_"
      },
      "source": [
        "# Empirically set thresholds of 'valid' poem sentence pieces\n",
        "MAX_SENTENCE_LENGTH = 30\n",
        "MIN_SENTENCE_LENGTH = 8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q8ugRGsViBET"
      },
      "source": [
        "clean_sent_pd = sent_pd[sent_pd.piece_len <= MAX_SENTENCE_LENGTH].copy()\n",
        "clean_sent_pd = clean_sent_pd[clean_sent_pd.piece_len >= MIN_SENTENCE_LENGTH].copy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h1SVBtcoiHFW",
        "outputId": "f819b934-b309-42c3-9caf-c6ec18c2417c"
      },
      "source": [
        "my_df = clean_sent_pd.copy()\n",
        "len(my_df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1354444"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3eNMwcDQyGqJ",
        "outputId": "a7ab2b6b-9bba-416d-fbcf-5c7c8d6a9afc"
      },
      "source": [
        "# omit bad chars\n",
        "OMIT_CHARS = \"()（）[]［］●⿰〔〕〖〗［］Ｂ=/…「」x{}《》、”：0123456789○『』\"\n",
        "\n",
        "def trim_author_fn(row):\n",
        "  return row.s_author[:4]\n",
        "\n",
        "def trim_title_fn(row):\n",
        "  trimed_title = row.s_title[:12].replace(\" \", \"\").replace(\"(\", \"\").replace(\")\", \"\")\n",
        "  return trimed_title\n",
        "\n",
        "def trim_piece_fn(row, feature):\n",
        "  trimed_content = row[feature]\n",
        "  for token in OMIT_CHARS:\n",
        "    trimed_content = trimed_content.replace(token, \"\")\n",
        "  return trimed_content\n",
        "\n",
        "# Trim the size\n",
        "my_df['s_author'] = my_df.apply(trim_author_fn, axis=1)\n",
        "my_df['s_title'] = my_df.apply(trim_title_fn, axis=1)\n",
        "my_df['s_content'] = my_df.apply(lambda r : trim_piece_fn(r, 's_content'), axis=1)\n",
        "my_df['piece'] = my_df.apply(lambda r : trim_piece_fn(r, 'piece'), axis=1)\n",
        "\n",
        "print(\"before filter:\", len(my_df))\n",
        "my_df = my_df[my_df.s_author != '']\n",
        "my_df = my_df[my_df.s_title != '']\n",
        "my_df = my_df[my_df.s_content != '']\n",
        "my_df = my_df[my_df.piece != '']\n",
        "my_df = my_df[~my_df.s_content.str.contains(\"□\")] # unrecognized chars, ignore\n",
        "print(\"after filter:\", len(my_df))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "before filter: 1354444\n",
            "after filter: 1340301\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oegAoUWctSOS"
      },
      "source": [
        "## HuggingFace AutoModel to load guwen bert"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 183,
          "referenced_widgets": [
            "d7e095275dff4309a79bef080fcfe3e3",
            "ff070ece72ba4315895d16dc5848cbb8",
            "a03c308c29f84bae8d80433023bc5745",
            "c41512860f014e7dbb9ecd2bbaa0e145",
            "80836c9367504f08a50a76f920383c17",
            "83997112934345169c14eae5bed613d5",
            "ad179e3f9bdb4a1aac0497ec598d5221",
            "09965f59b5d4401b9d4c8694c8b99da6",
            "24ec8e1691f84189b1750a826312971c",
            "d1637f7d80494a4ca354ec58e7731fa8",
            "ca34e05a674b4e828f0459d541d6df42",
            "766f659cfb5d4e8682f045ce65469845",
            "c9bb67f1a3d747d886ea3453ed514a91",
            "a5d1b0755c7f4726a0ec9e19f5a3528d",
            "5807417077f74541ba85f30388a8da53",
            "f3b5cb6c582e4023829e7cc45446ea4e",
            "7f2e06cc5e524164bd9f3ffd3ec9fa47",
            "a43e2247711240118d55d66e7f43bb26",
            "95526d25a2fe4e57b072034eb33c6a16",
            "cb87bbc8ac8e45aba8a7ede3694b5999",
            "144843fbeff24104b867dd85672e8d0a",
            "8e9e9f18b9c94d75ae8b084b79a0d573",
            "ab62bb42114348a29a4bd892b2f96460",
            "62dd82a5caa94bec928dbd9c45f78f2e",
            "8da2456942d0469ead0b080866028eff",
            "2af326464adc4780833e882298d65654",
            "3a8395b78e2b49a2b7bdc0cbc6c6448d",
            "afcbbf74944b45259bdd7771fa6992f4",
            "2e1b79a637244be4a33ceabf5b873fd8",
            "c489d75c37b044a2bdde71f3a0634bf8",
            "9314230a17bf4413b8c46af688e23296",
            "5a99cdae60ca4706bb8528fd1f5ff36f",
            "8efab33f22b8486598723f08a3f5328c"
          ]
        },
        "id": "NPc_HOtQ1ksa",
        "outputId": "00ac45e9-1087-401c-db76-415813cd15f8"
      },
      "source": [
        "# See https://github.com/ethan-yt/guwenbert/blob/main/README_EN.md\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"ethanyt/guwenbert-base\")\n",
        "model = AutoModel.from_pretrained(\"ethanyt/guwenbert-base\", output_hidden_states=True)\n",
        "gpu_model = model.to('cuda')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d7e095275dff4309a79bef080fcfe3e3",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/519 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "766f659cfb5d4e8682f045ce65469845",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/93.5k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ab62bb42114348a29a4bd892b2f96460",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/418M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at ethanyt/guwenbert-base were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.bias', 'lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4fZHLTsit0tw"
      },
      "source": [
        "## Fetch all sentence embeddings by calling guwenBERT model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hVyh42S6wEO1"
      },
      "source": [
        "# Change to smaller number such as 100k if you see OOM (or colab just restarts)\n",
        "SAMPLE_SIZE = len(my_df) # 100_000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8uJEj_NKu3Nq"
      },
      "source": [
        "# use last n layers' sum and average per token, so each string has\\\n",
        "# 768 * n * #token size embeddings.\n",
        "# literature says last concat last 4 is best, last 1 is a good start here.\n",
        "LAST_N = 1\n",
        "last_layers = [-i for i in range(1, LAST_N + 1)]  # -1, -2, ....\n",
        "\n",
        "def get_bert_embedding(in_strs):\n",
        "  inputs = tokenizer(in_strs, padding=True, truncation=True,\n",
        "      return_tensors=\"pt\", max_length=MAX_SENTENCE_LENGTH+2).to('cuda:0')\n",
        "  with torch.no_grad(): # absolutely required to avoid OOM\n",
        "    outputs = gpu_model(**inputs)\n",
        "  last_n_layers = [outputs['hidden_states'][i] for i in last_layers]\n",
        "  cat_hidden_states = torch.cat(tuple(last_n_layers), dim=-1)\n",
        "  cat_sentence_embedding = torch.mean(cat_hidden_states, dim=1).squeeze()\n",
        "  return cat_sentence_embedding"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "9c70f870e9904902b80e82164ddc86bb",
            "cc355189d55a4c6287bb2833170d0fe4",
            "3cf09109ef4048cd896701e51c9c75b2",
            "c4ac33d857374095b107ab988cb007ad",
            "99152cb0a847437e8e40369484f9b3b9",
            "5eb551a593da41698c9a2eb5c4bb9f35",
            "b84f72f590b5431780eb396b30e03c05",
            "6ce259fc74844c7798847c3c78476881",
            "8c9163e362074af094d511794eb04865",
            "6c406df96ed64ef3a6a2d2208b190f62",
            "9222b79cab814a43992137215de2acfc"
          ]
        },
        "id": "iJyKhaCjPBue",
        "outputId": "8cbe8cd6-1cc7-406c-8674-ee65fd9e038c"
      },
      "source": [
        "# Fetch embedding for 1.3m sentences\n",
        "\n",
        "# About 15 mins to fetch embeddings for 1.3M poem sentence pieces\n",
        "# so roughly 1300k/1200 = 1.1k record inferences per second (Tesla P100), \n",
        "# CPU is too low, TPU does not support\n",
        "batch_size = 2048 # ~max batch size for gpu-ram=16G\n",
        "all_pieces = list(my_df.piece.values)\n",
        "all_pieces = all_pieces[:SAMPLE_SIZE]\n",
        "\n",
        "out_embedding_np = None\n",
        "pbar = tqdm(total=len(all_pieces), desc=\"Fetch embeddings\")\n",
        "for i in range(0, len(all_pieces), batch_size):\n",
        "  start_idx = i\n",
        "  end_idx = min(start_idx + batch_size, len(all_pieces))\n",
        "  embeddings = get_bert_embedding(all_pieces[start_idx:end_idx])\n",
        "  if out_embedding_np is None:\n",
        "    out_embedding_np = embeddings.detach().cpu().numpy()\n",
        "  else:\n",
        "    out_embedding_np = np.append(\n",
        "        out_embedding_np, embeddings.detach().cpu().numpy(), axis=0)\n",
        "  pbar.update(end_idx - start_idx)\n",
        "  # Additional safeguard to avoid OOM\n",
        "  gc.collect()\n",
        "  torch.cuda.empty_cache()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9c70f870e9904902b80e82164ddc86bb",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Fetch embeddings:   0%|          | 0/1340301 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ikCg3dyiv3j_"
      },
      "source": [
        "assert out_embedding_np.shape == (len(all_pieces), 768 * LAST_N)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tZtX-u7Mf2Lp"
      },
      "source": [
        "## [optional] Persist embedding for reuse to save time for embedding lookup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VBnl9V-2uy-a"
      },
      "source": [
        "# drive.mount('/content/gdrive')\n",
        "# !mkdir -p /content/gdrive/MyDrive/ML/Data/ch_poem_search_sent_embed_20210727\n",
        "# my_df.to_pickle('/content/gdrive/MyDrive/ML/Data/ch_poem_search_sent_embed_20210727/mydf.pickle')\n",
        "# np.save('/content/gdrive/MyDrive/ML/Data/ch_poem_search_sent_embed_20210727/embedding.npy', out_embedding_np)\n",
        "# !ls -l /content/gdrive/MyDrive/ML/Data/ch_poem_search_sent_embed_20210727\n",
        "\n",
        "# If reload embedding and dataframe from savings\n",
        "# my_df = pd.read_pickle('drive/MyDrive/ML/Data/ch_poem_search_sent_embed_20210727/mydf.pickle')\n",
        "# out_embedding_np = np.load('drive/MyDrive/ML/Data/ch_poem_search_sent_embed_20210727/embedding.npy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HbmtAayIsAf3"
      },
      "source": [
        "## Tensorboard to visualize 10k samples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JU8MA2-y3kOE"
      },
      "source": [
        "VIZ_SAMPLE = 10000\n",
        "LOG_DIR = '/logs/embedding_projection/'  # Tensorboard log dir\n",
        "random_selected = np.zeros(len(my_df), dtype=int)\n",
        "random_selected[:VIZ_SAMPLE] = 1\n",
        "np.random.shuffle(random_selected)\n",
        "random_selected_mask = random_selected.astype(bool)\n",
        "embedding_samples = out_embedding_np[random_selected_mask]\n",
        "label_samples = my_df.piece.values[random_selected_mask]\n",
        "assert embedding_samples.shape[0] == len(label_samples)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "id": "QQF45xNUsDrf",
        "outputId": "5b128e00-2d66-4eac-b8f0-9fd905bccbdd"
      },
      "source": [
        "# see hack from https://github.com/tensorflow/tensorboard/issues/2471#issuecomment-580423961\n",
        "def register_embedding(embedding_tensor_name, meta_data_fname, log_dir):\n",
        "    config = projector.ProjectorConfig()\n",
        "    embedding = config.embeddings.add()\n",
        "    embedding.tensor_name = embedding_tensor_name\n",
        "    embedding.metadata_path = meta_data_fname\n",
        "    projector.visualize_embeddings(log_dir, config)\n",
        "\n",
        "def save_labels_tsv(labels, filepath, log_dir):\n",
        "    with open(os.path.join(log_dir, filepath), 'w') as f:\n",
        "        for label in labels:\n",
        "            f.write('{}\\n'.format(label))\n",
        "\n",
        "if not os.path.exists(LOG_DIR):\n",
        "    os.makedirs(LOG_DIR)\n",
        "META_DATA_FNAME = 'meta.tsv'  # Labels will be stored here\n",
        "EMBEDDINGS_TENSOR_NAME = 'embeddings'\n",
        "EMBEDDINGS_FPATH = os.path.join(LOG_DIR, EMBEDDINGS_TENSOR_NAME + '.ckpt')\n",
        "STEP = 0\n",
        "\n",
        "register_embedding(EMBEDDINGS_TENSOR_NAME, META_DATA_FNAME, LOG_DIR)\n",
        "save_labels_tsv(label_samples, META_DATA_FNAME, LOG_DIR)\n",
        "\n",
        "tensor_embeddings = tf.Variable(embedding_samples, name=EMBEDDINGS_TENSOR_NAME)\n",
        "saver = tf.compat.v1.train.Saver([tensor_embeddings])  # Must pass list or dict\n",
        "saver.save(sess=None, global_step=STEP, save_path=EMBEDDINGS_FPATH)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Saver is deprecated, please switch to tf.train.Checkpoint or tf.keras.Model.save_weights for training checkpoints. When executing eagerly variables do not necessarily have unique names, and so the variable.name-based lookups Saver performs are error-prone.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'/logs/embedding_projection/embeddings.ckpt-0'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j4D4Ngzkyuxr"
      },
      "source": [
        "# After load, pick Projector, UMAP seems to have best results\n",
        "%tensorboard --logdir {LOG_DIR}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ql1shKue3PLM"
      },
      "source": [
        "## ScaNN for fast nearest neighbor search"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "unIzPBhbxJWj"
      },
      "source": [
        "def create_searcher(normalized_embeddings, # size matters, 1/2 size results in 2/3 time\n",
        "                    num_leaves=1000, # 1/2 size = 2/3 time, recommend ~sqrt(size)\n",
        "                    num_leaves_to_search=100, # not affect build time\n",
        "                    reorder_size=20, # not affect build time\n",
        "                    min_partition_size=50, # not affect build time\n",
        "                    training_iterations=12, # slightly affect build time\n",
        "                    neighbor_size=10, # not affect build time\n",
        "                    search_func=\"dot_product\"): # seems the best option after tried\n",
        "  \"\"\"Creates scann searcher. Params briefly tested, comments may be wrong.\"\"\"\n",
        "  search_builder = scann.scann_ops_pybind.builder(\n",
        "      normalized_embeddings, neighbor_size, search_func)\n",
        "  search_builder = search_builder.tree(\n",
        "      num_leaves=num_leaves,\n",
        "      num_leaves_to_search=num_leaves_to_search,\n",
        "      training_sample_size=normalized_embeddings.shape[0],\n",
        "      min_partition_size=min_partition_size,\n",
        "      training_iterations=training_iterations)\n",
        "  search_builder = search_builder.score_ah(\n",
        "      2, anisotropic_quantization_threshold=0.2)\n",
        "  search_builder = search_builder.reorder(reorder_size)\n",
        "  searcher = search_builder.build()\n",
        "  return searcher"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "REnTLbdDZCkR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9bd7fc74-0945-42c1-a69a-ff3ef09283f2"
      },
      "source": [
        "%%time\n",
        "print(\"Est time: \", 2 * (out_embedding_np.shape[0] // 1000), \"secs\")\n",
        "normalized_embedding_np = out_embedding_np / np.linalg.norm(out_embedding_np, axis=1)[:, np.newaxis]\n",
        "scann_searcher = create_searcher(normalized_embedding_np)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Est time:  2680 secs\n",
            "CPU times: user 31min 15s, sys: 11.3 s, total: 31min 26s\n",
            "Wall time: 8min 17s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KY7Ant3Syr9p"
      },
      "source": [
        "## Sup method for inference Test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i8YwRdjjyucH"
      },
      "source": [
        "sents = my_df.piece.values\n",
        "authors = my_df.s_author.values\n",
        "titles = my_df.s_title.values\n",
        "contents = my_df.s_content.values\n",
        "\n",
        "def query_poem_now(input_str,\n",
        "                   scann_searcher=scann_searcher,\n",
        "                   size=10):\n",
        "  \"\"\"Fetch embedding, ask scann the nearest neighbors and format.\"\"\"\n",
        "  inp_embedding = get_bert_embedding(input_str)\n",
        "  neighbors, distances = scann_searcher.search(\n",
        "      inp_embedding.detach().cpu().numpy(), final_num_neighbors=size)\n",
        "  print(f\"输入：\\x1b[34m{input_str}\\x1b[0m\")\n",
        "  id = 1\n",
        "  for n, d in zip(neighbors, distances):\n",
        "    content = contents[n]\n",
        "    if content.find(sents[n]) > 40:\n",
        "      content = \"...\" + content[content.find(sents[n]):]\n",
        "    content = content.replace(sents[n], f\"[\\x1b[31m{sents[n]}\\x1b[0m]\")\n",
        "    if (len(content) > 80):\n",
        "      content = content[:80] + \"...\"\n",
        "    print(f\"{id}: {authors[n]} [{titles[n]}]\\n{content}\")\n",
        "    id += 1\n",
        "  print()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAxCsiHr0v4r"
      },
      "source": [
        "## Inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9ctebYq3zBjF",
        "outputId": "f75f751f-0635-4886-9105-146a2d7ff73c"
      },
      "source": [
        "%%time\n",
        "for query in [\"你看那长江的水从天上来\", \"忆长安\", \"月有圆缺\", \"九歌\"]:\n",
        "  query_poem_now(query, size=3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "输入：\u001b[34m你看那长江的水从天上来\u001b[0m\n",
            "1: 李白 [鼓吹曲辞将进酒]\n",
            "[\u001b[31m君不见黄河之水天上来，奔流到海不复回\u001b[0m]。君不见高堂明镜悲白发，朝如青丝暮成雪。人生得意须尽欢，莫使金尊空对月。天生我材必有用，千金散尽还复来...\n",
            "2: 马之纯 [新亭其二]\n",
            "[\u001b[31m新亭见说在山头，看见江河衮衮流\u001b[0m]。何事后人轻变改，不教遗址且存留。怜他一代称贤相，说此诸人似楚囚。若使有人来访旧，一番人见一番羞。\n",
            "3: 释善果 [偈其五]\n",
            "[\u001b[31m苏州有，常州有，吸尽西江只一口\u001b[0m]。百八数珠数不尽，须知天长与地久。腾今焕古作嘉祥，一一面南看北斗。\n",
            "\n",
            "输入：\u001b[34m忆长安\u001b[0m\n",
            "1: 宋祁 [农阁]\n",
            "菌阁俯江干，西南蜀塞宽。[\u001b[31m看云记巫峡，望日省长安\u001b[0m]。钿崒峰头碧，霞皴荔子丹。比来秋物好，谁伴数凭栏。\n",
            "2: 裴说 [咏鹦鹉]\n",
            "常贵西山鸟，衔恩在玉堂。语传明主意，衣拂美人香。缓步寻珠网，高飞上画梁。[\u001b[31m长安频道乐，何日从君王\u001b[0m]。\n",
            "3: 徐凝 [寄白司马]\n",
            "三条九陌花时节，万户千车看牡丹。[\u001b[31m争遣江州白司马，五年风景忆长安\u001b[0m]。\n",
            "\n",
            "输入：\u001b[34m月有圆缺\u001b[0m\n",
            "1: 王义山 [赠心月相士]\n",
            "...[\u001b[31m嗟彼天上月，有圆阙阴晴\u001b[0m]。惟有心月月，天者常清明。持此以鉴人，妍媸奚所遁。此月不在天，月在尔方寸。\n",
            "2: 崔萱 [古意]\n",
            "灼灼叶中花，夏萎春又芳。[\u001b[31m明明天上月，蟾缺圆复光\u001b[0m]。未如君子情，朝违夕已忘。玉帐枕犹暖，纨扇思何长。愿因西南风，吹上玳瑁牀。娇眠锦衾裏，展转双...\n",
            "3: 李曾伯 [题冯司法水月书堂其一]\n",
            "[\u001b[31m水能清亦能浊，月有满亦有亏\u001b[0m]。清浊满亏区别，冯君书以知之。\n",
            "\n",
            "输入：\u001b[34m九歌\u001b[0m\n",
            "1: 范成大 [浮湘行]\n",
            "...[\u001b[31m九歌凄悲不可听，愿赓楚调归和平\u001b[0m]。\n",
            "2: 宋庠 [屈原其二]\n",
            "[\u001b[31m司命湘君各有情，九歌愁苦荐新声\u001b[0m]。如何不救沉江祸，枉解堂中许目成。\n",
            "3: 不详 [郊庙歌辞中宗祀昊天乐章]\n",
            "[\u001b[31m九成爰奏，三献式陈\u001b[0m]。钦承景福，恭托明禋。\n",
            "\n",
            "CPU times: user 54.7 ms, sys: 36 µs, total: 54.7 ms\n",
            "Wall time: 57.9 ms\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xmnJ48bTGjBw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "736ff136-52e6-456f-85cf-b3eb7f61ee07"
      },
      "source": [
        "%%time\n",
        "for query in ['今日看到美女，惊呆了！', '想骂人', '我要打人',\n",
        "              '祝贺朋友结婚', '做人要开心', '载歌在谷', '自由而无用', ]:\n",
        "  query_poem_now(query, size=5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "输入：\u001b[34m今日看到美女，惊呆了！\u001b[0m\n",
            "1: 释广闻 [行素长老请自赞]\n",
            "无狮子教儿之诀，无老猫上树之机。世无所容其拙，人或见谓之慈。知之已熟，画出何爲。[\u001b[31m后三十年扬在𡏖𡒁堆头，便令无光怪发现，未免起傍观按剑之疑\u001b[0m]。\n",
            "2: 王大烈 [绝句]\n",
            "弧矢重悬旧礼仪，郎君又産玉麟儿。[\u001b[31m便烦着眼从头看，的似徐卿二子奇\u001b[0m]。\n",
            "3: 陆游 [春雨绝句六首其三]\n",
            "天公似欲败蚕麰，雨冒南山暮不收。[\u001b[31m呆女癡儿那念此，贪看科斗满清沟\u001b[0m]。\n",
            "4: 范纯仁 [安之家庭甘结实三首其]\n",
            "遮藏霜雪免摧残，绿实初垂未满栏。[\u001b[31m想见主人珍惜意，一回出户一回看\u001b[0m]。\n",
            "5: 释宗琏 [赞月庵]\n",
            "...[\u001b[31m只因会春园裏失却眼睛，从此恶名滔滔流遍天下\u001b[0m]。高挂虚堂兮如师子全威，一任百怪千妖暗中惊讶。灯禅灯禅，第一不得容易与伊点化。\n",
            "\n",
            "输入：\u001b[34m想骂人\u001b[0m\n",
            "1: 释道济 [与张提点共饮席间作]\n",
            "每日终朝醉似泥，未尝一日不昏迷。[\u001b[31m细君发怒将言骂，道是人间吃酒儿\u001b[0m]。莫要管，你休癡，人生能有几多时。杜康曾唱莲花落，刘伶好饮舞啰哩，陶渊明赏菊...\n",
            "2: 唐士耻 [两溪]\n",
            "...[\u001b[31m叱石而兴真戯耳，可羡人家好弟兄\u001b[0m]。西则三洞足仙灵，奇哉石穴声铿铿。瀑泉喷写资照耀，俨如冰蚕之所成。夕阳斜带九华度，何媿芙蓉峥且嵘。鼎湖...\n",
            "3: 杨万里 [晨炊翫鞭亭二首其二]\n",
            "[\u001b[31m问着无声是阿兄，坐看家贼只吞声\u001b[0m]。戮尸大放经纶手，长柄判将锡茂弘。\n",
            "4: 陈师道 [赠知命]\n",
            "黑头居士元方弟，不肯作公称法嗣。[\u001b[31m外人怪笑那得知，他日灵山亲授记\u001b[0m]。学诗初学杜少陵，学书不学王右军。黄尘扶杖笑邻女，白衫骑馿惊市人。静中作业此...\n",
            "5: 赵必𤩪 [赠相士桂月岩]\n",
            "昔有道人张岩电，口不言钱似王衍。[\u001b[31m却言姓钱不姓张，滑稽玩世名犹香\u001b[0m]。昔有岩电今月岩，月眼神舌和天谈。江湖剩结贵人知，虽不爱钱犹爱诗。\n",
            "\n",
            "输入：\u001b[34m我要打人\u001b[0m\n",
            "1: 释师观 [颂古十七首其六]\n",
            "[\u001b[31m张打油，李打油，不打浑身只打头\u001b[0m]。今朝有酒今朝醉，明日愁来明日愁。\n",
            "2: 释师体 [偈颂十八首其一○]\n",
            "[\u001b[31m一拳也是打爷来，未有输赢莫放开\u001b[0m]。割舍拍盲穷性命，觜喎鼻塌见全材。\n",
            "3: 释元静 [偈三首其二]\n",
            "行年七十老躘踵，眼目精明耳不聋。[\u001b[31m忽地有人欺负我，一拳打倒过关东\u001b[0m]。\n",
            "4: 释道枢 [颂古三十九首其二五]\n",
            "朴寔头禅无伎俩，一句分明如扑相。[\u001b[31m客来只是叫担板，不知的当谁担板\u001b[0m]。\n",
            "5: 李之仪 [题黄蘖泉]\n",
            "住庵老子不复见，传法阿师追不回。[\u001b[31m饮水能知大愚意，一拳真是打爷来\u001b[0m]。\n",
            "\n",
            "输入：\u001b[34m祝贺朋友结婚\u001b[0m\n",
            "1: 王十朋 [陈商霖挽词]\n",
            "畴昔游从分不轻，十年乡校读书灯。[\u001b[31m德星会聚三兄弟，诗社追随八友朋\u001b[0m]。去国悲凉老王粲，还乡牢落失陈登。诵君生死炎凉句，斜日寒天涕满膺。\n",
            "2: 王禹偁 [赠密直张谏议]\n",
            "先皇忧蜀辍枢臣，独冒兵戈出剑门。万里辞家堪下泪，四年归阙似还魂。[\u001b[31m弟兄齿序元投分，儿女情亲又结婚\u001b[0m]。且喜相逢开口笑，甘陈功业不须论。\n",
            "3: 韦元甫 [木兰歌]\n",
            "...[\u001b[31m亲戚持酒贺父母，始知生女与男同\u001b[0m]。门前旧军都，十年共崎岖。本结弟兄交，死战誓不渝。今者见木兰，言声虽是颜貌殊。惊愕不敢前，叹息徒嘻吁。...\n",
            "4: 陆游 [初冬杂咏八首其二]\n",
            "[\u001b[31m古寿书来言得婿，东阳人到报生孙\u001b[0m]。一家三处俱强健，且拨闲愁近酒尊。\n",
            "5: 韩淲 [庆元庚申二月药局书满七月]\n",
            "[\u001b[31m儿女长须婚嫁，朋友来须应酬\u001b[0m]。应酬重费青眼，婚嫁未了白头。\n",
            "\n",
            "输入：\u001b[34m做人要开心\u001b[0m\n",
            "1: 邵雍 [笺年老逢春八首谁让万金]\n",
            "笺云白日偏催处，黄金欲尽时。[\u001b[31m侈心都用了，始得一开眉\u001b[0m]。\n",
            "2: 邵雍 [老去吟其一]\n",
            "老去无成不入时，中年养病只吟诗。[\u001b[31m因乘意思要舒放，肯把语言生事治\u001b[0m]。世上闲愁都一致，人间何务更能爲。携筇晚步天津畔，爲报沙鸥慎勿飞。\n",
            "3: 邵雍 [县尉廨宇莲池]\n",
            "县尉小斋前，水清池有莲。岂唯观菡萏，兼可听潺湲。宛类江湖上，殊非尘土边。[\u001b[31m古人用心处，料得不徒然\u001b[0m]。\n",
            "4: 李曾伯 [别权泸帅朱杞材太社]\n",
            "...[\u001b[31m度泸要了经营事，心放公平手放开\u001b[0m]。\n",
            "5: 释普宁 [偈颂二十一首其一三]\n",
            "...[\u001b[31m必竟事作麽生，但看人情若好，自然吃水也肥\u001b[0m]。\n",
            "\n",
            "输入：\u001b[34m载歌在谷\u001b[0m\n",
            "1: 司马光 [奉和江邻几六月七日文德殿]\n",
            "...[\u001b[31m退复歌天保，期于采颂声\u001b[0m]。\n",
            "2: 夏竦 [奉和御制奉先歌]\n",
            "...[\u001b[31m思文成颂兮迪先谋，多黍申歌兮昭乐岁\u001b[0m]。参侔开阖已无爲，虔巩盈成弥若厉。惟寤寐兮符通，实天人兮合契。降仙驭兮自三清，谕灵源兮逾百世。遐锡...\n",
            "3: 郊庙朝会 [祭九鼎十二首亚、终献用]\n",
            "工祝致辞，黄流协鬯。[\u001b[31m爰登清歌，载期神享\u001b[0m]。噫予诚心，精禋是虔。嘉予陈祀，丰盈豆笾。\n",
            "4: 魏了翁 [次韵李参政和薛秘书诗见寄]\n",
            "书来寄我薛公诗，磊落襟期尉我思。[\u001b[31m所谓伊人在空谷，公侯逸豫镇无期\u001b[0m]。\n",
            "5: 陈着 [寿马裕斋观文东山，二章]\n",
            "凤凰于飞，出自幽谷。[\u001b[31m载好其音，集于灌木\u001b[0m]。\n",
            "\n",
            "输入：\u001b[34m自由而无用\u001b[0m\n",
            "1: 白居易 [何处堪避暑]\n",
            "...[\u001b[31m从心至百骸，无一不自由\u001b[0m]。拙退是其分，荣耀非所求。虽被世间笑，终无身外忧。此语君莫怪，静思吾亦愁。如何三伏月，杨尹谪虔州。\n",
            "2: 邵雍 [秋怀三十六首其一]\n",
            "七月夜初长，星斗争煌煌。庭除经小雨，枕簟生微凉。照物无遁形，虚鉴自有光。[\u001b[31m照事无遁情，虚心自有常\u001b[0m]。\n",
            "3: 释正觉 [禅人并化主写真求赞其]\n",
            "[\u001b[31m青山之主兮体本平等，白云之宾兮用能自然\u001b[0m]。应有余兮神不可测，得其简兮妙不可传。\n",
            "4: 黄庭坚 [至乐词寄黄几复]\n",
            "...[\u001b[31m廓宇宙以爲量兮，奚自适而不通，遂风休而水释\u001b[0m]。\n",
            "5: 邵雍 [观棋大吟]\n",
            "...[\u001b[31m治乱不自己，因革徒从宜\u001b[0m]。与贤不与子，贤愚生瑕玼。与子不与贤，子孙生疮痍。或苗民逆命，或有扈阻威。或羿浞起衅，或管蔡造疑。或商人征葛，...\n",
            "\n",
            "CPU times: user 87.8 ms, sys: 12 ms, total: 99.8 ms\n",
            "Wall time: 95.2 ms\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}