wikihow_intent_code.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "wikihow_intent_code.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_Y2PiDf2not"
      },
      "source": [
        "# \"Reasoning about Goals, Steps, and Temporal Ordering with WikiHow\" - codes\n",
        "\n",
        "Codes to reproduce the results in the paper \"[Intent Detection with WikiHow](https://arxiv.org/abs/2009.05781)\". \n",
        "\n",
        "Adapted from the [Huggingface Transformers repository](https://colab.research.google.com/github/ViktorAlm/notebooks/blob/master/MPC_GPU_Demo_for_TF_and_PT.ipynb).\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Sj_Y8Rm3Phq"
      },
      "source": [
        "## Install Huggingface transformers :)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8klpyPsOv3hS"
      },
      "source": [
        "!git clone https://github.com/huggingface/transformers\n",
        "%cd transformers\n",
        "!git checkout b42586ea560a20dcadb78472a6b4596f579e9043\n",
        "!pip install .\n",
        "!pip install -r ./examples/requirements.txt\n",
        "%cd .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6piL7gWD3IGu"
      },
      "source": [
        "## Prepare the directories and data\n",
        "\n",
        "\n",
        "*   Make two directories, `./data` and `./output`.\n",
        "*   Obtain the data [here](https://drive.google.com/drive/folders/1147ZJ7ZZs0InrNrUslUMG2X4FqFdkDJK?usp=sharing) , and put the 3 resulting folders (`goal/`, `step/`, `order/`) under `./data`. You can do this by:\n",
        " - Save the 3 folders to your Google Drive, and mount your drive using the `mount drive` function of Colab.\n",
        " - Download the 3 folders to your local machine, and then upload them to `./data` in the current session.\n",
        " - Download them directly to the current session using `gdown`.\n",
        "\n",
        " We will demonstrate the third way. Note: in the folder for each target dataset, `train.csv` is our wikiHow training data, while `val.csv` is actually the test set of each original dataset. The reason for this naming is that out-of-the-box Transformers libary does not permit a file named `test.csv` to have labels. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "37fO5EO3TG9i"
      },
      "source": [
        "!mkdir data\n",
        "!mkdir output\n",
        "!gdown https://drive.google.com/uc?id=1gsEhosbZ5bQKEgsuyjweu9sGxSSjfcFL\n",
        "!unzip wikihow-intent-data.zip -d ./data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHtKtmky6XCk"
      },
      "source": [
        "## Set the configuration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iDJG3_ZV6Wlq"
      },
      "source": [
        "import os\n",
        "\n",
        "# the benchmark task you want to train/evaluate on. 'goal': goal inference, 'step': step inference, 'order': step ordering.\n",
        "task = ['sgd', 'snips', 'fb-en', 'fb-es', 'fb-th'][0]\n",
        "\n",
        "# the name of the model you want to train/evaluate on.\n",
        "modelName = ['intent_enwh_rl', 'intent_enwh_xlmr', 'intent_eswh', 'intent_thwh', 'intent_snips_wh_id', 'intent_snips_id', 'intent_sgd_wh_id', 'intent_sgd_id', 'intent_fb-en_wh_id_rl', 'intent_fb-en_id_rl', 'intent_fb-es_wh_id', 'intent_fb-es_enwh_id', 'intent_fb-es_id', 'intent_fb-th_wh_id', 'intent_fb-th_enwh_id', 'intent_fb-th_id'][0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "feSe82xh216x"
      },
      "source": [
        "## Evaluate our pretrained models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "49gbdgjKBvS_"
      },
      "source": [
        "os.environ['MODEL_NAME'] = f'zharry29/{modelName}'\n",
        "os.environ['DATA_DIR'] = f'./data/{task}/'\n",
        "os.environ['OUTPUT_DIR'] = f'./output/{task}_{modelName}'\n",
        "\n",
        "!python ./transformers/examples/multiple-choice/run_multiple_choice.py \\\n",
        "  --task_name swag \\\n",
        "  --model_name_or_path $MODEL_NAME \\\n",
        "  --do_eval \\\n",
        "  --data_dir $DATA_DIR \\\n",
        "  --max_seq_length 200 \\\n",
        "  --per_gpu_eval_batch_size=16 \\\n",
        "  --output_dir $OUTPUT_DIR \\\n",
        "  --overwrite_output\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2MJTzqABFI6t"
      },
      "source": [
        "##Train and evaluate the models yourself"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LYvCYvTy7srp"
      },
      "source": [
        "os.environ['MODEL_NAME'] = f'zharry29/{modelName}'\n",
        "os.environ['DATA_DIR'] = f'./data/{task}/'\n",
        "os.environ['OUTPUT_DIR'] = f'./output/{task}_{modelName}'\n",
        "\n",
        "!python ./transformers/examples/multiple-choice/run_multiple_choice.py \\\n",
        "  --task_name swag \\\n",
        "  --model_name_or_path $MODEL_NAME \\\n",
        "  --do_train \\\n",
        "  --do_eval \\\n",
        "  --data_dir $DATA_DIR \\\n",
        "  --learning_rate 5e-5 \\\n",
        "  --num_train_epochs 3 \\\n",
        "  --max_seq_length 200 \\\n",
        "  --per_gpu_eval_batch_size=16 \\\n",
        "  --per_gpu_train_batch_size=16 \\\n",
        "  --gradient_accumulation_steps 2 \\\n",
        "  --output_dir $OUTPUT_DIR \\\n",
        "  --overwrite_output"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}