Example_dataset_production.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Example_dataset_production.ipynb",
      "version": "0.3.2",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wv4Nv8_bxMYA",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "# imports\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_1sINpM4xC6G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e9b139ff-e2e2-4f60-f419-a60f6ded9ec5"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive', force_remount=True)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gggvA6mC3lHY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "5c366333-e9e0-49f7-c448-db355d846024"
      },
      "source": [
        "%pwd"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'/content'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qvJMSZGSxKee",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b617bcef-7765-4bf8-c13e-6e444b7c4162"
      },
      "source": [
        "%cd /content/drive/My \\Drive/masters_project/python_modules/my_stuff"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masters_project/python_modules/my_stuff\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MtTjh53W16xx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from conflict_lstm.latest_run import *\n",
        "from conflict_lst.hpc_construct import *"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "25Trni-7xSTs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd ../..\n",
        "\n",
        "%cd data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Urh0JUgYxSwe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.backends.cudnn.benchmark = True\n",
        "torch.backends.cudnn.enabled = True\n",
        "device = 'cuda"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRIdCCTrxdSh",
        "colab_type": "text"
      },
      "source": [
        "# Example usecase to prepare a hdf5 image sequence dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZH2GzZ_1xafX",
        "colab_type": "text"
      },
      "source": [
        "# Loading in data\n",
        "\n",
        "You must download the latest versions of the UCDP GED dataset : \n",
        "\n",
        "and the latest version of the PRIO V2 GRID: \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8aomfAwwxU6s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# First we load the data into dataframes from csv.\n",
        "data_prio = pd.read_csv(\"data/PRIO-GRID Yearly Variables for 1946-2014 - 2019-07-26.csv\")\n",
        "data_ucdp = pd.read_csv(\"data/ged191.csv\")\n",
        "\n",
        "data_ucdp = regional_selection(data_ucdp)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B6_9WB20y6I-",
        "colab_type": "text"
      },
      "source": [
        "We wish to produce a dataset constructed using 4 PRIO variables. (see PRIO grid codebook for detail on variables).\n",
        "The PRIO variables we wish to use are \"petroleum_y\",\"drug_y\", \"prec_gpcp\". \n",
        "Observing the variables we note that petroleum_y and drug_y encode the positive variable (i.e the presence of drug or oil production in a prio grid cell as 0, and the non presence as NAN.) We can correct this below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kfOHSU6vy5at",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "nan_to_one(data_prio, \"petroleum_y\")\n",
        "nan_to_one(data_prio, \"drug_y\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yt4TwhjH0wln",
        "colab_type": "text"
      },
      "source": [
        "We now sort the UCDP event data into monthly bins"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fftwUIs1ymbP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "date_column(data_ucdp)\n",
        "binary_event_column(data_ucdp)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R10ciecl0-F5",
        "colab_type": "text"
      },
      "source": [
        "We define which predictors we would like to extract from the conflict data dataframes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8eR7pnkb02G1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "key_list_ucdp = ['binary_event', 'best']\n",
        "key_list_prio = [\"petroleum_y\",\"drug_y\", \"prec_gpcp\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_RXEIlGs9DKp",
        "colab_type": "text"
      },
      "source": [
        "Finally we construct the dataset. This stores the produced image dataset as a hdf5 file in the local directory. Please note this is a lengthy process due to size of datafile being produced ($\\approx 15 GB$)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GWRwWvLy2iSa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "construct_dataset(\"test_dataset\", data_prio, data_ucdp, key_list_prio, key_list_ucdp)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}