school_tiles_download.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Workflow for training dataset generation for ML UNICEF school detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will work through few steps we took to generate training dataset for machine learning work of school detection. \n",
    "\n",
    "- Training dataset validation and cleaning by Development Seed Data Team, a group of eight expert mappers. \n",
    "  - The reviewed schools were classified into three groups against DG vivid base satellite layer by the expert mappers;\n",
    "    - confirmed schools\n",
    "    - unreconginized schools\n",
    "    - not-schools\n",
    "- we added few more objects, including hospitals, courthouses, marketplaces, parks, and farms, to not-school class to balace the training classes;\n",
    "\n",
    "- we genrate tile grids with DevSeed Geokit from schools and not-schools points;\n",
    "\n",
    "- we wrote python function to download the tiles for both classes. \n",
    "\n",
    "At the end these are the number we have:\n",
    "\n",
    "|Tasks|Confirmed | Unreconginized | not-schools| Total |\n",
    "| ----| -------- | -------------- | ---------- | ----- |\n",
    "| Data Cleaning  |  6,663   | 11,774         | 2,268      | 20,705| \n",
    "| Tile generation  | 5452     | N/A            | 3,953 | 9,405|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding more objects from OSM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "- Step one, download [OSM Colombia dataset](http://download.geofabrik.de/south-america/colombia.html) from Geofabric;\n",
    "\n",
    "- Step two, split training dataset into confirmed schools and not-schools from the valided/cleaned points;\n",
    "\n",
    "- Step three, using [Development Seed Geokit](https://github.com/developmentseed/geokit) to exact hospitals, courthouses, marketplaces, parks and farms;\n",
    "  - using hospitals extraction from OSM as an example:\n",
    "  \n",
    "    - run 'docker run --rm -v ${PWD}:/app developmentseed/geokit osmfilter colombia.osm --keep=\"amenity=hospital\" -o=hospital_r_colombia.osm'\n",
    "    \n",
    "    - run \"docker run --rm -v ${PWD}:/app developmentseed/geokit osmtogeojson hospital_r_colombia.osm > hospitals_c.geojson\"\n",
    "    \n",
    "    - from QGIS only selected points for the training dataset as hospitals_c_final.geojson.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create tiles from points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ### Merge all the geojsons\n",
    "\n",
    "After adding all the objects, e.g. hospitals, parks, farms, courthouses, and marketplaces, you will need to merge all of the objects' geojsons into `not_schools_final.geojson`.\n",
    "\n",
    "Run 'docker run --rm -v ${PWD}:/app developmentseed/geokit geojson-merge input1.geojson input2.geojson > output.geojson', remember to replace all the geojson names accordingly.\n",
    "\n",
    "- ### Genrate not-school tile-grid from not-school points\n",
    "\n",
    "Run 'docker run --rm -v ${PWD}:/app developmentseed/geokit point2tile data/combined_not_schools_final.geojson --zoom=17 --buffer=0.001 > data/not_schools_tiles_1m.geojson'\n",
    "\n",
    "- ### generate school tile-grid from school points;\n",
    "\n",
    "Run 'docker run --rm -v ${PWD}:/app developmentseed/geokit point2tile data/confirmed_schools_final.geojson  --zoom=17 --buffer=0.001 > data/schools_tiles_1m.geojson'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Down all the tiles for school and not-school\n",
    "\n",
    "Use following script to download tiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Remember to replace your token with \"TOKEN\" list following\n",
    "%%file unicef_school_tiles.json\n",
    "\n",
    "{\"school\": \"schools_tiles_1m.geojson\", \n",
    "\"not_school\": \"not_schools_tiles_1m.geojson\",\n",
    "\"school_url\":\"https://a.tiles.mapbox.com/v4/digitalglobe.2lnpeioh/{z}/{x}/{y}.tif?access_token=TOKEN\",\n",
    "\"not_school_url\": \"https://a.tiles.mapbox.com/v4/digitalglobe.2lnpeioh/{z}/{x}/{y}.png?access_token=TOKEN\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from urllib.parse import urlparse, parse_qs\n",
    "import requests\n",
    "import os\n",
    "from os import makedirs, path as op\n",
    "import rasterio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tile(geojson, base_url):\n",
    "    \"\"\"\n",
    "    Function to download tiles for school and not-school. \n",
    "    The tile index was created using DevSeed Geokit with 1m buffer to the geolocation points for school and not-school classes;\n",
    "    :param geojson: geojson for tile and tile index from geokit (poin2tile);\n",
    "    :param base_url: url to access DG vivid and given the token to download the tiles.\n",
    "    \n",
    "    :return tiles: a list of tiles \n",
    "    \"\"\"\n",
    "    # open geojson and get tile index\n",
    "    with open(geojson, 'r') as data:\n",
    "        tile_geojson = json.load(data)\n",
    "    features = tile_geojson[\"features\"]\n",
    "    # get the tile index as x, y, z formats.\n",
    "    xyz = [features[i]['properties']['tiles'] for i in range(len(features))]\n",
    "    \n",
    "    # create tile folder\n",
    "    tiles_folder = op.splitext(geojson)[0].split(\"/\")[0]\n",
    "    if not op.isdir(tiles_folder):\n",
    "        makedirs(tiles_folder)\n",
    "        \n",
    "    # download and get the list of tiles \n",
    "    tiles = list()\n",
    "    for i in range(len(xyz)):\n",
    "#         x, y, z = str(xyz[i])\n",
    "        x=str(xyz[i][0])\n",
    "        y=str(xyz[i][1])\n",
    "        z=str(xyz[i][2])\n",
    "        url = base_url.replace('{x}', x).replace('{y}', y).replace('{z}', z)\n",
    "        o = urlparse(url)\n",
    "        _, image_format = op.splitext(o.path)\n",
    "        tile_bn =\"{}-{}-{}{}\".format(z, x, y,image_format)\n",
    "        r = requests.get(url)\n",
    "        tile= op.join(tiles_folder, tile_bn)\n",
    "        tiles.append(tile)\n",
    "        with open(tile, 'wb')as w:\n",
    "            w.write(r.content)\n",
    "    return tiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"unicef_school_tiles.json\", 'r') as config:\n",
    "    all_data = json.load(config)\n",
    "    \n",
    "school_geojson = all_data[\"school\"]\n",
    "school_turl = all_data[\"school_url\"]\n",
    "not_school_geojson= all_data[\"not_school\"]\n",
    "not_school_turl = all_data[\"not_school_url\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download all the school tiles\n",
    "school_tiles = get_tile(school_geojson, school_turl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download all the none school tiles\n",
    "not_school_tiles = get_tile(not_school_geojson, not_school_turl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}