get_effechecka_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lists of Species by Country "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code uses the effechecka API to get a list of taxa that have been reported in each country. The API taxes a polygon (points are lat/lon coordinates) and returns observations within that polygon from several species occurrance databases. To use this notebook, you need a list of geonames ids and a json file with geonames polygons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below, we import the necessary libraries and data files. The input file, test_country.txt, contains the geonames ID for the country. There is only two in the file at any given time. This helps to avoid overloading the server. The code is written so that more countries can be included if in the future the server can handle more queries at once. The file low_res_countries.json are the polygons from geonames that have been reduced in resolution so they can fit in the URL API call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import urllib.error\n",
    "import json\n",
    "\n",
    "in_file = open('test_country.txt', 'r')\n",
    "shape_file = open('low_res_countries.json','r')\n",
    "\n",
    "shapes = json.load(shape_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below takes the country-shaped polygon and forms the URL to query the API. Each query will take two hits. The first gets effechecka started on the query and the second (done a day later) will grab the results. If the query has been submitted before, then you will not need to do the second query. The json results returned by the API are written to the out_files. Each country has a separate out_file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99237\n",
      "http://api.effechecka.org/checklist.tsv?traitSelector=&wktString=GEOMETRYCOLLECTION%28POLYGON%20%28%2848.567%2029.916%2C%2048.21%2030.033%2C%2047.95%2030.061%2C%2047.709%2030.096%2C%2047.181%2030.026%2C%2046.556%2029.103%2C%2044.72%2029.206%2C%2043.611%2030.022%2C%2041.444%2031.378%2C%2039.203%2032.158%2C%2039.26%2032.356%2C%2038.986%2032.478%2C%2038.796%2033.368%2C%2041.238%2034.785%2C%2041.283%2035.486%2C%2041.381%2035.835%2C%2041.295%2036.356%2C%2041.828%2036.593%2C%2042.364%2037.109%2C%2042.799%2037.377%2C%2043.167%2037.374%2C%2043.8%2037.23%2C%2044.03%2037.325%2C%2044.279%2037.236%2C%2044.202%2037.098%2C%2044.351%2037.049%2C%2044.771%2037.167%2C%2044.921%2037.02%2C%2044.861%2036.784%2C%2045.072%2036.691%2C%2045.108%2036.419%2C%2045.284%2036.383%2C%2045.279%2036.253%2C%2045.387%2036.085%2C%2045.557%2036.001%2C%2046.093%2035.861%2C%2046.349%2035.809%2C%2046.013%2035.678%2C%2046.156%2035.287%2C%2046.203%2035.198%2C%2046.189%2035.108%2C%2046.064%2035.036%2C%2045.883%2035.031%2C%2045.799%2034.91%2C%2045.701%2034.812%2C%2045.748%2034.542%2C%2045.532%2034.492%2C%2045.503%2034.327%2C%2045.588%2034.303%2C%2045.413%2033.972%2C%2045.777%2033.623%2C%2045.907%2033.626%2C%2045.871%2033.491%2C%2046.051%2033.374%2C%2046.205%2033.18%2C%2046.05%2033.121%2C%2046.102%2032.97%2C%2046.737%2032.762%2C%2047.177%2032.452%2C%2047.449%2032.401%2C%2047.567%2032.224%2C%2047.648%2032.084%2C%2047.699%2031.4%2C%2048.031%2030.994%2C%2048.17%2030.423%2C%2048.267%2030.336%2C%2048.383%2030.127%2C%2048.567%2029.916%29%29%29\n",
      "Service Unavailable\n",
      "2963597\n",
      "http://api.effechecka.org/checklist.tsv?traitSelector=&wktString=POLYGON((-8.063%2055.379%2C%20-7.22%2055.46%2C%20-6.923%2055.237%2C%20-8.096%2054.438%2C%20-7.551%2054.125%2C%20-7.253%2054.203%2C%20-7.042%2054.361%2C%20-6.108%2054.007%2C%20-6.053%2053.729%2C%20-5.91%2053.37%2C%20-6.002%2052.967%2C%20-6.075%2052.476%2C%20-6.119%2052.14%2C%20-6.481%2052.066%2C%20-6.932%2052.059%2C%20-7.426%2051.923%2C%20-8.041%2051.713%2C%20-8.459%2051.529%2C%20-8.997%2051.399%2C%20-9.667%2051.316%2C%20-10.272%2051.522%2C%20-10.722%2051.883%2C%20-10.722%2052.153%2C%20-9.941%2052.561%2C%20-9.84375%2053.021%2C%20-10.173%2053.291%2C%20-10.404%2053.572%2C%20-10.404%2053.93%2C%20-10.25%2054.29%2C%20-9.92%2054.399%2C%20-8.833%2054.367%2C%20-8.975%2054.622%2C%20-8.8%2054.908%2C%20-8.481%2055.272%2C%20-8.063%2055.379))\n",
      "Service Unavailable\n",
      "294640\n",
      "http://api.effechecka.org/checklist.tsv?traitSelector=&wktString=GEOMETRYCOLLECTION%28POLYGON%20%28%2835.033%2029.631%2C%2034.856%2029.739%2C%2034.547%2030.4%2C%2034.27%2031.217%2C%2034.39%2031.394%2C%2034.491%2031.597%2C%2034.963%2032.821%2C%2035.115%2033.09%2C%2035.534%2033.121%2C%2035.625%2033.248%2C%2035.847%2033.2%2C%2035.87%2033.056%2C%2035.756%2032.726%2C%2035.568%2032.39%2C%2035.405%2032.508%2C%2035.076%2032.468%2C%2035.006%2032.028%2C%2035.057%2031.852%2C%2035.243%2031.751%2C%2034.89%2031.377%2C%2035.384%2031.486%2C%2035.39%2031.244%2C%2035.415%2030.949%2C%2035.202%2030.575%2C%2035.192%2030.347%2C%2035.176%2030.119%2C%2035.033%2029.631%29%29%29\n",
      "Service Unavailable\n",
      "complete\n"
     ]
    }
   ],
   "source": [
    "out_files = ['output1.tsv','output2.tsv','output3.tsv']\n",
    "#all of the code to line 47 is about reading the input json and forming the URL for the API query\n",
    "for index, line in enumerate(in_file):\n",
    "    line = line.strip()\n",
    "    row = line.split('\\t')\n",
    "    geonamesid = row[2]\n",
    "    iso = row[1]\n",
    "    print(geonamesid) #print the id so you know what country you are on\n",
    "    country = row[0]\n",
    "    polygons = shapes['features']\n",
    "    for polygon in polygons:\n",
    "        geoid = polygon['properties']['geoNameId']\n",
    "        if geonamesid == geoid: #use the geonames id to find the right polygon in the shapes file\n",
    "            shape_type = polygon['geometry']['type']\n",
    "            if shape_type == 'Polygon': #some country polygons are multiple polygons. Need a different procedure\n",
    "                p = []\n",
    "                wkt = polygon['geometry']['coordinates'][0]\n",
    "                for i in wkt:\n",
    "                    z = []\n",
    "                    lat = i[1]\n",
    "                    lon = i[0]\n",
    "                    z.append(str(lon))\n",
    "                    z.append(str(lat))\n",
    "                    m = '%20'.join(z)\n",
    "                    p.append(str(m))\n",
    "                q = '%2C%20'.join(p)\n",
    "                url = 'http://api.effechecka.org/checklist.tsv?traitSelector=&wktString=POLYGON((' + str(q) + '))'\n",
    "                z = 'POLYGON((' + str(q) + '))'\n",
    "            elif shape_type == 'MultiPolygon':\n",
    "                q = ''\n",
    "                url = 'http://api.effechecka.org/checklist.tsv?traitSelector=&wktString=GEOMETRYCOLLECTION%28POLYGON%20%28%28'\n",
    "                wkt = polygon['geometry']['coordinates']\n",
    "                for k in wkt:\n",
    "                    k = k[0]\n",
    "                    if len(k) == 0: #the process of shortening the polygons left a lot of blank coordinates. They get removed here.\n",
    "                        continue\n",
    "                    p = []\n",
    "                    for i in k:\n",
    "                        z = []\n",
    "                        for j in i:\n",
    "                            z.append(str(j))\n",
    "                        m = '%20'.join(z)\n",
    "                        p.append(str(m))\n",
    "                    q = q + '%2C%20'.join(p) + '%29%29%2CPOLYGON%20%28%28'\n",
    "                url = url + q\n",
    "                url = url.strip('%2CPOLYGON%20%28%28')\n",
    "                url = url + '%29'\n",
    "                z = 'GEOMETRYCOLLECTION%28POLYGON%20%28%28' + q.strip('%2CPOLYGON%20%28%28')\n",
    "                z = z + '%29'\n",
    "            print(url)\n",
    "            try: urllib.request.urlretrieve(url, out_files[index]) #This is where the url is submitted to the API and results are read\n",
    "            except urllib.error.URLError as e:\n",
    "                 print(e.reason)\n",
    "            with open(out_files[index], 'a') as u:\n",
    "                u.write('\\ncountry\\t' + country + '\\n')\n",
    "                u.write('country_uri\\t' + geonamesid + '\\n')\n",
    "                u.write('polygon\\t' + z + '\\n')\n",
    "                u.close()\n",
    "print('complete') #make sure the code gets to the end\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}