examples/add_property.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Adding-a-Property-to-wikirepo\" data-toc-modified-id=\"Adding-a-Property-to-wikirepo-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Adding a Property to wikirepo</a></span><ul class=\"toc-item\"><li><span><a href=\"#Adding-single-column-property\" data-toc-modified-id=\"Adding-single-column-property-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Adding single column property</a></span></li><li><span><a href=\"#Adding-a-single-column-property-that-spans-time\" data-toc-modified-id=\"Adding-a-single-column-property-that-spans-time-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Adding a single column property that spans time</a></span></li><li><span><a href=\"#Adding-a-multi-column-property\" data-toc-modified-id=\"Adding-a-multi-column-property-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Adding a multi-column property</a></span></li></ul></li><li><span><a href=\"#Adding-a-Property-to-Wikidata\" data-toc-modified-id=\"Adding-a-Property-to-Wikidata-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Adding a Property to Wikidata</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Adding Properties**\n",
    "\n",
    "In this example we'll show how to add properties to wikirepo. See [examples/add_data](http://localhost:8888/notebooks/programming/wikirepo/examples/add_data.ipynb) for (eventually) how to leverage wikirepo to add data to Wikidata.\n",
    "\n",
    "Adding properties to wikirepo can be as simple as finding a wikirepo data module that queries a similar data structure, copying this module to the appropriate data directory for the new property (see next note), renaming the module to what the user should enter to query it, and assigning appropriate values to the variables that make up the module header: `pid`, `sub_pid`, `col_name`, `col_prefix`, `ignore_char` and `span`. To fully detail this, we're going to pretend that the following properties can't already be accessed by wikirepo:\n",
    "\n",
    "- ['P1082' (population)](https://www.wikidata.org/wiki/Property:P1082)\n",
    "- ['P6' (head of government)](https://www.wikidata.org/wiki/Property:P6)\n",
    "- ['P172' (ethnicity)](https://www.wikidata.org/wiki/Property:P172)\n",
    "\n",
    "The final modules for each of these can be found in [data/demographic/population](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/demographic/population.py), [data/political/executive](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/political/executive.py), and [data/demographic/ethnic_div](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/demographic/ethnic_div.py) respectively. The focus will be how to add a property that already exists on [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page) to wikirepo, with later versions covering the process of adding a property to Wikidata as well.\n",
    "\n",
    "**Note:** by \"the appropriate data directory for the new property\" we mean that a new module should go into the [wikirepo/data](https://github.com/andrewtavis/wikirepo/tree/main/wikirepo/data) directory that matches a Wikidata sub-page. Sometimes data isn't on the location's page itself, but rather on a sub-page. An example is that certain economic properties for [Germany](https://www.wikidata.org/wiki/Q183) are found on the page [economy of Germany](https://www.wikidata.org/wiki/Q8046). wikirepo checks for a property on the main page of a location first, and if the property is not found then the package checks the sub-page associated with the module's directory (the user is notified that the property does not exist for the given location if it is found in neither page). Properties are often moved from main pages to sub-pages, so even current main page property modules need to be organized based on where they could be re-indexed. Worst comes to worst, put the module in [data/misc](https://github.com/andrewtavis/wikirepo/tree/main/wikirepo/data/misc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:32.138356Z",
     "start_time": "2020-12-08T11:29:31.696107Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:99% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from wikirepo.data import time_utils, wd_utils\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:99% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use ['Q183' (Germany)](https://www.wikidata.org/wiki/Q183) for this example. First we'll initialize an `EntitiesDict` and the QID, and then we'll load in the entity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:36.673058Z",
     "start_time": "2020-12-08T11:29:34.630060Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Germany']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ents_dict = wd_utils.EntitiesDict()\n",
    "qid = 'Q183'\n",
    "\n",
    "ent = wd_utils.load_ent(ents_dict=ents_dict, pq_id=qid)\n",
    "ents_dict.key_lbls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a Property to wikirepo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding single column property"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "['P1082' (population)](https://www.wikidata.org/wiki/Property:P1082) is an example of a property that goes in a single column, which also only occurs once at any given time.\n",
    "\n",
    "Let's start by defining our property and checking an element of the population data for Germany:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:36.679579Z",
     "start_time": "2020-12-08T11:29:36.674835Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mainsnak': {'snaktype': 'value',\n",
       "  'property': 'P1082',\n",
       "  'datavalue': {'value': {'amount': '+80500000',\n",
       "    'unit': '1',\n",
       "    'upperBound': '+80500500',\n",
       "    'lowerBound': '+80499500'},\n",
       "   'type': 'quantity'},\n",
       "  'datatype': 'quantity'},\n",
       " 'type': 'statement',\n",
       " 'qualifiers': {'P585': [{'snaktype': 'value',\n",
       "    'property': 'P585',\n",
       "    'hash': 'd071256bb4b9260491239bfad2cc561ad8bf870c',\n",
       "    'datavalue': {'value': {'time': '+2012-12-31T00:00:00Z',\n",
       "      'timezone': 0,\n",
       "      'before': 0,\n",
       "      'after': 0,\n",
       "      'precision': 11,\n",
       "      'calendarmodel': 'http://www.wikidata.org/entity/Q1985727'},\n",
       "     'type': 'time'},\n",
       "    'datatype': 'time'}]},\n",
       " 'qualifiers-order': ['P585'],\n",
       " 'id': 'Q183$3c493715-464a-6bad-ce6c-0f204e655157',\n",
       " 'rank': 'normal',\n",
       " 'references': [{'hash': '1112211b516d0ce090dfd3dd197bf7b7a4b88eaf',\n",
       "   'snaks': {'P143': [{'snaktype': 'value',\n",
       "      'property': 'P143',\n",
       "      'datavalue': {'value': {'entity-type': 'item',\n",
       "        'numeric-id': 764739,\n",
       "        'id': 'Q764739'},\n",
       "       'type': 'wikibase-entityid'},\n",
       "      'datatype': 'wikibase-item'}],\n",
       "    'P854': [{'snaktype': 'value',\n",
       "      'property': 'P854',\n",
       "      'datavalue': {'value': 'https://www.destatis.de/DE/ZahlenFakten/GesellschaftStaat/Bevoelkerung/Bevoelkerung.html',\n",
       "       'type': 'string'},\n",
       "      'datatype': 'url'}],\n",
       "    'P813': [{'snaktype': 'value',\n",
       "      'property': 'P813',\n",
       "      'datavalue': {'value': {'time': '+2014-06-14T00:00:00Z',\n",
       "        'timezone': 0,\n",
       "        'before': 0,\n",
       "        'after': 0,\n",
       "        'precision': 11,\n",
       "        'calendarmodel': 'http://www.wikidata.org/entity/Q1985727'},\n",
       "       'type': 'time'},\n",
       "      'datatype': 'time'}]},\n",
       "   'snaks-order': ['P143', 'P854', 'P813']}]}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_pid = 'P1082'\n",
    "\n",
    "pop_0_entry = wd_utils.get_prop(ents_dict=ents_dict, qid=qid, pid=pop_pid)[0]\n",
    "pop_0_entry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The big thing to notice in that is that the qualifier ['P585' (point in time)](https://www.wikidata.org/wiki/Property:P585) is present. That and that `prop_0_entry['mainsnak']['datavalue']['value']['amount']` is a single value tells us that this property should go into a single column. \n",
    "\n",
    "Let's check this value, as well as get its date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:37.212298Z",
     "start_time": "2020-12-08T11:29:37.208646Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'+80500000'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_0_val = pop_0_entry['mainsnak']['datavalue']['value']['amount']\n",
    "pop_0_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:37.644350Z",
     "start_time": "2020-12-08T11:29:37.640642Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'+2012-12-31T00:00:00Z'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_0_t = pop_0_entry['qualifiers']['P585'][0]['datavalue']['value']['time']\n",
    "pop_0_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From that we see that we could have a character that needs to be ignored - specifically the `+`. We actually don't though, as wikirepo will convert this value to an integer, and `int('+string_number')` gets rid of the `+` for us.\n",
    "\n",
    "**Note:** wikirepo will also take care of the date for us. The package will first format the date, and then it will use a provided `time_lvl` variable's value to truncate this formatted `datetime.date` object to an appropriate level. Here's a quick demo of this assuming that the `time_lvl` of our query is `yearly`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:38.241261Z",
     "start_time": "2020-12-08T11:29:38.237517Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2012, 12, 31)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pop_0_t_formatted = wd_utils.format_t(pop_0_t)\n",
    "pop_0_t_formatted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:38.518157Z",
     "start_time": "2020-12-08T11:29:38.514295Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2012'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_utils.truncate_date(d=pop_0_t_formatted, time_lvl='yearly')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value itself will be included if the above year is included in the `timespan` value passed. If no `time_lvl` variable is passed, then the full date will be maintained, and its value will be queried if it's the most recent, with the date then being appended as a string for documentation of when the value comes from.\n",
    "\n",
    "Final notes on the property module: the value in question can be accessed directly instead of through another property, so this tells us that we have no need for the `sub_pid` variable (more on this later); as the value goes into one column, we use the `col_name` variable instead of `col_prefix` (more on this later as well); and the value occurs at only one time, so we keep the `span` variable as `False` (more on this later too).\n",
    "\n",
    "We now have all the information needed to make the **population** module's header:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:39.325093Z",
     "start_time": "2020-12-08T11:29:39.322385Z"
    }
   },
   "outputs": [],
   "source": [
    "pid = 'P1082'\n",
    "sub_pid = None\n",
    "col_name = 'population'\n",
    "col_prefix = None\n",
    "ignore_char = ''\n",
    "span = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final module can again be found in [data/demographic/population](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/demographic/population.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding a single column property that spans time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An executive via ['P6' (head of government)](https://www.wikidata.org/wiki/Property:P6) is an example of a property that goes in a single column that further occurs over a span of time.\n",
    "\n",
    "Let's start again by defining the pid and loading in an entry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:40.996529Z",
     "start_time": "2020-12-08T11:29:40.989802Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mainsnak': {'snaktype': 'value',\n",
       "  'property': 'P6',\n",
       "  'datavalue': {'value': {'entity-type': 'item',\n",
       "    'numeric-id': 567,\n",
       "    'id': 'Q567'},\n",
       "   'type': 'wikibase-entityid'},\n",
       "  'datatype': 'wikibase-item'},\n",
       " 'type': 'statement',\n",
       " 'qualifiers': {'P580': [{'snaktype': 'value',\n",
       "    'property': 'P580',\n",
       "    'hash': 'ad8007db4be39b05f62a2bf5821d32c5464bb183',\n",
       "    'datavalue': {'value': {'time': '+2005-11-22T00:00:00Z',\n",
       "      'timezone': 0,\n",
       "      'before': 0,\n",
       "      'after': 0,\n",
       "      'precision': 11,\n",
       "      'calendarmodel': 'http://www.wikidata.org/entity/Q1985727'},\n",
       "     'type': 'time'},\n",
       "    'datatype': 'time'}]},\n",
       " 'qualifiers-order': ['P580'],\n",
       " 'id': 'q183$d0db3461-4291-0b36-4092-c40d14699212',\n",
       " 'rank': 'preferred',\n",
       " 'references': [{'hash': '7c5619b50f5af5766a660bda2eb09605dee4df72',\n",
       "   'snaks': {'P143': [{'snaktype': 'value',\n",
       "      'property': 'P143',\n",
       "      'datavalue': {'value': {'entity-type': 'item',\n",
       "        'numeric-id': 317027,\n",
       "        'id': 'Q317027'},\n",
       "       'type': 'wikibase-entityid'},\n",
       "      'datatype': 'wikibase-item'}],\n",
       "    'P854': [{'snaktype': 'value',\n",
       "      'property': 'P854',\n",
       "      'datavalue': {'value': 'http://www.bundeskanzlerin.de/Webs/BKin/EN/AngelaMerkel/Biography/biography_node.html;jsessionid=D04CEAD1827AC93F21CB6E908B31A5AA.s4t1',\n",
       "       'type': 'string'},\n",
       "      'datatype': 'url'}],\n",
       "    'P813': [{'snaktype': 'value',\n",
       "      'property': 'P813',\n",
       "      'datavalue': {'value': {'time': '+2014-03-28T00:00:00Z',\n",
       "        'timezone': 0,\n",
       "        'before': 0,\n",
       "        'after': 0,\n",
       "        'precision': 11,\n",
       "        'calendarmodel': 'http://www.wikidata.org/entity/Q1985727'},\n",
       "       'type': 'time'},\n",
       "      'datatype': 'time'}],\n",
       "    'P1683': [{'snaktype': 'value',\n",
       "      'property': 'P1683',\n",
       "      'datavalue': {'value': {'text': 'since November 2005 [Angela Merkel is] Chancellor of the Federal Republic of Germany',\n",
       "        'language': 'en'},\n",
       "       'type': 'monolingualtext'},\n",
       "      'datatype': 'monolingualtext'}]},\n",
       "   'snaks-order': ['P143', 'P854', 'P813', 'P1683']}]}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exec_pid = 'P6'\n",
    "\n",
    "exec_0_entry = wd_utils.get_prop(ents_dict=ents_dict, qid=qid, pid=exec_pid)[0]\n",
    "exec_0_entry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly we can see that the value in question cannot be directly subscripted for, as it is a QID entity itself. wikirepo will access the variable for us and derive its label, but let's find out who it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:41.713928Z",
     "start_time": "2020-12-08T11:29:41.709914Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Q567'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exec_0_qid = exec_0_entry['mainsnak']['datavalue']['value']['id']\n",
    "exec_0_qid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:43.069363Z",
     "start_time": "2020-12-08T11:29:42.108621Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Angela Merkel'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wd_utils.get_lbl(ents_dict=ents_dict, pq_id=exec_0_qid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That this entity is a span can be seen by the fact that it does not have ['P585' (point in time)](https://www.wikidata.org/wiki/Property:P585), but rather ['P580' (start time)](https://www.wikidata.org/wiki/Property:P580). Values in this property can also have the property ['P582' (end time)](https://www.wikidata.org/wiki/Property:P582). \n",
    "\n",
    "**Note:** wikirepo assumes that an entity that has a start time and lacks an end time is the current subject for the property, so the latest date in the `timespan` argument for query functions will be used. The opposite is true for if an end time is present without a start time - the first date in the `timespan` will be used based on the assumption that this is the first subject of the property.\n",
    "\n",
    "Having values or subjects with start and end times implies that the `span` variable for the module header should in this case be `True`. We still are putting our results in a single column, so we use `col_name` instead of `col_prefix` (this is covered in the next section), and we can again the ignore `sub_pid` variable (also covered in the next section).\n",
    "\n",
    "From this we have all the information we need for the **executive** module's header:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:43.089259Z",
     "start_time": "2020-12-08T11:29:43.086580Z"
    }
   },
   "outputs": [],
   "source": [
    "pid = 'P6'\n",
    "sub_pid = None\n",
    "col_name = 'executive'\n",
    "col_prefix = None\n",
    "ignore_char = ''\n",
    "span = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting module can again be found in [data/political/executive](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/political/executive.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding a multi-column property "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ethnic diversity via ['P172' (ethnic group)](https://www.wikidata.org/wiki/Property:P172) is an example of a property that should be split over multiple columns. Rather than put all the information into a single column for the user to then split, wikirepo instead prefixes each potential element and creates columns for them for their respective data.\n",
    "\n",
    "Let's look at the first element of German ethnicity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:44.177540Z",
     "start_time": "2020-12-08T11:29:44.172241Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mainsnak': {'snaktype': 'value',\n",
       "  'property': 'P172',\n",
       "  'datavalue': {'value': {'entity-type': 'item',\n",
       "    'numeric-id': 42884,\n",
       "    'id': 'Q42884'},\n",
       "   'type': 'wikibase-entityid'},\n",
       "  'datatype': 'wikibase-item'},\n",
       " 'type': 'statement',\n",
       " 'qualifiers': {'P1107': [{'snaktype': 'value',\n",
       "    'property': 'P1107',\n",
       "    'hash': '17752ff515cf871f1b4e82ae5ee4e1cea61556ff',\n",
       "    'datavalue': {'value': {'amount': '+0.915', 'unit': '1'},\n",
       "     'type': 'quantity'},\n",
       "    'datatype': 'quantity'}]},\n",
       " 'qualifiers-order': ['P1107'],\n",
       " 'id': 'Q183$c3db8ed3-4346-945d-75d7-de9ff7181e83',\n",
       " 'rank': 'normal',\n",
       " 'references': [{'hash': '35ad938ca5a2b12719ee2b3fbe70f8bf27e77284',\n",
       "   'snaks': {'P248': [{'snaktype': 'value',\n",
       "      'property': 'P248',\n",
       "      'datavalue': {'value': {'entity-type': 'item',\n",
       "        'numeric-id': 11191,\n",
       "        'id': 'Q11191'},\n",
       "       'type': 'wikibase-entityid'},\n",
       "      'datatype': 'wikibase-item'}],\n",
       "    'P813': [{'snaktype': 'value',\n",
       "      'property': 'P813',\n",
       "      'datavalue': {'value': {'time': '+2017-09-28T00:00:00Z',\n",
       "        'timezone': 0,\n",
       "        'before': 0,\n",
       "        'after': 0,\n",
       "        'precision': 11,\n",
       "        'calendarmodel': 'http://www.wikidata.org/entity/Q1985727'},\n",
       "       'type': 'time'},\n",
       "      'datatype': 'time'}]},\n",
       "   'snaks-order': ['P248', 'P813']}]}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ethnic_div_pid = 'P172'\n",
    "\n",
    "ethnic_div_0_entry = wd_utils.get_prop(ents_dict=ents_dict, qid=qid, pid=ethnic_div_pid)[0]\n",
    "ethnic_div_0_entry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the values for this property is an entity, and the values are stored within sub PIDs. As before, let's check some QIDs of this value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:45.068913Z",
     "start_time": "2020-12-08T11:29:45.065127Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Q42884'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ethnic_div_0_qid = ethnic_div_0_entry['mainsnak']['datavalue']['value']['id']\n",
    "ethnic_div_0_qid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:46.092155Z",
     "start_time": "2020-12-08T11:29:45.474176Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Germans'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ethnic_div_0_lbl = wd_utils.get_lbl(ents_dict=ents_dict, pq_id=ethnic_div_0_qid)\n",
    "ethnic_div_0_lbl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:46.614310Z",
     "start_time": "2020-12-08T11:29:46.094634Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Turks'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ethnic_div_1_entry = wd_utils.get_prop(ents_dict=ents_dict, qid=qid, pid=ethnic_div_pid)[1]\n",
    "ethnic_div_1_qid = ethnic_div_1_entry['mainsnak']['datavalue']['value']['id']\n",
    "ethnic_div_1_lbl = wd_utils.get_lbl(ents_dict=ents_dict, pq_id=ethnic_div_1_qid)\n",
    "ethnic_div_1_lbl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value itself needs to be subsetted for using ['P1107' (proportion)](https://www.wikidata.org/wiki/Property:P1107). wikirepo will do this for us, but let's subset for the first value anyway:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:46.868401Z",
     "start_time": "2020-12-08T11:29:46.864730Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'+0.915'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ethnic_div_0_val = ethnic_div_0_entry['qualifiers']['P1107'][0]['datavalue']['value']['amount']\n",
    "ethnic_div_0_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this property we thus need to use a `sub_pid` variable that tells wikirepo where to look for the value. \n",
    "\n",
    "**None:** another use of `sub_pid` is to set its value to `bool`. This tells wikirepo to assign `True` if the property is present. An example of this is [data/institutional/org_membership](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/institutional/org_membership.py) where a boolean value is assigned to columns based on if a location is a member of an organization at a given time. Values of `False` need to be filled afterwards, and some values are replaced for organizations that are widely known. This is thus an example of a property that requires a bit more work than simply setting the module header.\n",
    "\n",
    "Continuing, as we want the values to be put into separate columns where the QIDs labels for the entries get prefixed, we need to use the `col_prefix` variable and set the `col_name` variable to `None`. Let's choose `eth` for `col_prefix`, meaning that columns produced will be `eth_germans`, `eth_turks`, etc (an underscore is added automatically). To complete the needed information, the values themselves are only present at individual times, so in this case we can set `span` to `False`.\n",
    "\n",
    "From here we have the full information for the header of the **ethnic_div** module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-08T11:29:48.197124Z",
     "start_time": "2020-12-08T11:29:48.194308Z"
    }
   },
   "outputs": [],
   "source": [
    "pid = 'P172'\n",
    "sub_pid = 'P1107'\n",
    "col_name = None\n",
    "col_prefix = 'eth'\n",
    "ignore_char = ''\n",
    "span = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final version of this module can be found in [data/demographic/ethnic_div](https://github.com/andrewtavis/wikirepo/blob/main/wikirepo/data/demographic/ethnic_div.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a Property to Wikidata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}