Partial block measurements.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Partial block measurements\n",
    "\n",
    "This work is tracked in [T209403](https://phabricator.wikimedia.org/T209403).\n",
    "\n",
    "Originally, these measurements are geared towards being run on a monthly basis, using the Data Lake as the source. Currently, partial blocks are deployed on a low number of wikis and have not been deployed for long, meaning that monthly measurements makes little sense. Secondly, data on partial blocks are not yet available in the Data Lake.\n",
    "\n",
    "Until further notice, we'll grab the data from the replicated MediaWiki databases and explore whether daily measurements make sense.\n",
    "\n",
    "We have the following measurements defined in our master document, defined on a monthly basis:\n",
    "\n",
    "* Number of partial blocks for users/IPs where the block has a start timestamp within the month of interest.\n",
    "* Number of partial blocks for users/IPs where the duration of the partial block intersects with the month of interest and the user/IP makes ≥1 non-reverted edits during the intersection.\n",
    "* Number of partial blocks of users/IPs where:\n",
    "   * The partial block has a duration intersecting with the month of interest, and the block does not have pages added during the intersection.\n",
    "   * The partial block has a duration intersecting with the month of interest, and the block is not replaced by a sitewide block during the intersection.\n",
    "   * The partial block has an expiry timestamp within the month of interest and the expiration timestamp is not altered to increase the duration of the partial block.\n",
    "\n",
    "The first part of this notebook explores the first measurement, making plots of number of partial blocks set (for both registered users and IPs) per day for all wikis. Once we have data for partial blocks in the Data Lake, we'll turn these into monthly metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You can find the source for `wmfdata` at https://github.com/neilpquinn/wmfdata\n"
     ]
    }
   ],
   "source": [
    "## Load Python libraries\n",
    "\n",
    "import re\n",
    "import datetime as dt\n",
    "\n",
    "import pandas as pd\n",
    "import phpserialize as ps\n",
    "import pymysql\n",
    "\n",
    "from wmfdata import hive\n",
    "from growth import db, utils\n",
    "\n",
    "import mwreverts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Load the RPython library so we can use R for graphs\n",
    "\n",
    "%load_ext rpy2.ipython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: \n",
      "Attaching package: ‘zoo’\n",
      "\n",
      "\n",
      "R[write to console]: The following objects are masked from ‘package:base’:\n",
      "\n",
      "    as.Date, as.Date.numeric\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%R\n",
    "library(ggplot2);\n",
    "library(data.table);\n",
    "library(zoo);\n",
    "library(tidyr);\n",
    "library(RColorBrewer);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Some configuration variables.\n",
    "## Start and end timestamps allow us to speed up database queries.\n",
    "\n",
    "start_time = dt.datetime(2019, 1, 1, 0, 0, 0)\n",
    "end_time = dt.datetime(2019, 7, 22, 0, 0, 0)\n",
    "\n",
    "## Mapping from wiki DB name to host/port information\n",
    "dbhost_map = dict()\n",
    "\n",
    "## Mapping from wiki DB name to database connection\n",
    "dbconn_map = dict()\n",
    "\n",
    "## Format strings:\n",
    "## MediaWiki database timestamp format\n",
    "mw_format = \"%Y%m%d%H%M%S\"\n",
    "hive_format = \"%Y-%m-%dT%H:%M:%S\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## See T218626 for the status of deployment of partial blocks\n",
    "\n",
    "wikis = ['arwiki', 'metawiki', 'mediawikiwiki', 'itwiki', 'fawiki',\n",
    "         'plwiki', 'frwiki', 'zhwiki', 'tewiki', 'bnwiki',\n",
    "         'huwiki', 'jawiki', 'hewiki', 'srwiki', 'enwikivoyage', 'enwiktionary',\n",
    "         'enwikisource', 'hewikivoyage']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Some configuration:\n",
    "\n",
    "## Dates that we will use to make plots, they'll show from start date and up to,\n",
    "## but not including, the end date\n",
    "start_display_date = dt.date(2019, 1, 1)\n",
    "end_display_date = dt.date(2019, 7, 1)\n",
    "\n",
    "## We'll gather data from a year prior to the start date so we can have a yearly running\n",
    "## average across the whole graph\n",
    "start_data_date = dt.date(start_display_date.year - 1,\n",
    "                          start_display_date.month,\n",
    "                          start_display_date.day)\n",
    "end_data_date = end_display_date\n",
    "\n",
    "## Some statistics require a 28-day period, so in those cases we'll truncate to the previous\n",
    "## month, which we can get at with some date math (assuming that end_display_date is the\n",
    "## first of the month):\n",
    "truncated_end_date = (end_display_date - dt.timedelta(days = 1)).replace(day = 1)\n",
    "\n",
    "## The name of the database snapshot we're using, which should be the year and month\n",
    "## of the truncated end date\n",
    "snapshot = \"{}\".format(truncated_end_date.strftime(\"%Y-%m\"))\n",
    "\n",
    "## Because rpy2 doesn't convert datetime.date objects, we'll make the display dates\n",
    "## strings to help with that:\n",
    "start_display_date_str = str(start_display_date)\n",
    "end_display_date_str = str(end_display_date)\n",
    "truncated_end_date_str = str(truncated_end_date)\n",
    "\n",
    "## Filenames/paths to datasets that can be shared\n",
    "partial_monthly_filename = \"public_datasets/partial_blocks_monthly.tsv\"\n",
    "partial_constructive_filename = \"public_datasets/partial_blocks_constructive_edits.tsv\"\n",
    "partial_expired_filename = \"public_datasets/sitewide_expired_blocks.tsv\"\n",
    "partial_expired_noreblock_filename = \"public_datasets/sitewide_expired_not-reblocked.tsv\"\n",
    "partial_expired_nonreverted_filename = \"public_datasets/sitewide_expired_non-reverted.tsv\"\n",
    "partial_expired_nonreverted_ns0_filename = \"public_datasets/sitewide_expired_non-reverted_ns0.tsv\"\n",
    "\n",
    "def to_tsv(df, output_file):\n",
    "    df.to_csv(output_file, sep='\\t', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The second function needs dnspython to work\n",
    "import dns.resolver\n",
    "import glob\n",
    "\n",
    "def get_mediawiki_section_dbname_mapping(mw_config_path, use_x1):\n",
    "    db_mapping = {}\n",
    "    if use_x1:\n",
    "        dblist_section_paths = [mw_config_path.rstrip('/') + '/dblists/all.dblist']\n",
    "    else:\n",
    "        dblist_section_paths = glob.glob(mw_config_path.rstrip('/') + '/dblists/s[0-9]*.dblist')\n",
    "    for dblist_section_path in dblist_section_paths:\n",
    "        with open(dblist_section_path, 'r') as f:\n",
    "            for db in f.readlines():\n",
    "                db_mapping[db.strip()] = dblist_section_path.strip().rstrip('.dblist').split('/')[-1]\n",
    "\n",
    "    return db_mapping\n",
    "\n",
    "\n",
    "def get_dbstore_host_port(db_mapping, use_x1, dbname):\n",
    "    if dbname == 'staging':\n",
    "        shard = 'staging'\n",
    "    elif use_x1:\n",
    "        shard = 'x1'\n",
    "    else:\n",
    "        try:\n",
    "            shard = db_mapping[dbname]\n",
    "        except KeyError:\n",
    "            raise RuntimeError(\"The database {} is not listed among the dblist files of the supported sections.\"\n",
    "                               .format(dbname))\n",
    "    answers = dns.resolver.query('_' + shard + '-analytics._tcp.eqiad.wmnet', 'SRV')\n",
    "    host, port = str(answers[0].target), answers[0].port\n",
    "    return (host,port)\n",
    "\n",
    "wikidb_map = get_mediawiki_section_dbname_mapping('/srv/mediawiki-config', False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "for wiki in wikis:\n",
    "    dbhost_map[wiki] = get_dbstore_host_port(wikidb_map, False, wiki)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "for wiki in wikis:\n",
    "    dbconn_map[wiki] = db.get_db_conn(wiki)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Code form wmfdata to decode bytestrings returned from the database into UTF-8 strings\n",
    "\n",
    "def try_decode(cell):\n",
    "    try:\n",
    "        return cell.decode(encoding = \"utf-8\")\n",
    "    except AttributeError:\n",
    "        return cell\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_params(row):\n",
    "    '''\n",
    "    Extract relevant block parameters from the given row of partial block log\n",
    "    data, and return a new `pandas.Series` that can be used to update a data frame\n",
    "    with columns for those parameters.\n",
    "    '''\n",
    "    params = ps.loads(row['log_params'].encode('utf-8'), decode_strings=True)\n",
    "    \n",
    "    try:\n",
    "        duration = params['5::duration']\n",
    "    except KeyError:\n",
    "        duration = None\n",
    "        \n",
    "    try:\n",
    "        flags = params['6::flags']\n",
    "    except KeyError:\n",
    "        flags = None\n",
    "\n",
    "    ## Restrictions can either refer to pages or namespaces (or both, I suspect).\n",
    "    ## We therefore have to check and act accordingly.\n",
    "    try:     \n",
    "        num_pages = 0\n",
    "        num_namespaces = 0\n",
    "        \n",
    "        restrictions = params['7::restrictions']\n",
    "        if 'pages' in restrictions:\n",
    "            num_pages = len(restrictions['pages'])\n",
    "            \n",
    "        if 'namespaces' in restrictions:\n",
    "            num_namespaces = len(restrictions['namespaces'])\n",
    "    except KeyError:\n",
    "        num_pages = None\n",
    "        num_namespaces = None\n",
    "    \n",
    "    return(pd.Series([duration, flags, num_pages, num_namespaces]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "## SQL query to get data on partial blocks, adapted from\n",
    "## https://github.com/dayllanmaza/wikireplicas-reports/blob/master/generators/partial_blocks.py\n",
    "\n",
    "def get_partial_blocks(wikis, dbconns, start_timestamp, end_timestamp):\n",
    "    pb_query = '''\n",
    "    SELECT DATABASE() AS wiki,\n",
    "           log_timestamp,\n",
    "           log_params,\n",
    "           log_user_text AS blocker,\n",
    "           log_title AS blockee,\n",
    "           comment_text AS reason\n",
    "    FROM {wiki}.logging\n",
    "    LEFT JOIN {wiki}.comment\n",
    "    ON log_comment_id=comment_id\n",
    "    WHERE log_timestamp >= \"{start_timestamp}\"\n",
    "    AND log_timestamp < \"{end_timestamp}\"\n",
    "    AND log_type = \"block\"\n",
    "    AND log_action = \"block\" -- only interested in initial blocks created\n",
    "    AND log_params LIKE '%\"sitewide\";b:0;%'\n",
    "    '''\n",
    "    \n",
    "    pbs = []\n",
    "    for wiki in wikis:\n",
    "        print('getting partial blocks from {}'.format(wiki))\n",
    "        df = pd.read_sql_query(\n",
    "            pb_query.format(\n",
    "                wiki = wiki,\n",
    "                start_timestamp = start_timestamp.strftime(mw_format),\n",
    "                end_timestamp = end_timestamp.strftime(mw_format)),\n",
    "            dbconns[wiki])\n",
    "        df = df.applymap(try_decode).rename(columns = try_decode)\n",
    "        \n",
    "        if len(df) == 0:\n",
    "            continue\n",
    "        \n",
    "        ## Turn the timestamps into datetime objects, and add a log_date string for convenience\n",
    "        df['log_timestamp'] = pd.to_datetime(df['log_timestamp'], format=mw_format, utc=True)\n",
    "        df['log_date'] = df['log_timestamp'].apply(lambda x: str(x.date()))\n",
    "        \n",
    "        df[['block_duration', 'block_flags', 'block_num_pages', 'block_num_namespaces']] = df.apply(extract_params, axis=1)\n",
    "        \n",
    "        pbs.append(df)\n",
    "    \n",
    "    return(pd.concat(pbs))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting partial blocks from arwiki\n",
      "getting partial blocks from metawiki\n",
      "getting partial blocks from mediawikiwiki\n",
      "getting partial blocks from itwiki\n",
      "getting partial blocks from fawiki\n",
      "getting partial blocks from plwiki\n",
      "getting partial blocks from frwiki\n",
      "getting partial blocks from zhwiki\n",
      "getting partial blocks from tewiki\n",
      "getting partial blocks from bnwiki\n",
      "getting partial blocks from huwiki\n",
      "getting partial blocks from jawiki\n",
      "getting partial blocks from hewiki\n",
      "getting partial blocks from srwiki\n",
      "getting partial blocks from enwikivoyage\n",
      "getting partial blocks from enwiktionary\n",
      "getting partial blocks from enwikisource\n",
      "getting partial blocks from hewikivoyage\n"
     ]
    }
   ],
   "source": [
    "partial_blocks = get_partial_blocks(wikis, dbconn_map, start_time, end_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R -i partial_blocks\n",
    "\n",
    "partial_blocks = data.table(partial_blocks);\n",
    "partial_blocks[, log_date := as.Date(log_date)];\n",
    "\n",
    "## Some configuration variables\n",
    "graph_dir = 'graphs/';\n",
    "pb_graph_prefix = 'partial_blocks_per_day_';\n",
    "pb_graph_suffix = '.png';\n",
    "\n",
    "make_pb_graphs = function(pbs, graph_dir, prefix, suffix) {\n",
    "    wikis = unique(pbs$wiki);\n",
    "    \n",
    "    for(w in wikis) {\n",
    "        ## Grab the subset for this wiki, as we have different date ranges for each\n",
    "        wiki_blocks = pbs[wiki == w];\n",
    "        \n",
    "        ## Make a date sequence from the first to the last date, and left join against\n",
    "        ## the data to fill in any dates with 0 blocks.\n",
    "\n",
    "        dates = seq.Date(min(wiki_blocks$log_date), max(wiki_blocks$log_date), by='day');\n",
    "        dates = data.table(log_date = dates);\n",
    "\n",
    "        blocks_per_day = wiki_blocks[, list(num_blocks=sum(.N)), by=log_date];\n",
    "        blocks_per_day = blocks_per_day[dates, on = 'log_date'];\n",
    "        blocks_per_day[is.na(num_blocks), num_blocks := 0];\n",
    "\n",
    "        ## Add the moving averages\n",
    "        blocks_per_day[\n",
    "            , num_blocks_1wma := rollapply(\n",
    "                num_blocks,\n",
    "                width = 7,\n",
    "                FUN = mean,\n",
    "                na.rm = TRUE,\n",
    "                fill = 0,\n",
    "                align = 'right')];\n",
    "        blocks_per_day[\n",
    "            , num_blocks_2wma := rollapply(\n",
    "                num_blocks,\n",
    "                width = 14,\n",
    "                FUN = mean,\n",
    "                na.rm = TRUE,\n",
    "                fill = 0,\n",
    "                align = 'right')];\n",
    "        \n",
    "        ## Tidy up and make the plot\n",
    "        blocks_per_day_long = blocks_per_day %>% gather(measure, num_blocks, 2:4);\n",
    "        blocks_per_day_long = data.table(blocks_per_day_long);\n",
    "        blocks_per_day_long[measure == 'num_blocks', measure := 'raw data'];\n",
    "        blocks_per_day_long[measure == 'num_blocks_1wma', measure := '1-week MA'];\n",
    "        blocks_per_day_long[measure == 'num_blocks_2wma', measure := '2-week MA'];\n",
    "        blocks_per_day_long[\n",
    "            , measure := ordered(measure, rev(c('raw data', '1-week MA', '2-week MA')))];\n",
    "\n",
    "        ## Choose blues with some contrast, with the raw data getting the strongest color\n",
    "        b_palette = brewer.pal('Blues', n = 7)[c(3,5,7)];\n",
    "        \n",
    "        block_day_plot = ggplot(blocks_per_day_long,\n",
    "                                aes(x=log_date, y=num_blocks, color=measure)) +\n",
    "        scale_x_date(date_breaks = \"1 month\", date_labels = \"%d %b\") +\n",
    "        scale_y_continuous() +\n",
    "        scale_colour_manual(values = b_palette) +\n",
    "        expand_limits(y = 0) +\n",
    "        labs(title = paste0('Partial blocks created per day - ', w),\n",
    "             x = 'Date',\n",
    "             y = 'Number of blocks') +\n",
    "        theme_light(base_size = 14) +\n",
    "        geom_line();\n",
    "\n",
    "        ggsave(paste0(graph_dir, prefix, w, suffix),\n",
    "           plot = block_day_plot, width = 30, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "    }\n",
    "}\n",
    "\n",
    "make_pb_graphs(partial_blocks, graph_dir, pb_graph_prefix, pb_graph_suffix);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Partial blocks per month\n",
    "\n",
    "\"Number of partial blocks for users/IPs where the block has a start timestamp within the month of interest.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pb_monthly_query = '''\n",
    "SELECT pbs.wiki_db,\n",
    "       DATE_FORMAT(pbs.start_timestamp, 'yyyy-MM-01') AS log_month,\n",
    "       count(*) AS num_blocks\n",
    "FROM (\n",
    "    SELECT wiki_db,\n",
    "    start_timestamp,\n",
    "    COALESCE(LAG(inferred_from, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp),\n",
    "        \"block\") AS prev_inferred,\n",
    "    LAG(caused_by_event_type, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS prev_event_type,\n",
    "    LAG(end_timestamp, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS prev_end_timestamp\n",
    "    FROM wmf.mediawiki_user_history\n",
    "    WHERE snapshot = \"{snapshot}\"\n",
    "      AND wiki_db IN ({wikis})\n",
    "      AND start_timestamp BETWEEN \"{start_date}\" AND \"{end_date}\"\n",
    "      AND caused_by_event_type = 'alterblocks'\n",
    "      AND inferred_from IS NULL -- not an unblock\n",
    "      AND source_log_params['sitewide'] = 'false' -- don't forget that's a string, not boolean!\n",
    "    ) AS pbs\n",
    "WHERE pbs.prev_end_timestamp IS NULL\n",
    "   OR pbs.prev_inferred IS NULL\n",
    "   OR NOT (-- prev event is also a block, meaning this is a reblock\n",
    "    pbs.prev_end_timestamp = pbs.start_timestamp\n",
    "    AND pbs.prev_inferred = \"block\"\n",
    "    AND pbs.prev_event_type = 'alterblocks')\n",
    "GROUP BY pbs.wiki_db,\n",
    "DATE_FORMAT(pbs.start_timestamp, 'yyyy-MM-01')\n",
    "LIMIT 10000\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "monthly_counts = hive.run(pb_monthly_query.format(\n",
    "    snapshot = snapshot,\n",
    "    start_date = start_data_date,\n",
    "    end_date = end_data_date,\n",
    "    wikis = \",\".join(['\"{}\"'.format(w) for w in wikis])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've verified that the monthly counts (197 as of the April 2019 snapshot) matches the actual data, as there were 220 blocks in total and 23 of them were reblocks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n",
      "R[write to console]: geom_path: Each group consists of only one observation. Do you need to adjust\n",
      "the group aesthetic?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%R -i monthly_counts,start_display_date_str,end_display_date_str\n",
    "\n",
    "## Some configuration variables\n",
    "graph_dir = 'graphs/';\n",
    "pb_monthly_prefix = 'partial_blocks_per_month_';\n",
    "pb_graph_suffix = '.png';\n",
    "\n",
    "## FIXME: mapping codes to names needs a scalable solution, PBs are deployed to\n",
    "## many more wikis, so we'll either have to switch back to the short name,\n",
    "## or figure out a better way.\n",
    "wiki_names = list(\n",
    "    arwiki = 'Arabic Wikipedia',\n",
    "    metawiki = 'Meta Wiki',\n",
    "    mediawikiwiki = 'MediaWiki Wiki',\n",
    "    itwiki = 'Italian Wikipedia',\n",
    "    fawiki = 'Persian Wikipedia',\n",
    "    plwiki = 'Polish Wikipedia',\n",
    "    frwiki = 'French Wikipedia',\n",
    "    zhwiki = 'Chinese Wikipedia',\n",
    "    tewiki = 'Telugu Wikipedia',\n",
    "    bnwiki = 'Bengali Wikipedia',\n",
    "    huwiki = 'Hungarian Wikipedia',\n",
    "    jawiki = 'Japanese Wikipedia',\n",
    "    hewiki = 'Hebrew Wikipedia',\n",
    "    srwiki = 'Serbian Wikipedia',\n",
    "    enwikivoyage = 'English Wikivoyage',\n",
    "    enwiktionary = 'English Wiktionary',\n",
    "    enwikisource = 'English Wikisource',\n",
    "    hewikivoyage = 'Hebrew Wikivoyage'\n",
    ");\n",
    "\n",
    "monthly_counts = data.table(monthly_counts);\n",
    "monthly_counts[, log_month := as.Date(log_month)];\n",
    "\n",
    "## Add 3-month and 12-month running average\n",
    "monthly_counts[\n",
    "    , num_blocks_3mma := rollapply(\n",
    "        num_blocks,\n",
    "        width = 3,\n",
    "        FUN = mean,\n",
    "        na.rm = TRUE,\n",
    "        fill = 0,\n",
    "        align = 'right')];\n",
    "monthly_counts[\n",
    "    , num_blocks_1yma := rollapply(\n",
    "        num_blocks,\n",
    "        width = 12,\n",
    "        FUN = mean,\n",
    "        na.rm = TRUE,\n",
    "        fill = 0,\n",
    "        align = 'right')];\n",
    "\n",
    "monthly_counts_long = monthly_counts %>% gather(measure, num_blocks, 3:5);\n",
    "monthly_counts_long = data.table(monthly_counts_long);\n",
    "monthly_counts_long[measure == 'num_blocks', measure := 'raw data'];\n",
    "monthly_counts_long[measure == 'num_blocks_3mma', measure := '3-month MA'];\n",
    "monthly_counts_long[measure == 'num_blocks_1yma', measure := '1-year MA'];\n",
    "monthly_counts_long[\n",
    "    , measure := ordered(measure, rev(c('raw data', '3-month MA', '1-year MA')))];\n",
    "\n",
    "make_ma_plot = function(dt, wiki, start_date, end_date, title, breaks) {\n",
    "    b_palette = brewer.pal('Blues', n = 7)[c(3,5,7)];\n",
    "    ggplot(\n",
    "        dt[log_month >= start_date &\n",
    "           log_month < end_date &\n",
    "           wiki_db == wiki],\n",
    "        aes(x=log_month, y=num_blocks, colour = measure)) +\n",
    "    scale_x_date(date_breaks = \"1 month\", date_labels = \"%Y-%m\") +\n",
    "    scale_y_continuous(breaks = breaks) +\n",
    "    expand_limits(y = 0) +\n",
    "    scale_colour_manual(values = b_palette) +\n",
    "    labs(title = title,\n",
    "         x = 'Date',\n",
    "         y = 'Number of blocks') +\n",
    "    geom_line() +\n",
    "    theme_light(base_size = 14);\n",
    "}\n",
    "\n",
    "for(wiki in monthly_counts_long$wiki_db) {\n",
    "    g = make_ma_plot(monthly_counts_long, wiki, start_display_date_str, end_display_date_str,\n",
    "                     paste(\"Partial blocks created per month -\", wiki_names[[wiki]]), c(0:10*10));\n",
    "    ggsave(paste0(graph_dir, pb_monthly_prefix, wiki, pb_graph_suffix),\n",
    "           plot = g, width = 40, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "};rm(wiki);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Export the partial blocks monthly count dataset\n",
    "\n",
    "to_tsv(monthly_counts, partial_monthly_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Partially blocked user/IP makes constructive edits\n",
    "\n",
    "The second measurement we have defined is:\n",
    "\n",
    "* Number of partial blocks for users/IPs where the duration of the partial block intersects with the month of interest and the user/IP makes ≥1 non-reverted edits during the intersection.\n",
    "\n",
    "Because this is about block overlap with months, and collapsing edits during the overlap, it's probably easier to do this with programming than with SQL.\n",
    "\n",
    "**TODO:** Reconsider whether infinite blocks cannot be part of this analysis. In the sitewide block analysis, we disregard infinite blocks because they're mainly used to shut down vandalism-only accounts. A partial block is a different type of tool so counting them makes more sense. For example, they might be used to enforce an interaction ban. Also, I suspect they are somewhat rare.\n",
    "\n",
    "For now, I've removed the \"can't be an infinite block\" restriction as I don't think it makes sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pb_constructive_query = '''\n",
    "SELECT pbs.wiki_db,  pbs.start_timestamp, pbs.end_timestamp,\n",
    "       revisions.event_user_id,\n",
    "       revisions.event_timestamp AS rev_timestamp\n",
    "FROM (\n",
    "    SELECT wiki_db, user_id, start_timestamp, end_timestamp\n",
    "    FROM (\n",
    "        SELECT wiki_db, user_id,\n",
    "        start_timestamp, end_timestamp,\n",
    "        COALESCE(LAG(inferred_from, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp),\n",
    "            \"block\") AS prev_inferred,\n",
    "        LAG(caused_by_event_type, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "            AS prev_event_type,\n",
    "        LAG(end_timestamp, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "            AS prev_end_timestamp\n",
    "        FROM wmf.mediawiki_user_history\n",
    "        WHERE snapshot = \"{snapshot}\"\n",
    "          AND wiki_db IN ({wikis})\n",
    "          AND start_timestamp BETWEEN \"{start_date}\" AND \"{end_date}\"\n",
    "          AND caused_by_event_type = 'alterblocks'\n",
    "          AND inferred_from IS NULL -- not an unblock\n",
    "          AND source_log_params['sitewide'] = 'false' -- don't forget that's a string, not boolean!\n",
    "    ) AS a\n",
    "    WHERE a.prev_end_timestamp IS NULL\n",
    "       OR a.prev_inferred IS NULL\n",
    "       OR NOT (\n",
    "          -- prev event is also a block, meaning this is a reblock\n",
    "          a.prev_end_timestamp = a.start_timestamp\n",
    "          AND a.prev_inferred = \"block\"\n",
    "          AND a.prev_event_type = 'alterblocks')\n",
    ") AS pbs\n",
    "JOIN (\n",
    "    SELECT wiki_db, event_user_id, event_timestamp, event_entity\n",
    "    FROM wmf.mediawiki_history\n",
    "    WHERE snapshot = \"{snapshot}\"\n",
    "    AND wiki_db IN ({wikis})\n",
    "    AND event_entity = \"revision\"\n",
    "    AND event_type = \"create\"\n",
    "    AND event_timestamp > \"{start_date}\"\n",
    "    AND (revision_is_identity_reverted = FALSE\n",
    "         OR revision_seconds_to_identity_revert > 60*60*48)\n",
    ") AS revisions\n",
    "ON pbs.wiki_db = revisions.wiki_db\n",
    "   AND pbs.user_id = revisions.event_user_id\n",
    "WHERE (pbs.end_timestamp IS NULL\n",
    "       AND revisions.event_timestamp > pbs.start_timestamp)\n",
    "OR (revisions.event_timestamp BETWEEN pbs.start_timestamp AND pbs.end_timestamp)\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "constructive_pb_edits = hive.run(pb_constructive_query.format(\n",
    "    snapshot = snapshot,\n",
    "    start_date = start_data_date,\n",
    "    end_date = end_data_date,\n",
    "    wikis = \",\".join(['\"{}\"'.format(w) for w in wikis])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_block_id(row):\n",
    "    '''Variant of block ID creator that supports None-timestamps'''\n",
    "    \n",
    "    start_ts = row.start_timestamp\n",
    "    if not start_ts:\n",
    "        start_ts = '1970-01-01 00:00:00.0'\n",
    "        \n",
    "    end_ts = row.end_timestamp\n",
    "    if not end_ts:\n",
    "        end_ts = '2038-01-01 00:00:00.0'\n",
    "    \n",
    "    return(str(row.event_user_id) + \\\n",
    "           dt.datetime.strptime(start_ts, '%Y-%m-%d %H:%M:%S.0').strftime('%Y%m%d%H%M%S') + \\\n",
    "           dt.datetime.strptime(end_ts, '%Y-%m-%d %H:%M:%S.0').strftime('%Y%m%d%H%M%S'))\n",
    "\n",
    "def count_constructive(df, start_timestamp, end_timestamp):\n",
    "    '''\n",
    "    For the given `pandas.DataFrame` `df` containing constructive (non-reverted)\n",
    "    edits made by partially blocked users during their block, and for every month\n",
    "    between `start_timestamp` and `end_timestamp`, calculate the number of blocks\n",
    "    with constructive edits. Group these by wiki.\n",
    "    '''\n",
    "    \n",
    "    ## I need a way to identify each block so that we don't count blocks multiple times\n",
    "    ## during each month. To do that, I'll concatenate user ID, start, and end timestamps.\n",
    "    if not 'block_id' in df:\n",
    "        df['block_id'] = df.apply(make_block_id, axis = 1)\n",
    "        \n",
    "    if not 'rev_ts' in df:\n",
    "        df['rev_ts'] = pd.to_datetime(df['rev_timestamp'], format = \"%Y-%m-%d %H:%M:%S.0\")\n",
    "    \n",
    "    ## Then, iterate over wiki and month, identify all constructive edits within that month,\n",
    "    ## then collapse to count the number of unique block IDs.\n",
    "    date_range = pd.date_range(start_timestamp, end_timestamp, freq = \"MS\")\n",
    "    \n",
    "    res_df = pd.DataFrame(columns = ['wiki_db', 'log_month', 'num_blocks'])\n",
    "\n",
    "    ## Note that we're only looking at the revision timestamp for identifying these blocks,\n",
    "    ## it's the HQL query above that is responsible for properly handling infinite blocks.\n",
    "    for wiki in df.wiki_db.unique():\n",
    "        for month_idx in range(len(date_range) -1): ## iterate but exclude the last month\n",
    "            n_blocks = df.loc[(df.wiki_db == wiki) &\n",
    "                              (df.rev_ts >= date_range[month_idx]) &\n",
    "                              (df.rev_ts < date_range[month_idx + 1])]['block_id'].nunique()\n",
    "            res_df = res_df.append(\n",
    "                pd.DataFrame([[wiki, date_range[month_idx], n_blocks]],\n",
    "                             columns = ['wiki_db', 'log_month', 'num_blocks']\n",
    "                )\n",
    "            )\n",
    "\n",
    "    res_df['num_blocks'] = res_df['num_blocks'].astype(int)\n",
    "            \n",
    "    return(res_df)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_blocks_with_constr_edits = count_constructive(constructive_pb_edits, start_time, end_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R -i num_blocks_with_constr_edits,start_display_date_str,end_display_date_str\n",
    "\n",
    "pb_constructive_prefix = 'partial_blocks_constructive_per_month_';\n",
    "\n",
    "num_blocks_with_constr_edits = data.table(num_blocks_with_constr_edits);\n",
    "num_blocks_with_constr_edits[, log_month := as.Date(log_month)];\n",
    "\n",
    "## Add 3-month and 12-month running average\n",
    "num_blocks_with_constr_edits[\n",
    "    , num_blocks_3mma := rollapply(\n",
    "        num_blocks,\n",
    "        width = 3,\n",
    "        FUN = mean,\n",
    "        na.rm = TRUE,\n",
    "        fill = 0,\n",
    "        align = 'right')];\n",
    "num_blocks_with_constr_edits[\n",
    "    , num_blocks_1yma := rollapply(\n",
    "        num_blocks,\n",
    "        width = 12,\n",
    "        FUN = mean,\n",
    "        na.rm = TRUE,\n",
    "        fill = 0,\n",
    "        align = 'right')];\n",
    "\n",
    "num_blocks_with_constr_edits_long = num_blocks_with_constr_edits %>% gather(measure, num_blocks, 3:5);\n",
    "num_blocks_with_constr_edits_long = data.table(num_blocks_with_constr_edits_long);\n",
    "num_blocks_with_constr_edits_long[measure == 'num_blocks', measure := 'raw data'];\n",
    "num_blocks_with_constr_edits_long[measure == 'num_blocks_3mma', measure := '3-month MA'];\n",
    "num_blocks_with_constr_edits_long[measure == 'num_blocks_1yma', measure := '1-year MA'];\n",
    "num_blocks_with_constr_edits_long[\n",
    "    , measure := ordered(measure, rev(c('raw data', '3-month MA', '1-year MA')))];\n",
    "\n",
    "for(wiki in num_blocks_with_constr_edits_long$wiki_db) {\n",
    "    g = make_ma_plot(num_blocks_with_constr_edits_long, wiki, start_display_date_str, end_display_date_str,\n",
    "                     paste(\"Partial blocks w/constructive edits per month -\", wiki_names[[wiki]]),\n",
    "                     c(0:20*5));\n",
    "    ggsave(paste0(graph_dir, pb_constructive_prefix, wiki, pb_graph_suffix),\n",
    "           plot = g, width = 40, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "};rm(wiki);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Export the constructive edits monthly dataset\n",
    "\n",
    "to_tsv(num_blocks_with_constr_edits, partial_constructive_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partial blocks that are not altered\n",
    "\n",
    "The last part of our measurements look at whether partial blocks are altered, either by having pages added, being replaced with a sitewide block, or by having the duration increased. Note that these go together with a logical \"or\", so we're reporting only one measurement:\n",
    "\n",
    "* Number of partial blocks of users/IPs where:\n",
    "  * The partial block has a duration intersecting with the month of interest, and the block does not have pages added during the intersection.\n",
    "  * The partial block has a duration intersecting with the month of interest, and the block is not replaced by a sitewide block during the intersection.\n",
    "  * The partial block has an expiry timestamp within the month of interest and the expiration timestamp is not altered to increase the duration of the partial block.\n",
    "  \n",
    "### Notes\n",
    "\n",
    "After digging deeper into the data, it looks like the restrictions are stored as a string-representation of a Map, rather than an actual Map. This means that I might want to grab all partial blocks and enough information to answer all three questions, and write Python to do the analysis.\n",
    "\n",
    "The information I need for that is:\n",
    "* Value of restrictions for this block and the next block\n",
    "* Value of sitewide for this block and next block (current block has to be false, next block has to be true)\n",
    "* Value of the expiration date for this block and the next block.\n",
    "\n",
    "Regarding expiry timestamp: `caused_by_block_expiration` is set to \"infinite\" for indef blocks, and it's set to a numeric value if the block had an expiry timestamp set (note that it looks like MediaWiki operates with 30-day months for simplicity). So if the next event is a reblock and the expiry timestamp is extended, we can exclude it.\n",
    "\n",
    "**TODO:** The specification requires the upgrade to occur during the intersection with the month of interest. So the code below needs to be modified to take the time of upgrade into account as well so that a block is not excluded if it's in effect during a month but does not get altered that month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## This query retrieves all the necessary information to calculate\n",
    "## our final measurement.\n",
    "\n",
    "pb_candidates_query = '''\n",
    "SELECT wiki_db, user_id, start_timestamp, end_timestamp, source_log_params, next_params,\n",
    "       caused_by_block_expiration, next_expiration\n",
    "FROM (\n",
    "    SELECT wiki_db, user_id,\n",
    "    start_timestamp, end_timestamp, caused_by_block_expiration, source_log_params,\n",
    "    COALESCE(LAG(inferred_from, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp),\n",
    "        \"block\") AS prev_inferred,\n",
    "    LAG(caused_by_event_type, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS prev_event_type,\n",
    "    LAG(end_timestamp, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS prev_end_timestamp,\n",
    "    LEAD(start_timestamp, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_start_timestamp,\n",
    "    LEAD(caused_by_event_type, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_event_type,\n",
    "    COALESCE(LEAD(inferred_from, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp),\n",
    "        \"block\") AS next_inferred,\n",
    "    LEAD(source_log_params, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_params,\n",
    "    LEAD(caused_by_block_expiration, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_expiration\n",
    "    FROM wmf.mediawiki_user_history\n",
    "    WHERE snapshot = \"{snapshot}\"\n",
    "      AND wiki_db IN ({wikis})\n",
    "      AND start_timestamp BETWEEN \"{start_date}\" AND \"{end_date}\"\n",
    "      AND caused_by_event_type = \"alterblocks\"\n",
    "      AND inferred_from IS NULL -- not an unblock\n",
    "      AND source_log_params[\"sitewide\"] = \"false\" -- don't forget that's a string, not boolean!\n",
    ") AS a\n",
    "WHERE a.prev_end_timestamp IS NULL\n",
    "   OR a.prev_inferred IS NULL\n",
    "   OR NOT (\n",
    "      -- prev event is also a block, meaning this is a reblock\n",
    "      a.prev_end_timestamp = a.start_timestamp\n",
    "      AND a.prev_inferred = \"block\"\n",
    "      AND a.prev_event_type = \"alterblocks\")\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pb_candidates = hive.run(pb_candidates_query.format(\n",
    "    snapshot = snapshot,\n",
    "    start_date = start_data_date,\n",
    "    end_date = end_data_date,\n",
    "    wikis = \",\".join(['\"{}\"'.format(w) for w in wikis])\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_block_id(row):\n",
    "    '''Variant of block ID creator that supports None-timestamps'''\n",
    "    \n",
    "    start_ts = row.start_timestamp\n",
    "    if not start_ts:\n",
    "        start_ts = '1970-01-01 00:00:00.0'\n",
    "        \n",
    "    end_ts = row.end_timestamp\n",
    "    if not end_ts:\n",
    "        end_ts = '2038-01-01 00:00:00.0'\n",
    "    \n",
    "    return(str(row.user_id) + \\\n",
    "           dt.datetime.strptime(start_ts, '%Y-%m-%d %H:%M:%S.0').strftime('%Y%m%d%H%M%S') + \\\n",
    "           dt.datetime.strptime(end_ts, '%Y-%m-%d %H:%M:%S.0').strftime('%Y%m%d%H%M%S'))\n",
    "\n",
    "def count_nonmodified(df, start_timestamp, end_timestamp):\n",
    "    '''\n",
    "    For the given `pandas.DataFrame` `df` containing information on initial partial\n",
    "    blocks, identify blocks that were in effect during a given month and remove those\n",
    "    that meet either of the following criteria:\n",
    "    \n",
    "    * Had pages or namespaces added to it.\n",
    "    * Was replaced with a sitewide block.\n",
    "    * Had its duration extended.\n",
    "    '''\n",
    "    \n",
    "    ## Iterate over wiki and month, and for each month identify all\n",
    "    ## partial blocks in effect that month that do not meet either\n",
    "    ## of the exclusion criteria.\n",
    "    date_range = pd.date_range(start_timestamp, end_timestamp, freq = \"MS\")\n",
    "    \n",
    "    res_df = pd.DataFrame(columns = ['wiki', 'log_month', 'n_blocks'])\n",
    "    \n",
    "    ## I need a way to identify each block, which I'll use to identify blocks that\n",
    "    ## need to be excluded.\n",
    "    if not 'block_id' in df:\n",
    "        df['block_id'] = df.apply(make_block_id, axis = 1)\n",
    "        \n",
    "    ## Turn the timestamps into datetime objects since we're comparing them to other timestamps\n",
    "    df['start_ts'] = pd.to_datetime(df['start_timestamp'], format=hive_format)\n",
    "    df['end_ts'] = pd.to_datetime(df['end_timestamp'], format=hive_format)\n",
    "        \n",
    "    for wiki in df.wiki_db.unique():\n",
    "        for month_idx in range(len(date_range) -1): ## iterate but exclude the last month\n",
    "            \n",
    "            ## Candidates are all blocks that started before the end of the month,\n",
    "            ## and where the end timestamp is after the start of the month or NULL (infinite block)\n",
    "            candidates = df.loc[(df.wiki_db == wiki) &\n",
    "                                (df.start_ts < date_range[month_idx + 1]) &\n",
    "                                (df.end_ts.isna() |\n",
    "                                (df.end_ts >= date_range[month_idx]))]\n",
    "            \n",
    "            ## block IDs of IDs that were upgraded, we'll exclude those later\n",
    "            excluded_ids = set()\n",
    "            \n",
    "            for row in candidates.itertuples():\n",
    "                cand_params = eval(row.source_log_params)\n",
    "                try:\n",
    "                    next_params = eval(row.next_params)\n",
    "                except TypeError:\n",
    "                    # block is not followed by another block, or the next block doesn't have parameters, skip\n",
    "                    continue\n",
    "                    \n",
    "                ## Was the block upgraded from partial to sitewide?\n",
    "                if cand_params['sitewide'] == 'false' and next_params['sitewide'] == 'true':\n",
    "                    excluded_ids.add(row.block_id)\n",
    "                    \n",
    "                    ## print('block {} on {} upgraded from partial to sitewide'.format(row.block_id, wiki))\n",
    "                    continue\n",
    "            \n",
    "                ## Was the block upgraded to indefinite?\n",
    "                if row.caused_by_block_expiration != 'indefinite' and row.next_expiration == 'indefinite':\n",
    "                    excluded_ids.add(row.block_id)\n",
    "                    \n",
    "                    ## print('block {} on {} upgraded to an indef block'.format(row.block_id, wiki))\n",
    "                    continue\n",
    "            \n",
    "                ## Was the block upgraded from blocking specific pages to a whole namespace? (can't happen if it already blocks both)\n",
    "                try:\n",
    "                    if not ('pages' in cand_params['7::restrictions'] and 'namespaces' in cand_params['7::restrictions']) \\\n",
    "                    and 'pages' in cand_params['7::restrictions'] and 'namespaces' in next_params['7::restrictions']:\n",
    "                        excluded_ids.add(row.block_id)\n",
    "            \n",
    "                        ## print('block {} on {} upgraded from pages to namespaces'.format(row.block_id, wiki))\n",
    "                \n",
    "                        continue # skip further checking of this block\n",
    "                except KeyError:\n",
    "                    ## one or both of the blocks don't have restrictions set, skip it because we can't do the\n",
    "                    ## next test either\n",
    "                    continue\n",
    "                    \n",
    "                ## Did the block have pages or namespaces added to it?\n",
    "                ## This boils down to counting the number of '->' in the restrictions map string\n",
    "                if cand_params['7::restrictions'].count('->') < next_params['7::restrictions'].count('->'):\n",
    "                    excluded_ids.add(row.block_id)\n",
    "                    \n",
    "                    ## print('block {} on {} had pages or namespaces added to it'.format(row.block_id, wiki))\n",
    "\n",
    "            ## Add the wiki, month, and number of non-modified blocks to the results.\n",
    "            res_df = res_df.append(\n",
    "                pd.DataFrame([[wiki, date_range[month_idx], len(candidates.loc[~candidates.block_id.isin(excluded_ids)])]],\n",
    "                             columns = ['wiki', 'log_month', 'n_blocks']\n",
    "                )\n",
    "            )\n",
    "\n",
    "    return(res_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I looked through the partial blocks on Italian Wikipedia, as that is the wiki where blocks have been used most. There, I find that alterations of the partial blocks happens very rarely. While I didn't go through all of them carefully, I found only one block where this happened. That block was altered two times, one to change the duration from \"infinite\" to \"1 year\" and at the same time add one page to the block list, then again to add another page (but keep the duration)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_not_modified = count_nonmodified(pb_candidates, start_time, end_time)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "SELECT wiki_db, user_id, start_timestamp, end_timestamp, source_log_params, next_params,\n",
    "       caused_by_block_expiration, next_expiration\n",
    "FROM (\n",
    "    SELECT wiki_db, user_id,\n",
    "    start_timestamp, end_timestamp, caused_by_block_expiration, source_log_params,\n",
    "    COALESCE(LAG(inferred_from, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp),\n",
    "        \"block\") AS prev_inferred,\n",
    "    LAG(caused_by_event_type, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS prev_event_type,\n",
    "    LAG(end_timestamp, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS prev_end_timestamp,\n",
    "    LEAD(start_timestamp, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_start_timestamp,\n",
    "    LEAD(caused_by_event_type, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_event_type,\n",
    "    COALESCE(LEAD(inferred_from, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp),\n",
    "        \"block\") AS next_inferred,\n",
    "    LEAD(source_log_params, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_params,\n",
    "    LEAD(caused_by_block_expiration, 1) OVER (PARTITION BY wiki_db, user_id ORDER BY start_timestamp)\n",
    "        AS next_expiration\n",
    "    FROM wmf.mediawiki_user_history\n",
    "    WHERE snapshot = \"2019-06\"\n",
    "      AND wiki_db = \"itwiki\"\n",
    "      AND user_id = 1774029\n",
    "      AND start_timestamp BETWEEN \"2019-01-01\" AND \"2019-07-01\"\n",
    "      AND caused_by_event_type = \"alterblocks\"\n",
    "      AND inferred_from IS NULL -- not an unblock\n",
    "      AND source_log_params[\"sitewide\"] = \"false\" -- don't forget that's a string, not boolean!\n",
    ") AS a;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_not_modified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measurements for the Research Showcase\n",
    "\n",
    "There are some measurements I'd like to be able to present at the Research Showcase:\n",
    "\n",
    "1. Proportion of partial blocks that are of IPs on Italian Wikipedia. We know they often use partial blocks to block IPs from editing, so we'd like to know that proportion.\n",
    "2. Proportion of partial blocks that are of namespaces. I've seen examples on Italian Wikipedia where they block an entire namespace, so I'd like to know to what extent that's common.\n",
    "3. Plot of total number of partial blocks per month on Italian Wikipedia, counting both IP and user blocks.\n",
    "\n",
    "## Proportion of partial blocks that are IPs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "## From https://stackoverflow.com/a/81899\n",
    "import socket\n",
    "\n",
    "def check_ip(ip_string):\n",
    "    try:\n",
    "        ## quick check if it looks like an IPv6 string\n",
    "        if '::' in ip_string:\n",
    "            socket.inet_pton(socket.AF_INET6, ip_string)\n",
    "        else:\n",
    "            socket.inet_pton(socket.AF_INET, ip_string)\n",
    "        return True\n",
    "    except socket.error:\n",
    "        return False\n",
    "    \n",
    "partial_blocks['blockee_is_ip'] = partial_blocks.blockee.map(check_ip)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "partial_blocks['log_month'] = partial_blocks['log_timestamp'].map(lambda x: x.strftime('%Y-%m-01'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "it_ips_monthly = (partial_blocks.loc[partial_blocks.wiki == 'itwiki']\n",
    "                  .groupby(['log_month', 'blockee_is_ip'])['wiki']\n",
    "                  .count().reset_index()\n",
    "                  .rename(columns = {'wiki' : 'n'}))\n",
    "\n",
    "it_ips_monthly['percent'] = (\n",
    "    100 * it_ips_monthly['n'] /\n",
    "    it_ips_monthly.groupby(['log_month'])['n'].transform('sum'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>log_month</th>\n",
       "      <th>blockee_is_ip</th>\n",
       "      <th>n</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-01-01</td>\n",
       "      <td>False</td>\n",
       "      <td>21</td>\n",
       "      <td>46.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-01-01</td>\n",
       "      <td>True</td>\n",
       "      <td>24</td>\n",
       "      <td>53.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01</td>\n",
       "      <td>False</td>\n",
       "      <td>33</td>\n",
       "      <td>47.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-02-01</td>\n",
       "      <td>True</td>\n",
       "      <td>37</td>\n",
       "      <td>52.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>63.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>True</td>\n",
       "      <td>35</td>\n",
       "      <td>36.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>False</td>\n",
       "      <td>20</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>True</td>\n",
       "      <td>20</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2019-05-01</td>\n",
       "      <td>False</td>\n",
       "      <td>20</td>\n",
       "      <td>46.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2019-05-01</td>\n",
       "      <td>True</td>\n",
       "      <td>23</td>\n",
       "      <td>53.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2019-06-01</td>\n",
       "      <td>False</td>\n",
       "      <td>31</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2019-06-01</td>\n",
       "      <td>True</td>\n",
       "      <td>19</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2019-07-01</td>\n",
       "      <td>False</td>\n",
       "      <td>17</td>\n",
       "      <td>65.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2019-07-01</td>\n",
       "      <td>True</td>\n",
       "      <td>9</td>\n",
       "      <td>34.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     log_month  blockee_is_ip   n  percent\n",
       "0   2019-01-01          False  21     46.7\n",
       "1   2019-01-01           True  24     53.3\n",
       "2   2019-02-01          False  33     47.1\n",
       "3   2019-02-01           True  37     52.9\n",
       "4   2019-03-01          False  60     63.2\n",
       "5   2019-03-01           True  35     36.8\n",
       "6   2019-04-01          False  20     50.0\n",
       "7   2019-04-01           True  20     50.0\n",
       "8   2019-05-01          False  20     46.5\n",
       "9   2019-05-01           True  23     53.5\n",
       "10  2019-06-01          False  31     62.0\n",
       "11  2019-06-01           True  19     38.0\n",
       "12  2019-07-01          False  17     65.4\n",
       "13  2019-07-01           True   9     34.6"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "it_ips_monthly.round(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAAC+lBMVEUAAAABAQECAgIDAwMEBAQFBQUGBgYHBwcICAgJCQkLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///9Obj0VAAAaMUlEQVR4nO2dC3xUxb3HF+sDa0kIlaAFoyIitEqpV+vVthiloq0bAYtieNxSjAXa0t6Wcq3Gd2pbja9i1Vqu1Hp9IGqtKFrA1ipSSqyWh48GCI+ERxYSkmx2s6+cz+ees4+zj3NmdvbMzO7s8vt+dHfP/8yZ/5z55sxZktlZlwZKGlehGwDkAsElTuEEvzuKPXdmSeuRhLqIKVzEAvJ6hLXmLOVya6BEwVmq/sqL7MeXjODslScLkcoyVZNamr2sy1U5+6CoWgf1kWslCNWLDqicvodNi4tSEaNgVy7nbCXTBKlX2ocaXTFZ/99/YjulXJZq+EvrZVun1YqqNaWLLbWSBGtapO0nX8mrYOZztsIqWKtZrmmHjvJo2tOTqeWyVMNd2ii7f4j+fN/wAVrfD4dV/lD/yXP9cmjF9wJaclvfqf/Yx07ODD5SNfCCzbFa4rFYmWStn1xdUTbFk3G8GUxp7OHj06oO1Z9Sfm800nTyA9qa8QOrHo+VjdURTxyvyExqNptYUzxf4pzN9PpDPEnkzlMHz/HGuyOZOLW/YvlSziJ1K9knL16paa+4Xta0y/4YL2S0wJVsZSJVYEFF5a/SqxEt+MBn9eer2jStfuKePdW36BsTW1sn3paybexMdIUZnNzivf2iWC1mLL3Wc970Hf7+9RnHm0GzaGTf4glpVd918baOHxmRV078o6adtKJv19xUHfHEGRUlm02sKV5F4pyTNZpJGi/d0THzx/EWpyRO7a/oISnJU88p2SfBkw5o/zPpp1rrsGCsULQFrmQrE6lundi6pzq9GhZprOhl26ZP15936hunb9W0zSP1Df15y8iU7Z0pXWEG92la7/GxWsxYRq06XcMzjjeD8aIGg7ekVX1G7CJwLTn5H/rTiId2p51ZauKUipLNJtYUryKldWar4knO+li/uKviLU5JnNpfZr548tRzSmnawvu1r757kfbzH8UKxVrgSrYykWqk0dT0arKR45usoTP0dwGufn1joF9/TzBQ34g/m9v9KV1hBlNSpcWStW68ZLDL9amM481gsrHtN12SVrXxbOw6/Ubj6b3JQ0a9lnJmhIqSzSbWFD8wcc6plcWTHG/8uB0Vb3FqYleyudFXKckzmxIv+f74wNnamL7RH8QKxVrgSrYykSra1PRqskrLXsRSNvpsXogfatrWkRkX5oB4sbRg/HCbK9hg5O8PhTtcGcenBeNP3cenVT0qft3tOeNX0Rf9K4fFCg8wj0ipaIB5LvFmU2pKvwR1BvZq2j6XmWR0S9oJmIlT+2tAxqllnlOi+nGPXa99+94vxnPGWuBKtjKRaqS+vSW9mmw4FnyTfue65GZ9Y1Jr62W3pGwbO0/8MFYsLRg/PC2WDA97qW/71a6M49OCsaeD9RekVd1w8fbYnbNt9M81bfrWwMqTYoUTdaRVFA1GY/FmE2uKH5jaPxfd7t1R4zKTPDDxw8Cma+NFUhKn9lc0X8pZZJ5Tovr7hjylLTvhgUTOaAtcyVYmUtXr2xPTq2GVxkKaEv8PKit/oI8Y+hu9wfP7UraNnY3lsVakBeOHp8WS4VdHH3PKQ66M49OC0cHPVVGzLa3q4M+GD74vGtk35g7tmdEDx78ZK5yoI62iaDAaizebWJOW6Owkmy8YWPWIHogniTx01nFnvxwvkpI4tb+i+VLOIvOcEtW3H9OqtRzTbuY0WuBKtjKRqm/e4KH3pFfDKs0x3BUAMgI6F4JVBoJLHBUEA7WB4BIHgkscCC5xILjEgeASJwfBXZ0WQl5rTCx+2Qm8YdkZOrul91JnsNcaO5Sr4A6PhWC3NSYWn+wEXSHZGTyd0nvJ09djE4RgAwiGYF4gmAIEMwHBRCAYgnmBYAoQzAQEE4FgCOYFgilAMBMQTASCIZgXCKYAwUxAMBEIhmBeFBTsdusPO2+4YWfiEYI5UFCwZgi+889/vjPxCMEcqCp4Rnf3zMSj9tHq1e3dFkI+a0wsAdkJfGHZGbq90nupO+S3xjqyCZ4cCU9OPGrPzJ/fGrTQH7bGxBKRnSDULztDMCS9l4IRmxQ9OV3BGKKdo+oQjXuwIBQU7DbeRrfU1bUkHiGYAwUF2wLBDoFgChDMBAQTgWAI5gWCKUAwExBMBIIhmBcIpgDBTEAwEQiGYF4gmAIEMwHBRCAYgnmBYAoQzAQEE4FgE5+VSMAmKJSQ7ASBiOwMvj7pveQLB62xLlzBBriCIZgXCKYAwUxAMBEIhmBeIJgCBDMBwUQgGIJ5gWAKEMwEBBOBYAjmBYIpQDATEEwEgiGYFwimAMFMOBL81sz5e7DKjgBUFTyzubkBn/AXgKqCa5ubZ2GNDgGoKvi1a++Zaq6yc391dUu/Bc0aKjpK4RxsT6I3m2BN2zLXvII7W1sPdlgIeq0xsfhlJ+gJyc7Q0SW9lzoCvdbYwayCu+v/F/dgAag6RLun3ePHKjsCUFVwJhDsEAimAMFMQDARCIZgXiCYAgQzAcFEIBiCeYFgChDMBAQTgWAI5gWCKUAwExBMBIIhmBcIpgDBTEAwEQg28fZYCPutMbEEZCfwhWVn6OmV3ks9oT5rrDNXwYets0IwZYeFblWn7GSAIdohxTJEQ7BDIJgCBDMBwUQgGIJ5gWAKEMwEBBOBYAjmBYIpQDATEEwEgiGYFwimAMFMQDARCIZgXiCYAgQz4Ujw6pkzVmMZJQGoKnh6c/N0LOEgAFUFL2hu/r65CMv6Zcv2ey2E+6wxsQRlJ/BHZGfw+qT3kjdkk+JwNsHvT536gbmM0qqGhr1+C5GgNSaWkOwEgYjsDP6A9F7yh21SZP3ms+ubm+uwEJoAVB2ia5uba3EPFoCqgt+orX0DyygJQFXBmUCwQyCYAgQzAcFEIBiCeYFgChDMBAQTgWAI5gWCKUAwExBMBIIhmBcIpgDBTEAwEQiGYF4gmAIEMwHBRCDYxGclErAJCiUkO0EgIjuDr096L/nCQWss65QdXMGCKJYrGIIdAsEUIJgJCCYCwRDMCwRTgGAmIJgIBEMwLxBMAYKZgGAiEAzBvEAwBQhmAoKJQDAE8wLBFCCYCUeC3W73tVhlRwCqCta0tffhE/4CUFfwLzZgjQ4BKCs4PCdorrJz97nnbqf9LADl6I0+0gR/0KiZV7C/q+vQQQvBbmtMLH7ZCbpCsjMcPNwjPUXAaxPMKvhxfYTGPZgfZYfoH+kjNFbZ4UdZwRlAsEMgmAIEMwHBRCAYgnmBYAoQzAQEE4FgCOYFgilAMBMQTASCIZgXCKYAwUxAMBEIhmBeIJgCBDMBwUQg2MRmcResssMCVtmhgCuYCQzRRCAYgnmBYAoQzAQEE4FgCOYFgilAMBMQTASCIZgXCKYAwUxAMBEIhmBeIJgCBDMBwUQgGIJ5UVWw/95pbqyyIwBVBS+t79bwCX8BqCp47lbjEavscKOq4Cm/vPZ1c5WdVQ0Ne/0WIkFrTCwh2QkCEdkZ/AHpveQP26TIOmVnZnNzrXkFr1+2bL/XQrjPGhNLUHYCf0R2Bq9Pei95QzYpDmcTfFdz8yzcgwWg6hC9e96Mt7DKjgBUFZwJBDsEgilAMBMQTASCIZgXCKYAwUxAMBEIhmBeIJgCBDMBwUQgGIJ5gWAKEMwEBBOBYAjmBYIpQDAT2QW7stuGYIcoI3hHGVVwp/WbeeR/b5JPdoJ8fG+S9F7K+r1JriiDbqcK7rXOCsGUHRaUmLLDcC/GEO0QJYZoBiDYIUoI/tOoo/RBGoJloITgk5eHcQVLQgnBZV4M0U448M6Shr1Zyigh+I7b/A4Ev1c+rOrUceMumFBdU3Pd7LkLF95Uf2dj4++WPrVixZo165r+2dzcxtlylQU3Pf69C08YXP3F6hZ6OSUEx/6hlKvgnk1vrXl9xYplSx9pbLy1fvHChbNnX1PjnjDhvHFfqKr6bPmnjTqPLi8fXnXGuHFfnfD1mppZs29Y+MP6+rsbH1i69NkVL69Zs6FpN7Xligre9OSPqytOuHDB400eT6v77C3UwkoIZsDREL2jeWtT09o1r654funSXzfeU1//44ULZs+eWnPFhAnjx42pqiovG7mZdrx6gj957sbLhx33H9cveedAPHLguyPW0Y4oacHZWUg1rJTgnSsbrhl9zOhrGlZm3HkaKv5EOUwJwRvOOUq7bnkhBHt+cAZlhFNF8N63l8wefVRlTcNK21vKsrLfko9VQvDZr7u03acVRHD79aO2EncqIHjf20vqzj+28rJFT/2bXOi1IbcQ9ykh+IQ+l3agoiCCPe3fGfUhaV+BBTfpbgcabj/KVs87I+oOEHYpIfiby1xt02cURrCnfc7YTwi7Cid481OLLhs86Py6JW+zVbTlnCv32O9RQvD+6UMqZhwqkGBP+7c/TzBcEMGG2yGfNty251DTzkvPsz8JJQQzIPM3We2zv2B/g8u34O36G+VT9DfKjW+TBlwy+2aN/qddXFXBbp28LaN04NqzbQ3nUTDxH0HMtC+qfNMmrIRgmyk77uhjvpZw2D/tnGabcH4EZ/lHEDsPlj1rDSoh+DifpvUeayM4vghLZ2vrwQ4LQa815hTPlPN2W6N+cQns6d54/6yzjz51yh1/ssmeO0+XPWKJdQnsJQKBXmvsYJrgS37a0fGT6nTB19RtNJdRur+6uqXfgmYNOSc47aJukfUxcdA99Ko7VnnEVbh+aIO4ypixE9GbJnjXVeXlU/ak34TD62fndSG0/VPO35kZkzxEr66a3im4yqaRtfvSI0oM0XZENs7O7zJKe7/x5UzDUgW3N3ymQfwf/D85//L0O7kSggdY/bpr6jbkeRmlvVdcsCs9IlPwDvcZb8mY0bF70pfSfvGlhODPtVMu5bwJ9rRNmpD+6yCJgldXTdspZ8rO/jlVf0/ZVELwfXWdKgj2tF12cWvqtjzBjZ9pMJ7kzMlqqHg1uaGEYGczOiTMyWr7enWqYVmCd9SM/Gv0haRJdw+XPWW+VkIwA3madLdnwiUpv0mSJHjNqVduj72SNavyhcG/SryE4HT2fPXSpGE5guPDs4G0abN/O7ku/qcKJQQXcEaHhd1fmWgaliF4x1Xx4dlA3rzoTZ+/NjahVgnBBZzRYWX3RVcm5hpLELzWHJ4NJE58b77wa9FESggu5IwOK7suvDL+2yDxglOGZwOZn2xomzLmXx5FBBd0RoeVXf/pjhkWLXjH5NP/khaQ+tGV9kUnvaWI4MLO6LCy49yrooYFC1572pXb0iOSP5tkTKhVQjAD+f1s0vYvTd7vES14SXlDZkj2h8+Wlf0Wgm3ZPn76AbGCWyaPeMMSlP7pwlVDbi284I3jjx//nmKCPdu+eN0BkYLfPO2b26xR+R8fXTdiXu7zu3Ikm+BzH/Q+eJ5qgj3N42oPiBNsMzwb5OHzwZ+MI02oFUY2wccGtL5jaXILItjz8ZiZvYKqapky4nXbHfn4APg+0oRaYWQT7NIY7sY+K5GATVAgu8fO7RVS0fqR7n32ewIRIQlo9AW654z5WGqKcNAaS/3mM5dLmb8mpfPRmBtEVEMYng3ys4RD+6LKtTJTFOO76Bi7Rn+Xu46WqYTh2SBfa3TYTqgVRvEK9n145jzOKv5y+jfsZlzHydsiLMvL75OXoogFe/512gKuGvThmfYJo/ytsvPnExdJS1HMgj3/OvV7zo/Xh+dV1AJ5XEap6YzMCbXCKGrBng+qHP/ovzP6CsrwbJDPdbL+fX62BXmcUtyCPe+fstjZ0UsG1Wf7AGheF0LbPWl81k+SO6LIBXveH3Gjg2N3Xj2cPjwb5Helu/3fSZtQK4xiF+z554if5XzourMuzzI8G+R7KcO0CbXCKHrBng0n3ZTjkQzDs0He16pMnVArjOIX7Nkw7OZcjtv5reGvMRXM/2KkLyYn1AqjBAR7/l5JXqjIwroxl1PWPEqlAKvN/u1zdbms/cFCKQj2rK+8m/UoxuHZoBDLCW9OTKgVRkkI9rw79BdMx+yZ+Tm24dmgIOtFJybUCqM0BHveGvJLhkPWjZnEODwbFGZB8Lap0Qm1wnAo+KX8rbJDIn1Gx18r7sl6xJJBi3KZIVOgFd9jE2qF4Uxw13+78/sJfxsypuz8peJeevk9s3IYng0KtqR/Q8XL4lI4E/ybte68rtFhR+acrDcrltCKvzvmshyGZ4PCfWcDdYXaHHEkeOe8iNtcZeelxYtb+yxEgtaYWMKZgfVDniCXXlp2sy/HBMFIjgfkTjBkH//bZ+8SlSJik6I7m+Cb12rJK/j9F17Y32Mh7LfGxBK0RN4s/x2hbPuc4atzTuAL53xIrvSSeqnplPldYlKE+qyxzmyC3dG1DNW6Bxu8VvaobVF9eHYwd7GgX6uz9ZxviplQ6/SfSe48r7JjxW5e9Gu2N6+Hc3v3nKCw35tEXKE2R0rk38EJXi3/fWaote5kZ3+lKfAXY+2bdabtCrU5UmKCPSvLn0wPrB87gbhoPJ1Cf/OZmAm1pSbY80r5H1I3f+NseDYotGCP5yEBE2pLTrDn+UHJv6rqw/NKxwkKL1jEhNrSE+x5btDy+Kt3x1zK8UZFAcGeN04cOfX2P+7gSFGCgj3PDFoRff5N2c08f11VQbBn1yt3TTvzaA7LpSjY81z5C9Hh+RWuBEoIjhL9SoFPGd/hk/uAVJKCPc+Wv/T3z3+N/KVaTKgjOIpDy6Up2PNEWdlNvJNfFBMcJWr56NOn3PYS67SAEhXsWc4/BVVFwVF26ZbP0i3f+iKD5VIVLABlBUdpM783kT5iQzARtQVHYbAMwUSKQHCUpOWPbfZCMJFiERxlL8kyBBMpKsFR7CwLEdzVaSHktcbE4pedwBuWnaGzW0Ivta9/5LtfPrZy0uJnmo3NYK+1yKFcBfd0Wwj5rDGxBGQn8IVlZ+juldVLhzY8Nu+C44ZdfuNze/3WvR25CsYQ7RC5i5FGR+wzcQ8mUfSCo3ghmERpCMa7aCIQDMG8QDAFCGYCgolAMATzAsEUIJgJCCYCwRDMCwRTgGAmIJgIBEMwLxBMAYKZcCT41drvvKPYMkoSOIIF/9a7aYZqyyiJ5wgWrGlbF5iLsLRs2OA5bCHUa42JpU92Am9YdobDPdJ76XDQZ41ln9Hhvvp9cxmlJ2bO3B2y0B+2xsQSkZ0g3C87QygsvZdC/Tb91JNVcP+mucothCacI3iIXhF4bxbuwQJQVvC0ug1KLqMklCNYcCYQ7BAIpgDBTEAwEQiGYF4gmAIEMwHBRCAYgnmBYAoQzAQEE4FgCOYFgilAMBMQTASCIZgXCKYAwUxAMBEINvFZiQRsgkIJyU4QiMjO4OuT3ku+cNAa68IVbIArGIJ5gWAKEMwEBBOBYAjmBYIpQDATEEwEgiGYFwimAMFMQDARCIZgXiCYAgQzAcFEIBiCeYFgChDMhCPBq2bMWYdVdgSgquDHejZglR0RqCpY0w7dgjU6BKCs4P5HD2OVHREpVF1l59H3NKyTJQBV18l6crXxiHswN6oO0W6dAFbZ4UdVwZlAsEMgmAIEMwHBRCAYgnmBYAoQzAQEE4FgCOYFgilAMBMQTASCIZgXCKYAwUxAMBEIhmBeIJgCBDMBwUQg2KSn20LIZ42JJSA7gS8sO0N3r/Re6g75rbHs33yWQVenhZDXGhOLX3YCb1h2hs5u6b3UGey1xg7lKhhDtEOKZYiGYIdAMAUIZgKCiUAwBPMCwRQgmAkIJgLBEMwLBFOAYCYgmAgEQzAvEEwBgpmAYCIQDMG8QDAFCGbCmWC3W8MySgJQVrBmCMYSDtyoLTi+CMtHq1e328wUwZQdBrzKTtkxBMeXUXpm/vzWoIX+sDUmlojsBKF+2RmCIem9FIzYpMi+jFLqFYwh2jlqD9G4B3OjrGC38TYayyhxo6zgDCDYIRBMAYKZgGAiEAzBvEAwBQhmAoKJQDAE8wLBFCCYCQgmAsEQzAsEU4BgJiCYCARDMC8QTAGCmYBgIhBs4rMSCdgEhRKSnSAQkZ3B1ye9l3zhoDXWhSvYAFcwBPMCwRQgmAkIJgLBEMwLBFOAYCYgmAgEQzAvEEwBgpmAYCIQDMG8QDAFCGYCgolAMATzAsEUIJgJDsFYZYcbtQXjE/7cqC0Ya3Rwo7bg+Co791dXt/Rb0KyhoqMUzsH2JHpzuoI7W1sPdlgIeq0xsfhlJ+gJyc7Q0SW9lzoCvdbYQTbBuAdzo/YQjVV2uFFbcBIIdggEU4BgJiCYCARDMC8QTAGCmYBgIhAMwbxAMAUIZgKCiUAwBPMCwRQgmAkIJgLBEMwLBFOAYCYgmAgEm3h7LOw7ZI2JJSA7QUeb7Aw9vT7pKfYftMY6cxVsw4I3uA5XgXX/VegWiGDxC6Q9EAzBFP5vK9fhKrDtiUK3QAQvvEfawycYKA8Elw5uu6BTwbaVFRf919f1F7oNInBbXqRyBAve2tj4YaHbIAJ5gt3TVukPj0x73mEdBeZ3WzYv1dt//3Xv6o+PTS50cxzjTvnPCtcVvG2O/uIf22sdt62g3Br9z71x4zz9cW3xjtbSBD9X667RX/QX6XDd7NbZprlDwan6WYQL3RznRNX2SxA87aPNlJ8c5Xlynaa98wfN3dQ0r1hPIYbe9trtGwQL7tdvWQ9f82gxC56/T9P2LkjcgwvdGucYKp6f9qhgwdvm8rRJJYpYbYwsKhwKvuZ1Z8epR9ELzqICv8kqcSC4xIHgEgeCUynB3ijBU8qG61n94WnLibu0kuyNEjylbLjGRrTIWAguWVzjn9WeHq+f+M4JJ1y8S9++o2z4q5pLJ/6ypDgSBa8ZGxq7Rj9x941diyfr2/Xe5aPiV3DsZSlxJArWLv/WFcaJl3dqHYP1bb8WPiouOPaylDgiBW86ZnNccOdg8+abuAeXWI+U2Omw4Eo8um/uvnGyaXXQDgguDUzBLRM+HX2TFdu6bZALgkHRAcElDgSXOBBc4kBwiQPBJQ4ElzgQXOL8P+/4VnwzEBQbAAAAAElFTkSuQmCC\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -i it_ips_monthly\n",
    "\n",
    "it_ips_monthly = data.table(it_ips_monthly);\n",
    "it_ips_monthly[, log_month := as.Date(log_month)]\n",
    "\n",
    "g = ggplot(it_ips_monthly[blockee_is_ip == TRUE],\n",
    "           aes(x = log_month, y = percent)) +\n",
    "scale_x_date(\"Month\") +\n",
    "scale_y_continuous(\"Percent\", lim = c(0, 100), breaks = c(0:10)*10) +\n",
    "ggtitle(\"Proportion of Partial Blocks that block IP users - Italian Wikipedia\") +\n",
    "geom_line()\n",
    "ggsave(\"graphs/itwiki_partial_prop_ip.png\",\n",
    "       plot = g, width = 30, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proportion of partial blocks that block namespaces\n",
    "\n",
    "Similarly to how we counted how many partial blocks where of IP users, we can also count how many partial blocks have at least one namespace set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "it_namespaces_monthly = (partial_blocks.loc[partial_blocks.wiki == 'itwiki']\n",
    "                         .groupby(['log_month', 'block_num_namespaces'])['wiki']\n",
    "                         .count().reset_index()\n",
    "                         .rename(columns = {'wiki' : 'n'}))\n",
    "\n",
    "it_namespaces_monthly['percent'] = (\n",
    "    100 * it_namespaces_monthly['n'] /\n",
    "    it_namespaces_monthly.groupby(['log_month'])['n'].transform('sum'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>log_month</th>\n",
       "      <th>block_num_namespaces</th>\n",
       "      <th>n</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-01-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>52</td>\n",
       "      <td>91.228070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "      <td>8.771930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49</td>\n",
       "      <td>76.562500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12</td>\n",
       "      <td>18.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3.125000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.562500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19</td>\n",
       "      <td>61.290323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12</td>\n",
       "      <td>38.709677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2019-05-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29</td>\n",
       "      <td>74.358974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2019-05-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "      <td>23.076923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2019-05-01</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.564103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2019-06-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28</td>\n",
       "      <td>65.116279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2019-06-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "      <td>34.883721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2019-07-01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14</td>\n",
       "      <td>58.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2019-07-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "      <td>37.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2019-07-01</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.166667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     log_month  block_num_namespaces   n     percent\n",
       "0   2019-01-01                   0.0  40  100.000000\n",
       "1   2019-02-01                   0.0  52   91.228070\n",
       "2   2019-02-01                   1.0   5    8.771930\n",
       "3   2019-03-01                   0.0  49   76.562500\n",
       "4   2019-03-01                   1.0  12   18.750000\n",
       "5   2019-03-01                   2.0   2    3.125000\n",
       "6   2019-03-01                   3.0   1    1.562500\n",
       "7   2019-04-01                   0.0  19   61.290323\n",
       "8   2019-04-01                   1.0  12   38.709677\n",
       "9   2019-05-01                   0.0  29   74.358974\n",
       "10  2019-05-01                   1.0   9   23.076923\n",
       "11  2019-05-01                   2.0   1    2.564103\n",
       "12  2019-06-01                   0.0  28   65.116279\n",
       "13  2019-06-01                   1.0  15   34.883721\n",
       "14  2019-07-01                   0.0  14   58.333333\n",
       "15  2019-07-01                   1.0   9   37.500000\n",
       "16  2019-07-01                   3.0   1    4.166667"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "it_namespaces_monthly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAAC6FBMVEUAAAAAujgBAQECAgIDAwMFBQUGBgYHBwcICAgJCQkLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY4ODg6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NEREREzG1GRkZHR0dISEhJSUlLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxeXl5fX19gYGBhYWFhnP9jY2NkZGRlZWVmZmZoaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Pz8/Q0NDS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4dm34+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///8/MNpFAAAViUlEQVR4nO2df5wU9XnHBwlykejdoSCGQBAJimKLqbWmApZqmu7l1GpFTJvoIUnU/KiUFGIIMW3S+KO2qSa2atPUWJFEA1KL1qS1agi2CYhtChzIIcqPO25v75bdvb39/tt5dmf2ZmafnZn9zvd7O7v3eb9gf8w883yemTc7u8fd7hkCNDVGvQcAeoHgJkeJ4MAmr86rIcdwN63csnqvmnLUETY0oE7L7GMj+CM/rKFFBMHBOVowwhxHo2LPpNrUSriOxkcLfsWBTU7P1DBCBMG+Oerwmqg20JFpNE+n+Td91hGfuoA2kQgpePlf+xUHNvEtUCh4jE7QYQWLjqeEOD7xqBBPdPrWBbSJQkjBfR/aLUZ3jK7/ds6pF7782LyWy94073xjWvtnskKMbPhg2ydT5v37Zp5S3C5z59nT78wIw6TUxyr83+vaz7jmaKmwtNJwLBSuWqPcRgyvm9X6reKS1895QLy4qGX2I4LPeWh2y2U7HS2d09pTWtt7Z7JCPFWeg1ZKckzsvGeFm/zw94XYbDwrxFXPWEU0uFF5xLKfbp/+TXcbRYQULH7yGzm34I/tSd3zvo69qXsWm3eW9fQs+4oQ9/7Ovt6bP2/e//ih0nbrlh08uPTLjkejVXjxS0MnPnurVWi3LC901xqjbb62ZE/v52jJ5rOeEWLG05kDt1TJ6exOrb/c0dI5rT2ltb13JivEU+U9aMUkx8TO+a1wk9yMd8WfXn2X6Dk7VyoqDm5UHrG7l/UcXOpuo4iwgsUX17gFvy3EoHHYvHiveecNIXbNFeL8/xHindnm/f3WdueaK3bOdRx4q5Don2kV2i3LC921xmib80qPC+Nvzvm5efWBB98S1XKswcotndPaU1rbe2eyQjxV3sNRPm7WxM75HeF33C9++9XLxdc/VyoqDW5UHrG5tAPuNooILTh7ycsuweWN6U7afBnRIsR76cw10bxfsLZrsVaUD7x1f8eVbYbxHqvQ7lJe6K41RtvQNa05dw1d/Vfn1Hlbq+RwLe0Le0pre+9MVoinqritMfryoHjL0Z4NE+K/F2UXigsy839RKioNzhyx4g642ygitGCx64J+OtaDQhyuEGw+Qb9h/nuc3+1pWvHIsgrn/sPxfK/d5RRrE9fC0VrHI3ie9Qg+eN43izcKW86uksO1tC/sKa3tvTNZIZ4q7+EoDu1oz4aZ/Np3bhV//K1ft45TaXDmiM017+9yt1FEeMHigT8yry5fn9rXUSH46p6eq75sVizbnf3lHzqa/pn53HjlWseBtwrP/lFm73V2l7Os12+uhaO1xmibe5bsLT0HH5r/dSFufCO7ZUaVHMG0tC/sKa3tvTNZIZ4q7+EoDu1oz4aZ3Df1++LxKQ9Yx6k0OHPE1pn3l7nbKKIGwYWrzaudl7XMfqhC8Demta0yX+WOPHj+5IXPOpqmb58+/fa048Bbhc/NnzTrQbvLva2lW66Fo7XGaJvcl2a23VdccviCr4ofzG9Z9FKVHOvC1dK+sKe0tvfOZIV4qryHozi0oz0bZnJkUo/onmR9EWwNzhyxzG1t0/7S3UYR6jrFLy0cdZlp7EIhGIIbNi0cEAwaGQhuciC4yYHgJgeCmxwIbnJCCe49ypBNckslGFLUJ5lV1EjZREcHfEfS7bYIBDNAMAHBwUAwAcE6gWAGCCYgOBgIJiBYJxDMAMEEBAcDwQQE6wSCGSCYgOBgIJiAYJ1AMAMEExAcDAQTEKwTXnAiUbwoXkGwPLEVLBLWXwiOREMIfvOFF44kGYbT3FIJsor6pIcVNVI2kf9IfXUXfEPXDvPqB6tW9eQYCnluqQQjivrkC4oasROddBB+JL99S9VbsMi/9onSPZyijx51Cg7dKN6naDGyA4LLNJfg4uvnREfXdgi2aS7BHiAYgiMAwTqBYAYIJiA4GAgmIFgnEMwAwQQEBwPBBATrBIIZIJiA4GAgmIBgnUAwAwQTEBwMBBMQrBMIZoBgAoKDgWACgnUCwQwQTEBwMBBMQLBOIJhh3AkeTDHkM9xSCXKK+mTyihqxEzkFqxmpX7fbIqEE9x1jyA1wSyUYUtRnIKuoETuRU7CikXS7LYJTNMO4O0VDMARHAIJ1AsEMEExAcDAQTECwTiCYAYIJCA4GggkI1gkEM0AwAcHBQDABwTqBYAYIJiA4GAgmIFgnEMwAwQQEBwPBBATrBIIZIJiA4GAgmIBgnUAwAwQTEBwMBBMQrJNQgoc48ll2ce0MK+qTzStqxE7kFKxmpBi9swGP4CZ/BEMwBEcAgnUCwQwQTEBwMBBMQLBOIJgBggkIDgaCCQjWCQQzQDABwcFAMAHBOoFgBggmIDgYCCYgWCcQzADBBAQHA8EEBOsEghkgmIDgYCCYgGCdQDADBBMQHAwEExCsk1CC+/sYcoPcUgnSivoM5hQ1YidyClYz0nHdbouEEjyQZBhOc0slyCaTzqN3UqJFxM2ZidREpId9VvbpdlskHqdol2CJFhE3ZyZSE9Eop2gIhuAIQLBOIJifSE0EBBMQrBMI5idSEwHBBATrBIL5idREQDABwTqBYH4iNREQTECwTiCYn0hNBAQTEKwTCOYnUhMBwQQE6wSC+YnUREAwAcE6gWB+IjUREExAsE4gmJ9ITQQEExCsEwjmJ1ITAcEEBOsEgvmJ1EQ0iuDBFEM+wy2VIJdKuQRLtIi4OTORmohM3mflePpIfzyCtQHB/ERqIiCYgGCdQDA/kZoICCYgWCcQzE+kJgKCCQjWCQTzE6mJgGACgnUCwfxEaiIgmIBgnUAwP5GaCAgmIFgnEMxPpCYCggkI1gkE8xOpiYBgAoJ1AsH8RGoiIJiAYJ1AMD+RmggIJiBYJxDMT6QmIraCEwnzYv/KlfshOFJEbAULErxh27YNEBwpIt6CVySTN5tXh3fvPsp9Ir2yj/TPuD8vP/wH5o8ScXNmIjUR8f1IfxLcOZLvNK8e6ujYn2cojHBLJSjk8y7BEi0ibs5MpCZihOtkM1hvwdYjWOAULR0R71M0noMjR8RWcIJeRnd3dXVDcKSI2Ar2AMEQHAEI1gkE8xOpiYBgAoJ1AsH8RGoiIJiAYJ1AMD+RmggIJiBYJxDMT6QmAoIJCNYJBPMTqYmIkeBHbhHLpz4LwfZEaiJiJHj2gW0feWUhBNsTqYmIkeDJuVV/lW2BYHsiNRExErzgPy7Z96vzINieSE1EjARvav8T0fVdCLYnUhMRI8FFuqsUQXDjC/4n8+++ORBsT6QmIkaCf3Or2DvnexBsT6QmIkaC3130/XM3ViuC4MYXLA7M21y1CIIbXLBhU6VoIMkwnOaWSpBNJl2CJVpE3JyZSE1EethnZd/YCQ6in3vnhbK3rqTH8q0roSrTUSKchfF964qbJjpFh6qMdIp2FsblFG1fV7MNwY0vePKQEIOnQrA9kXxETAVfeVdv7xeWQrA9kXxETAUf+Hhr6zUHIdieSD4ipoJ9geCxEWxcF9YIBEeqrJvgWT/TJvhAZ1tb5wEItieSj4gi+NErSkaMU+b8q3m5oXX6pvVt52wV4kjifQu3C7HpwsmzHzFXrD7t0n3lsoPLJhveikrBS9b09a5eAsH2RPIRUQSPXLSlZGTkp+ebd9elNk5eN7BxvhDLH89svViI9n/PHbjFXLG2f01HuazzC/RhEO6KSsGnJYXonwLB9kTyEVEEi80LR0wj2xaearzHvJsWefpr3pxqGLTk85eseoHq+kRfa7mstZe2dVdUCl68pq9v7WIItieSj4gkWFzxmHkx88fppGGdrEt/2w+VKvY98FsryoKtspJgd0Wl4O5Ea2tiPwTbE8lHRBP82izzYupPk3e6Ba9YceSt5UKs/FXulXY6RSfpFG2VXXPXkPBWVAr2BYLHTLC41rx4/MwZ692Cj9/UNucpIb47t+XCH5VfZFllB5dOqqiA4FgKDkstX0mF+n4wBDe44CAgOF6CawGC+YnkI2IqeF9Ha2vHPgi2J5KPiKngS82vg9dcCsH2RPIRMRVc/J+s0yDYnkg+IqaCV6/t61u7GoLtieQj5AUPn/SgTDC+TKqYSD5CXnBWm+AgIBiCIwDBdRc8xJHPsotrZ3hoyLVbEi3Cbx6qcjhKhLMwm/cp7I+PYDyC6/sIrvriCILlKuMmOMrPadnvaPBtAcENLvj9rxYg2D2RfEQcBT89ZyK+DnZPJB8RR8FBLSC48QX7AsH1fhUtbxg/+M5PJB+hQXAU8IPv/ETyETEVjB98d08kHxFTwfjBd/dE8hExFYwffHdPJB8hL9j7u2hHVAr2BYLHRHA+6yYHwTbNITg7wQME20AwBNdSCcEEBAcnxEjwKRDsmkg+Qr3gh8/8wNORBb//CAQ7J5KPUC/49t4fnxlZ8H1dPh9+CsF1PkUfuiqyYHw/2D2RfIQGwYXP+J1fwwn2BYLrK/izz0v7heAqE8lHqBf8pUfl/ZYFb794olj+FATbE8lHqBdMT57pqIIXPm+It6r9Wp2+Ywy5AW6pBEPHjrkES7QIv3moyqEoEW7BfpVhBUfCFjwlY4h326sUDaYY8hluqQS5VMolWKJF+M1DVeaiRDgLM3mfwsp3NugU/LHHjUM3rqhShFN04/9P1js3Tm1fUe3XREBw4wv2BYLH5vvBv+sBgm2aQ7DWn+jAp+y4J5KPkBec/ZQHlYLxKTvuieQjYioYn7Ljnkg+IqaC8Sk77onkI+IoGJ+yUzGRfIR6wd9un1X1N/+GExwEBNdV8B19P4n+DX8Idk8kH6FesBAvL4gseMuHJuIU7ZhIPkKDYKNlW2TBMzbmqxdBcH0FF176YGTBbSmfIgiuq+C/SD8/PbLg9Xf7fEsZgusreMrcZyML3jQZXyY5J5KPUC84ErbTaZvwHOycSD4ipoJn4DnYNZF8REwF4znYPZF8REwF478q3RPJR8RUsC8QPCaCtQDB/ETyETEVjFO0eyL5iJgKNhla9zUItieSj4ivYDFwRpWi1ABDPs0tlSA7MOASLNEi/OahKrNRIpyFmWGfwhNj4dcpOP/kzCpFJ3oZciluqQTp3l6XYIkW4TcPVZmOEuEsHMz5FFb7KXQ9gs0n4InznqtShFN0c5yiqwLBEBwBCNZJSTB+6M47kXxELAUTI4/NugaC7YnkI+IqeMtFV7xWrQiCG17wz664aHP1IghudMHXznrU761sENzogvEiyzuRfEQsBQcAwRAcAQjWCQTzE8lHQLAXCNYJBPMTyUdAsBcI1gkE8xPJR0CwFwjWCQTzE8lHQLAXCNYJBPMTyUdAsBcI1omP4IQJBNcc0UCCy7cguJkFp/v7j3MfSK/sI/3TsftI/3SUCLdgv8q6C76ha4d59ecf/vDeyDHOvQ5YLfN7r8NvLh0UesPQhdK/Z6Mm/F5k5V/7ROlG9FO0c6+963CKrpfgkR0QXHtE4whOdHRth+CaIxpH8CgQDMEBQDCDbrdFIJgBggkIDk6AYAKCdQLBDBBMQHBwAgQTEKwTCGaAYAKCgxMgmIBgnUAwAwQTEBycAMEEBOsEghkgmIDg4AQIJiBYJxDMAMEEBAcnQDABwTqBYIZxJzj6J74799q7Dp/4rpNQgqP/zgbXP2sP+J0NOsEpmmHcnaIhGIIDgGAG3W6LQDADBBMQHJwAwQQE6wSCGSCYgODgBAgmIFgnEMwAwQQEBydAMAHBOoFgBggmIDg4AYIJCNYJBDNAMAHBwQkQTECwTiCYAYIJCA5OgGACgnUCwQwQTEBwcAIEExCsk1CChzjyWXYxj3OvveuG3asr1kdsL1E5HCXCWZjN+xT263ZbBI9ghnH3CIZgCA4Aghl0uy0CwQwQTEBwcAIEExCsEwhmgGACgoMTIJiAYJ1AMAMEExAcnADBBATrBIIZIJiA4OAECCYgWCcQzADBBAQHJ0AwAcE6gWAGCCYgODgBggkI1gkEM0AwAcHBCRBMQLBOIJhh3AkeTDHkM9zSKjj32rsu5159MjXBQfT2EpW5KBHOwkzep3D8vrPBKTh6e4nKcfcIhmAIDgCCGXS7LQLBDBBMQHBwAgQTEKwTCGaAYAKCgxMgmIBgnUAwAwQTEBycAMEEBOsEghkgmIDg4AQIJiBYJxDMAMEEBAcnQDABwTqBYAYIJiA4OAGCCQjWCQQzQDABwcEJEExAsE5CCU6eYBge5JZWwbnX3nUZ9+qTJ5yCo7eXqMxEiXAWDuV8Cnt1uy0Sf8GfciDT3qdyQpV/SgoF+0wfI8F1PUU7D5FMe5/KCe4k50TyEc7Cgazf9LrdFoFgCIbgmiIg2LsOgnUCwRAMwTVFQLB3HQTrBIIhGIJrioBg7zoI1gkEQzAE1xQBwd51EKwTCIZgCK4pAoK96yBYJxAMwRBcUwQEe9dBsE4gGIIhuKaIxhG8f+XK/RBcc0TjCN6wbdsGCK45onEEr0gmb4bgmiMaR3DnSL7TvLp/6dLuAoPgFlbDudcBq08WnIe9UHAeIsn21SonuJOi7EGVQuEz/VC9BVuP4L6enmO9DLkUt1SCtKI+qZyiRsom6h30G+l4vQWrfA72gzshypD0fadXLaia6Gi833zW3dXVDcGRiLfgUSBYEggmIFgnEMwAwQQEBwPBBATrBIIZIJiA4GAgmIBgnUAwAwQTEBwMBBMQrBMIZoBgAoKDgWACgnUCwQwQTBx6R9FRSCnq884hRY2UTXT03bf91up2WySUYJbbtyocQwXP3VHvCSp45ov1ngCCtdLQgp/cpXAMFex8st4TVPCLjfWeIIJg0BBAsHYSdU2XEVzfiRkKt3YV6j1DBYmKG3WhKQS/ce+9u+s9QwUNLjhx/b+YFw9dX/9XESZ/t2vn35vT3L/8VfPyO531HqdEwvGnjkg/gvd80rzx8703KZ5HiruLfxI7dtxmXv5bTM7WjS34n29KdJg3CrE4Xf9fwmSPSAznrjVnytd7HIui2kKjCr7+zZ1x+OdZ4nuvCPGf/ygSr79+WzwGKmIOctPe7XU/RBKCC+aT3LdveDg+glcdFuLtT9vPwfWexoKO0sbrH677IZIQvOcW9WOoIC5qS8TlKEkIvuF59WOoIF6C43KU8D9ZTQ4ENzkQ3OQ0seAm3rUaaPSjYNA3gZ+o2AtDNP6uqaHRj4KxYESMLIDgqjT6UTAWPSmeWGTuxf7FU5YcMO9/9YyZzwnDxLo53ml4wS8uGF7workXiTX9qzvN++tST82zHsGlm+OchhcsPvoHv0d70donetvM+2mRn2gJLt0c5zS+4F9O2mkJ7msrP/naz8GNvnvRafQjYNiXibXJNZ1lq6fvg+ASjX4EyoK7F59WfJFVuveV0w0ILoIj0ORAcJMDwU0OBDc5ENzkQHCTA8FNDgQ3Of8PDAVnheoeWi4AAAAASUVORK5CYII=\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -i it_namespaces_monthly\n",
    "\n",
    "it_namespaces_monthly = rbind(data.table(it_namespaces_monthly),\n",
    "                              list('2019-01-01', 1, 0, 0))\n",
    "it_namespaces_monthly[, log_month := as.Date(log_month)]\n",
    "\n",
    "\n",
    "g = ggplot(it_namespaces_monthly[block_num_namespaces > 0],\n",
    "           aes(x = log_month, y = n, fill = factor(block_num_namespaces))) +\n",
    "scale_x_date(\"Month\") +\n",
    "scale_y_continuous(\"Number of blocks\", limit = c(0, 15), breaks = c(0:4)*5,\n",
    "                   minor_breaks = c(0:20)) +\n",
    "scale_fill_discrete(\"Namespaces\") +\n",
    "ggtitle(\"Number of partial blocks of namespaces - Italian Wikipedia\") +\n",
    "geom_col(position = position_dodge2(width = 0.9, preserve = \"single\"));\n",
    "\n",
    "ggsave(\"graphs/itwiki_partial_namespaces.png\",\n",
    "       plot = g, width = 30, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of blocks per month - Italian Wikipedia\n",
    "\n",
    "Just plotting the total number of partial blocks per month on Italian Wikipedia, for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "it_blocks = partial_blocks.loc[partial_blocks.wiki == 'itwiki']\n",
    "it_counts_monthly = (it_blocks.groupby('log_month')['wiki']\n",
    "                     .count().reset_index()\n",
    "                     .rename(columns = {'wiki': 'n'}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAADAFBMVEUAAAABAQECAgIDAwMEBAQFBQUGBgYHBwcICAgJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7////isF19AAAgAElEQVR4nO2dCXzUVP7AR/6rFIXSVi4VUEBEwAtcWDwQEBZ1dWpFgVJQ1uUQUJGVVRahixzFVZbDYwVPUBFdDtHlUkBXRUUERTnFUlqgpbSBlrbTY67mP0dmOjPJS17eMZOZ/L4fnUleXvJ7ky8vmSZvfrGIQEJjiXUDAL6A4ASHoWDNTX3XGT9aZE35mkz/aWJtDDeiRr3o9qloCr71I/wtGFSwdr2GSqi6WJthhp5Qljvr1VbR3FRynVTRYmn90Bn1lS3Bque1zjzJU7AFubFIE6iIpa28HyzD839ty1Ltlhm3B1tGvKy2iuamLA3vhUOz1Fdu6AXuoqm3GluwmL5aFM82EkRxVQZGywwsuPyqQ2LD5/a+v9rhgu47lndO6nPYM/N8q7RH7R4jc69IfdjmmV/U9jzfenVPtGn9RJ23N1qCMU9fLB65P635fYK/nn+Z579gYUjzzjXxTUibEZ3Z7VP+5SvZc+kScXuPpMvfkCqjGhDWzMBmLEsvT+qzX2qVNBO5b/wLQ9oUOtewykf3iOIGyyeiOPhjqZK3ZRZ5i+yT0lq/EL4Z3ugSLP7v945wwXcfteU0S8+z5fTzzAwqLBz0rCguHHisbNSTnvl7i/zrZQ86eXLAP8J6sFjSQrzui5pzj42V6gW2GCwMVnUXT+vnmwhsZl7/o2VTvCUbWn4sipesrTs+RqqMakBYM4Otyci3zb4lEFmaidw3voUhbQptYcMqjktKxL/f8bRY2Mbhr+RrmUXeolmDCk8OCN8Mb/QJFv82PVzwKVGsthR7Xryd7KAoHugkilf/6umgl3vmC6T1OnoW7O8UJrgoM9M3WdFWqhd6oPQVSlW9pB7wzQc2c6W/21heufQHz1u7l04E24dqQFgzg60JNNtXQ5qJ/LzB3SO1KbSFIatMXiz2/e4Wcf4UfyV/yyzyFnXyxg7fDG90Crb33BEmOLgJ70yt51tGkig28Upp5Jmvl9ZLkhYEBVssrUaWirtvT7VY/k+qF9hIsLCheaUzbvdNBDbjffcu6jjd+/ZjxsWdN0uVUQ0Ia2ZYayI/g9S84Gq+qZA2RbZQqrm3h/1asWtdl5/9lfwtU9glvtjhm+GNTsHiga4V3p1ULYrFMsGeE/RBzz/XLvkRm1bowV46vXPWVRbYyHnSgrBC6a2ySVgP7iz14JNXvuCbqN/YRqqMakBYM8NaY2mILN8XgYUhbYpsYWCV618bK/75XzdIO8LfMoVd0skzfyB8M7zRK1hcMtrzdsts27F0meA7CgsHe05uSwYdsu8bHrLpGZ6z3u0zZYLbrK/Luz+wkZbS17ewQv/bmew+vonAZnL65/nPwUVd5oti5kH7xkukyqgGhDUzrDWWhsiKgn0LQ9oU2cLAKosuXimuaLpE2hH+linskmzP/KDwzfBGt+D6Ozxv+/skXb5UJvj5VqkTPV9P3S9d3fjaT0I2Xft469aP18oEb+pyfvuXAhtZmOKfCiv0HS4taelHffOBzTieaZu6yFdS3HWO+EGXpB5fSJVRDQhrZlhrLA2RFQX7Foa0KbKFgVVKzy8U88+X/giWWqawS+ompLZaEL4Z3kQhRNQw3mcxQIsM0ARmGO+zGKBFBmgCM4z3WQzQIgM0AeAJCE5wQHCCA4ITHBCc4IDgBEeH4IpyGU6bvIwttbwD2Fy8I5RXct9L5Y5qedlZvYLLBBmOSnkZW2p4B6hw8o4glHPfS0JdlUIhCPYCgkEwLSBYBRCMBQhGAoJBMC0gWAUQjAUIRgKCQTAtIFgFEIwFCEYCgkEwLSBYBRCMBQhGAoJBMC0GFGy1el4Kxo8vCLyCYAoMKFj0Cp67devcwCsIpsCogkdWVo4KvIqHt20rrZThrJGXscXOO0CNi3eEShv3vVTprJWXlWkJznC7MgKv4gcTJxY6ZNS75GVscfMO4KznHcHh5L6XHG6FEFW6ejAcoskx6iEazsGMMKBgq/drdP64cfmBVxBMgQEFKwKCCQHBKoBgLEAwEhAMgmkBwSqAYCxAMBIQDIJpAcEqgGAsQDASEAyCaQHBKoBgLEAwEhAMgmkBwSqAYCwSVfDJYuoAIDhIjRy3XaGQKU61hdV9n6EOYHdTb0KLOu57qcblkJdVxH8PXpR8FXUA6MHGFXwodXXKd7QBQLBxBWeMFh6YQRsABBtW8IetcoXlPWgDgGCjCj7R/i3Py4V7KQOAYKMKnjDQ+/qn+ZQBQLBBBW9r9qP37ZVbKAOAYGMKLr4ux/d+NOlXugAg2JiCZ/U47Z/o9yJdABBsSME/JX8hTT0/mC4ACDak4MGTA1MHk45RBQDBRhS8tN3x4HSvN6gCgGADCv6t5eqGmVkZVAFAsAEFjxgaMvND00KaACDYeII/STscOtt1FU0AEwv+atTEk0bMslPU+eWw+adG0gQwseBRubk5RvyF/19vLg2b/zKNZlyHiQVn5eY+aMAcHV83jbwHfMXHFAFMLHjz8AVDgll2Fg8YkF8vQ5QX8cZ1y7zIoqmTaTYYg8/AHqUPUa0lWBQPjAn24PLCwjNlMhw2eRlbaiMLXuhaElm05dKz5AGqnOTrYlLBfS+V2avlZWc0BVdmv224c/CB1M2yOiWtt5EHMPEh2jp0Qa3hsuzcPVah0p+nkAcwseBIjCB4VeujCpVWdyIPAIKNJDj/0neUKp1K+ZY4AAg2kuCx9yjXGko+uBIEG0jwZ81/Ua61/AbiACDYOIKLr3keUevEhT+SBgDBxhE848YSVLW7c0gDgGDDCP6h6ZfIav++mTQACDaM4H5PoqsdTTqMXqgKCDaK4Jc7qt3Y77+EMAAINojgIy3WqdV74Y+EAUCwQQQPzVKtd4h0cCUINobg9WlH1Cv2ep0sAAg2hOCTVyzTqPjsvWQBQLAhBD8+QKvij4SDK0GwEQR/lbxbs2a394kCgOAgtioZrlp5GVvs3peK3vO0a05/kChAjYtoNT1Uc99LVc46eVm5XsHn5KNCojRkJ6e7bJiOnB1ppSQBojBkp9KoQ3YiiNkhem/yFpyqndaTBIBDdOwF3zkBq+qjSqN5NAHBMRf8ZtsCrKqbLynVriQDBMdacN4l7+FVLWn9GUEAEBxrwQ9h/zz04ScIAoDgGAvekrIft+6ajgQBQHBsBZ/rsgi77qnUb/QHAMGxFTyjj45vTsOm6w8AgmMq+PtkPZ1yxXX6I4DgWAou7atrwPNJgsGVIDiWghddqRBUhXvm6Q4BgmMo+FDqen0P5Xj1Jt0xQHAMBWeM1vnUlaOND+qNAYJjJ9ib8VvnY3UG4P9RJQGCYybYl/Fbp+AFA/VGMbHgbaNGbotlGiVfxm+dgg81Vvr9sBomFpyZm5sZwxQOXyZ7/+bR++Sz3q/pXMHEgifl5j4WTMKyc8WK0zYZrjp5GSPO3fCC982hc7X59+lcodatcwX91PDbSwGcCiHOaQneO2TIz8E0Sltyck7VynA75GWMmN/T5n1z6lzt16Zl+lawu3VG0I+d314K4FIIofnks7G5ueNilgjtp+TPfe+6H07ZHfPucQATH6KzcnOzYnYOHvy4/1234Kcz9dU3seDPsrI+i1UapWDGb92Cv9aZudLEgiOJouCGjN/6nx/c6SNd1UFwLAQ3ZPzWL/hxfYMrQXAMBIdk/NYveIu+wZUgOPqCQzN+6xdceumneqqD4OgLDs34rV+w8JfJ2nUaAMFRF7wjNOM3geC1ugZXguBoCy7pHTpMh0BwcdoOHbVBcLQFP9/1VMgcgWAh8+86KoPgKAuOyPhNIvjda3VUBsFRFnxP+J+xJIJPXrQHvzIIjq7gyIzfJIKFe+bi1wXBURUsy/hNJHhpH/y6IDiqgmUZv4kE5+kYXAmCoylYnvGbSLBw+0LsqiA4ioIVMn6TCV54O3ZVEBykRo7brlBIzrO9bJFFTqINFSSdwq1qdxNF0EMd272khMshL9McshP1HqyU8ZusBwt/0Mp8GAR6cPQEK2X8JhQ8x4pbEwRHTbBixm9CwT9deAKzJgiOluAjLdYqlBIKFq59F7MiCI6W4KEjlEpJBf8dd3AlCI6S4PVpvyoVkwr+uvkp7UpeQHB0BJ+8YqliOalg4UrVRzw0AIKjIxiV8ZtY8OQxePVAcFQEIzN+Ewv+tA3e4EoQHA3BJb1mIZYQCy69DCsHMQiOiuCcbqivRMSChTGPY1UDwVEQ/DM64ze54HUdsKqB4CgIvgud8Ztc8Om0r3GqgWD+gt9SyfhNLljInIZTCwRzF3xMLeM3heD3rsGpZWLBVqt1eDSy7IxWy/hNIbiwKc7gShMLFsXPF0XhF/7qGb8pBAvWORiVTC34n7v45+goUs/4TSN42R8wKplZsOthRzDLznM33pin9m+BmBl967ls18O5pGJem44Hqn2vaoJ/XigGe3BtRcXZMzIclfIyfWxO/kF1eS3Nxgct1K5T4aSJgMW5Ku4h7DaFQk3Bb3iO0JzPwcc7vqRegeYQLSzSfGapuQ/RUzxHaM5ZdkbcpVGBSvDhxrmadcwsOAIOgt9pcUijBpVgoY/yTeZQQDBHwYdbrtKqQid4buQvYeSAYH6CS/+ofVOeTvBPTTQHV4JgfoKf66w9tpVOsHDdO1o1QDA3wd81w3iYJKXg6cO1aoBgXoKLe+I8E4lS8I7mRRo1QDAvwVN6ncaoRSlYuFJpNH0oIJiT4E3Ju3Cq0Qp+4i8aFUAwH8EFHV7WriTQC/6sVYl6BRDMR3Dmn/Dq0QouvWyzegUQzEXwOy0Pa1fyQitYGPuY+nIQzEPw/rQPMGtSC/5IY3AlCOYguHQQdtJuasFagytBMAfBz12F+/NsesHCiKdVF4Ng9oK/bbYVuy694JXdVReD4CAKyV2IsuxU3DgbvzJZlp1QypMPqC2GLDvMe/ATvXEuYUnQ92AhfbbaUujBrAVvSv5BR20Ggl/rrbYUBDMWjHsJS4KB4PykAypLQTBjwZl366rOQLAwaIHKQhDMVvCKNkd01WcheLHa4EoQzFQw/iUsCRaCjySpDK4EwSwF67iEJcFCsHDTq+hlIJil4PlXndTZciaC56kMrjSL4DfGiCMu/oSzYD2XsCSYCP6pyXHkMrMIvvz41lu/vZav4FM9ZupuORPBwvUrkIvMIrixY+KL9iS+gifruYQlwUbwM8OQi8wiuNvXPY/9diVXwfouYUmwEfwNenClWQSvS3tKHPc6T8F57V4haDkbwULnNaglZhHsI5+n4GF3k7SckeApD6OWmEXw+57/j3XgKFjvJSwJRoK3tkQNrjSL4N6bxbwO74YJrf3XUCuzLDv70z4kajkjwaVtUYMrzSK4pMfKjmvCe+xb2ZUiq1/4lw4cT9ZyRoKFsY8iFphFsHi884aIQ/KYg95XNll2cnRfwpJgJXh9O8QCUwi2BAgTfN/zwz8NZtnZkpNzqlaG2yEvU+Kn5G/wKspwEq4Xia3FLuUFdjejCGjsmHuJApdCCM0hO6Nyc7OCPXjnihWnbTJcdfIyBcp7zsGqp4CDdMVIHnpGubzWzSoCkhq8vUSDUyHEOS3B83JzH2RzDn78D/ovYUmwOkQL73dTLjfFITp4qA4TfGLCyK+YZNnZmPITccuZCS5q9r1iuVkEN64RxeoL0N2ZQnBeu3+Tt5yZYOFe5acGmEXw7U+XlU0dwEXwUO1UN2jYCX69l2KxWQQfvzcl5b6TPAQvJ7uEJcFOcEETxay2ZhGMAZngfYSXsCTYCRb++IJSKQimE1w68BGqljMUvKS/UqlZBB/PSE3NOM5e8LwuhJewJBgKPtL4N4VSswjuP728bFp/5oK/Sf4fXcsZChZuVrohbRbBF1WKYkVT1oKLuv+DsuUsBeco5QUxi+B+08vLZ/ZjLfgx8ktYEiwF/6I0uNIsgvOtKSnWAsaCN6SSX8KSYClYuGG5vMwsgjHQLTivncovCjBhKnjGUHkZCCYX/ADNJSwJpoK/T5YPrjSFYOX7wbSCl1+i9GeJTpgKFq5aLSsyhWAuPZjyEpYEW8FTRsuKQDCh4NLb0Y8U1QFbwdtayr7Vm0XwsfSUlPRjqoLL5U/mUXlu0pwuhSweCFTDYiNBhLabIoui8dwk6qdLaaL93KRenr+Dp/dSFVwtHxWCHrKzJ3knk7EozIbs+Jk0ObLELEN2fFeyLmJ2iC7q/iybYw/bQ7TwiWxwpVkO0dNmlpfPnMZM8KN9aC9hSTAWfLpF5LVxUwhm/mcSg0tYEowFCyP/FlFgCsF44AvOa7eMVctZC17VNaIABBMIvp/BJSwJ1oJlgytBsH7BTC5hSbAWLNwXcQMTBOsWvC9tHbuWMxf85u/D50GwXsElfScybDlzwQVJv4TNm0KwBUs0puDZlKOwwmEuWBj8fNisKQRf9l09M8E7kr9k2XL2gl/sFzZrCsFrOzRi9XdwUXfV9Nu6YS/4SOOwgfimECyyO0RPYnUJS4K9YOHWsITVZhGMAY5gdpewJDgIfu6u0DmzCGYz8D2v7WuMW85B8L6wwZVmEcxm4PsQdpewJDgIFnqEDq40i2CFge9WD/rSKC1jeAlLgofgGQ+EzJhFsMLAd6vvVUcKh31pHzFvOQ/Bu0IHV5pFsMLAd79gKQlLeWHhmTIZDlvIzJnbJslr0FLLfpNlZV3WNExXOXlECKPCplUjnzaEvVpeFj5kRwHrsHG7g2mUFg8YkF8vQwydWXBtrbyGIZkxPmRGRFaLEq5NGb9bSLkNpQ9RrSVYdO18SEcitK+afcXh2MPjEC1sD8lcGetD9P7p7Tpkr2mm/ohUTcj+Dnbvfgj/HFzUbQ5dG5XhIlhovzE4GVPBJWvTL0xfWyoIsygv4BMJtqaP24WfRmniTah8rlTwEfxIww2vGAo+kN2+Q7b/SeglfekSIfC/kvXf1L1ULUTBR/B/25UGJmMlONh5/VD+DkRb8Hl0go8yv4QlwUdwSasvApOxERzSeSXokhFpC76slErwkPspWqcGH8HCqKmBqRgIjui8EsOV8g/goi140bhyCsFLL2V+CUuCk+APgoMroy5Y3nn9FHR4kTwExu1CmvvBv3C4hCXBSXBRs53SVHQFK3deP5uSlbNp4sD3S1ZJ30eJW6YFJ8HCkGxpIpqCUZ1X4skbTpGG4Ct41tWFpA3ThJfgt26UJqImWK3z+im+8SnSENqCd13XSByxmkjwjmQel7AkeAkuaPKzfyJKgjU6r5/dzVGPDtFCW/C1n1rEE0SP1SnqNpewVTjwEizc8U//ezQEn9PsvBKL2h8jC6EtuGmdRSxJIxH8CJ9LWBLcBL90m/+dv+ADsy7X7rwS6SPIYmgLvnuFpShzJIHgdSl8LmFJcBMcGFzJWbD3zJvxCUbn9XOk9RtEYbQFn868OG3kWf2Cj7Z9nahFuHATLPT1D67kKvhAdruO2Yf13PBfTXbJl9+36PsUcouxhJ/g5+70vfETHPzarGtEx9h+JGc8boLfuDSXoDk64Cd4X1K+942XYH/n9U3qEkz2nZVJlh0Fwflp6wlaowd+goWeb3tfuQgO/5tX35gsot/+aAvGyLJTI+fjGQqFTHHy2/Sc4d5Xu5v5hnNntO08/0TDfJ1d1+rPdSvTHdLlkJeFP/mMcZYdZnDswbuaegdXsu7BChesdI6qLLlN/83/qGfZYQZHwcLV3nvsbAWHnHkb0Dtsdl/aB3rjxiQZKRN4Cn7yQYGpYNTVZt3jope3xLwuEiSGj9WhhKfgz1ucZih4v1Ln9aF/4Lvum/8gWJH2G1gJVr1VpF9wQccl+lbQFrzxqkamO0QLEyawEezrvL8iFxP8dGWzzpv/2oIvWeMyXw/e0K6UXnDue3c0GfKx2tVmkt8mTb2xWE91bcGpNhMeok+3/JxGcNFXSycPvOx33WdrjIckEVzcMzLnoiragmfPqjWfYOHBJwkF71+bM+z6xim9xy3ceEKzMtGvC3cn/1dHbW3B6xqb788kQfjwat2Cj21/ZVy/iy/oMix75UHMVch+PrpYz81/bcGt1pnwHCycar4TX3DxnpVPpXdp1Hrw5Fd26LrjQ/j74PRM/LoYX7LMeA4WhPtnYgnO3ZjzUO+klN4P5WxUeHiaFoSC9dz8h3Mwgrd7agjO3/7K5MEtfUfkA6RBSH/hvwZ/sAznge8c4Sz4xIWHUIJDj8h0ub+IUziMuxn3VABXslDcuUBBsO+I3KS594hcwCAGsWD831wTCl6vM8sOB3gLfvm2MMFFOzxH5Fbnd0l/auUeZjHIk7Bg3/wnO0RX/NWqK8sOD3gL/i0pcMVo/8rsYf4j8nb5Ew6poMiyg5u6F6sH12TPCxf86udWHTk6+MBbsNDvTc8ReeG43hcme4/I+TxCUAguuW08Vj28Q3RV8zC/BRPc1mCWnfXTphXWyXA75GVscfEO8GLrlhfckDV/wzF+IRxO8nXz0jbgVHMrhKiMFOz6sG2Y4Jmfiw09eO+6daerZLhq5WVscfAOUPqf3eWcQ1TT7KV32xRg1HLWycvKI8/BjTpvChNs9eUyTPBzcMzTKGmSeZd2HeI/k6z4WXY4AYKxbv7D38FIDC9Y2Bz5qCcFtASbc9CdD+MLFqZer/nLf5we7F7e/j4QzANawcW/17z5jyF44zW37YRDNBeo0wnvaf6JRg1Nwd/fds0Gdb0gmBj6fNGaN/+1BA9p/7Zbyy8IJoVBQnCtm//wJQtJfAjWuvkPfyYhiQ/Bwhr1X/6DYCRxIlgYr5rrBgQjiRfBRd3UnhkIgpHEi2D1m/8gGEncCBbmqNz8B8FI4kdw6cBxyGUgGEn8CBb2pa1CLQLBSOJIsLCixSHEEiaCK8plOG3yMrbU8g5gc/GOUF7JbC9l/QmxwFEtLzurV3BVpQxnjbyMLXbeAWpcvCNUVjPbS6c7vay8wFkrLyvTKxgO0YQwfDjl1uY7FcvhHIwkvgQLf1O++Q+CkcSZ4OJeU5WKQTCSOBMs7ElWuvkPgpHEm2BhidLNfxCMJO4EC/cOl5eBYCTxJzj3MnmSfRCMJP4EK/3yHwQjiUPBwvibIlMOgGAk8Si4qNuzESUgGEk8ChZ2JP8vvAAEI4lLwbKb/yAYSXwKLh04NmweBCOJT8HC/vCb/yAYSZwKjrj5TyR4U9Zfvkn4NEpxK1gYEfrLfyLBr9v2jUz4NErxK/h4x0UNM4SH6IOTgklY8nftEs7JcFbLy9hSxzuAzcU7wrkqPnvpi+a7g9OOGvly7REd1vv3BtMoLR816oRTRr1LXsYWN+8ArnreEZwuTnspu2dNYLJeYT9VaQqu3zcm8ROhxe0h2nvz/8nAJNEheq39xwfhHMwAXoJDfvlPJnjouF2QRokB3AQLLwZu/sPfwUjiWnDw5j8IRhLfggM3/0EwkvgWHLj5D4KRxLlg4ZE+3pv/IBhJvAsu6j5LAMEqxLtgYUezrSBYhbgXLMy96iQIRhP/gr03/0EwkvgX7L35D4KRJIBg4Z0WJ0AwikQQLGTdoRQCBHtJCMEFA3IVSkGwl4QQDF+y0IBgEExLvAiukeO2KxQyxck7gN3NO0JNHfe9VONyyMsqoAd7gR4MgmkBwSqAYCxAMBIQDIJpAcEqgGAsQDASEAyCaQHBKoBgLEAwEhAMgmkBwSqAYCxAMBIQDIJpAcEqgGAsiARvGfnwt5BlhwFGFfxa1S7IssMCowoWxbP/gBwdDDCs4Ppl5yDLDosQ3PcSYZadZT+KkCeLAZzyZIVClCfr3W3eVzgHU2PUQ7TVgx2y7NBjVMGRgGBCQLAKIBgLEIwEBINgWkCwCiAYCxCMBASDYFpAsAogGAsQjAQEg2BaQLAKIBgLEIwEBINgWkCwCiAYCxCMBAQHqaqU4ayRl7HFzjtAjYt3hMpq7nup0lkrL9N+8lkEFeUynDZ5GVtqeQewuXhHKK/kvpfKHdXysrN6BcMhmpB4OUSDYEJAsAogGAsQjAQEg2BaQLAKIBgLEIwEBINgWkCwCiAYCxCMBASDYFpAsAogGAsQjAQEg2BaQLAKIBgLMsFWqwhplBhgWMGiVzCkcKDG2IKlJCyHt20rVRgpAkN2MLAZdsiOV7CURumDiRMLHTLqXfIytrh5B3DW847gcHLfSw63QgjtNEqhPRgO0eQY+xAN52BqDCvY6v0aDWmUqDGs4AhAMCEgWAUQjAUIRgKCQTAtIFgFEIwFCEYCgkEwLSBYBRCMBQhGAoJBMC0gWAUQjAUIRgKCQTAtIFgFEIwFCEYCgoPUyHHbFQqZ4uQdwO7mHaGmjvteqnE55GUV0IO9QA8GwbSAYBVAMBYgGAkIBsG0gGAVQDAWIBgJCAbBtIBgFUAwFiAYCQgGwbSAYBVAMBYgGAkIBsG0gGAVQDAWFIIhyw41xhYMv/CnxtiCIUcHNcYWLGXZWTxgQH69DFFeFHckwmdQ/BDVunpweWHhmTIZDpu8jC21vANUOXlHKKvgvpfK7NXysjN4guEcTI2xD9GQZYcaYwtuAAQTAoJVAMFYgGAkIBgE0wKCVQDBWIBgJCAYBNMCglUAwViAYCQgGATTAoJVAMFYgGAkIBgE0wKCVQDBWIBgJCA4iK1KRvFZeRlb7LwDlBXxjlBVXcM9xOkz8rJyvYIVmPQZ1epG4NvRsW4BC6atQy0BwaNj3QIW8BL8/kGq1Y3A0eWxbgEL1v2IWkInGDA8IDhxsCoVkgpW3Fh8UT92XH2s28ACq2wiFBMLPrhw4aFYt4EF/ARbh27xvCwduoZwGzHmzQP73/K0f/GI7zyvr2XEujnEWEP+k0PVg48+7Jn4IS+LuG0xZZbvP+vu3RM8r5/H79Gam+D/ZFnTPRP1cXq4zrV6OCpanY4hnk/hinVzyPGprecgeOjh/Sr/cgzPu9+K4jfvidY9eybE60fw42l7Vt4uxoLrPaesfw9bFs+CJxaL4qlJgXNwrFtDjlfFmqHLGAs+OoamTUYijtX60VBBKHjYp2TrGY+4F2tWBDEAAADXSURBVKyhAq5kJTggOMEBwQkOCA4lAfdGAn4kLSwfel5WyT64RUzIvZGAH0kLSze36O4GghMWS48PxVU9PB+8oF/T/sc983Oat90kWjxIkwmFGQVv7+bstt3zwa3TK6ZleOazbas7Sz3YP5lImFGweOcDd3k/eEq5WJbqma8VXY0kwf7JRMKUgvedv18SXJ4aPPkGzsEJtkcS7OPgYAm8WmdWTs8IWk0+BoITg6Dg/H4X+b5k+eeeTbaAYCDuAMEJDghOcEBwggOCExwQnOCA4AQHBCc4/w+A93wDaz9D6QAAAABJRU5ErkJggg==\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -i it_counts_monthly\n",
    "\n",
    "it_counts_monthly = data.table(it_counts_monthly);\n",
    "it_counts_monthly[, log_month := as.Date(log_month)]\n",
    "\n",
    "g = ggplot(it_counts_monthly,\n",
    "           aes(x = log_month, y = n)) +\n",
    "scale_x_date(\"Month\") +\n",
    "scale_y_continuous(\"Number of blocks\", lim = c(0, 100), breaks = c(0:10)*10) +\n",
    "ggtitle(\"Number of Partial Blocks per month - Italian Wikipedia\") +\n",
    "geom_line()\n",
    "ggsave(\"graphs/itwiki_partial_per_month.png\",\n",
    "       plot = g, width = 30, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution of namespaces\n",
    "\n",
    "For blocks that block a namespace, what is the distribution of namespaces?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "def count_namespaces(df):\n",
    "    '''\n",
    "    For the given `pandas.DataFrame` `df` with partial blocks data,\n",
    "    loop through the log parameters of all blocks, identify if they\n",
    "    block a namespace and count the number of blocks per namespace.\n",
    "    '''\n",
    "    \n",
    "    ns_counts = defaultdict(int)\n",
    "\n",
    "    for row in df.itertuples(index = False):\n",
    "        params = ps.loads(row.log_params.encode('utf-8'), decode_strings=True)\n",
    "    \n",
    "        try:     \n",
    "            restrictions = params['7::restrictions']\n",
    "            namespaces = restrictions['namespaces']\n",
    "            for ns in namespaces.values():\n",
    "                ns_counts[int(ns)] += 1\n",
    "        except KeyError:\n",
    "            pass\n",
    "       \n",
    "    return(pd.DataFrame(ns_counts.items(), columns = ['namespace', 'count']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "it_ns_counts = count_namespaces(partial_blocks.loc[partial_blocks.wiki == 'itwiki'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAAC31BMVEUAAAABAQECAgIDAwMFBQUGBgYHBwcICAgJCQkLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERGRkZHR0dISEhLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZoaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKkpKSlpaWmpqanp6eoqKiqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5ubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7////QTiPbAAAT1UlEQVR4nO3diX8c5X2A8cUBO5DGRzkM1MYYx1AwLbS0lAQwdWgqRxwNtpS0xEhAk5Kj4Lq1SakgLSlX3ZazDQ5JCNgByhEQISFNgLgUp4Bpg4Ml2wLjQ9dqpT20u+8f0Nnd2d+r3RnNzvuylnfHz/PB0q7mfeedma93V7JlEVMU6WIH+wDowAZwxKsLcM2dvLwoVjHQOyHMR6assEvXGNcQD56pAf749ysHNj5w7dX1oMnGhtrNgS7cEcT+KB80uOZOfi1VObDBgKslJlt67zGF82h1fiWP3hswrsZuprSQwCv+OWhwzZ1UwzYpsFr+iFL90/Yp9d3WwHE1djOVhQQe/NhbSl+Iwvu7Fxxx+k8fXDTj3P917tx6zOw/TyuV6zpp1pUJ5/4dJx5WnJe67rhjr0upmFNpP+7AmGxS4+vmzbyt+JH/Pv4u9cOzZsx/QE0c+4mHnds75g4VP3TP/BnnvqHULy+b/dFL9lUdRHlxdxflpcpj3XXKo6ouQukIZb/F/+Seu6rT9/9YqSdjTyj1ycfdQYWDjnmvQPra2cd+o3I3B6mQwOrHv5upBP7UrxK3fGT5O4lbLnDuLOvrW/a3St3+h9sH2r/i3P/0u6V565bt2rX0xgmPXHdgTG+6+cJfDXyp8JEnj35cqbkbUztWVYx99rScUqv+ofSh1p7ETecpdeaPxoa+cFXVQZQXd3dRXqo81l2nPKr6IhSPUPZb/E/uuas6ZebuUX918Q2q77hMaVDxoGPeK/C1ZX27llbu5iAVFlj95ZpK4PeUGo3tdt582LmzVak3Fyp16v8p9f58536vO+9kZ8MbCycAuwNjetMppcdG7F+O/y/n3W+s36mqxp7zHfX2CaOlD7nrFRo+seogyou7uyhPL4911ymPqj49uQ7F/cr94r0Jq/7FneoTL5+nvv6l0qDSQce8V2Bh4ewqd3OQCg2cPvunFcAyuXAn6XzaMUOpDxee6aY59/PuvBnuBgHW98ubCu8LW05eU3j3WuucRc9Ujn18cXbF+gkH67x59aJZsdiHqg6ivLi7i/L08lh3nfKo4qSY/jS4eGvCfn1XUWrLWekl6rTU4l+UBpUO2ucKFM+ucjcHqdDA6s3ThgsuzmNptwfYeYHe6vz+XdxTtVPPI9gdOOERvMh9BO865RvFG/mnjqscmz/z+pMmfhLuvFn4rf7sQPVBlBd3d1GeXh7rrqNHVZ5e8XOGCfv1XcXpt+67Sv3Zbb/tnnfpoH2uwELn/puVuzlIhQdWd/2p8+68mxLbl3uAL+7r+6TzenrXsrfSr39mwk7/xnmhvWjtBGB3YExvuuXCd0qvwe8u/rpSV2xNPzW3cqz6XuzfJh6G8+a4x1LvXFZ9EOXF3V2Up5fHuuuUR1Wf3tGFTyIn7Nd3Fac75nxbbTjqLve8SwftcwXWOfeXVe7mIGUAnL/YeffGuTPm3+MBvvWYWdc4j7Lc+lOnL3liwk6TXzz22C8mJwC7A2N6U+avT5x1R/Eju0/7O/Xw4hln/ahyrHp00fjEw3DePL348Hnrqw+ivLi7i/L08lh3nfKo6tO7fWblfn1Xcdp7eJ/qOdz9Itg9aJ8rkLp61jH/WLmbg9TBWzl0Ld+1mTW1J9a4l7Fxj8wtd98ZOZt5AJdq3CNzi530qt28Oh9HI61mUuMeGdUlgCMewBEP4IgHcMQDOOKFAh7YF9hwJni7f9lBi0kjaZul8jVOwLdE0mYptd9i0tioxaR+VfNYAA4KYAlgHcASwBLAEsASwDqAAfYGsASwDmAJYAlgCWAJYB3AAHsDWAJYB7AEsASwBLAEsA5ggL0BLAGsA1gCWGoY4BWGBS4PsASwBLAEsA5ggL0BDLBPAEsASwBLAEsA6wAG2BvAAPsEsASwNOXALU5K9XZ29gJsWJMAF992dXd3AWxYMwG3xePtABvWJMB/0vGqUq25bOHnmv94/fr3xkqZAo8FlU8FbvYvk7WYNJZP2iw1brOUsllq3GappKoxYHhyYJV95XPyCAbYpCYBzr36OV6Do/wUvbxjs1I9HR09ABvWHMDVARw6gCWAJYAlgCWAdQAD7A1ggH0CWAJYAlgCWAJYBzDA3gAG2CeAJYAlgCWAJYB1AAPsDWCAfQJYAlgCWAJYAlgHMMDeAAbYJ4AlgCWAJYAlgHUAA+wNYIB9AlgCWAJYAlgCWAcwwN4ABtgngCWAJYAlgKV6Ao/ES5kCx4PKjQZu9i85bjEpnk9YTEplbJaSa2VSOm0xaUTVGDAQHnh4sJQp8GBQuZHAzf6NZiwmDeaHa4/xlEzbLKWGLCalkhaThlSNAf08RQd2KD1FAxw6gCWAJYAlgCWAdQAD7A1ggH0CWAJYAlgCWAJYBzDA3gAG2CeAJYAlgCWAJYB1AAPsDWCAfQJYAlgCWAJYAlgHMMDeAAbYJ4AlgCWAJYAlgHUAA+wNYIB9AlgCWAJYAlgCWAcwwN4ABtinZgF+rEWp3s7OXoANaxLg4S87wF3d3V0AG9YkwHe/4AC3xePtABvWHMC9V+cc4NZctrXwbL16dV+qlClwKqh8JnCzf+M5i0kplbZZKmu1lM2krM1S6VpLxScHXvuC0o/gVzZseD9RyhQ4EVRuLHCzf6lxi0mJvM1Saaul1KjFpEzGYtKoqjFgaHLglkK8Btss1RxP0QVjpXo6OnoANqxpgCsCOHQASwBLAEsASwDrAAbYG8AA+wSwBLAEsASwBLAOYIC9AQywTwBLAEsASwBLAOsABtgbwAD7BLAEsASwBLAEsA5ggL0BDLBPAEsASwBLAEsA6wAG2BvAAPsEsASwBLAEsASwDmCAvQEMsE8ASwBLAEsAS/UEToyUMgUeCSo3FrjZv9S4xaSR/KjFpHTGZim5ViZl0haTEqrGgMHwwEMDpUyBB4LKxgM3+zeasZg0kB+qPcbTWMpmKTVoMSmZtJg0pGoM2M9TdGCH0lM0wKEDWAJYAlgCWAJYBzDA3gAG2CeAJYAlgCWAJYB1AAPsDWCAfQJYAlgCWAJYAlgHMMDeIgb8wCq1Ys4TAOsiBjx/R/fHX1oCsC5iwNMz1/xTegbAuogB/+Z/nr397VMA1kUMeNPs61XH/QDrIgZcrAdgXcSAv+P82r4AYF3EgH/vGfXOgocA1kUMeM9Z3z750Ul8AQ5fwwKrHYuenMwX4PA1JnCsHMC6KAHXCuDQASwBLE0FcCxIG+DQNSzw9DGlRo8AWBcx4ItuGBj46lKAdRED3vHpmTMv2eXeeXrl53+mVG9nZy/AhjUscEX3J15vU6qru7sLYMOaA1iprdcq1RaPtwNsWMMC72idNat1R9m35bItSrXmsq3O7YevuaYvU8oUOBNUfjxws3/ZnMWkjJrCpWwm5Q7IUiMVwBeuGRxYfWEZOP/6KnkEb9m06X3357aYAgf+EBh+yo40FT9l58i4UsNHuXc2pl/7LK/B0XqKvmDN4ODaC8rAl3dsVqqno6MHYMMaFrinZebMll7/z7gADl/DAgcGcOgAlgCW+PtgCWCJR7AOYIC9RQx4+/KZM5dvB1gXMeBznK+D15wDsC5iwMU/yToSYF3EgFevHRxcuxpgXZSA+TLJpygB1wrg0AEsASwBLAEsAawDGGBvUQKO1YAGOHSNCXzCy3mAq4sS8MYF0/g6uLooASueor1FDDgwgEPXsMCV3/gOcOSAK7/xHeDIAVd84zvA+yIHXPGN7wDvixww3/heXcSAAwM4dABLAEsASwBLAOsABthbxIAPA7iqiAGfsBfgyiIGfEfH4OTAo4lSpsCJoHJjgZv9S41bTErkbZZKWy0l18qkTMZi0qiqMWCoAjjw74MH95cyBd4fVHYocLN/I2mLSfvzg7XHeBpN2iyl+i0mjY1ZTBpQNY9lInBgPEWHrmGfogGuLmLAm8+cplY8ArAuYsBLno2pnfxvdSYUMeCjUjG1ZzbAuogBf2pD7N0r2gDWRQz4/SvmzG7rB1gXMeDAAA4dwBLAEj9lRwJY4qfs6ADmp+x4ixgwP2WnuigB81N2fIoScK0ADh3AEsDSVAA/9bFpPEVXFDHguY9meQRXFDHgWQmeoiuLGPBNX0sCXFHEgDdN58ukyiIGfMwmXoMrixjwXF6Dq4oYMK/B1UUMmD+qrC5iwIEBHDqAJYAlnqIlgCXzR/DYupsB1kUPWI18FGBd9ICz3zsRYF3EgJ0X4GmLngZYFzHgin7QduVLSvV2dvYCbFhzAN83srlNqa7u7i6ADWtQYM833fXfqFRbPN4OsGENClwo9+C8S8q38/cOKdWay7Y6tx9sb985XsoUeDyofDZws3+5vMWkcWW1VM5qKZtJB2apkUrgp844/xXBvvc1JY/gns2b9w2VMgUeCio3ErjZv7GMxaShfNxiUjJts5QatpiUSllMGlY1BgxMBP75+Wc8qV+DH3q+8JbXYIsa9Cn60nnfzE34JKvFKa16Ojp6ADasQYH5lw3eIgVcI4BDB7AEsASwBLAEsA5ggL0BDLBPAEsASwBLAEsA6wAG2BvAAPsEsASwBLAEsASwDmCAvQEMsE8ASwBLAEsASwDrAAbYG8AA+wSwBLAEsASwBLAOYIC9AQywTwBLAEsASwBLAOsABtgbwAD7BLAEsHTQgcfcTIHHgsqnAjf7l8laTBrLJ22WGrdZStksNW6zVFLVGDDMIziwQ+kRDHDoAJYAlgCWAJYA1gEMsDeAAfYJYAlgCWAJYAlgHcAAewMYYJ8AlgCWAJYAlgDWAQywN4AB9glgCWAJYAlgCWAdwAB7AxhgnwCWAJYAlgCWANYBDLA3gAH2CWAJYAlgCWAJYB3AAHsDGGCfAJYAlqYeuKXFedPb2dkLsGFNAqwKwF3d3V0AG9ZMwG3xeDvAhjUTcGsu2+q8u2f58t5sKVPgbFAqF7jZv1zeYlJW2UyawqXyB2SpRMhH8O633to3WMoUeDCo3EjgZv9GMxaTBvPDFpOSaZul1JDFpFTSYtKQqjGgn9fgwCL9FN1S+DS6p6OjB2DDmgS4KoBDB7AEsASwBLAEsA7gQxq4rmcFsASwBLAEsASwDmCAAS4EMMAAWxwfwBLAEsA6gAEGuBDAAANscXwASwBLAOsABhjgQgADDLDF8QEsASwBrAMYYIALAQwwwBbHB7AEsASwDmCAAS4EMMAAWxwfwBLAEsA6gGs05mZ6KcaCyqcCN/uXyVpMGssng7bW9axU4FKTND5uMSmpagwY5hFcqq5n1ZyPYIABBrgUwBLAEsASwBLAOoABBrgQwB8QuJ5LAawDGOB6nRXAEsAfbCmAdQADXK+zAlgC+IMtBbDuAACbTrKbFXTYAOsABtj0rEwnARwugCWAdQADbHpWppMADhfAEsA6gA8ocD3Pqgaw1VJTMgngcGcFMMDGS03JJIDDnRXAABsvNSWTAA53VgADbLzUlEwCuGpWoy9lOmkKlwK4HkuZTgK4alajL2U6CeCqWY2+lOmkRgTu7ezsBbhOkxoRuKu7uwvgOk1qROC2eLwd4DpNakTg1ly21Xl359KlPflSpkt9gFmNvpTppClcysXKj4Z8BA/29e0fCGxkPHi7f9m4xaTRjM1S+SGLSWMpm6XUoMWkZNJi0pCqMWC/+WvwJA1ngrf71zB/4T9JiaTNUs3074N7Ojp6ADasmYB1AIcOYAlgCWAJYAlgHcAAewNYAlgHsASwBLAEsASwDmCAvQEsAawDWAJYAlgCWAJYBzDA3gCW9vRZHN++vj0Wk/a8Z7PUDpuldtsttddmqd02S+2sNSA8cI1evMpm1qX/YzHpsettllq602LSg7dYTMr9zojFrDv/1WJS/++HGgbwZAEs/fJbNrPu6bOYtGWjzVK391tMeulpi0m5W1IWs374osWkxN+HGlYPYGrgAI54dQDW/4bJqJYW8zlPr/z8zyyWUo+Zr5W87XLjScVTMr0c7nUIf4jPt7c9r9QP2q58qXw7qDoA6++fNssC+P7E620WKw1/2Xytf18XN1+osIzx5Sgem8EhXrFt2xVK3Teyua18O6g6AOt/w2SWBbBSW6+1mHT3C+ZrrdpqsVBhGePLUTw2g0O8dtu2LxTe99+ob09eHYDdf8NknA1wy2VbzCf1Xp0zX+uSWz/zrPGkwjLGl6P4vG5wiFsuvfQXzrv8vUNyO6AmewTnX19lPmntCxZrtW/bttJ4kvUj2OQQr9q2rcN5d+9r+nZAzfUavDH92mctFipkOunmbdvMl7J+DTY5xJXF33oPPa9vB1UHYP1vmIyyuOhq4+Udmy2WsvnNtPPqtp8Yr1I4JdPLUb4OoQ/xuZUrnyv9lki7t4Pi6+CIB3DEAzjiARzxAI54AEe8qAPHLlPRP8mgon7usXk/j/5JBhX1c4998/zSScYOW/Cc87Zr5rGbbpp1/DNK7W35yJLNSm06ffr8B5wNq488Z7sM27VsekxGNHeRB86d8VTpJHMvnurcXZd4dPq6kUcXK7ViQ+qZM5Wa/ZPMjlXOhrXDa5bLsNavFn6KmDuiuYs8sHpySc45ye4lR8Q+5NxNqmzhl3NzTixW+MhXzr6m8Ke6sUE1OFOGzRwozHVHNHfRB1bnP+i8OfE/kvGY+2Rd+jX73dKI7Xf9QZsAu8NKwOURTd0hAPzKPOfNnBfj11UCt7Xt3blCqc63My/NLjxFxwtP0e6wS24YUzKiuTsEgNWlzpsNvz73pkrg/pWzFjyi1P0LZ5z+mHyS5Q7btfRwPaK5izpw2CJ7HSJ7YoZF9jpE9sSoFMARD+CIB3DEAzjiARzx/h/F8K7fi0GbVgAAAABJRU5ErkJggg==\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -i it_ns_counts\n",
    "\n",
    "g = ggplot(it_ns_counts, aes(x = factor(namespace), y = count)) +\n",
    "ggtitle(\"Number of blocks by namespace - Italian Wikipedia\") +\n",
    "scale_x_discrete(\"Namespace\") +\n",
    "scale_y_continuous(\"Number of blocks\") +\n",
    "geom_col()\n",
    "\n",
    "ggsave(\"graphs/itwiki_blocks_per_namespace.png\",\n",
    "       plot = g, width = 30, height = 20, units = \"cm\", dpi = \"screen\");\n",
    "g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}