1_2_explore_flight_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Mapping quarterly flight path data\n",
    "Here we make use public data from the Airline Origin and Destination Survey (DB1B) from Bureau of Transportation Statistics (transtats.bts.gov). See https://github.com/mapequation/airline-data for scripts to download and generate this data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Path data for this tutorial is available on `data/air2015_{q}_paths.net` for each q in [1,2,3,4] and looks like this:\n",
    "\n",
    "```\n",
    "*vertices 345\n",
    "11618 \"Newark, NJ: Newark Liberty International\"\n",
    "11057 \"Charlotte, NC: Charlotte Douglas International\"\n",
    "11617 \"New Bern/Morehead/Beaufort, NC: Coastal Carolina Regional\"\n",
    "...\n",
    "*paths\n",
    "11618 11618 11057 11617 11057 22\n",
    "11618 11618 11057 10994 11057 47\n",
    "11618 11618 11057 12323 11057 52\n",
    "11618 11618 11057 13495 11057 82\n",
    "...\n",
    "```\n",
    "The last column is assumed to be the weight of the path, unless `--unweighted-paths` is provided to Infomap."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Generate state networks of first and second order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import infomap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def generateStateNetworkFromPaths(inputFilename, outputFilename, markovOrder):\n",
    "    network = infomap.Network(infomap.Config(\"--directed --path-markov-order {}\".format(markovOrder)))\n",
    "    network.readInputData(inputFilename) \n",
    "    network.writeStateNetwork(outputFilename)\n",
    "    print(\"State network of order {} written to {}\".format(markovOrder, outputFilename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for quarter in [1,2,3,4]:\n",
    "    inputFilename = \"data/air2015_{}_paths.net\".format(quarter)\n",
    "    for order in [1,2]:\n",
    "        outputFilename = \"output/air2015_{}_order_{}.net\".format(quarter, order)\n",
    "        generateStateNetworkFromPaths(inputFilename, outputFilename, order)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Mapping change in first-order networks with Alluvial Diagram\n",
    "\n",
    "![alluvial-diagram](http://www.mapequation.org/assets/img/neuroAlluvial2001-2007.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def createMap(inputFilename, flags = \"--directed\"):\n",
    "    print(\"Cluster '{}'...\".format(inputFilename))\n",
    "    name = inputFilename.rsplit(\".\", maxsplit=1)[0].split('/')[-1]\n",
    "    infomap1 = infomap.Infomap(flags)\n",
    "    infomap1.network().readInputData(inputFilename)\n",
    "    infomap1.run()\n",
    "    print(\" -> Found {} top modules with codelength {}\".format(infomap1.numTopModules(), infomap1.codelength()))\n",
    "    mapFilename = \"output/{}.map\".format(name)\n",
    "    infomap1.writeMap(mapFilename)\n",
    "    print(\" -> Wrote .map file to '{}'\".format(mapFilename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for quarter in [1,2,3,4]:\n",
    "    inputFilename = \"output/air2015_{}_order_1.net\".format(quarter)\n",
    "    createMap(inputFilename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### No structure in first-order networks?\n",
    "In first-order, the networks are so densely connected that they contain no or only little modular structure. That can be solved by using paths in second-order networks, where physical nodes can have overlapping modules. But we may uncover some modular structure even in the first-order networks by scaling down the Markov time to increase the code rate. In this way, the flow persistence in local structures increases and Infomap identifies more and smaller modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Reveal first-order structure in tightly connected networks using shorter Markov time\n",
    "Re-run with `--markov-time 0.75` to descover the modular structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for quarter in [1,2,3,4]:\n",
    "    inputFilename = \"output/air2015_{}_order_1.net\".format(quarter)\n",
    "    createMap(inputFilename, flags=\"--directed --markov-time 0.75\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Mapping second-order networks\n",
    "The current Alluvial Generator doesn't support overlapping or multi-level modular structure. However, we can explore such networks individually using the [Network Navigator](http://navigator.mapequation.org). It uses the `.ftree` format as input.\n",
    "\n",
    "![Network Navigator](http://www.mapequation.org/assets/img/InfomapNetworkNavigator.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def createFlowTree(inputFilename, flags = \"--directed\"):\n",
    "    print(\"Cluster '{}'...\".format(inputFilename))\n",
    "    name = inputFilename.rsplit(\".\", maxsplit=1)[0].split('/')[-1]\n",
    "    infomap2 = infomap.Infomap(flags)\n",
    "    infomap2.network().readInputData(inputFilename)\n",
    "    infomap2.run()\n",
    "    print(\" -> Found {} top modules with codelength {}\".format(infomap2.numTopModules(), infomap2.codelength()))\n",
    "    ftreeFilename = \"output/{}.ftree\".format(name)\n",
    "    infomap2.writeFlowTree(ftreeFilename)\n",
    "    print(\" -> Wrote .ftree file to '{}'\".format(ftreeFilename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for quarter in [1,2,3,4]:\n",
    "    inputFilename = \"output/air2015_{}_order_2.net\".format(quarter)\n",
    "    createFlowTree(inputFilename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}