examples/mapper_quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with Mapper\n",
    "\n",
    "In this notebook we explore a few of the core features included in giotto-tda's implementation of the [Mapper algorithm](https://research.math.osu.edu/tgda/mapperPBG.pdf). \n",
    "\n",
    "## Useful references\n",
    "\n",
    "* [An introduction to Topological Data Analysis: fundamental and practical aspects for data scientists](https://arxiv.org/abs/1710.04019)\n",
    "* [An Introduction to Topological Data Analysis for Physicists: From LGM to FRBs](https://arxiv.org/abs/1904.11044)\n",
    "\n",
    "#### License: AGPLv3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data wrangling\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# data viz\n",
    "import plotly.graph_objects as go\n",
    "\n",
    "# tda magic\n",
    "from gtda.mapper import (\n",
    "    CubicalCover,\n",
    "    make_mapper_pipeline,\n",
    "    Projection,\n",
    "    plot_static_mapper_graph,\n",
    "    plot_interactive_mapper_graph\n",
    ")\n",
    "from gtda.mapper.utils.visualization import set_node_sizeref\n",
    "\n",
    "# from gtda.mapper.utils.visualization import set_node_sizeref\n",
    "\n",
    "# ml tools\n",
    "from sklearn import datasets\n",
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "import warnings\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate and visualise data\n",
    "As a simple example, let's generate a two-dimensional point cloud of two concentric circles. The goal will be to examine how Mapper can be used to generate a topological graph that captures the salient features of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, _ = datasets.make_circles(n_samples=5000, noise=0.05, factor=0.3, random_state=42)\n",
    "\n",
    "fig = go.Figure(\n",
    "    data=go.Scatter(x=data[:, 0], y=data[:, 1], mode=\"markers\"),\n",
    "    layout={\"autosize\": False},\n",
    ")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the Mapper pipeline\n",
    "Given a dataset ${\\cal D}$ of points $x \\in \\mathbb{R}^n$, the basic steps behind Mapper are as follows:\n",
    "\n",
    "1. Map ${\\cal D}$ to a lower-dimensional space using a _**filter function**_ $ f: \\mathbb{R}^n \\to \\mathbb{R}^m $. Common choices for the filter function include projection onto one or more axes via PCA or density-based methods. In giotto-tda, you can import a variety of filter functions as follows:\n",
    "\n",
    "```python\n",
    "from gtda.mapper.filter import FilterFunctionName\n",
    "```\n",
    "\n",
    "2. Construct a cover of the filter values ${\\cal U} = (U_i)_{i\\in I}$, typically in the form of a set of overlapping intervals which have constant length. As with the filter, a choice of cover can be imported as follows:\n",
    "\n",
    "```python\n",
    "from gtda.mapper.cover import CoverName\n",
    "```\n",
    "\n",
    "3. For each interval $U_i \\in {\\cal U}$ cluster the points in the preimage $f^{-1}(U_i)$ into sets $C_{i,1}, \\ldots , C_{i,k_i}$. The choice of clustering algorithm can be any of scikit-learn's [clustering methods](https://scikit-learn.org/stable/modules/clustering.html) or an implementation of agglomerative clustering in giotto-tda:\n",
    "\n",
    "```python\n",
    "# scikit-learn method\n",
    "from sklearn.cluster import ClusteringAlgorithm\n",
    "# giotto-tda method\n",
    "from gtda.mapper.cluster import FirstSimpleGap\n",
    "```\n",
    "\n",
    "4. Construct the topological graph whose vertices are the cluster sets $(C_{i,j})_{i\\in I, j \\in \\{1,\\ldots,k_i\\}}$ and an edge exists between two nodes if they share points in common: $C_{i,j} \\cap C_{k,l} \\neq \\emptyset$. This step is handled automatically by giotto-tda.\n",
    "\n",
    "These four steps are implemented in the `MapperPipeline`  object that mimics the `Pipeline` class from scikit-learn. We provide a convenience function `make_mapper_pipeline()` that allows you to pass the choice of filter function, cover, and clustering algorithm as arguments. For example, to project our data onto the $x$- and $y$-axes, we could setup the pipeline as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define filter function - can be any scikit-learn transformer\n",
    "filter_func = Projection(columns=[0, 1])\n",
    "# define cover\n",
    "cover = CubicalCover(n_intervals=10, overlap_frac=0.3)\n",
    "# choose clustering algorithm - default is DBSCAN\n",
    "clusterer = DBSCAN()\n",
    "\n",
    "# configure parallelism of clustering step\n",
    "n_jobs = 1\n",
    "\n",
    "# initialise pipeline\n",
    "pipe = make_mapper_pipeline(\n",
    "    filter_func=filter_func,\n",
    "    cover=cover,\n",
    "    clusterer=clusterer,\n",
    "    verbose=False,\n",
    "    n_jobs=n_jobs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualise the Mapper graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the Mapper pipeline at hand, it is now a simple matter to visualise it. To warm up, let's examine the graph in two-dimensions using the default arguments of giotto-tda's plotting function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_static_mapper_graph(pipe, data)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the figure we can see that we have captured the salient topological features of our underlying data, namely two holes!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the coloring of the Mapper graph\n",
    "By default, the nodes of the Mapper graph are colored by the mean value of the points that belong to a given node. However, in this example it is more instructive to colour by the $x$- and $y$-axes. This can be achieved by toggling the `color_by_columns_dropdown`, which calculates the coloring for each column in the input data array. At the same time, let's configure the choice of colorscale:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plotly_kwargs = {\n",
    "    'node_trace_marker_colorscale':'Blues'\n",
    "}\n",
    "fig = plot_static_mapper_graph(pipe, data, color_by_columns_dropdown=True, plotly_kwargs=plotly_kwargs)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the dropdown menu, the entry `color_variable` refers to a user-defined quantity to color by - by default it is the average value of the points in each node. In general, one can configure this quantity to be an array, a scikit-learn transformer, or a list of indices to select from the data. For example, coloring by a PCA component can be implemented as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise estimator to color graph by\n",
    "pca = PCA(n_components=1).fit(data)\n",
    "\n",
    "fig = plot_static_mapper_graph(pipe, data, color_by_columns_dropdown=True, color_variable=pca)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pass a pandas DataFrame as input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to feed `plot_static_mapper_graph()` a pandas DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_df = pd.DataFrame(data, columns=['x', 'y']); data_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before plotting we need to update the Mapper pipeline to know about the projection onto the column names. This can be achieved using the `set_params()` method as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.set_params(filter_func=Projection(columns=['x', 'y']));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_static_mapper_graph(pipe, data_df, color_by_columns_dropdown=True)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change the layout algorithm\n",
    "By default, `plot_static_mapper_graph()` uses the Kamada–Kawai algorithm for the layout; however any of the layout algorithms defined in python-igraph are supported (see [here](https://igraph.org/python/doc/igraph.Graph-class.html) for a list of possible layouts). For example, we can switch to the Fruchterman–Reingold layout as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# rest back to numpy projection\n",
    "pipe.set_params(filter_func=Projection(columns=[0,1]));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_static_mapper_graph(pipe, data, layout='fruchterman_reingold', color_by_columns_dropdown=True)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change the layout dimension\n",
    "It is also possible to visualise the Mapper graph in 3-dimensions by configuring the `layout_dim` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_static_mapper_graph(pipe, data, layout_dim=3, color_by_columns_dropdown=True)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the Mapper pipeline\n",
    "Behind the scenes of `plot_static_mapper_graph()` is a `MapperPipeline` object `pipe` that can be used like a typical scikit-learn estimator. For example, to extract the underlying graph data structure we can do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph = pipe.fit_transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting graph is an [python-igraph](https://igraph.org/python/) object that contains metadata that is stored in the form of dictionaries. We can access this data as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph['node_metadata'].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here `node_id` is a globally unique node identifier used to construct the graph, while `pullback_set_label` and `partial_cluster_label` refer to the interval and cluster sets described above. The `node_elements` refers to the indices of our original data that belong to each node. For example, to find which points belong to the first node of the graph we can access the desired data as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "node_id, node_elements = graph['node_metadata']['node_id'], graph['node_metadata']['node_elements']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Node Id: {}, \\nNode elements: {}, \\nData points: {}'.format(node_id[0], node_elements[0], data[node_elements[0]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `node_elements` are handy for situations when we want to customise e.g. the size of the node scale. In this example, we use the utility function `set_node_sizeref()` and pass the function as a plotly argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# configure scale for node sizes\n",
    "plotly_kwargs = {\n",
    "    \"node_trace_marker_sizeref\": set_node_sizeref(node_elements, node_scale=30)\n",
    "}\n",
    "fig = plot_static_mapper_graph(pipe, data, layout_dim=3, color_by_columns_dropdown=True, plotly_kwargs=plotly_kwargs)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting graph is much easier to decipher with the enlarged node scaling!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating custom filter functions\n",
    "In some cases, the list of filter functions provided in `filter.py` or scikit-learn may not be sufficient for the task at hand. In such cases, one can pass any callable to the pipeline that acts _row-wise_ on the input data. For example, we can project by taking the sum of the $(x,y)$ coordinates as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_func = np.sum\n",
    "\n",
    "pipe = make_mapper_pipeline(\n",
    "    filter_func=filter_func,\n",
    "    cover=cover,\n",
    "    clusterer=clusterer,\n",
    "    verbose=True,\n",
    "    n_jobs=n_jobs,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_static_mapper_graph(pipe, data, plotly_kwargs=None)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, any callable (i.e. function) that operates **_row-wise_** can be passed. For example we can filter by the ratio of $x$- and $y$-coordinates as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_xy_ratio(row):\n",
    "    return row[0] / row[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_mapper_pipeline(\n",
    "    filter_func=calculate_xy_ratio,\n",
    "    cover=cover,\n",
    "    clusterer=clusterer,\n",
    "    verbose=True,\n",
    "    n_jobs=n_jobs,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plot_static_mapper_graph(pipe, data, plotly_kwargs=None)\n",
    "# display figure\n",
    "fig.show(config={\"scrollZoom\": True})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualise the 2D Mapper graph interactively\n",
    "In general, buidling useful Mapper graphs requires some iteration through the various parameters in the cover and clustering algorithm. To simplify that process, giotto-tda provides an interactive figure that can be configured in real-time. If invalid parameters are selected, the _Show logs_ checkbox can be used to see what went wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_mapper_pipeline()\n",
    "\n",
    "# generate interactive plot\n",
    "plot_interactive_mapper_graph(pipe, data, color_by_columns_dropdown=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}