MNIST_Dataset_and_Databases.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Dataset & Database\n",
    "\n",
    "In the [MNIST tutorial](https://github.com/caffe2/caffe2/blob/master/caffe2/python/tutorials/MNIST.ipynb) we use an lmdb database. You can also use leveldb or even minidb by changing the type reference when you get ready to read from the dbs. In this tutorial, we will go over how to download, extract, and generate lmdb and leveldb variants of the MNIST dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset:\n",
    "\n",
    "You can download the raw [MNIST dataset](https://download.caffe2.ai/datasets/mnist/mnist.zip), g/unzip the dataset and labels, and make the database yourself. \n",
    "\n",
    "\n",
    "## Databases:\n",
    "\n",
    "We provide a few database formats for you to try with the MNIST tutorial. The default is lmdb. \n",
    "\n",
    "* [MNIST-nchw-lmdb](https://download.caffe2.ai/databases/mnist-lmdb.zip) - contains both the train and test lmdb MNIST databases in NCHW format\n",
    "* [MNIST-nchw-leveldb](https://download.caffe2.ai/databases/mnist-leveldb.zip) - contains both the train and test leveldb MNIST databases in NCHW format\n",
    "* [MNIST-nchw-minidb](https://download.caffe2.ai/databases/mnist-minidb.zip) - contains both the train and test minidb MNIST databases in NCHW format\n",
    "\n",
    "\n",
    "## Tools:\n",
    "\n",
    "### make_mnist_db\n",
    "\n",
    "If you like LevelDB you can use Caffe2's `make_mnist_db` binary to generate leveldb databases. This binary is found in `/caffe2/build/caffe2/binaries/` or depending on your OS and installation, in `/usr/local/bin/`.\n",
    "\n",
    "Here is an example call to `make_mnist_db`:\n",
    "\n",
    "```\n",
    "./make_mnist_db --channel_first --db leveldb --image_file ~/Downloads/train-images-idx3-ubyte --label_file ~/Downloads/train-labels-idx1-ubyte --output_file ~/caffe2/caffe2/python/tutorials/tutorial_data/mnist/mnist-train-nchw-leveldb\n",
    "\n",
    "./make_mnist_db --channel_first --db leveldb --image_file ~/Downloads/t10k-images-idx3-ubyte --label_file ~/Downloads/t10k-labels-idx1-ubyte --output_file ~/caffe2/caffe2/python/tutorials/tutorial_data/mnist/mnist-test-nchw-leveldb\n",
    "```\n",
    "Note leveldb can get deadlocked if more than one user attempts to open the leveldb at the same time. This is why there is logic in the Python below to delete LOCK files if they're found.\n",
    "\n",
    "\n",
    "### Python script\n",
    "\n",
    "You can use the Python in the code blocks below to download and extract the dataset with `DownloadResource`, call the `make_mnist_db` binary, and generate your database with `GenerateDB`. \n",
    "\n",
    "First, we will define our functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "from __future__ import unicode_literals\n",
    "\n",
    "import os\n",
    "\n",
    "def DownloadResource(url, path):\n",
    "    '''Downloads resources from s3 by url and unzips them to the provided path'''\n",
    "    import requests, zipfile, StringIO\n",
    "    print(\"Downloading... {} to {}\".format(url, path))\n",
    "    r = requests.get(url, stream=True)\n",
    "    z = zipfile.ZipFile(StringIO.StringIO(r.content))\n",
    "    z.extractall(path)\n",
    "    print(\"Completed download and extraction.\")\n",
    "\n",
    "    \n",
    "def GenerateDB(image, label, name):\n",
    "    '''Calls the make_mnist_db binary to generate a leveldb from a mnist dataset'''\n",
    "    name = os.path.join(data_folder, name)\n",
    "    print('DB: ', name)\n",
    "    if not os.path.exists(name):\n",
    "        syscall = \"/usr/local/bin/make_mnist_db --channel_first --db leveldb --image_file \" + image + \" --label_file \" + label + \" --output_file \" + name\n",
    "        # print \"Creating database with: \", syscall\n",
    "        os.system(syscall)\n",
    "    else:\n",
    "        print(\"Database exists already. Delete the folder if you have issues/corrupted DB, then rerun this.\")\n",
    "        if os.path.exists(os.path.join(name, \"LOCK\")):\n",
    "            # print \"Deleting the pre-existing lock file\"\n",
    "            os.remove(os.path.join(name, \"LOCK\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our functions for loading, extracting, and generating our dbs, we will put these functions to use and generate the MNIST data in both lmdb and leveldb formats (if they do not already exist).\n",
    "\n",
    "First, we **download and extract the MNIST dataset (train and test) in lmdb format** using:\n",
    "\n",
    "```python\n",
    "DownloadResource(\"http://download.caffe2.ai/databases/mnist-lmdb.zip\", data_folder)\n",
    "```\n",
    "\n",
    "\n",
    "Next, we focus on **downloading, extracting, and generating MNIST train and test leveldbs**. We start by downloading and extracting the raw MNIST dataset (in ubyte format). This will ultimately extract four files, consisting of training images and labels, and testing images and labels.\n",
    "\n",
    "```python\n",
    "DownloadResource(\"http://download.caffe2.ai/datasets/mnist/mnist.zip\", data_folder)\n",
    "```\n",
    "\n",
    "\n",
    "Finally, we **generate the leveldb train and test databases** (or regenerate; it can get locked with multi-user setups or abandoned threads). We do this by passing our `GenerateDB` function the names of the corresponding ubyte files along with an output file name.\n",
    "\n",
    "```python\n",
    "GenerateDB(image_file_train, label_file_train, \"mnist-train-nchw-leveldb\")\n",
    "GenerateDB(image_file_test, label_file_test, \"mnist-test-nchw-leveldb\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_folder = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks')\n",
    "data_folder = os.path.join(current_folder, 'tutorial_data', 'mnist')\n",
    "\n",
    "# If the data_folder does not already exist, create it\n",
    "if not os.path.exists(data_folder):\n",
    "    os.makedirs(data_folder)   \n",
    "\n",
    "# Downloads and extracts the lmdb databases of MNIST images - both test and train\n",
    "if not os.path.exists(os.path.join(data_folder,\"mnist-train-nchw-lmdb\")):\n",
    "    DownloadResource(\"http://download.caffe2.ai/databases/mnist-lmdb.zip\", data_folder)\n",
    "else:\n",
    "    print(\"mnist-lmdb already downloaded and extracted\")\n",
    "\n",
    "# Downloads and extracts the MNIST data set\n",
    "if not os.path.exists(os.path.join(data_folder, \"train-images-idx3-ubyte\")):\n",
    "    DownloadResource(\"http://download.caffe2.ai/datasets/mnist/mnist.zip\", data_folder)\n",
    "else:\n",
    "    print(\"Raw mnist ubyte data already downloaded and extracted\")\n",
    "\n",
    "# (Re)generate the leveldb database (it can get locked with multi-user setups or abandoned threads)\n",
    "# Requires the download of the dataset (mnist.zip) - see DownloadResource above.\n",
    "# You also need to change references in the MNIST tutorial code where you train or test from lmdb to leveldb\n",
    "image_file_train = os.path.join(data_folder, \"train-images-idx3-ubyte\")\n",
    "label_file_train = os.path.join(data_folder, \"train-labels-idx1-ubyte\")\n",
    "image_file_test = os.path.join(data_folder, \"t10k-images-idx3-ubyte\")\n",
    "label_file_test = os.path.join(data_folder, \"t10k-labels-idx1-ubyte\")\n",
    "GenerateDB(image_file_train, label_file_train, \"mnist-train-nchw-leveldb\")\n",
    "GenerateDB(image_file_test, label_file_test, \"mnist-test-nchw-leveldb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Changes for Other DBs\n",
    "\n",
    "If you chose to use a format other than lmdb you will need to change a couple lines of code. When you use `ModelHelper` to instantiate the CNN, you pass in the `db` parameter with a path and the `db_type` with the type of db. You would need to update both of these values. Since you create two networks, one for training and one for testing, you would need to update the code for both of these.\n",
    "\n",
    "**Default code using lmdb**\n",
    "```python\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "```\n",
    "\n",
    "**Updated code using leveldb**\n",
    "```python\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-leveldb'),\n",
    "    db_type='leveldb')\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}