pickle-tut01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn Python Series (#28) - Using Pickle and Shelve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repository\n",
    "https://github.com/python/cpython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What will I learn?\n",
    "\n",
    "- In this episode of the `Learn Python Series` you will learn about two additional ways to serialize and de-serialize Python objects for persistent storage: `pickle` and `shelve`,\n",
    "- you will learn when to (not) use `pickle` over JSON, and when to (not) use `shelve` over a \"real\" database environment,\n",
    "- also we'll discuss some dangers to be aware of when using `pickle`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements\n",
    "\n",
    "- A working modern computer running macOS, Windows or Ubuntu;\n",
    "- An installed Python 3(.6) distribution, such as (for example) the Anaconda Distribution;\n",
    "- The ambition to learn Python programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Difficulty\n",
    "- Beginner / Intermediate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Curriculum (of the `Learn Python Series`):\n",
    "\n",
    "- [Learn Python Series - Intro](https://steemit.com/utopian-io/@scipio/learn-python-series-intro)\n",
    "- [Learn Python Series (#2) - Handling Strings Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-2-handling-strings-part-1)\n",
    "- [Learn Python Series (#3) - Handling Strings Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-3-handling-strings-part-2)\n",
    "- [Learn Python Series (#4) - Round-Up #1](https://steemit.com/utopian-io/@scipio/learn-python-series-4-round-up-1)\n",
    "- [Learn Python Series (#5) - Handling Lists Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-5-handling-lists-part-1)\n",
    "- [Learn Python Series (#6) - Handling Lists Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-6-handling-lists-part-2)\n",
    "- [Learn Python Series (#7) - Handling Dictionaries](https://steemit.com/utopian-io/@scipio/learn-python-series-7-handling-dictionaries)\n",
    "- [Learn Python Series (#8) - Handling Tuples](https://steemit.com/utopian-io/@scipio/learn-python-series-8-handling-tuples)\n",
    "- [Learn Python Series (#9) - Using Import](https://steemit.com/utopian-io/@scipio/learn-python-series-9-using-import)\n",
    "- [Learn Python Series (#10) - Matplotlib Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-10-matplotlib-part-1)\n",
    "- [Learn Python Series (#11) - NumPy Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-11-numpy-part-1)\n",
    "- [Learn Python Series (#12) - Handling Files](https://steemit.com/utopian-io/@scipio/learn-python-series-12-handling-files)\n",
    "- [Learn Python Series (#13) - Mini Project - Developing a Web Crawler Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-13-mini-project-developing-a-web-crawler-part-1)\n",
    "- [Learn Python Series (#14) - Mini Project - Developing a Web Crawler Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-14-mini-project-developing-a-web-crawler-part-2)\n",
    "- [Learn Python Series (#15) - Handling JSON](https://steemit.com/utopian-io/@scipio/learn-python-series-15-handling-json)\n",
    "- [Learn Python Series (#16) - Mini Project - Developing a Web Crawler Part 3](https://steemit.com/utopian-io/@scipio/learn-python-series-16-mini-project-developing-a-web-crawler-part-3)\n",
    "- [Learn Python Series (#17) - Roundup #2 - Combining and analyzing any-to-any multi-currency historical data](https://steemit.com/utopian-io/@scipio/learn-python-series-17-roundup-2-combining-and-analyzing-any-to-any-multi-currency-historical-data)\n",
    "- [Learn Python Series (#18) - PyMongo Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-18-pymongo-part-1)\n",
    "- [Learn Python Series (#19) - PyMongo Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-19-pymongo-part-2)\n",
    "- [Learn Python Series (#20) - PyMongo Part 3](https://steemit.com/utopian-io/@scipio/learn-python-series-20-pymongo-part-3)\n",
    "- [Learn Python Series (#21) - Handling Dates and Time Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-21-handling-dates-and-time-part-1)\n",
    "- [Learn Python Series (#22) - Handling Dates and Time Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-22-handling-dates-and-time-part-2)\n",
    "- [Learn Python Series (#23) - Handling Regular Expressions Part 1](https://steemit.com/utopian-io/@scipio/learn-python-series-23-handling-regular-expressions-part-1)\n",
    "- [Learn Python Series (#24) - Handling Regular Expressions Part 2](https://steemit.com/utopian-io/@scipio/learn-python-series-24-handling-regular-expressions-part-2)\n",
    "- [Learn Python Series (#25) - Handling Regular Expressions Part 3](https://steemit.com/utopian-io/@scipio/learn-python-series-25-handling-regular-expressions-part-3)\n",
    "- [Learn Python Series (#26) - pipenv & Visual Studio Code](https://steemit.com/utopian-io/@scipio/learn-python-series-26-pipenv-and-visual-studio-code)\n",
    "- [Learn Python Series (#27) - Handling Strings Part 3 (F-Strings)](https://steemit.com/utopian-io/@scipio/learn-python-series-27-handling-strings-part-3-f-strings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional sample code files\n",
    "\n",
    "The full - and working! - iPython tutorial sample code file is included for you to download and run for yourself right here:\n",
    "https://github.com/realScipio/learn-python-series/blob/master/strings-tut03.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GitHub Account\n",
    "https://github.com/realScipio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn Python Series (#28) - Using Pickle and Shelve\n",
    "Welcome to episode #28 of the `Learn Python Series`! In [episode #15](https://steemit.com/utopian-io/@scipio/learn-python-series-15-handling-json) we focused our attention on the JSON file format and reading from and writing to `.json` files. JSON is language- and platform-independent and human-readable as well. It can be used to serialize / deserialize JSON data to and from Python objects, and because `.json` files can be stored on disk they can also be shared among processes and computer systems. However, great as JSON might be, for serializing / de-serializing Python objects, it does have its limitations, for not all Python object formats can be \"JSON-i-fied\": JSON for example doesn't properly differentiate between lists and tuples, object keys are required to be strings, and datetime objects could be customized to work with JSON but not \"out-of-the-box\" (requires custom (de)serialization). Also, there are situations where \"human-readable\" could be considered a security risk, ergo not in every situation using JSON is preferred.\n",
    "\n",
    "### Pickle vs JSON\n",
    "`Pickle`, which as a module is part of most Python distributions (including Anaconda), can also be used for serializing and de-serializing Python objects. \"Pickling\" out-of-the-box converts **almost any** Python object (apart from a few edge-case scenarios, like generators and lambda functions, which we haven't discussed yet in earlier episodes) into a character stream that can be saved to disk, where the character stream contains all the information that's needed to rebuild the object by the same or another Python program.\n",
    "\n",
    "You could pickle the following object types:\n",
    "\n",
    "- normal and unicode strings\n",
    "- integers, floats, complex numbers\n",
    "- lists, dictionaries, tuples, sets\n",
    "- None, True and False\n",
    "- (built-in) functions and classes defined at a module's top level\n",
    "\n",
    "**Nota bene:**\n",
    "As opposed to JSON however, Pickle is **not** platform independent (it can even vary per Python version), it's rather slow, uses a binary format (ergo not human-readable), and could be a security risk for executing arbitrary code, contained in the pickle, while de-serializating. So while this last sentence might not sound like a great sales-pitch to make a case for using Pickle, if you don't have language interoperability requirements for exchanging serialized objects, if you don't have to deal with untrusted data sources and if a binary format is OK or even preferred, then Pickle works great!\n",
    "\n",
    "Let's see how Pickle works!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with `pickle`\n",
    "In order to work with pickle, first import it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serializing (pickling, dumping)\n",
    "Now, say, we want to pickle a list of our favorite cryptos, like these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "fav_cryptos = [\n",
    "    \"Steem\",\n",
    "    \"Steem Backed Dollars\",\n",
    "    \"Bitcoin\",\n",
    "    \"IoTeX\",\n",
    "    \"Litecoin\",\n",
    "    \"Stellar\",\n",
    "    \"Byteball\",\n",
    "    \"Tether\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like json, pickle also has two main methods:\n",
    "\n",
    "- `dump`, to serialize and \"dump\" a Python object to file, and\n",
    "- `load`, to de-serialize a pickled file object.\n",
    "\n",
    "For writing the `fav_cryptos` list to file via pickle, we need to first specify the filename:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"fav_cryptos.p\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define the file object, we open the file for writing via the `open()` function, to which we pass in two arguments: the `filename`, and `wb` for writing in binary mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileobject = open(filename, 'wb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the file is opened for writing to, use `pickle.dump()` and pass in the object you want to pickle (in this case our `fav_cryptos` list) as its first argument, and the `fileobject` as its second argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(fav_cryptos, fileobject)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then close the file object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileobject.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, the object `fav_cryptos` is saved on disk as the pickled file `fav_cryptos.p`!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could also use the following shorthand notation via `with`, that will automatically close the file for you (for this example I'll use another list and pickle dumpfile):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "colors = [\n",
    "    'Green', \n",
    "    'Yellow', \n",
    "    'Orange', \n",
    "    'Red',\n",
    "    'Blue',\n",
    "    'Brown',\n",
    "    'White',\n",
    "    'Black'\n",
    "]\n",
    "with open('colors.p', 'wb') as f:\n",
    "    pickle.dump(colors, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### De-serializing (unpickling, loading)\n",
    "Unpickling a pickled file is quite similar: `open()` the file again but now use the `rb` flag (for **r**eading in **b**inary mode), and use `pickle.load()` to assign it to a new variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> ['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether']\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "fileobject = open('fav_cryptos.p', 'rb')\n",
    "unpickled_cryptos = pickle.load(fileobject)\n",
    "fileobject.close()\n",
    "\n",
    "print(type(unpickled_cryptos), unpickled_cryptos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, again use the shorthand notation using `with`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> ['Green', 'Yellow', 'Orange', 'Red', 'Blue', 'Brown', 'White', 'Black']\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "with open('colors.p', 'rb') as f:\n",
    "    unpickled_colors = pickle.load(f)\n",
    "    \n",
    "print(type(unpickled_colors), unpickled_colors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, while printing the unpickled object, I also checked to see the unpickled file types, and they're both correctly typed as a `list`. You could go a step further and compare the original to the unpickled objects to see if they are the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(fav_cryptos == unpickled_cryptos)\n",
    "print(colors == unpickled_colors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Again, a word of caution using `pickle` with untrusted data sources\n",
    "As explained, because functions and classes could also be pickled and **executed** while unpickling, as a rule of thumb, simply never use pickle with unknown systems. Yet if you must for some reason, make sure to use an encrypted network connection, and/or cryptographically sign and verify the pickle, and/or restrict file system permissions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with `shelve`\n",
    "`shelve` is built on top of `pickle`, and it acts somewhat like a database. In fact, you can use `shelve` as a persistent Python object store when you don't want to or can't use a \"real\" database. Shelved objects are also pickled, but via `shelve` the objects are associated with a string key. This means you can access your pickled objects via their key, just like you would with a Python dictionary! `shelve` is pretty convenient when serializing many objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to work with `shelve` first import it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shelve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serializing (shelving, dumping)\n",
    "The `shelve` syntax is pretty similar to `pickle`. Let's shelve the objects that we unpickled before!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelf['cryptos'] = unpickled_cryptos\n",
    "    shelf['colors'] = unpickled_colors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, a shelved database file is stored (`test_shelf.db` on macOS, but on other systems, depending on the specific DBM implementation that is used, you might get output files with no extension, or with the extensions `.bak`, `.dat`, `.dir`, or `.pag`.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### De-serializing (ununshelving, loading)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to access the shelved data, just open the shelf via `shelve.open()` and use it like you would with a \"normal\" Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Green', 'Yellow', 'Orange', 'Red', 'Blue', 'Brown', 'White', 'Black']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelved_colors = shelf['colors']\n",
    "\n",
    "print(shelved_colors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't know the keys that exist within the shelf, you can of course list them, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cryptos', 'colors']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    print(list(shelf.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Listing the key values (although quite slow), can be used via the `values()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether'], ['Green', 'Yellow', 'Orange', 'Red', 'Blue', 'Brown', 'White', 'Black']]\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    print(list(shelf.values()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since shelves behave like dictionaries, if you want to iterate over all shelved items, you can:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cryptos ['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether']\n",
      "colors ['Green', 'Yellow', 'Orange', 'Red', 'Blue', 'Brown', 'White', 'Black']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    for key in shelf:\n",
    "        print(key, shelf[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Updating / modifying shelves\n",
    "By default, a shelf doesn't track any updates / modifications on a de-serialized object.\n",
    "So if you would try to do the following (to add `Monero` to the shelved list of cryptos), the shelf itself isn't persisently updated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelf['cryptos'].append('Monero')\n",
    "\n",
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelved_cryptos = shelf['cryptos']\n",
    "\n",
    "print(shelved_cryptos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, after re-loading the shelf `Monero` is **not** contained in the shelved cryptos list.\n",
    "\n",
    "You can of course do so, by two ways:\n",
    "\n",
    "1) de-serialize the shelf, create a copy, append the item (`Monero`) to the copy, and then store the entire copied item back to the shelf using its key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether', 'Monero']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    cryptos = shelf['cryptos']\n",
    "    cryptos.append('Monero')\n",
    "    shelf['cryptos'] = cryptos\n",
    "\n",
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelved_cryptos = shelf['cryptos']\n",
    "\n",
    "print(shelved_cryptos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now `Monero` is added to the persistent shelf.\n",
    "\n",
    "2) The second way, which is less verbose but is slower and demands more RAM usage, is by opening the shelve including the flag `writeback=True`, and directly appending the new item to the shelf (let's now add `Dash` since `Monero` is already added):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether', 'Monero', 'Dash']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf', writeback=True) as shelf:\n",
    "    shelf['cryptos'].append('Dash')\n",
    "\n",
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelved_cryptos = shelf['cryptos']\n",
    "\n",
    "print(shelved_cryptos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Removing `Dash` is of course as simple as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Steem', 'Steem Backed Dollars', 'Bitcoin', 'IoTeX', 'Litecoin', 'Stellar', 'Byteball', 'Tether', 'Monero']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf', writeback=True) as shelf:\n",
    "    shelf['cryptos'].remove('Dash')\n",
    "\n",
    "with shelve.open('test_shelf') as shelf:\n",
    "    shelved_cryptos = shelf['cryptos']\n",
    "\n",
    "print(shelved_cryptos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting elements from a shelf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to complete remove a shelf element, for example all cryptos stored in `shelf['cryptos']`, then use the `del` keyword:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['colors']\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('test_shelf') as shelf:\n",
    "    del shelf['cryptos']\n",
    "\n",
    "with shelve.open('test_shelf') as shelf:\n",
    "    print(list(shelf.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shelf concurrency and Read-Only using `flag=r`\n",
    "Please note that the underlying DBM module powering shelf databases, doesn't support concurrent writing, for example when multiple applications try to write to a shalve database at the same time / when opened.\n",
    "\n",
    "DBM does however support concurrent reads, so a smart thing to do if you want to use concurrency, is to let the client that only wants to read from the shelve do so in read-only mode by passing `flag=r` while opening the shelf.\n",
    "\n",
    "For demonstration purposes, I'll now explicitly import the `dbm` module as well, so that it can print an error message when the read-only shelve does try to write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Woops! There's an error: cannot add item to database\n"
     ]
    }
   ],
   "source": [
    "import dbm\n",
    "\n",
    "with shelve.open('test_shelf', flag='r') as shelf:\n",
    "    try:\n",
    "        colors = shelf['colors']\n",
    "        colors.append('Pink')\n",
    "        shelf['colors'] = colors\n",
    "        print(shelf['colors'])\n",
    "    except dbm.error as err:\n",
    "        print(f\"Woops! There's an error: {err}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What did we learn, hopefully?\n",
    "Using `pickle` and `shelve`, although they should be treated with care (i.e. security issues when executing pickled code), is both pretty powerful and convenient, to me at least. If you would scroll back to my earlier [#14 episode, on developing a mini-Steem crawler for account discovery](https://steemit.com/utopian-io/@scipio/learn-python-series-14-mini-project-developing-a-web-crawler-part-2), just imagine how much easier it would have been to simply apply a shelf and update the `todo`, `done` and `all` files!\n",
    "\n",
    "I hope you enjoyed this tutorial as much as I have writing it!\n",
    "### Thank you for your time!\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}