5_Chapter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Chapter 5. Files and I/O](http://chimera.labs.oreilly.com/books/1230000000393/ch05.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All programs need to perform input and output.  \n",
    "This chapter covers common idioms for working with different kinds of files, including text and binary files, file encodings, and other related matters.  \n",
    "Techniques for manipulating filenames and directories are also covered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Reading and Writing Text Data](http://chimera.labs.oreilly.com/books/1230000000393/ch05.html#_reading_and_writing_text_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to read or write text data, possibly in different text encodings such as ASCII, UTF-8, or UTF-16."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `open()` function with mode `rt` to read a text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python Ipsum: Your source for Python-flavored placeholder text.\n",
      "http://pythonipsum.com/\n",
      "\n",
      "Lambda raspberrypi beautiful test script. Kwargs integration itertools dict reduce egg import cython.\n",
      "\n",
      "Django integration functools unit object kwargs functools dictionary cython. Cython integration exception. Lambda integration diversity bdfl. Return integration exception self dunder. Python integration mercurial bdfl python lambda generator. Kwargs raspberrypi decorator unit cython import. Cython raspberrypi exception unit future klass exception. Python integration community. Object raspberrypi community bdfl cython import method.\n",
      "\n",
      "Method raspberrypi diversity 2to3 return yield unit yield guido. Method integration mercurial unit import python exception dictionary. Django raspberrypi functools self import. Python integration mercurial dict return klass. Lambda integration mercurial 2to3 cython zen.\n",
      "\n",
      "Import raspberrypi community pypi reduce dunder pyladies functools. Lambda raspberrypi decorator bdfl. Python integration generator pypy script. Gevent integration generator pypy lambda jinja decorator arduino dict zip.\n",
      "\n",
      "Method integration coroutine unit import cython implicit django python. Return raspberrypi functools pypi method gil dunder. Dunder raspberrypi decorator list reduce numpy. Future raspberrypi coroutine bdfl python egg gevent python functools descriptor.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Read the entire file as a single string:\n",
    "with open('python_ipsum.txt', 'rt') as f:\n",
    "    data = f.read()\n",
    "f.close()\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 364,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python Ipsum: Your source for Python-flavored placeholder text.\n",
      "\n",
      "http://pythonipsum.com/\n",
      "\n",
      "\n",
      "\n",
      "Lambda raspberrypi beautiful test script. Kwargs integration itertools dict reduce egg import cython.\n",
      "\n",
      "\n",
      "\n",
      "Django integration functools unit object kwargs functools dictionary cython. Cython integration exception. Lambda integration diversity bdfl. Return integration exception self dunder. Python integration mercurial bdfl python lambda generator. Kwargs raspberrypi decorator unit cython import. Cython raspberrypi exception unit future klass exception. Python integration community. Object raspberrypi community bdfl cython import method.\n",
      "\n",
      "\n",
      "\n",
      "Method raspberrypi diversity 2to3 return yield unit yield guido. Method integration mercurial unit import python exception dictionary. Django raspberrypi functools self import. Python integration mercurial dict return klass. Lambda integration mercurial 2to3 cython zen.\n",
      "\n",
      "\n",
      "\n",
      "Import raspberrypi community pypi reduce dunder pyladies functools. Lambda raspberrypi decorator bdfl. Python integration generator pypy script. Gevent integration generator pypy lambda jinja decorator arduino dict zip.\n",
      "\n",
      "\n",
      "\n",
      "Method integration coroutine unit import cython implicit django python. Return raspberrypi functools pypi method gil dunder. Dunder raspberrypi decorator list reduce numpy. Future raspberrypi coroutine bdfl python egg gevent python functools descriptor.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Iterate over the lines of the file:\n",
    "with open('python_ipsum.txt', 'rt') as f:\n",
    "    for line in f:\n",
    "        print(line)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, to write a text file, use `open()` with mode `wt` to write a file, clearing and overwriting the previous contents (if there were any, at least)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write chunks of text data:\n",
    "with open('text_data.txt', 'wt') as f:\n",
    "    f.write(\"This is the first sentence.\\n\")\n",
    "    f.write(\"This is the second sentence.\\n\")\n",
    "    f.write(\"Spicy Jalapeño\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 366,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "Spicy Jalapeño\n"
     ]
    }
   ],
   "source": [
    "with open('text_data.txt', 'rt') as f:\n",
    "    text_data = f.read()\n",
    "f.close()\n",
    "print(text_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try a redirected `print()` call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 367,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('print_statement.txt', 'wt') as f:\n",
    "    print(\"This is the first sentence.\", file=f)\n",
    "    print(\"This is the second sentence.\", file=f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 368,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('print_statement.txt', 'rt') as f:\n",
    "    print_statement = f.read()\n",
    "f.close()\n",
    "print(print_statement)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To append to the end of an existing file, use `open()` with mode `at`.  \n",
    "By default, files are read/written using the system default text encoding, as can be found in `sys.getdefaultencoding()`.  \n",
    "On most machines, this is set to `utf-8`.  \n",
    "If you know that the text you are reading or writing is in a different encoding, supply the optional encoding parameter to `open()`."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "with open('somefile.txt', 'rt', encoding='latin-1') as f:\n",
    "    ... do something ...\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python understands several hundred possible text encodings.  \n",
    "However, some of the more common encodings are `ascii, latin-1, utf-8, and utf-16`.  \n",
    "`UTF-8` is usually a safe bet if working with web applications.  \n",
    "`ascii` corresponds to the 7-bit characters in the range `U+0000` to `U+007F`.  \n",
    "`latin-1` is a direct mapping of bytes 0-255 to Unicode characters `U+0000` to `U+00FF`.  \n",
    "`latin-1` encoding is notable in that it will never produce a decoding error when reading text of a possibly unknown encoding.  \n",
    "Reading a file as `latin-1` might not produce a completely correct text decoding, but it still might be enough to extract useful data out of it.   \n",
    "Also, if you later write the data back out, the original input data will be preserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 369,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "Spicy JalapeÃ±o\n"
     ]
    }
   ],
   "source": [
    "with open('text_data.txt', 'rt', encoding='latin-1') as f:\n",
    "    text_data = f.read()\n",
    "f.close()\n",
    "print(text_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading and writing text files is typically very straightforward.  \n",
    "However, there are a number of subtle aspects to keep in mind.  \n",
    "First, the use of the `with` statement in the examples establishes a context in which the file will be used.  \n",
    "<b>When control leaves the `with` block, the file will be closed automatically</b>.  \n",
    "You don’t need to use the `with` statement, but if you don’t use it, make sure you remember to close the file, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 370,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "Spicy Jalapeño\n"
     ]
    }
   ],
   "source": [
    "f = open('text_data.txt', 'rt')\n",
    "text_data = f.read()\n",
    "f.close()\n",
    "print(text_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another minor complication concerns the recognition of newlines, which are different on Unix and Windows (i.e., `\\n` versus `\\r\\n`).  \n",
    "By default, Python operates in what’s known as \"universal newline\" mode.  \n",
    "In this mode, all common newline conventions are recognized, and newline characters are converted to a single `\\n` character while reading.  \n",
    "Similarly, the newline character `\\n` is converted to the system default newline character on output.  \n",
    "If you don’t want this translation, supply the `newline=''` argument to `open()`, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 371,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "Spicy Jalapeño\n"
     ]
    }
   ],
   "source": [
    "# Read a file with a disabled newline translation:\n",
    "with open('text_data.txt', 'rt', newline='') as f:\n",
    "    text_data = f.read()\n",
    "print(text_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate the difference, here’s what you will see on a Unix machine if you read the contents of a Windows-encoded text file containing the raw data `hello world!\\r\\n`:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> # Newline translation enabled (default):\n",
    ">>> f = open('hello.txt', 'rt')\n",
    ">>> f.read()\n",
    "'hello world!\\n'\n",
    "\n",
    ">>> # Newline translation disabled:\n",
    ">>> g = open('hello.txt', 'rt', newline='')\n",
    ">>> g.read()\n",
    "'hello world!\\r\\n'\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A final issue concerns possible encoding errors in text files.  \n",
    "When reading or writing a text file, you might encounter an encoding or decoding error."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> f = open('sample.txt', 'rt', encoding='ascii')\n",
    ">>> f.read()\n",
    "Traceback (most recent call last):\n",
    "  File \"<stdin>\", line 1, in <module>\n",
    "  File \"/usr/local/lib/python3.3/encodings/ascii.py\", line 26, in decode\n",
    "    return codecs.ascii_decode(input, self.errors)[0]\n",
    "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position\n",
    "12: ordinal not in range(128)\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get this error, it usually means that you’re not reading the file in the correct encoding.  \n",
    "You should carefully read the specification of whatever it is that you’re reading and check that you’re doing it right (e.g., reading data as `UTF-8` instead of `Latin-1` or whatever it needs to be).  \n",
    "If encoding errors are still a possibility, you can supply an optional `errors` argument to `open()` to deal with the errors.  \n",
    "Here are two samples of common error handling schemes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "Spicy Jalape��o\n"
     ]
    }
   ],
   "source": [
    "# Replace bad characters with Unicode U+fffd replacement characters:\n",
    "with open('text_data.txt', 'rt', encoding='ascii', errors='replace') as f:\n",
    "    text_data = f.read()\n",
    "print(text_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 373,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first sentence.\n",
      "This is the second sentence.\n",
      "Spicy Jalapeo\n"
     ]
    }
   ],
   "source": [
    "# Ignore bad characters entirely:\n",
    "f = open('text_data.txt', 'rt', encoding='ascii', errors='ignore')\n",
    "text_data = f.read()\n",
    "f.close()\n",
    "print(text_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you’re constantly fiddling with the `encoding` and `errors` arguments to `open()` and doing lots of hacks, you’re probably making life more difficult than it needs to be.  \n",
    "The number one rule with text is that you simply need to make sure you’re always using the proper text encoding.  \n",
    "When in doubt, use the default setting (typically UTF-8)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Printing to a File](http://chimera.labs.oreilly.com/books/1230000000393/ch05.html#_printing_to_a_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to redirect the output of the `print()` function to a file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the [`file` keyword argument](https://docs.python.org/3/library/functions.html#print) to `print()`, like this:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "with open('example.txt', 'rt') as f:\n",
    "    print('This is an example!', file=f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's not much more to printing to a file other than this.  \n",
    "However, make sure that the file is opened in text mode.  \n",
    "Printing will fail if the underlying file is in binary mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Printing with a Different Separator or Line Ending"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to output data using `print()`, but you also want to change the separator character or the line ending."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `sep` and `end` keyword arguments to `print()` to change the output as you see fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ACME 50 91.5\n",
      "ACME,50,91.5\n",
      "ACME!50!91.5\n",
      "ACME,50,91.5!!\n",
      "ACME!!\n",
      "50!!\n",
      "91.5,ACME!50!91.5!"
     ]
    }
   ],
   "source": [
    "print('ACME', 50, 91.5)\n",
    "print('ACME', 50, 91.5, sep=',')\n",
    "print('ACME', 50, 91.5, sep='!')\n",
    "print('ACME', 50, 91.5, sep=',', end ='!!\\n')\n",
    "print('ACME', 50, 91.5, sep='!!\\n', end =',')\n",
    "# Notice how the last one messes with the notebook when it's run after the other print statements.\n",
    "# If you run the next command in a cell by itself, it returns the correct output.\n",
    "# Hmmmmm...\n",
    "print('ACME', 50, 91.5, sep='!', end='!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use of the `end` argument is also how you suppress the output of newlines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 375,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01234"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print(i, end='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01234"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print(i, end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `print()` with a different item separator is often the easiest way to output data when you need something other than a space separating the items.  \n",
    "Sometimes you'll see programmers using `str.join()` to accomplish the same thing."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "print(','.join('ACME', '50', '91.5'))\n",
    "\n",
    ">>> ACME,50,91.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem with `str.join()` is that it only works with strings.  \n",
    "You may have to resort to some creative means to achieve your end."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "row = ('ACME', 50, 91.5)\n",
    "print(''.join(row))\n",
    "\n",
    ">>>\n",
    "---------------------------------------------------------------------------\n",
    "TypeError                                 Traceback (most recent call last)\n",
    "<ipython-input-63-26c0ab3c0043> in <module>()\n",
    "      1 row = ('ACME', 50, 91.5)\n",
    "----> 2 print(''.join(row))\n",
    "\n",
    "TypeError: sequence item 1: expected str instance, int found\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, you will have better luck with the following:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# https://docs.python.org/3.6/glossary.html#term-argument\n",
    "print(*row, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and Writing Binary Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to write or write [binary data](https://en.wikipedia.org/wiki/Binary_data#In_computer_science), like the kind found in image or sound files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `open()` function with mode set to `rb` or `wb` to read or write binary data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 378,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the entire file as a single byte string:\n",
    "with open('text_data.txt', 'rb') as f:\n",
    "    data = f.read()\n",
    "    \n",
    "# Write binary data to a file:\n",
    "with open('binary.bin', 'wb') as f:\n",
    "    f.write(b'Python Cookbook')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When reading binary, it is important to stress that all data returned will be in the form of [byte strings](https://docs.python.org/3/library/stdtypes.html#bytes), not text strings.  \n",
    "Similarly, when writing, you must supply data in the form of objects that expose data as bytes (e.g., byte strings, bytearray objects, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When reading binary data, the subtle semantic differences between byte strings and text strings pose a potential gotcha.  \n",
    "In particular, be aware that indexing and iteration return integer byte values instead of byte strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 379,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'H'"
      ]
     },
     "execution_count": 379,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Text String:\n",
    "t = 'Hello World'\n",
    "t[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H\n",
      "e\n",
      "l\n",
      "l\n",
      "o\n",
      " \n",
      "W\n",
      "o\n",
      "r\n",
      "l\n",
      "d\n"
     ]
    }
   ],
   "source": [
    "for c in t:\n",
    "    print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "72"
      ]
     },
     "execution_count": 381,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Byte String:\n",
    "b = b'Hello World'\n",
    "b[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 382,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72\n",
      "101\n",
      "108\n",
      "108\n",
      "111\n",
      "32\n",
      "87\n",
      "111\n",
      "114\n",
      "108\n",
      "100\n"
     ]
    }
   ],
   "source": [
    "for c in b:\n",
    "    print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you ever need to read or write text froma binary-mode file, remember to decode or encode it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 383,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first\n"
     ]
    }
   ],
   "source": [
    "with open('text_data.txt', 'rb') as f:\n",
    "    data = f.read(17)\n",
    "    text = data.decode('utf-8')\n",
    "    print(text)\n",
    "    \n",
    "with open('binary.bin', 'wb') as f:\n",
    "    text = 'Python Cookbook'\n",
    "    f.write(text.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lesser-known aspect of binary I/O is that objects such as arrays and C structures can be used for writing without any kind if intermediate conversion to a `bytes` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "metadata": {},
   "outputs": [],
   "source": [
    "import array\n",
    "\n",
    "nums = array.array('i', [1, 2, 3, 4])\n",
    "with open('array_data.bin', 'wb') as f:\n",
    "    f.write(nums)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This applies to any object that implements the so-called [\"buffer interface\"](https://docs.python.org/3/c-api/buffer.html), which exposes an underlying memory buffer to [operations](https://docs.python.org/3/c-api/buffer.html#buffer-related-functions) that can work with it.  \n",
    "Writing binary data is one such operation.  \n",
    "Many Python objects also allow binary data to be directly read into their underlying memory using the `readinto()` method of files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 385,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16\n"
     ]
    }
   ],
   "source": [
    "import array\n",
    "\n",
    "a = array.array('i', [0, 0, 0, 0, 0, 0, 0 ,0])\n",
    "with open('array_data.bin', 'rb') as f:\n",
    "    print(f.readinto(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 386,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array('i', [1, 2, 3, 4, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 386,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use caution when using this technique, because it is often platform specific and may depend on such things as the word size and [byte ordering](https://docs.python.org/3/library/struct.html#struct-alignment).  \n",
    "See Recipe 5.9 for another example of reading binary data into a mutable buffer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing to a File That Doesn’t Already Exist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to write data toa file, but only if it doesn't already exist on the filesystem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This problem can be handled by using the `x` mode in the function call to `open()` instead of `w` mode."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "with open('write_something.txt', 'wt') as f:\n",
    "    f.write('Sample 1\\n')\n",
    "    \n",
    "with open('write_something.txt', 'xt') as f:\n",
    "    f.write('Sample 2\\n')\n",
    "    \n",
    ">>>\n",
    "-----------------------\n",
    "FileExistsError                           Traceback (most recent call last)\n",
    "<ipython-input-17-65c8f2343f2d> in <module>()\n",
    "      2     f.write('Sample 1\\n')\n",
    "      3 \n",
    "----> 4 with open('write_something.txt', 'xt') as f:\n",
    "      5     f.write('Sample 2\\n')\n",
    "\n",
    "FileExistsError: [Errno 17] File exists: 'write_something.txt'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the file is binary, use mode `xb` instead of `xt`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative solution is to first test for the existence of the file like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 387,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "That file already exists, sucka.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "if not os.path.exists('write_something.txt'):\n",
    "    with open('write_something.txt', 'wt') as f:\n",
    "        f.write('Word Up!\\n')\n",
    "else:\n",
    "    print(\"That file already exists, sucka.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that using the `open()` function in `x` mode can only be done in Python 3.  \n",
    "No such mode exists in earlier Python versions or in the C libraries used in Python's implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing I/O Operations on a String"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to feed a text or binary string to code that's been written to operate on file-like objects instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `io.StringIO()` and `io.BytesIO()` classes to create file-like objects that operate on string data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 388,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 388,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import io\n",
    "\n",
    "s = io.StringIO()\n",
    "s.write('Hello World\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 389,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello World\\nThis is a test\\n'"
      ]
     },
     "execution_count": 389,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('This is a test', file=s)\n",
    "# Get all of the data written so far: \n",
    "s.getvalue()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 390,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hell'"
      ]
     },
     "execution_count": 390,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Wrap a file interface around an existing string:\n",
    "s = io.StringIO('Hello\\nWorld\\n')\n",
    "s.read(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 391,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'o\\nWorld\\n'"
      ]
     },
     "execution_count": 391,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `io.StringIO` class should only be used for text.  \n",
    "If you are using binary data, use the `io.BytesIO` class instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'binary data'"
      ]
     },
     "execution_count": 392,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = io.BytesIO()\n",
    "s.write(b'binary data')\n",
    "s.getvalue()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `StringIO` and `BytesIO` classes are most useful in scenarios where you need to mimic a normal file for some reason.  \n",
    "For example, in unit tests, you might use `StringIO` to create a file-like object containing test data that’s fed into a function that would otherwise work with a normal file.  \n",
    "Be aware that `StringIO` and `BytesIO` instances don’t have a proper integer file-descriptor.  \n",
    "Thus, they do not work with code that requires the use of a real system-level file such as a file, pipe, or socket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and Writing Compressed Datafiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to read or write data in a file with gzip or bz2 compression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, by golly, Python has both `gzip` and `bz2` modules that you can use.  \n",
    "Both modules provide an alternative implementation of `open()` that can be used."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Read compressed files as text:\n",
    "\n",
    "import gzip\n",
    "with gzip.open('somefile.gz', 'rt') as f:\n",
    "    text = f.read()\n",
    "    \n",
    "import bz2\n",
    "with bz2.opern('anotherfile.bz2', 'rt') as f:\n",
    "    text = f.read()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Write compressed data to a file:\n",
    "\n",
    "import gzip\n",
    "with gzip.open('somefile.gz', 'wt') as f:\n",
    "    f.write(text)\n",
    "    \n",
    "import bz2\n",
    "with bz2.open('anotherfile.bz2', 'wt') as f:\n",
    "    f.write(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown, all I/O will use text and perform Unicode encoding and decoding.  \n",
    "If you would rather use binary data, use the file modes `rb` or `wb`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the most part, reading or writing compressed data is straightforward.  \n",
    "However, be aware that choosing the correct file mode is critically important.  \n",
    "If you don’t specify a mode, the default mode is binary, which will break programs that expect to receive text.  \n",
    "Both `gzip.open()` and `bz2.open()` accept the same parameters as the built-in `open()` function, including encoding, errors, newline, and so forth.  \n",
    "When writing compressed data, the compression level can be optionally specified using the `compresslevel` keyword argument."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "with gzip.open('anyfile.gz', 'wt', compresslevel=5) as f:\n",
    "    f.write(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default level is 9, which provides the highest level of compression.  \n",
    "Lower levels offer better performance, but not as much compression.  \n",
    "Finally, `gzip.open()` and `bz2.open()` can be layered on top of an existing file opened in binary mode."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import gzip\n",
    "\n",
    "f = open('somefile.gz', 'rb')\n",
    "with gzip.open(f, 'rt') as g:\n",
    "    text = g.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using that neat little trick can come in handy when you work with various file-like objects including sockets, pipes, and in-memory files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating Over Fixed-Sized Records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of iterating over a file by lines, you want to iterate over a collection of fixed-size records."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `iter()` function and `functools.partial()` for this little trick:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from functools import partial\n",
    "\n",
    "RECORD_SIZE = 32\n",
    "\n",
    "with open('example.data', 'rb') as f:\n",
    "    records = iter(partial(f.read, RECORD_SIZE), b'')\n",
    "    for r in records:\n",
    "        do something\n",
    "        return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `records` object in this example is an iterable that will produce fixed-size chunks until the end of the file is reached.  \n",
    "However, be aware that the last item may have fewer bytes than expected if the file size is not an exact multiple of the record size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A little-known feature of the `iter()` function is that it can create an iterator if you pass it a callable and a sentinel value.  \n",
    "The resulting iterator simply calls the supplied callable over and over again until it returns the sentinel, at which point iteration stops.\n",
    "In the solution, the `functools.partial` is used to create a callable that reads a fixed number of bytes from a file each time it’s called.  \n",
    "The sentinel of `b''` is what gets returned when a file is read but the end of file has been reached.  \n",
    "Last, but not least, the solution shows the file being opened in binary mode.  \n",
    "For reading fixed-sized records, this would probably be the most common case.  \n",
    "For text files, reading line by line (the default iteration behavior) is more common."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading Binary Data into a Mutable Buffer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to read binary data directly into a mutable buffer without any intermediate copying.  \n",
    "Perhaps you want to mutate the data in-place and write it back out to a file?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read data into a mutable array, use the `readinto()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 393,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "\n",
    "def read_into_buffer(filename):\n",
    "    buf = bytearray(os.path.getsize(filename))\n",
    "    with open(filename, 'rb') as f:\n",
    "        f.readinto(buf)\n",
    "    return buf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what we can do with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 394,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('write_to_buffer.bin', 'wb') as f:\n",
    "    f.write(b'Hello World')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 395,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_buffer_object = read_into_buffer('write_to_buffer.bin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 396,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytearray(b'Hello World')"
      ]
     },
     "execution_count": 396,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_buffer_object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 397,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_buffer_object[0:5] = b'Haloo'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 398,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytearray(b'Haloo World')"
      ]
     },
     "execution_count": 398,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_buffer_object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 399,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('new_buffer_object.bin', 'wb') as f:\n",
    "    f.write(a_buffer_object)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to continue the above example in any way that amuses you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `readinto()` method of files can be used to fill any preallocated array with data.  \n",
    "This even includes arrays created from the array module or libraries such as `numpy`.  \n",
    "Unlike the normal `read()` method, `readinto()` fills the contents of an existing buffer rather than allocating new objects and returning them.  \n",
    "Thus, you might be able to use it to avoid making extra memory allocations.  \n",
    "For example, if you are reading a binary file consisting of equally sized records, you can write code like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 400,
   "metadata": {},
   "outputs": [],
   "source": [
    "record_size = 32\n",
    "\n",
    "buf = bytearray(record_size)\n",
    "with open('example.bin', 'rb') as f:\n",
    "    while True:\n",
    "        n = f.readinto(buf)\n",
    "        if n < record_size:\n",
    "            break\n",
    "        # Use the contents of buf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another interesting feature to check out is [memoryview](https://docs.python.org/3/library/stdtypes.html#memoryview), which you can use to make zero-copy slices of an existing buffer and even change its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 401,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytearray(b'Hello World\\n\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00')"
      ]
     },
     "execution_count": 401,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "buf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 402,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<memory at 0x10fe6c408>"
      ]
     },
     "execution_count": 402,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m1 = memoryview(buf)\n",
    "m1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 403,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<memory at 0x10fe6c4c8>"
      ]
     },
     "execution_count": 403,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m2 = m1[-5:]\n",
    "m2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 404,
   "metadata": {},
   "outputs": [],
   "source": [
    "m2[:] = b'WORLD'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 405,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytearray(b'Hello World\\n\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00WORLD')"
      ]
     },
     "execution_count": 405,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "buf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output you see above is different from what your interpreter will return, but you can get the idea from the example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One caution with using `f.readinto()` is that you must always make sure to check its return code, which is the number of bytes actually read.  \n",
    "If the number of bytes is smaller than the size of the supplied buffer, it might indicate truncated or corrupted data (e.g., if you were expecting an exact number of bytes to be read).  \n",
    "Finally, be on the lookout for other \"into\" related functions in various library modules (like `recv_into()` and `pack_into()`).  \n",
    "Many other parts of Python have support for direct I/O or data access that can be used to fill or alter the contents of arrays and buffers.  \n",
    "See Recipe 6.12 for a significantly more advanced example of interpreting binary structures and usage of memoryviews."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory Mapping Binary Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to memory map a binary file into a mutable byte array, possibly for random access to its contents or to make in-place modifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `mmap` module to map memory files.  \n",
    "Here is a utility function that shows how to open a file and memory map it in a portable manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 406,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import mmap\n",
    "\n",
    "def memory_map(filename, access=mmap.ACCESS_WRITE):\n",
    "    size = os.path.getsize(filename)\n",
    "    fd = os.open(filename, os.O_RDWR)\n",
    "    return mmap.mmap(fd, size, access)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use this function, you would need to have a file already created and filled with data.  \n",
    "Here is an example of how you could initially create a file and expand it to a desired size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will create 4KB of blank space on your disk.\n",
    "\n",
    "size = 1000000\n",
    "with open('some_data.bin', 'wb') as f:\n",
    "    f.seek(size-1)\n",
    "    f.write(b'\\x00')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of memory mapping the contents using the `memory_map()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 408,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000000"
      ]
     },
     "execution_count": 408,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = memory_map('some_data.bin')\n",
    "len(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 409,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"
      ]
     },
     "execution_count": 409,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 410,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 410,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what else we can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 411,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reassign a slice:\n",
    "m[0:11] = b'Hello World'\n",
    "m.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 412,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\n"
     ]
    }
   ],
   "source": [
    "# Let's see what we changed and how:\n",
    "with open('some_data.bin', 'rb') as f:\n",
    "    print(f.read(11))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that failed completely.  \n",
    "Maybe something to do with the Jupyter Notebook?  \n",
    "Oh well, let's try something else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `mmap` object returned by `mmap()` can also be used as a context manager, in which case the underlying file is closed automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000000\n",
      "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\n"
     ]
    }
   ],
   "source": [
    "with memory_map('some_data.bin') as m:\n",
    "    print(len(m))\n",
    "    print(m[0:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 414,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 414,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.closed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the `memory_map()` function shown opens a file for both reading and writing.  \n",
    "Any modifications made to the data are copied back to the original file.  \n",
    "If read-only access is needed instead, use `mmap.ACCESS_READ` for the `access` argument."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "m = memory_map(filename, mmap.ACCESS_READ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you intend to modify the data locally, but don't want those changes written back to the original file, use `mmap.ACCESS_COPY`:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "m = memory_map(filename, mmap.ACCESS_COPY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `mmap` to map files into memory can be an efficient and elegant means for randomly accessing the contents of a file.  \n",
    "For example, instead of opening a file and performing various combinations of `seek()`, `read()`, and `write()` calls, you can simply map the file and access the data using slicing operations.  \n",
    "Normally, the memory exposed by `mmap()` looks like a `bytearray` object.  \n",
    "However, you can interpret the data differently using a memoryview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 415,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x07\\x00\\x00\\x00'"
      ]
     },
     "execution_count": 415,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = memory_map('some_data.bin')\n",
    "# Memoryview of unsigned integers:\n",
    "v = memoryview(m).cast('I')\n",
    "v[0] = 7\n",
    "m[0:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That seemed to work okay.  \n",
    "Moving on ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "263"
      ]
     },
     "execution_count": 416,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m[0:4] = b'\\x07\\x01\\x00\\x00'\n",
    "v[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It should be emphasized that memory mapping a file does not cause the entire file to be read into memory.  \n",
    "That is, it’s not copied into some kind of memory buffer or array.  \n",
    "Instead, the operating system merely reserves a section of virtual memory for the file contents.  \n",
    "As you access different regions, those portions of the file will be read and mapped into the memory region as needed.  \n",
    "However, parts of the file that are never accessed simply stay on disk.  \n",
    "This all happens transparently, behind the scenes.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If more than one Python interpreter memory maps the same file, the resulting `mmap` object can be used to exchange data between interpreters.  \n",
    "That is, all interpreters can read/write data simultaneously, and changes made to the data in one interpreter will automatically appear in the others.  \n",
    "Obviously, some extra care is required to synchronize things, but this kind of approach is sometimes used as an alternative to transmitting data in messages over pipes or sockets.  \n",
    "As shown, this recipe has been written to be as general purpose as possible, working on both Unix and Windows. Be aware that there are some platform differences concerning the use of the `mmap()` call hidden behind the scenes.  \n",
    "In addition, there are options to create anonymously mapped memory regions.  \n",
    "For more information, [here is a link to the Python docs](https://docs.python.org/3/library/mmap.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manipulating Pathnames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to manipulate pathnames in order to find the base filename, directory name, absolute path, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To manipulate pathnames, use the functions in the `os.path` module.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "path = 'Users/benjamingrove/Desktop/punchcode_syllabus.pdf'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the last component of the path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 418,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'punchcode_syllabus.pdf'"
      ]
     },
     "execution_count": 418,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.basename(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the directory name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Users/benjamingrove/Desktop'"
      ]
     },
     "execution_count": 419,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.dirname(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Join path components together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 420,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tmp/data/punchcode_syllabus.pdf'"
      ]
     },
     "execution_count": 420,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.join('tmp', 'data', os.path.basename(path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I wonder if we can expand the user's home directory?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 421,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/benjamingrove/Data/data.csv'"
      ]
     },
     "execution_count": 421,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = '~/Data/data.csv'\n",
    "os.path.expanduser(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even split the file extension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 422,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('~/Data/data', '.csv')"
      ]
     },
     "execution_count": 422,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.splitext(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any manipulation of filenames, you should use the `os.path` module instead of trying to cook up your own code using the standard string operations.  \n",
    "In part, this is for portability.  \n",
    "The `os.path` module knows about differences between Unix and Windows and can reliably deal with filenames such as `Data/data.csv` and `Data\\data.csv`.  \n",
    "Second, you really shouldn’t spend your time reinventing the wheel.  \n",
    "It’s usually best to use the functionality that’s already provided for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing for the Existence of a File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to test whether or not a file or directory exists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `os.path` module to test for the existence of a file or directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 423,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "print(os.path.exists('/etc/passwd'))\n",
    "print(os.path.exists('/tmp/spam'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform further tests to see what kind of a file it might be.  \n",
    "These tests return `False` if the file in question doesn't exist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 424,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 424,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Regular file:\n",
    "os.path.isfile('/etc/passwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 425,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 425,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Directory:\n",
    "os.path.isdir('/etc/passwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 426,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 426,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Symlink:\n",
    "os.path.islink('/usr/local/bin/python3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 427,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6'"
      ]
     },
     "execution_count": 427,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Linked file:\n",
    "os.path.realpath('/usr/local/bin/python3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `os.path` to get metadata from a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 428,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6774"
      ]
     },
     "execution_count": 428,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.getsize('/etc/passwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1506990573.0"
      ]
     },
     "execution_count": 429,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.getmtime('/etc/passwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 430,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Mon Oct  2 17:29:33 2017'"
      ]
     },
     "execution_count": 430,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.ctime(os.path.getmtime('/etc/passwd'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "File testing is a straightforward operation using `os.path`.  \n",
    "Probably the only thing to be aware of when writing scripts is that you might need to worry about permissions, especially for the operations that get metadata."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> os.path.getsize('/Users/guido/Desktop/foo.txt')\n",
    "\n",
    "Traceback (most recent call last):\n",
    "File \"<stdin>\", line 1, in <module>\n",
    "File \"/usr/local/lib/python3.3/genericpath.py\", line 49, in getsize\n",
    "return os.stat(filename).st_size\n",
    "PermissionError: [Errno 13] Permission denied: '/Users/guido/Desktop/foo.txt'\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting a Directory Listing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to get a list of the files contained in a directory on the filesystem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `os.listdir()` function to obtain a list of the files in a directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['DevCareerGuide', 'CodeSchool', 'RoR', '.DS_Store', 'Python3', 'wordpress', 'Python2', 'Java', 'PHP', 'Sites', 'launchkey-python', 'aldryn_tutorial_cms_site', 'C_lang', 'Treehouse', 'Javascript', 'Apps', 'DSE', 'Shell_Scripts', 'R_Projects']\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "names = os.listdir('/Users/benjamingrove/Programming')\n",
    "print(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will give you the raw directory listing, including all files, subdirectories, symbolic links, and so forth.  \n",
    "If you need to filter the data in some way, consider using a [list comprehension](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions) combined with various functions in the `os.path` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 432,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.com.apple.timemachine.supported', '.DS_Store', '.localized']"
      ]
     },
     "execution_count": 432,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os.path\n",
    "\n",
    "# Get all regular files:\n",
    "filenames = [name for name in os.listdir('/Users/benjamingrove/Public')\n",
    "             if os.path.isfile(os.path.join('/Users/benjamingrove/Public', name))]\n",
    "filenames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Photo Booth Library', 'Photos Library.photoslibrary']"
      ]
     },
     "execution_count": 433,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dirnames = [name for name in os.listdir('/Users/benjamingrove/Pictures')\n",
    "            if os.path.isdir(os.path.join('/Users/benjamingrove/Pictures', name))]\n",
    "dirnames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `startswith()` and `endswith()` string methods can be useful for filtering the contents of a dirctory as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 434,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['vectorize_operations.py']"
      ]
     },
     "execution_count": 434,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyfiles = [name for name in os.listdir('/Users/benjamingrove/Programming/Python3/python3_machine_learning_projects/')\n",
    "           if name.endswith('.py')]\n",
    "pyfiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For filename matching, you may want to use the `glob` or `fnmatch` modules instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 435,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/Users/benjamingrove/Programming/Python3/python3_machine_learning_projects/vectorize_operations.py']"
      ]
     },
     "execution_count": 435,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "\n",
    "pyfiles = glob.glob('/Users/benjamingrove/Programming/Python3/python3_machine_learning_projects/*.py')\n",
    "pyfiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 436,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['vectorize_operations.py']"
      ]
     },
     "execution_count": 436,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from fnmatch import fnmatch\n",
    "\n",
    "pyfiles = [name for name in os.listdir('/Users/benjamingrove/Programming/Python3/python3_machine_learning_projects/')\n",
    "           if fnmatch(name, '*.py')]\n",
    "pyfiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting a directory listing is easy, but it only gives you the names of entries in the directory.  \n",
    "If you want to get additional metadata, such as file sizes, modification dates, and so forth, you either need to use additional functions in the `os.path` module or use the `os.stat()` function to collect the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 437,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/Users/benjamingrove/Programming/Python3/flask_app.py']"
      ]
     },
     "execution_count": 437,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import os.path\n",
    "import glob\n",
    "\n",
    "# Get a directory listing:\n",
    "pyfiles = glob.glob('/Users/benjamingrove/Programming/Python3/*.py')\n",
    "pyfiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 438,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/benjamingrove/Programming/Python3/flask_app.py 334 1507686106.0\n"
     ]
    }
   ],
   "source": [
    "# Get file sizes and modification dates:\n",
    "name_sz_date = [(name, os.path.getsize(name), os.path.getmtime(name))\n",
    "                for name in pyfiles]\n",
    "\n",
    "for name, size, mtime in name_sz_date:\n",
    "    print(name, size, mtime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 439,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/benjamingrove/Programming/Python3/flask_app.py 334 1507686106.0\n"
     ]
    }
   ],
   "source": [
    "# Here is another way to get file metadata:\n",
    "file_metadata = [(name, os.stat(name)) for name in pyfiles]\n",
    "for name, meta in file_metadata:\n",
    "    print(name, meta.st_size, meta.st_mtime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, but not least, be aware that there are subtle issues that can arise in filename handling related to encodings.  \n",
    "Normally, the entries returned by a function such as `os.list dir()` are decoded according to the system default filename encoding.  \n",
    "However, it’s possible under certain circumstances to encounter un-decodable filenames."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bypassing Filename Encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to perform file I/O operations using raw filenames that have not been decoded or encoded according to the default filename encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, all filenames are encoded and decoded according to the text encoding returned by `sys.getfilesystemencoding()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 440,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'utf-8'"
      ]
     },
     "execution_count": 440,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "sys.getfilesystemencoding()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to bypass this encoding for some reason, specify a filename using a raw byte string instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a file using a unicode filename:\n",
    "with open('jalape\\xf1o.txt', 'w') as f:\n",
    "    f.write('Spicy!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 442,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3_Chapter.ipynb',\n",
       " '4_Chapter.ipynb',\n",
       " 'some_data.bin',\n",
       " 'merged_file.txt',\n",
       " 'array_data.bin',\n",
       " 'write_something.txt',\n",
       " 'text_data.txt',\n",
       " 'python_ipsum.txt',\n",
       " 'new_buffer_object.bin',\n",
       " 'print_statement.txt',\n",
       " 'example.bin',\n",
       " 'write_to_buffer.bin',\n",
       " 'jalapeño.txt',\n",
       " '.ipynb_checkpoints',\n",
       " '5_Chapter.ipynb',\n",
       " '1_Chapter.ipynb',\n",
       " 'sorted_file_1.txt',\n",
       " 'binary.bin',\n",
       " 'sorted_file_2.txt',\n",
       " '2_Chapter.ipynb']"
      ]
     },
     "execution_count": 442,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Directory listing (decoded):\n",
    "import os\n",
    "\n",
    "os.listdir('.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 443,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'3_Chapter.ipynb',\n",
       " b'4_Chapter.ipynb',\n",
       " b'some_data.bin',\n",
       " b'merged_file.txt',\n",
       " b'array_data.bin',\n",
       " b'write_something.txt',\n",
       " b'text_data.txt',\n",
       " b'python_ipsum.txt',\n",
       " b'new_buffer_object.bin',\n",
       " b'print_statement.txt',\n",
       " b'example.bin',\n",
       " b'write_to_buffer.bin',\n",
       " b'jalape\\xc3\\xb1o.txt',\n",
       " b'.ipynb_checkpoints',\n",
       " b'5_Chapter.ipynb',\n",
       " b'1_Chapter.ipynb',\n",
       " b'sorted_file_1.txt',\n",
       " b'binary.bin',\n",
       " b'sorted_file_2.txt',\n",
       " b'2_Chapter.ipynb']"
      ]
     },
     "execution_count": 443,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Directory listing (raw):\n",
    "# Byte string\n",
    "os.listdir(b'.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 444,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spicy!\n"
     ]
    }
   ],
   "source": [
    "# Open the file with the raw filename:\n",
    "with open(b'jalapen\\xcc\\x83o.txt') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the last two operations, the filename handling changes ever so slightly when byte strings are supplied to file-related functions, such as `open()` and `os.listdir()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under normal circumstances, you shouldn’t need to worry about filename encoding and decoding.  \n",
    "Normal filename operations should just work.  \n",
    "However, many operating systems may allow a user, through accident or malice, to create files with names that don’t conform to the expected encoding rules.  \n",
    "Such filenames may mysteriously break Python programs that work with a lot of files.  \n",
    "Reading directories and working with filenames as raw undecoded bytes has the potential to avoid such problems, albeit at the cost of programming convenience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Printing Bad Filenames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your program received a directory listing, but when it tried to print the filenames, it crashed with a `UnicodeEncodeError` exception and a cryptic message about \"surrogates not allowed.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When printing filenames of unknown origin, use the following convention to avoid errors and confusion:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "def bad_filename(filename):\n",
    "    # https://docs.python.org/3/library/functions.html#repr\n",
    "    return repr(filename)[1:-1]\n",
    "\n",
    "try:\n",
    "    print(filename)\n",
    "except UnicodeEncodeError:\n",
    "    print(bad_filename(filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This recipe is about a potentially rare but very annoying problem regarding programs that must manipulate the filesystem.  \n",
    "By default, Python assumes that all filenames are encoded according to the setting reported by `sys.getfilesystemencoding()`.  \n",
    "However, certain filesystems don’t necessarily enforce this encoding restriction, thereby allowing files to be created without proper filename encoding.  \n",
    "It’s not common, but there is always the danger that some user will do something silly and create such a file by accident (e.g., maybe passing a bad filename to `open()` in some buggy code).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When executing a command such as `os.listdir()`, bad filenames leave Python in a bind.  \n",
    "On the one hand, it can’t just discard bad names.  \n",
    "On the other hand, it still can’t turn the filename into a proper text string.  \n",
    "Python’s solution to this problem is to take an undecodable byte value `\\xhh` in a filename and map it into a so-called \"surrogate encoding\" represented by the Unicode character `\\udchh`.  \n",
    "Here is an example of how a bad directory listing might look if it contained a filename *bäd.txt*, encoded as Latin-1 instead of UTF-8:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> import os\n",
    ">>> files = os.listdir('.')\n",
    ">>> files\n",
    "['spam.py', 'b\\udce4d.txt', 'foo.txt']\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have code that manipulates filenames or even passes them to functions such as `open()`, everything works normally.  \n",
    "It’s only in situations where you want to output the filename that you run into trouble (e.g., printing it to the screen, logging it, etc.).  \n",
    "Specifically, if you tried to print the preceding listing, your program will crash:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> for name in files:\n",
    "...     print(name)\n",
    "...\n",
    "spam.py\n",
    "Traceback (most recent call last):\n",
    "  File \"<stdin>\", line 2, in <module>\n",
    "UnicodeEncodeError: 'utf-8' codec can't encode character '\\udce4' in\n",
    "position 1: surrogates not allowed\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason it crashes is that the character `\\udce4` is technically invalid Unicode.  \n",
    "It’s actually the second half of a two-character combination known as a surrogate pair.  \n",
    "However, since the first half is missing, it’s invalid Unicode.  \n",
    "Thus, the only way to produce successful output is to take corrective action when a bad filename is encountered.  \n",
    "For example, changing the code to the recipe produces the following:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> for name in files:\n",
    "...     try:\n",
    "...         print(name)\n",
    "...     except UnicodeEncodeError: \n",
    "...         print(bad_filename(name))\n",
    "...\n",
    "spam.py\n",
    "b\\udce4d.txt\n",
    "foo.txt\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The choice of what to do for the `bad_filename()` function is largely up to you.  \n",
    "Another option is to re-encode the value in some way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 445,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bad_filename(filename):\n",
    "    temp = filename.encode(sys.getfilesystemencoding(), errors='surrogateescape')\n",
    "    return temp.decode('latin-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see what the output is:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> for name in files:\n",
    "...     try:\n",
    "...         print(name)\n",
    "...     except UnicodeEncodeError:\n",
    "...         print(bad_filename(name))\n",
    "...\n",
    "spam.py\n",
    "bäd.txt\n",
    "foo.txt\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This recipe will likely be ignored by most readers.  \n",
    "However, if you’re writing mission-critical scripts that need to work reliably with filenames and the filesystem, it’s something to think about.  \n",
    "Otherwise, you might find yourself called back into the office over the weekend to debug a seemingly inscrutable error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding or Changing the Encoding of an Already Open File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to add or change the Unicode encoding of an already open file without closing it first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to add Unicode encoding/decoding to an already existing file object that's been opened in binary mode, wrap it with an `io.TextIOWrapper()` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 446,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!doctype html>\\n<!--[if lt IE 7]>   <html class=\"no-js ie6 lt-ie7 lt-ie8 lt-ie9\">   <![endif]-->\\n<!--[if IE 7]>      <html class=\"no-js ie7 lt-ie8 lt-ie9\">          <![endif]-->\\n<!--[if IE 8]>      <html class=\"no-js ie8 lt-ie9\">                 <![endif]-->\\n<!--[if gt IE 8]><!--><html class=\"no-js\" lang=\"en\" dir=\"ltr\">  <!--<![endif]-->\\n\\n<head>\\n    <meta charset=\"utf-8\">\\n    <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\\n\\n    <link rel=\"prefetch\" href=\"//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js\">\\n\\n    <meta name=\"application-name\" content=\"Python.org\">\\n    <meta name=\"msapplication-tooltip\" content=\"The official home of the Python Programming Language\">\\n    <meta name=\"apple-mobile-web-app-title\" content=\"Python.org\">\\n    <meta name=\"apple-mobile-web-app-capable\" content=\"yes\">\\n    <meta name=\"apple-mobile-web-app-status-bar-style\" content=\"black\">\\n\\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\\n    <meta name=\"HandheldFriendly\" conte'"
      ]
     },
     "execution_count": 446,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "import io\n",
    "\n",
    "u = urllib.request.urlopen('https://www.python.org')\n",
    "f = io.TextIOWrapper(u, encoding='utf-8')\n",
    "text = f.read()\n",
    "text[:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to change the encoding of an already open text-mode file, use its `detach()` method to remove the existing text encoding layer before replacing it with a new one.  \n",
    "Here is an example of changing the encoding on `sys.stdout`.  \n",
    "Don't actually try this unless you feel like fixing your terminal output."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> import sys\n",
    ">>> sys.stdout.encoding\n",
    "'UTF-8'\n",
    ">>> sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')\n",
    ">>> sys.stdout.encoding\n",
    "'latin-1'\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The I/O system is built as a series of layers."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> f = open('sample.txt','w')\n",
    ">>> f\n",
    "<_io.TextIOWrapper name='sample.txt' mode='w' encoding='UTF-8'> >>> f.buffer\n",
    "<_io.BufferedWriter name='sample.txt'>\n",
    ">>> f.buffer.raw\n",
    "<_io.FileIO name='sample.txt' mode='wb'>\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, `io.TextIOWrapper` is a text-handling layer that encodes and decodes Unicode, `io.BufferedWriter` is a buffered I/O layer that handles binary data, and `io.FileIO` is a raw file representing the low-level file descriptor in the operating system.  \n",
    "Adding or changing the text encoding involves adding or changing the topmost `io.TextIOWrapper` layer.  \n",
    "As a general rule, it’s not safe to directly manipulate the different layers by accessing the attributes shown.  \n",
    "For example, see what happens if you try to change the encoding using this technique:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> f\n",
    "<_io.TextIOWrapper name='sample.txt' mode='w' encoding='UTF-8'>\n",
    ">>> f = io.TextIOWrapper(f.buffer, encoding='latin-1')\n",
    ">>> f\n",
    "<_io.TextIOWrapper name='sample.txt' encoding='latin-1'>\n",
    ">>> f.write('Hello')\n",
    "Traceback (most recent call last):\n",
    "    File \"<stdin>\", line 1, in <module>\n",
    "ValueError: I/O operation on closed file.\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't work because the original value of `f` got destroyed and closed the underlying file in the process.  \n",
    "The `detach()` method disconnects the topmost layer of a file and returns the next lower layer.  \n",
    "Afterward, the top layer will no longer be usable."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> f = open('sample.txt', 'w')\n",
    ">>> f\n",
    "<_io.TextIOWrapper name='sample.txt' mode='w' encoding='UTF-8'> >>> b = f.detach()\n",
    ">>> b\n",
    "<_io.BufferedWriter name='sample.txt'>\n",
    ">>> f.write('hello')\n",
    "Traceback (most recent call last):\n",
    "    File \"<stdin>\", line 1, in <module>\n",
    "ValueError: underlying buffer has been detached\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once detached, however, you can add a new top layer to the result that is returned."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> f = io.TextIOWrapper(b, encoding='latin-1')\n",
    ">>> f\n",
    "<_io.TextIOWrapper name='sample.txt' encoding='latin-1'>\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although changing the encoding has been shown, it is also possible to use this technique to change the line handling, error policy, and other aspects of file handling."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='ascii', errors='xmlcharrefreplace')\n",
    ">>> print('Jalape\\u00f1o')\n",
    "Jalape&#241;o\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the non-ASCII character ñ has been replaced by `&#241;` in the output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.17 Writing Bytes to a Text File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to write raw bytes to a file opened in text mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simply write the byte data to the files underlying `buffer`."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">>> import sys\n",
    ">>> sys.stdout.write(b'Hello\\n')\n",
    "Traceback (most recent call last):\n",
    "    File \"<stdin>\", line 1, in <module>\n",
    "TypeError: must be str, not bytes\n",
    ">>> sys.stdout.buffer.write(b'Hello\\n')\n",
    "Hello\n",
    "5\n",
    ">>>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, binary data can be read from a text file by reading from its `buffer` attribute instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The I/O system is built from layers.  \n",
    "Text files are constructed by adding a Unicode encoding/decoding layer on top of a buffered binary-mode file.  \n",
    "The buffer attribute simply points at this underlying file.  \n",
    "If you access it, you’ll bypass the text encoding/decoding layer.  \n",
    "The example involving `sys.stdout` might be viewed as a special case.  \n",
    "By default, `sys.stdout` is always opened in text mode.  \n",
    "However, if you are writing a script that actually needs to dump binary data to standard output, you can use the technique shown to bypass the text encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.18 Wrapping an Existing File Descriptor as a File Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}