io_module/notebook_linux.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IO, CSV, JSON, Pickle and Shelve Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Input Output module ```io``` is used for reading and writing data to a file. The Comma Seperated Values module ```csv``` is used in conjunction with the input output module ```io``` for reading and writing data to a ```CSV``` file and other basic formats used for spreadsheets. The ```json``` module is used for reading and writing data from a JSON ```str``` to a Python ```dict```. The ```pickle``` module is used for serialising data and the ```shelve``` module is for storing serialised data in a database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The modules need to be imported before identifiers from them can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import csv\n",
    "import json\n",
    "import pickle\n",
    "import shelve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorize_Identifiers Module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will use the following functions ```dir2```, ```variables``` and ```view``` in the custom module ```categorize_identifiers``` which is found in the same directory as this notebook file. ```dir2``` is a variant of ```dir``` that groups identifiers into a ```dict``` under categories and ```variables``` is an IPython based a variable inspector. ```view``` is used to view a ```Collection``` in more detail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from categorize_identifiers import dir2, variables, view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Files (.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A text file called ```text.txt``` can be created in the same folder as the Interactive Python Notebook File:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Baa, baa, black sheep,\n",
    "Have you any wool?\n",
    "Yes, sir, yes, sir,\n",
    "Three bags full.\n",
    "\n",
    "One for my master,\n",
    "One for my dame,\n",
    "And one for the little boy\n",
    "Who lives down the lane.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text files can be viewed in Notepad++ with View → Show Symbol → Show All Characters:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_001.png' alt='img_001' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that there is a ```CRLF``` at the end of each line instructing to move onto the next row. This stands for carriage return and line feed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```open``` function in the ```io``` module is used for opening text and binary files. Its docstring can be viewed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mfile\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnewline\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mclosefd\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mopener\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Open file and return a stream.  Raise OSError upon failure.\n",
       "\n",
       "file is either a text or byte string giving the name (and the path\n",
       "if the file isn't in the current working directory) of the file to\n",
       "be opened or an integer file descriptor of the file to be\n",
       "wrapped. (If a file descriptor is given, it is closed when the\n",
       "returned I/O object is closed, unless closefd is set to False.)\n",
       "\n",
       "mode is an optional string that specifies the mode in which the file\n",
       "is opened. It defaults to 'r' which means open for reading in text\n",
       "mode.  Other common values are 'w' for writing (truncating the file if\n",
       "it already exists), 'x' for creating and writing to a new file, and\n",
       "'a' for appending (which on some Unix systems, means that all writes\n",
       "append to the end of the file regardless of the current seek position).\n",
       "In text mode, if encoding is not specified the encoding used is platform\n",
       "dependent: locale.getencoding() is called to get the current locale encoding.\n",
       "(For reading and writing raw bytes use binary mode and leave encoding\n",
       "unspecified.) The available modes are:\n",
       "\n",
       "========= ===============================================================\n",
       "Character Meaning\n",
       "--------- ---------------------------------------------------------------\n",
       "'r'       open for reading (default)\n",
       "'w'       open for writing, truncating the file first\n",
       "'x'       create a new file and open it for writing\n",
       "'a'       open for writing, appending to the end of the file if it exists\n",
       "'b'       binary mode\n",
       "'t'       text mode (default)\n",
       "'+'       open a disk file for updating (reading and writing)\n",
       "========= ===============================================================\n",
       "\n",
       "The default mode is 'rt' (open for reading text). For binary random\n",
       "access, the mode 'w+b' opens and truncates the file to 0 bytes, while\n",
       "'r+b' opens the file without truncation. The 'x' mode implies 'w' and\n",
       "raises an `FileExistsError` if the file already exists.\n",
       "\n",
       "Python distinguishes between files opened in binary and text modes,\n",
       "even when the underlying operating system doesn't. Files opened in\n",
       "binary mode (appending 'b' to the mode argument) return contents as\n",
       "bytes objects without any decoding. In text mode (the default, or when\n",
       "'t' is appended to the mode argument), the contents of the file are\n",
       "returned as strings, the bytes having been first decoded using a\n",
       "platform-dependent encoding or using the specified encoding if given.\n",
       "\n",
       "buffering is an optional integer used to set the buffering policy.\n",
       "Pass 0 to switch buffering off (only allowed in binary mode), 1 to select\n",
       "line buffering (only usable in text mode), and an integer > 1 to indicate\n",
       "the size of a fixed-size chunk buffer.  When no buffering argument is\n",
       "given, the default buffering policy works as follows:\n",
       "\n",
       "* Binary files are buffered in fixed-size chunks; the size of the buffer\n",
       "  is chosen using a heuristic trying to determine the underlying device's\n",
       "  \"block size\" and falling back on `io.DEFAULT_BUFFER_SIZE`.\n",
       "  On many systems, the buffer will typically be 4096 or 8192 bytes long.\n",
       "\n",
       "* \"Interactive\" text files (files for which isatty() returns True)\n",
       "  use line buffering.  Other text files use the policy described above\n",
       "  for binary files.\n",
       "\n",
       "encoding is the name of the encoding used to decode or encode the\n",
       "file. This should only be used in text mode. The default encoding is\n",
       "platform dependent, but any encoding supported by Python can be\n",
       "passed.  See the codecs module for the list of supported encodings.\n",
       "\n",
       "errors is an optional string that specifies how encoding errors are to\n",
       "be handled---this argument should not be used in binary mode. Pass\n",
       "'strict' to raise a ValueError exception if there is an encoding error\n",
       "(the default of None has the same effect), or pass 'ignore' to ignore\n",
       "errors. (Note that ignoring encoding errors can lead to data loss.)\n",
       "See the documentation for codecs.register or run 'help(codecs.Codec)'\n",
       "for a list of the permitted encoding error strings.\n",
       "\n",
       "newline controls how universal newlines works (it only applies to text\n",
       "mode). It can be None, '', '\\n', '\\r', and '\\r\\n'.  It works as\n",
       "follows:\n",
       "\n",
       "* On input, if newline is None, universal newlines mode is\n",
       "  enabled. Lines in the input can end in '\\n', '\\r', or '\\r\\n', and\n",
       "  these are translated into '\\n' before being returned to the\n",
       "  caller. If it is '', universal newline mode is enabled, but line\n",
       "  endings are returned to the caller untranslated. If it has any of\n",
       "  the other legal values, input lines are only terminated by the given\n",
       "  string, and the line ending is returned to the caller untranslated.\n",
       "\n",
       "* On output, if newline is None, any '\\n' characters written are\n",
       "  translated to the system default line separator, os.linesep. If\n",
       "  newline is '' or '\\n', no translation takes place. If newline is any\n",
       "  of the other legal values, any '\\n' characters written are translated\n",
       "  to the given string.\n",
       "\n",
       "If closefd is False, the underlying file descriptor will be kept open\n",
       "when the file is closed. This does not work when a file name is given\n",
       "and must be True in that case.\n",
       "\n",
       "A custom opener can be used by passing a callable as *opener*. The\n",
       "underlying file descriptor for the file object is then obtained by\n",
       "calling *opener* with (*file*, *flags*). *opener* must return an open\n",
       "file descriptor (passing os.open as *opener* results in functionality\n",
       "similar to passing None).\n",
       "\n",
       "open() returns a file object whose type depends on the mode, and\n",
       "through which the standard file operations such as reading and writing\n",
       "are performed. When open() is used to open a file in a text mode ('w',\n",
       "'r', 'wt', 'rt', etc.), it returns a TextIOWrapper. When used to open\n",
       "a file in a binary mode, the returned class varies: in read binary\n",
       "mode, it returns a BufferedReader; in write binary and append binary\n",
       "modes, it returns a BufferedWriter, and in read/write mode, it returns\n",
       "a BufferedRandom.\n",
       "\n",
       "It is also possible to use a string or bytearray as a file for both\n",
       "reading and writing. For strings StringIO can be used like a file\n",
       "opened in a text mode, and for bytes a BytesIO can be used like a file\n",
       "opened in a binary mode.\n",
       "\u001b[0;31mType:\u001b[0m      builtin_function_or_method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "io.open?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```ipython``` also includes this function in the global namespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mfile\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnewline\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mclosefd\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mopener\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Open file and return a stream.  Raise OSError upon failure.\n",
       "\n",
       "file is either a text or byte string giving the name (and the path\n",
       "if the file isn't in the current working directory) of the file to\n",
       "be opened or an integer file descriptor of the file to be\n",
       "wrapped. (If a file descriptor is given, it is closed when the\n",
       "returned I/O object is closed, unless closefd is set to False.)\n",
       "\n",
       "mode is an optional string that specifies the mode in which the file\n",
       "is opened. It defaults to 'r' which means open for reading in text\n",
       "mode.  Other common values are 'w' for writing (truncating the file if\n",
       "it already exists), 'x' for creating and writing to a new file, and\n",
       "'a' for appending (which on some Unix systems, means that all writes\n",
       "append to the end of the file regardless of the current seek position).\n",
       "In text mode, if encoding is not specified the encoding used is platform\n",
       "dependent: locale.getencoding() is called to get the current locale encoding.\n",
       "(For reading and writing raw bytes use binary mode and leave encoding\n",
       "unspecified.) The available modes are:\n",
       "\n",
       "========= ===============================================================\n",
       "Character Meaning\n",
       "--------- ---------------------------------------------------------------\n",
       "'r'       open for reading (default)\n",
       "'w'       open for writing, truncating the file first\n",
       "'x'       create a new file and open it for writing\n",
       "'a'       open for writing, appending to the end of the file if it exists\n",
       "'b'       binary mode\n",
       "'t'       text mode (default)\n",
       "'+'       open a disk file for updating (reading and writing)\n",
       "========= ===============================================================\n",
       "\n",
       "The default mode is 'rt' (open for reading text). For binary random\n",
       "access, the mode 'w+b' opens and truncates the file to 0 bytes, while\n",
       "'r+b' opens the file without truncation. The 'x' mode implies 'w' and\n",
       "raises an `FileExistsError` if the file already exists.\n",
       "\n",
       "Python distinguishes between files opened in binary and text modes,\n",
       "even when the underlying operating system doesn't. Files opened in\n",
       "binary mode (appending 'b' to the mode argument) return contents as\n",
       "bytes objects without any decoding. In text mode (the default, or when\n",
       "'t' is appended to the mode argument), the contents of the file are\n",
       "returned as strings, the bytes having been first decoded using a\n",
       "platform-dependent encoding or using the specified encoding if given.\n",
       "\n",
       "buffering is an optional integer used to set the buffering policy.\n",
       "Pass 0 to switch buffering off (only allowed in binary mode), 1 to select\n",
       "line buffering (only usable in text mode), and an integer > 1 to indicate\n",
       "the size of a fixed-size chunk buffer.  When no buffering argument is\n",
       "given, the default buffering policy works as follows:\n",
       "\n",
       "* Binary files are buffered in fixed-size chunks; the size of the buffer\n",
       "  is chosen using a heuristic trying to determine the underlying device's\n",
       "  \"block size\" and falling back on `io.DEFAULT_BUFFER_SIZE`.\n",
       "  On many systems, the buffer will typically be 4096 or 8192 bytes long.\n",
       "\n",
       "* \"Interactive\" text files (files for which isatty() returns True)\n",
       "  use line buffering.  Other text files use the policy described above\n",
       "  for binary files.\n",
       "\n",
       "encoding is the name of the encoding used to decode or encode the\n",
       "file. This should only be used in text mode. The default encoding is\n",
       "platform dependent, but any encoding supported by Python can be\n",
       "passed.  See the codecs module for the list of supported encodings.\n",
       "\n",
       "errors is an optional string that specifies how encoding errors are to\n",
       "be handled---this argument should not be used in binary mode. Pass\n",
       "'strict' to raise a ValueError exception if there is an encoding error\n",
       "(the default of None has the same effect), or pass 'ignore' to ignore\n",
       "errors. (Note that ignoring encoding errors can lead to data loss.)\n",
       "See the documentation for codecs.register or run 'help(codecs.Codec)'\n",
       "for a list of the permitted encoding error strings.\n",
       "\n",
       "newline controls how universal newlines works (it only applies to text\n",
       "mode). It can be None, '', '\\n', '\\r', and '\\r\\n'.  It works as\n",
       "follows:\n",
       "\n",
       "* On input, if newline is None, universal newlines mode is\n",
       "  enabled. Lines in the input can end in '\\n', '\\r', or '\\r\\n', and\n",
       "  these are translated into '\\n' before being returned to the\n",
       "  caller. If it is '', universal newline mode is enabled, but line\n",
       "  endings are returned to the caller untranslated. If it has any of\n",
       "  the other legal values, input lines are only terminated by the given\n",
       "  string, and the line ending is returned to the caller untranslated.\n",
       "\n",
       "* On output, if newline is None, any '\\n' characters written are\n",
       "  translated to the system default line separator, os.linesep. If\n",
       "  newline is '' or '\\n', no translation takes place. If newline is any\n",
       "  of the other legal values, any '\\n' characters written are translated\n",
       "  to the given string.\n",
       "\n",
       "If closefd is False, the underlying file descriptor will be kept open\n",
       "when the file is closed. This does not work when a file name is given\n",
       "and must be True in that case.\n",
       "\n",
       "A custom opener can be used by passing a callable as *opener*. The\n",
       "underlying file descriptor for the file object is then obtained by\n",
       "calling *opener* with (*file*, *flags*). *opener* must return an open\n",
       "file descriptor (passing os.open as *opener* results in functionality\n",
       "similar to passing None).\n",
       "\n",
       "open() returns a file object whose type depends on the mode, and\n",
       "through which the standard file operations such as reading and writing\n",
       "are performed. When open() is used to open a file in a text mode ('w',\n",
       "'r', 'wt', 'rt', etc.), it returns a TextIOWrapper. When used to open\n",
       "a file in a binary mode, the returned class varies: in read binary\n",
       "mode, it returns a BufferedReader; in write binary and append binary\n",
       "modes, it returns a BufferedWriter, and in read/write mode, it returns\n",
       "a BufferedRandom.\n",
       "\n",
       "It is also possible to use a string or bytearray as a file for both\n",
       "reading and writing. For strings StringIO can be used like a file\n",
       "opened in a text mode, and for bytes a BytesIO can be used like a file\n",
       "opened in a binary mode.\n",
       "\u001b[0;31mType:\u001b[0m      function"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "open?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```open``` function requires a file which can be specified directly when it is in the same folder as the interactive Python notebook file (or Python script file).\n",
    "\n",
    "If the file for example ```text.txt``` is in the same folder as the notebook it can be specified using:\n",
    "\n",
    "```python\n",
    "'text.txt'\n",
    "```\n",
    "\n",
    "Linux uses ```/``` as a file separator. This is also recognised as an alternative file seperator in Windows so the file can be selected using:\n",
    "\n",
    "```python\n",
    "'./text.txt'\n",
    "```\n",
    "\n",
    "If the file is in a parent folder this can be specified using ```../``` for example:\n",
    "\n",
    "```python\n",
    "'../text.txt'\n",
    "```\n",
    "\n",
    "If present in a subfolder on the same level as the notebook this can be specified using:\n",
    "\n",
    "```python\n",
    "'./files/text.txt'\n",
    "```\n",
    "\n",
    "A ```files_input``` and ```files_output``` subfolder will be used in this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```mode``` keyword input argument can be specified using a single letter:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|mode|definition|\n",
    "|---|---|\n",
    "|'r'|open an existing file and read existing content|\n",
    "|'w'|open an existing file and write over existing content|\n",
    "|'a'|open an existing file and append new content|\n",
    "|'x'|create a new file and write new content|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```encoding``` keyword argument is used to specify the encoding, which recall was discussed in detail when the ```bytes``` class examined in a previous notebook. The encoding has a default value of ```'utf-8'``` but if the data was processed with a Microsoft Product may require ```'utf-8-sig'``` in order to remove an unwanted BOM. To recap:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|encoding|bytes per character|bits per character|byte order|byte order marker BOM|\n",
    "|---|---|---|---|---|\n",
    "|'utf-8'|1, 2, 3, 4|8, 16, 24, 32|big endian| |\n",
    "|'utf-8-sig'|1, 2, 3, 4|8, 16, 24, 32|big endian|efbbbf|\n",
    "|'utf-32'|4|32|little endian|fffe0000|\n",
    "|'utf-32-le'|4|32|little endian| |\n",
    "|'utf-32-be'|4|32|big endian| |\n",
    "|'utf-16'|2|16|little endian|fffe|\n",
    "|'utf-16-le'|2|16|little endian| |\n",
    "|'utf-16-be'|2|16|big endian| |\n",
    "|'latin1'|1|8| ||\n",
    "|'ascii'|1|8| ||"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```newline``` keyword input argument can be used to specify the character that is used to represent a new line.\n",
    "\n",
    "On Linux this is normally just the new line character escape character ```'\\n'```. \n",
    "\n",
    "On Windows two escape characters carriage return and new line are used ```'\\n'```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```errors``` keyword input argument is used to handle errors, normally due to encoding issues and are set to ```'strict'``` by default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A file can be opened:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = open('./files_input/text.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the hood, this is actually the initialisation signature of the ```TextIOWrapper``` class and a new instance is created with the instance name ```file```. This can be seen when the datatype of ```file``` is checked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_io.TextIOWrapper"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```_io``` indicates that this class is from the ```io``` module. The prefix with an underscore means the module is internally being used here. The ```open``` function is essentially equivalent to the initialisation method of this class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m\n",
       "\u001b[0mio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextIOWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbuffer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnewline\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mline_buffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mwrite_through\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m     \n",
       "Character and line based layer over a BufferedIOBase object, buffer.\n",
       "\n",
       "encoding gives the name of the encoding that the stream will be\n",
       "decoded or encoded with. It defaults to locale.getencoding().\n",
       "\n",
       "errors determines the strictness of encoding and decoding (see\n",
       "help(codecs.Codec) or the documentation for codecs.register) and\n",
       "defaults to \"strict\".\n",
       "\n",
       "newline controls how line endings are handled. It can be None, '',\n",
       "'\\n', '\\r', and '\\r\\n'.  It works as follows:\n",
       "\n",
       "* On input, if newline is None, universal newlines mode is\n",
       "  enabled. Lines in the input can end in '\\n', '\\r', or '\\r\\n', and\n",
       "  these are translated into '\\n' before being returned to the\n",
       "  caller. If it is '', universal newline mode is enabled, but line\n",
       "  endings are returned to the caller untranslated. If it has any of\n",
       "  the other legal values, input lines are only terminated by the given\n",
       "  string, and the line ending is returned to the caller untranslated.\n",
       "\n",
       "* On output, if newline is None, any '\\n' characters written are\n",
       "  translated to the system default line separator, os.linesep. If\n",
       "  newline is '' or '\\n', no translation takes place. If newline is any\n",
       "  of the other legal values, any '\\n' characters written are translated\n",
       "  to the given string.\n",
       "\n",
       "If line_buffering is True, a call to flush is implied when a call to\n",
       "write contains a newline character.\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     UnclosingTextIOWrapper"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "io.TextIOWrapper?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be read from the ```TextIOWrapper``` instance for example using the files ```readlines``` method returning a list of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Baa, baa, black sheep,\\r\\n',\n",
       " 'Have you any wool?\\r\\n',\n",
       " 'Yes, sir, yes, sir,\\r\\n',\n",
       " 'Three bags full.\\r\\n',\n",
       " '\\r\\n',\n",
       " 'One for my master,\\r\\n',\n",
       " 'One for my dame,\\r\\n',\n",
       " 'And one for the little boy\\r\\n',\n",
       " 'Who lives down the lane.']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After working with a file, it should be closed. The file can be closed using the ```TextIOWrapper``` close method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```TextIOWrapper``` class has a number of attributes, most of these correspond to the input arguments provided when initialising the instance.\n",
    "\n",
    "The ```TextIOWrapper``` method ```readable``` will check whether a file is readable returning a boolean. The method ```read``` will read the entire file as a Unicode string, the method ```readline``` will read an individual line as a string and then advance, while the method ```readlines``` will read every line returning a list of Unicode strings corresponding to each line.\n",
    "\n",
    "The methods ```writable```, ```write``` and ```writelines``` are the write counterparts.\n",
    "\n",
    "The methods ```seekable``` and ```seek``` relate to the cursor position.\n",
    "\n",
    "The ```TextIOWrapper``` class has the additional datamodel methods ```__enter__``` (*dunder enter*) and ```__exit__``` (*dunder exit*) which are used when accessing the file with a ```with``` code block: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'attribute': ['buffer',\n",
      "               'closed',\n",
      "               'encoding',\n",
      "               'errors',\n",
      "               'line_buffering',\n",
      "               'mode',\n",
      "               'name',\n",
      "               'newlines',\n",
      "               'write_through'],\n",
      " 'method': ['close',\n",
      "            'detach',\n",
      "            'fileno',\n",
      "            'flush',\n",
      "            'isatty',\n",
      "            'read',\n",
      "            'readable',\n",
      "            'readline',\n",
      "            'readlines',\n",
      "            'reconfigure',\n",
      "            'seek',\n",
      "            'seekable',\n",
      "            'tell',\n",
      "            'truncate',\n",
      "            'writable',\n",
      "            'write',\n",
      "            'writelines'],\n",
      " 'datamodel_attribute': ['__dict__', '__module__'],\n",
      " 'datamodel_method': ['__del__',\n",
      "                      '__enter__',\n",
      "                      '__exit__',\n",
      "                      '__iter__',\n",
      "                      '__next__'],\n",
      " 'internal_attribute': ['_CHUNK_SIZE', '_finalizing'],\n",
      " 'internal_method': ['_checkClosed',\n",
      "                     '_checkReadable',\n",
      "                     '_checkSeekable',\n",
      "                     '_checkWritable']}\n"
     ]
    }
   ],
   "source": [
    "dir2(file, object, unique_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With Code Block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datamodel methods ```__enter__``` (*dunder enter*) and ```__exit__``` (*dunder exit*) are used by a ```with``` code block to open the file when the code block begins and close the file when the code block is exited respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./files_input/text.txt\n",
      "r\n",
      "utf-8\n",
      "strict\n",
      "readable:  True\n",
      "writeable:  False\n",
      "seekable:  True\n"
     ]
    }
   ],
   "source": [
    "with open('./files_input/text.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    print(file.name)\n",
    "    print(file.mode)\n",
    "    print(file.encoding)\n",
    "    print(file.errors)\n",
    "    print('readable: ', file.readable())\n",
    "    print('writeable: ', file.writable())\n",
    "    print('seekable: ', file.seekable())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```TextIOWrapper``` method ```read``` can be used to read the entire contents of the text file as a single string, notice that this includes carriage returns and new line escape characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_input/text.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data1 = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Baa, baa, black sheep,\\r\\nHave you any wool?\\r\\nYes, sir, yes, sir,\\r\\nThree bags full.\\r\\n\\r\\nOne for my master,\\r\\nOne for my dame,\\r\\nAnd one for the little boy\\r\\nWho lives down the lane.'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When printed this gives a similar display to the file opened in Notepad:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baa, baa, black sheep,\n",
      "Have you any wool?\n",
      "Yes, sir, yes, sir,\n",
      "Three bags full.\n",
      "\n",
      "One for my master,\n",
      "One for my dame,\n",
      "And one for the little boy\n",
      "Who lives down the lane.\n"
     ]
    }
   ],
   "source": [
    "print(data1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```TextIOWrapper``` method ```readline``` will only read a single line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_input/text.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data2 = file.readline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the end of this string includes the carriage return and new line escape characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Baa, baa, black sheep,\\r\\n'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These whitespace characters can be stripped using the string method ```strip```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Baa, baa, black sheep,'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```TextIOWrapper``` method ```readlines``` will instead output a list of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_input/text.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data3 = file.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the square brackets ```[]``` enclosing the list and comma delimiter between each line. Each line also includes the carriage return and newline character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Baa, baa, black sheep,\\r\\n',\n",
       " 'Have you any wool?\\r\\n',\n",
       " 'Yes, sir, yes, sir,\\r\\n',\n",
       " 'Three bags full.\\r\\n',\n",
       " '\\r\\n',\n",
       " 'One for my master,\\r\\n',\n",
       " 'One for my dame,\\r\\n',\n",
       " 'And one for the little boy\\r\\n',\n",
       " 'Who lives down the lane.']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These can be removed using a list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data4 = [line.strip() for line in data3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Baa, baa, black sheep,',\n",
       " 'Have you any wool?',\n",
       " 'Yes, sir, yes, sir,',\n",
       " 'Three bags full.',\n",
       " '',\n",
       " 'One for my master,',\n",
       " 'One for my dame,',\n",
       " 'And one for the little boy',\n",
       " 'Who lives down the lane.']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The length of the single string obtained using the ```TextIOWrapper``` method ```read``` is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "175"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Baa, baa, black sheep,\\r\\nHave you any wool?\\r\\nYes, sir, yes, sir,\\r\\nThree bags full.\\r\\n\\r\\nOne for my master,\\r\\nOne for my dame,\\r\\nAnd one for the little boy\\r\\nWho lives down the lane.'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 5th character and onwards can be seen by indexing into the string using the slice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'baa, black sheep,\\r\\nHave you any wool?\\r\\nYes, sir, yes, sir,\\r\\nThree bags full.\\r\\n\\r\\nOne for my master,\\r\\nOne for my dame,\\r\\nAnd one for the little boy\\r\\nWho lives down the lane.'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1[5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each character in the file also has a zero-ordered numeric index which the cursor can be placed at using the ```TextIOWrapper``` method ```seek```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_input/text.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    file.seek(5)\n",
    "    data5 = file.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives a similar result to slicing of the string as seen above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'baa, black sheep,\\r\\nHave you any wool?\\r\\nYes, sir, yes, sir,\\r\\nThree bags full.\\r\\n\\r\\nOne for my master,\\r\\nOne for my dame,\\r\\nAnd one for the little boy\\r\\nWho lives down the lane.'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the length of the string is ```175 - 5```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "170"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```TextIOWrapper``` method ```write``` can be used to write text to a file, note that this file is opened with ```mode='w'```. \n",
    "\n",
    "On Linux the keyword input argument should be ```newline='\\n'``` and the ```\\n``` should be incorporated in a string for a new line.\n",
    "\n",
    "On Windows the keyword input argument should be ```newline='\\n'``` however ```\\r\\n``` should be incorporated in a string for a new line. Each ```\\n``` used in the ```write``` method will be converted into ```\\n```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/text2.txt', mode='w', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    file.write('Hello World!\\nBye World!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file can then be read using the ```TextIOWrapper``` method ```read```, note that this file is opened with ```mode='r'``` and ```newline='\\n'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/text2.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data6 = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello World!\\nBye World!'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method ```writelines``` can be used to write a list of strings to a file. Note once again that this file is opened with ```mode='w'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello World!\\nBye World!\\n'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines = ['Hello World!', 'Bye World!']\n",
    "\n",
    "with open('./files_output/text4.txt', mode='w', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    file.writelines([line + '\\n' for line in lines])\n",
    "\n",
    "with open('./files_output/text4.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data8 = file.read()\n",
    "    \n",
    "data8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Append"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When ```mode='w'``` any content in an existing file is removed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'J'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_output/text4.txt', mode='w', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    file.write('J')\n",
    "\n",
    "with open('./files_output/text4.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data9 = file.read()\n",
    "    \n",
    "data9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When ```mode='a'``` any content in an existing file is instead appended:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello World!\\nBye World!Appended'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_output/text2.txt', mode='a', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    file.write('Appended')\n",
    "\n",
    "with open('./files_output/text2.txt', mode='r', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data10 = file.read()\n",
    "    \n",
    "data10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three variations of ```mode``` were seen ```'r'```, ```'w'``` and ```'a'``` and and have the alias ```'rt'```, ```'wt'``` and ```'at'``` indicating that this function is being used on a text file and a text ```.txt``` file uses a Unicode string with ```'utf-8'``` encoding by default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bin Files (.bin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```'rb'```, ```'wb'``` and ```'ab'``` instead indicate that this function is being used on a binary file and a binary ```.bin``` file uses a byte string. There is no ```encoding```, ```errors```, ```newline``` as raw ```bytes``` are used. When in binary mode, all the ```TextIOWrapper``` methods expect a byte string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Hello World!\\r\\nHello'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_output/text5.bin', mode='wb') as file:\n",
    "    file.write(b'\\x48\\x65\\x6c\\x6c\\x6f\\x20\\x57\\x6f\\x72\\x6c\\x64\\x21\\x0d\\x0a\\x48\\x65\\x6c\\x6c\\x6f')\n",
    "\n",
    "with open('./files_output/text5.bin', mode='rb') as file:\n",
    "    data11 = file.read()\n",
    "    \n",
    "data11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The binary file can be read into text mode if the correct encoding is supplied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Hello World!\\r\\n', 'Hello']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_output/text5.bin', mode='rt', encoding='utf-8', errors='strict', newline='\\n') as file:\n",
    "    data12 = file.readlines()\n",
    "    \n",
    "data12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comma Separated Values Files (.csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that a text file uses a ```CRLF``` at the end of each line instructing to move onto the next row. Notice all the data looks like a column:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_002.png' alt='img_002' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A comma seperated values files uses a comma to move onto the next column:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_003.png' alt='img_003' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a csv file is opened in a spreadsheet editor such as Excel or OnlyOffice Desktop Editors SpreadSheet, these are used to construct a grid:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_004.png' alt='img_004' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```readlines``` method of the ```TextIOWrapper``` instance can be used to read the data as a list of strings where each string corresponds to a row of text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['x,y\\r\\n', '0,0\\r\\n', '1,2\\r\\n', '2,4\\r\\n', '3,6\\r\\n', '4,8\\r\\n', '5,10\\r\\n']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_input/sheet1.csv', mode='r', encoding='utf-8', newline='\\n') as file:\n",
    "    data = file.readlines()\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Byte Order Marker (BOM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the 1st entry has the prefix ```\\ufeff``` then the file is saved using UTF-8 with BOM instead of UTF-8:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\ufeffx,y\\r\\n',\n",
       " '0,0\\r\\n',\n",
       " '1,2\\r\\n',\n",
       " '2,4\\r\\n',\n",
       " '3,6\\r\\n',\n",
       " '4,8\\r\\n',\n",
       " '5,10\\r\\n']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_input/sheet2.csv', mode='r', encoding='utf-8', newline='\\n') as file:\n",
    "    data = file.readlines()\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To handle the BOM, the encoding can be changed from ```'utf-8'``` to ```'utf-8-sig'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['x,y\\r\\n', '0,0\\r\\n', '1,2\\r\\n', '2,4\\r\\n', '3,6\\r\\n', '4,8\\r\\n', '5,10\\r\\n']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_input/sheet2.csv', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    data = file.readlines()\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV Reader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```reader``` function of the ```csv``` module can be used within the ```with``` code block created using the ```open``` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_input/sheet1.csv', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=',', quotechar='\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates an instance of the ```reader``` class from the ```csv``` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_csv.reader"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(csv_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its identifiers can be viewed. The datamodel identifier ```__next__``` (*dunder next*) is defined meaning the ```reader``` class is essentially an iterator. When ```next``` is used on this iterator it returns the current row as a list and each element in the list corresponds to a column. The attribute ```line_num``` corresponds to the zero-order indexed line number of the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'attribute': ['dialect', 'line_num'],\n",
      " 'datamodel_attribute': ['__module__'],\n",
      " 'datamodel_method': ['__iter__', '__next__']}\n"
     ]
    }
   ],
   "source": [
    "dir2(csv_file, object, unique_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```for``` loop can be used to create a ```dict``` instance where the keys correspond to the line numbers and the value is the data for each line in the form of a ```list```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_data = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['x', 'y'],\n",
       " 'row_2': ['0', '0'],\n",
       " 'row_3': ['1', '2'],\n",
       " 'row_4': ['2', '4'],\n",
       " 'row_5': ['3', '6'],\n",
       " 'row_6': ['4', '8'],\n",
       " 'row_7': ['5', '10']}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_input/sheet1.csv', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=',', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the relevent data can be indexed using the ```key``` which returns the row as a ```list```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '0']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data['row_2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this ```list``` can be indexed to select a column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data['row_2'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this is read in as a ```str``` instance and can be cast into an ```int``` using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int(csv_data['row_2'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV Writer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```writer``` function of the ```csv``` module can be used within the ```with``` code block created using the ```open``` function when the file is opened in write mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/sheet3.csv', mode='w', encoding='utf-8', newline='\\n') as file:\n",
    "    csv_file = csv.writer(file, delimiter=',', quotechar='\"') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates an instance of the ```writer``` class from the ```csv``` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_csv.writer"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(csv_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its identifiers can be viewed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'attribute': ['dialect'],\n",
      " 'method': ['writerow', 'writerows'],\n",
      " 'datamodel_attribute': ['__module__']}\n"
     ]
    }
   ],
   "source": [
    "dir2(csv_file, object, unique_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```writerow``` method can be used to write each row individually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/sheet3.csv', mode='w', encoding='utf-8', newline='\\n') as file:\n",
    "    csv_file = csv.writer(file, delimiter=',', quotechar='\"')\n",
    "    csv_file.writerow(['x', 'y', 'z'])\n",
    "    csv_file.writerow([1, 2, 3])\n",
    "    csv_file.writerow([2, 4, 6])    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be read into a ```dict``` instance using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['x', 'y', 'z'], 'row_2': ['1', '2', '3'], 'row_3': ['2', '4', '6']}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_output/sheet3.csv', mode='r', encoding='utf-8', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=',', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method ```writerows``` method can be used to write the data using a ```list``` of ```list``` instances. Each nested ```list``` instance is a row: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/sheet4.csv', mode='w', encoding='utf-8', newline='\\n') as file:\n",
    "    csv_file = csv.writer(file, delimiter=',', quotechar='\"')\n",
    "    csv_file.writerows([['w', 'x', 'y', 'z'],\n",
    "                        [0, 1, 2, 3],\n",
    "                        [0, 2, 4, 6]])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['w', 'x', 'y', 'z'],\n",
       " 'row_2': ['0', '1', '2', '3'],\n",
       " 'row_3': ['0', '2', '4', '6']}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_output/sheet4.csv', mode='r', encoding='utf-8', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=',', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```,``` is a special character in a CSV file. When it is incorporated into a cell within a CSV file. The cell contents have to be enclosed in double quotations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_005.png' alt='img_005' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file can be read using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['libraries', 'version'],\n",
       " 'row_2': ['python', '3,11,5'],\n",
       " 'row_3': ['numpy', '1,26,0'],\n",
       " 'row_4': ['pandas', '2,1,1'],\n",
       " 'row_5': ['matplotlib', '3,8,0'],\n",
       " 'row_6': ['seaborn', '0,13,0']}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_input/sheet5.csv', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=',', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### European CSV Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some European languages will use the semicolon as a delimiter instead of a colon:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_006.png' alt='img_006' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the CSV is read in using the wrong delimiter, each row will be read in as a single column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['libraries;version'],\n",
       " 'row_2': ['python;3', '11', '5'],\n",
       " 'row_3': ['numpy;1', '26', '0'],\n",
       " 'row_4': ['pandas;2', '1', '1'],\n",
       " 'row_5': ['matplotlib;3', '8', '0'],\n",
       " 'row_6': ['seaborn;0', '13', '0']}"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_input/sheet6.csv', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=',', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be fixed by specifying the correct delimiter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['libraries', 'version'],\n",
       " 'row_2': ['python', '3,11,5'],\n",
       " 'row_3': ['numpy', '1,26,0'],\n",
       " 'row_4': ['pandas', '2,1,1'],\n",
       " 'row_5': ['matplotlib', '3,8,0'],\n",
       " 'row_6': ['seaborn', '0,13,0']}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_input/sheet6.csv', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter=';', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Printer File (.prn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another file format is the print file format which uses a variable number of spaces to visually separate out the data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_007.png' alt='img_007' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When this is read in, each row is read in as a ```list``` containing a single ```str```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['name    x       y'],\n",
       " 'row_2': ['zero zer       0       0'],\n",
       " 'row_3': ['one two        1       2'],\n",
       " 'row_4': ['two four       2       4'],\n",
       " 'row_5': ['three si       3       6'],\n",
       " 'row_6': ['four eig       4       8'],\n",
       " 'row_7': ['five ten       5      10']}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_input/sheet7.prn', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter='\\t', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The string instance can be accessed from the ```row``` and the ```str``` method ```split``` can be used to split the data into columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['name', 'x', 'y'],\n",
       " 'row_2': ['zero zer', '0', '0'],\n",
       " 'row_3': ['one two', '1', '2'],\n",
       " 'row_4': ['two four', '2', '4'],\n",
       " 'row_5': ['three si', '3', '6'],\n",
       " 'row_6': ['four eig', '4', '8'],\n",
       " 'row_7': ['five ten', '5', '10']}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "csv_data = {}\n",
    "\n",
    "with open('./files_input/sheet7.prn', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter='\\t', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = re.split(r'\\s{2,}', row[0])\n",
    "        \n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Files Tab Delimited (.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tab limited text file uses a tab as a delimiter:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_008.png' alt='img_008' width='500'/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'row_1': ['x', 'y'],\n",
       " 'row_2': ['0', '0'],\n",
       " 'row_3': ['1', '2'],\n",
       " 'row_4': ['2', '4'],\n",
       " 'row_5': ['3', '6'],\n",
       " 'row_6': ['4', '8'],\n",
       " 'row_7': ['5', '10']}"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = {}\n",
    "\n",
    "with open('./files_input/sheet8.txt', mode='r', encoding='utf-8-sig', newline='\\n') as file:\n",
    "    csv_file = csv.reader(file, delimiter='\\t', quotechar='\"')\n",
    "    for row in csv_file:\n",
    "        csv_data[f'row_{csv_file.line_num}'] = row\n",
    "\n",
    "csv_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON Files (.json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON is very similar to a ```str``` of a Python ```dict``` instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapping = {True: {'red': (1, 0, 0),\n",
    "                  'green': (0, 0.5, 0),\n",
    "                  'blue': (0, 0, 1)},\n",
    "           False: {'black': (0, 0, 0)}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"{True: {'red': (1, 0, 0), 'green': (0, 0.5, 0), 'blue': (0, 0, 1)}, False: {'black': (0, 0, 0)}}\""
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JavaScript Object Notation (JSON) as the name suggests originally comes from JavaScript and has became a very popular format for storing data particularly on websites. Because it is from a different programming language, there are some subtle differences to a ```str``` of a Python ```dict```.\n",
    "\n",
    "The ```dumps``` function can be used to dump a Python ```object``` for example the ```dict``` instance ```mapping``` to a JSON ```str```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_string = json.dumps(mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"true\": {\"red\": [1, 0, 0], \"green\": [0, 0.5, 0], \"blue\": [0, 0, 1]}, \"false\": {\"black\": [0, 0, 0]}}'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON uses the double quotations. There the outer quotations that indicate a Python ```str``` instance are single and all the ```str``` instances that were in the Python ```dict``` instance are now enclosed in double quotations. \n",
    "\n",
    "JSON does not have a counterpart to a ```tuple``` and therefore the ```tuple``` instances that were present in the Python ```dict``` have in essence been cast to ```list``` instances. \n",
    "\n",
    "The representation of an ```int``` or ```float``` is unchanged.\n",
    "\n",
    "JSON uses a lowercase ```bool``` cast to a ```str``` instance for a ```bool```.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```loads``` function can be used to load a JSON ```str``` to a Python ```dict```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapping2 = json.loads(json_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'true': {'red': [1, 0, 0], 'green': [0, 0.5, 0], 'blue': [0, 0, 1]},\n",
       " 'false': {'black': [0, 0, 0]}}"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a similar form to before but the ```tuple``` instances are ```list``` instances because as previously mentioned ```JSON``` has no counterpart to a ```tuple```. The boolean values are now lowercase ```str``` instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because this JSON ```str``` is now a Python ```dict``` containing ```str```, ```list```, ```int```, ```float``` and ```dict``` instances, it can be accessed by indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'red': [1, 0, 0], 'green': [0, 0.5, 0], 'blue': [0, 0, 1]}"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping2['true']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 0, 0]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping2['true']['red']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```dump``` function can be used to dump a Python ```object``` to a ```.json``` file. The ```open``` function from ```io``` is used to open the file within a ```with``` code block in write mode. The ```dump``` function from ```json``` is used within this ```with``` code block and the file is closed after the ```with``` code block exits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open('./files_output/mapping.json', \"w\") as file:\n",
    "    json.dump(mapping, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```load``` function can be used to read the Python ```object``` from the ```.json``` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/mapping.json', \"r\") as file:\n",
    "    data = json.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```io```, ```csv```, ```json``` are used for basic reading and writing of data and it is important to understand the file formats above. For most cases, spreadsheets are read into a ```pd.DataFrame``` instance using the ```pd.read_csv``` option. The ```pd.DataFrame``` class is a datastructure that is very similar to a spreadsheet and the ```pandas``` library is based around this data structure. More details about ```pandas``` will be given in later tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pickle Files (.pkl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the file formats see before are high level which are human readable. There are some additional common file formats that are machine readible. These file formats normally use a ```bytes``` instance. Recall that the fundamental unit in a ```byte``` looks like the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_009.png' alt='img_009' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each bit in a ```byte``` is either ```0``` which can be represented as LOW in a digital voltage or ```1``` which can be represented as HIGH in a digital voltage:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_010.png' alt='img_010' width='500'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example time trace such as that shown above is normally transmitted over pin 3 of a Serial Port:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_011.png' alt='img_011' width='350'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Serial ports have fixed baud rates of 110, 300, 1200, 2400, 4800, 9600, 19200, 38400, 57600, or 115200 bits per second. \n",
    "\n",
    "If a baud rate of 9600 is used as an example. This means the digital trace for a bit will last:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'104.17 µs'"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f'{round(1/9600*1e6, 2)} µs'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for a byte, which is 8 bits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'833.33 µs'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f'{round(8/9600*1e6, 2)} µs'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The single ```byte``` above can be expressed in binary as: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0b01101000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This casts it to a decimal. The binary system is better at the machine level but is not human readable. Decimal is human readable but is harder to visualise what is going on at the hardware level. Therefore the hexadecimal system is commonly employed which splits a ```bytes``` up into two halfs and represents each half with a hexadecimal character:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_012.png' alt='img_012' width='500'/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0x68'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hex(104)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above concepts were covered when the ```bytes``` class was discussed. The ```pickle``` module is used to serialise a Python ```object``` into ```bytes``` and because the ```bytes``` instance isn't very human readable it is \"pickled\". The identifiers of the ```pickle``` module can be examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'attribute': ['bytes_types',\n",
      "               'compatible_formats',\n",
      "               'dispatch_table',\n",
      "               'format_version',\n",
      "               'maxsize'],\n",
      " 'constant': ['ADDITEMS',\n",
      "              'APPEND',\n",
      "              'APPENDS',\n",
      "              'BINBYTES',\n",
      "              'BINBYTES8',\n",
      "              'BINFLOAT',\n",
      "              'BINGET',\n",
      "              'BININT',\n",
      "              'BININT1',\n",
      "              'BININT2',\n",
      "              'BINPERSID',\n",
      "              'BINPUT',\n",
      "              'BINSTRING',\n",
      "              'BINUNICODE',\n",
      "              'BINUNICODE8',\n",
      "              'BUILD',\n",
      "              'BYTEARRAY8',\n",
      "              'DEFAULT_PROTOCOL',\n",
      "              'DICT',\n",
      "              'DUP',\n",
      "              'EMPTY_DICT',\n",
      "              'EMPTY_LIST',\n",
      "              'EMPTY_SET',\n",
      "              'EMPTY_TUPLE',\n",
      "              'EXT1',\n",
      "              'EXT2',\n",
      "              'EXT4',\n",
      "              'FALSE',\n",
      "              'FLOAT',\n",
      "              'FRAME',\n",
      "              'FROZENSET',\n",
      "              'GET',\n",
      "              'GLOBAL',\n",
      "              'HIGHEST_PROTOCOL',\n",
      "              'INST',\n",
      "              'INT',\n",
      "              'LIST',\n",
      "              'LONG',\n",
      "              'LONG1',\n",
      "              'LONG4',\n",
      "              'LONG_BINGET',\n",
      "              'LONG_BINPUT',\n",
      "              'MARK',\n",
      "              'MEMOIZE',\n",
      "              'NEWFALSE',\n",
      "              'NEWOBJ',\n",
      "              'NEWOBJ_EX',\n",
      "              'NEWTRUE',\n",
      "              'NEXT_BUFFER',\n",
      "              'NONE',\n",
      "              'OBJ',\n",
      "              'PERSID',\n",
      "              'POP',\n",
      "              'POP_MARK',\n",
      "              'PROTO',\n",
      "              'PUT',\n",
      "              'READONLY_BUFFER',\n",
      "              'REDUCE',\n",
      "              'SETITEM',\n",
      "              'SETITEMS',\n",
      "              'SHORT_BINBYTES',\n",
      "              'SHORT_BINSTRING',\n",
      "              'SHORT_BINUNICODE',\n",
      "              'STACK_GLOBAL',\n",
      "              'STOP',\n",
      "              'STRING',\n",
      "              'TRUE',\n",
      "              'TUPLE',\n",
      "              'TUPLE1',\n",
      "              'TUPLE2',\n",
      "              'TUPLE3',\n",
      "              'UNICODE'],\n",
      " 'module': ['codecs', 'io', 're', 'sys'],\n",
      " 'method': ['decode_long',\n",
      "            'dump',\n",
      "            'dumps',\n",
      "            'encode_long',\n",
      "            'load',\n",
      "            'loads',\n",
      "            'pack',\n",
      "            'unpack',\n",
      "            'whichmodule'],\n",
      " 'lower_class': ['islice', 'partial'],\n",
      " 'upper_class': ['FunctionType',\n",
      "                 'PickleBuffer',\n",
      "                 'PickleError',\n",
      "                 'Pickler',\n",
      "                 'PicklingError',\n",
      "                 'Unpickler',\n",
      "                 'UnpicklingError'],\n",
      " 'datamodel_attribute': ['__all__',\n",
      "                         '__builtins__',\n",
      "                         '__cached__',\n",
      "                         '__doc__',\n",
      "                         '__file__',\n",
      "                         '__loader__',\n",
      "                         '__name__',\n",
      "                         '__package__',\n",
      "                         '__spec__'],\n",
      " 'internal_attribute': ['_HAVE_PICKLE_BUFFER',\n",
      "                        '_compat_pickle',\n",
      "                        '_extension_cache',\n",
      "                        '_extension_registry',\n",
      "                        '_inverted_registry',\n",
      "                        '_tuplesize2code'],\n",
      " 'internal_method': ['_Framer',\n",
      "                     '_Pickler',\n",
      "                     '_Stop',\n",
      "                     '_Unframer',\n",
      "                     '_Unpickler',\n",
      "                     '_dump',\n",
      "                     '_dumps',\n",
      "                     '_getattribute',\n",
      "                     '_load',\n",
      "                     '_loads',\n",
      "                     '_test']}\n"
     ]
    }
   ],
   "source": [
    "dir2(pickle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice there are are large number of constants, these are ```bytes``` instances used to identify a protocol, and opticodes which identify the datatype and hence number of ```bytes``` to be expected for that instance and the end of a ```bytes``` stream:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x04K\\x01.'"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall a ```bytes``` that corresponds to a printable ASCII character will be displayed as the ASCII character. Non-printable characters will be represented as a hexadecimal escape sequence ```\\x``` followed by two hexadecimal characters e.g. ```\\x80```. Often ```bytes``` instances are cast into hexadecimal ```str``` instances for better readability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'80044b012e'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(1).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the constants are examined as hexadecimal ```str``` instances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ADDITEMS': '90',\n",
       " 'APPEND': '61',\n",
       " 'APPENDS': '65',\n",
       " 'BINBYTES': '42',\n",
       " 'BINBYTES8': '8e',\n",
       " 'BINFLOAT': '47',\n",
       " 'BINGET': '68',\n",
       " 'BININT': '4a',\n",
       " 'BININT1': '4b',\n",
       " 'BININT2': '4d',\n",
       " 'BINPERSID': '51',\n",
       " 'BINPUT': '71',\n",
       " 'BINSTRING': '54',\n",
       " 'BINUNICODE': '58',\n",
       " 'BINUNICODE8': '8d',\n",
       " 'BUILD': '62',\n",
       " 'BYTEARRAY8': '96',\n",
       " 'DEFAULT_PROTOCOL': 4,\n",
       " 'DICT': '64',\n",
       " 'DUP': '32',\n",
       " 'EMPTY_DICT': '7d',\n",
       " 'EMPTY_LIST': '5d',\n",
       " 'EMPTY_SET': '8f',\n",
       " 'EMPTY_TUPLE': '29',\n",
       " 'EXT1': '82',\n",
       " 'EXT2': '83',\n",
       " 'EXT4': '84',\n",
       " 'FALSE': '4930300a',\n",
       " 'FLOAT': '46',\n",
       " 'FRAME': '95',\n",
       " 'FROZENSET': '91',\n",
       " 'GET': '67',\n",
       " 'GLOBAL': '63',\n",
       " 'HIGHEST_PROTOCOL': 5,\n",
       " 'INST': '69',\n",
       " 'INT': '49',\n",
       " 'LIST': '6c',\n",
       " 'LONG': '4c',\n",
       " 'LONG1': '8a',\n",
       " 'LONG4': '8b',\n",
       " 'LONG_BINGET': '6a',\n",
       " 'LONG_BINPUT': '72',\n",
       " 'MARK': '28',\n",
       " 'MEMOIZE': '94',\n",
       " 'NEWFALSE': '89',\n",
       " 'NEWOBJ': '81',\n",
       " 'NEWOBJ_EX': '92',\n",
       " 'NEWTRUE': '88',\n",
       " 'NEXT_BUFFER': '97',\n",
       " 'NONE': '4e',\n",
       " 'OBJ': '6f',\n",
       " 'PERSID': '50',\n",
       " 'POP': '30',\n",
       " 'POP_MARK': '31',\n",
       " 'PROTO': '80',\n",
       " 'PUT': '70',\n",
       " 'READONLY_BUFFER': '98',\n",
       " 'REDUCE': '52',\n",
       " 'SETITEM': '73',\n",
       " 'SETITEMS': '75',\n",
       " 'SHORT_BINBYTES': '43',\n",
       " 'SHORT_BINSTRING': '55',\n",
       " 'SHORT_BINUNICODE': '8c',\n",
       " 'STACK_GLOBAL': '93',\n",
       " 'STOP': '2e',\n",
       " 'STRING': '53',\n",
       " 'TRUE': '4930310a',\n",
       " 'TUPLE': '74',\n",
       " 'TUPLE1': '85',\n",
       " 'TUPLE2': '86',\n",
       " 'TUPLE3': '87',\n",
       " 'UNICODE': '56'}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optcodes = {}\n",
    "for constant in dir2(pickle, print_output=False)['constant']:\n",
    "    if isinstance(getattr(pickle, constant), bytes):\n",
    "        optcodes[constant] = getattr(pickle, constant).hex()\n",
    "    else:\n",
    "        optcodes[constant] = getattr(pickle, constant)\n",
    "\n",
    "optcodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The picked hexadecimal ```str``` of the ```int``` ```1``` can be examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'80044b012e'"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(1).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means:\n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|4b|Type Int Optcode: BININT1|\n",
    "|01|Numeric value: 1|\n",
    "|2e|Stop OptoCode: STOP|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a ```bool``` instance is instead examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'8004882e'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(True).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means:\n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|88|Bool True Optcode: NEWTRUE|\n",
    "|2e|Stop OptoCode: STOP|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a ```str``` instance of ASCII characters is examined. Notice the data for the ```str``` is included in the pickled ```bytes``` instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x04\\x95\\t\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x05hello\\x94.'"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps('hello')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An equivalent ```bytes``` instance can be cast to a hexadecimal ```str``` instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'68656c6c6f'"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'hello'.hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The picked ```bytes``` sequence can also be a hexadecimal ```str``` instance and the data above can be seen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'80049509000000000000008c0568656c6c6f942e'"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps('hello').hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following means\n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|95|Frame OptCode: FRAME|\n",
    "|09 00 00 00 00 00 00 00|Frame Size: 9 bytes (0x9)|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|05|Length of String: len('hello')|\n",
    "|68 65 6c 6c 6f|String: 'hello'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|2e|Stop OptCode: STOP|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The frame size is the number of ```bytes``` the serialised ```str``` covers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.0"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len('8c0568656c6c6f942e') / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More complicated datatypes can be serialised for example a ```tuple```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'80049517000000000000008c0568656c6c6f948c05776f726c64948c01219487942e'"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(('hello', 'world', '!')).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means \n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|95|Frame OptCode: FRAME|\n",
    "|17 00 00 00 00 00 00 00|Frame Size: 23 bytes (0x17)|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|05|Length of String: len('hello')|\n",
    "|68 65 6c 6c 6f|String: 'hello'|\n",
    "|94|Memoize: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|05|Length of String: len('world')|\n",
    "|77 6f 72 6c 64|String: 'world'|\n",
    "|94|Memoize Optcode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|01|Length of String len('!')|\n",
    "|21|string: '!'|\n",
    "|94|Memoize Optcode: MEMOIZE|\n",
    "|87|Tuple OptCode: TUPLE3|\n",
    "|94|Memoize: MEMOIZE|\n",
    "|2e|Stop OptCode: STOP|\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```list```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'80049519000000000000005d94288c0568656c6c6f948c05776f726c64948c012194652e'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(['hello', 'world', '!']).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which means\n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|95|Frame OptCode: FRAME|\n",
    "|19 00 00 00 00 00 00 00|Frame Size: 25 (0x19)|\n",
    "|5d|Empty List OptCode: EMPTY_LIST|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|28|Mark OptCode: MARK|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|05|Length of String: len('hello')|\n",
    "|68 65 6c 6c 6f|String: 'hello'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|05|Length of String: len('world')|\n",
    "|77 6f 72 6c 64|String: 'world'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|01|Length of String len('!')|\n",
    "|21|string: '!'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|65|List Append OptCode: APPENDS|\n",
    "|2e|Stop OptCode: STOP|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```dict```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'80049526000000000000007d94288c0172948c03726564948c0167948c05677265656e948c0162948c04626c756594752e'"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps({'r': 'red', 'g': 'green', 'b': 'blue'}).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which means:\n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|95|Frame OptCode: FRAME|\n",
    "|26 00 00 00 00 00 00 00|Frame Size: 38 bytes (0x26)|\n",
    "|7d|Empty Dict OptCode: EMPTY_DICT|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|28|Mark OptCode: MARK|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|01|Length of String len('r')|\n",
    "|72|string: 'r'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|03|Length of String len('red')|\n",
    "|72 65 64|string: 'red'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|01|Length of String len('g')|\n",
    "|67|string: 'g'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|05|Length of String len('green')|\n",
    "|67 72 65 65 6e|string: 'green'|\n",
    "|94|Memoize OptiCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|01|Length of String len('b')|\n",
    "|62|string: 'b'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|8c|String OptCode: SHORT_BINUNICODE|\n",
    "|04|Length of String len('blue')|\n",
    "|62 6c 75 65|string: 'blue'|\n",
    "|94|Memoize OptCode: MEMOIZE|\n",
    "|75|Dict SetItem OptCode: SETITEMS|\n",
    "|2e|Stop OptCode: STOP|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Python ```float``` is a double precision floating point number and the IEEE representation can be used using the ```struct.pack```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3fb999999999999a'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import struct\n",
    "struct.pack('>d', 0.1).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The picked version of the ```float``` can then be examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'8004950a00000000000000473fb999999999999a2e'"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.dumps(0.1).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which means:\n",
    "\n",
    "|Hex Value|Meaning|\n",
    "|---|---|\n",
    "|80|Start OptCode: PROTO|\n",
    "|04|Protocol Version: DEFAULT_PROTOCOL|\n",
    "|95|Frame OptCode: FRAME|\n",
    "|0a 00 00 00 00 00 00 00|Frame Size: 10 bytes (0xa)|\n",
    "|2e|Stop OptCode: STOP|\n",
    "|47|BINFLOAT|\n",
    "|3f b9 99 99 99 99 9a|0.1 in 64 Bit IEEE|\n",
    "|2e|Stop OptCode: pickle.STOP.hex()|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, the fine details of the opticodes is not required:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "bytestream = pickle.dumps({'r': 'red', 'g': 'green', 'b': 'blue'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x04\\x95&\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x01r\\x94\\x8c\\x03red\\x94\\x8c\\x01g\\x94\\x8c\\x05green\\x94\\x8c\\x01b\\x94\\x8c\\x04blue\\x94u.'"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytestream"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the ```loads``` function is used to read the data back into a Python ```object```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'r': 'red', 'g': 'green', 'b': 'blue'}"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.loads(bytestream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like the ```json``` module, the ```pickle``` module has the counterparts ```dump``` and ```load``` which are used to dump data and load data to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./files_output/serialised.pkl', mode='wb') as file:\n",
    "    pickle.dump(obj=1, file=file)\n",
    "    pickle.dump(obj='hello', file=file)\n",
    "    pickle.dump(obj=('hello', 'world', '!'), file=file)\n",
    "    pickle.dump(obj=['hello', 'world', '!'], file=file)\n",
    "    pickle.dump(obj={'r': 'red', 'g': 'green', 'b': 'blue'}, file=file)\n",
    "    pickle.dump(obj=0.1, file=file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous files examined, the new line was used as a delimiter. In the ```.pkl``` file, there is only a new line character if it is present in the ```bytes``` instance by chance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'\\x80\\x04K\\x01.\\x80\\x04\\x95\\t\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x05hello\\x94.\\x80\\x04\\x95\\x17\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x05hello\\x94\\x8c\\x05world\\x94\\x8c\\x01!\\x94\\x87\\x94.\\x80\\x04\\x95\\x19\\x00\\x00\\x00\\x00\\x00\\x00\\x00]\\x94(\\x8c\\x05hello\\x94\\x8c\\x05world\\x94\\x8c\\x01!\\x94e.\\x80\\x04\\x95&\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x01r\\x94\\x8c\\x03red\\x94\\x8c\\x01g\\x94\\x8c\\x05green\\x94\\x8c\\x01b\\x94\\x8c\\x04blue\\x94u.\\x80\\x04\\x95\\n',\n",
       " b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00G?\\xb9\\x99\\x99\\x99\\x99\\x99\\x9a.']"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./files_output/serialised.pkl', mode='rb') as file:\n",
    "    data = file.readlines()\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```load``` function will only read out one value at a time using the Start Opticode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "hello\n"
     ]
    }
   ],
   "source": [
    "with open('./files_output/serialised.pkl', mode='rb') as file:\n",
    "    print(pickle.load(file))\n",
    "    print(pickle.load(file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file is essentially an iterator and there is no easy way to determine the number of items left in the file, therefore a ```for``` loop cannot be constructed. Instead a ```while``` loop can be constructed that is set to ```break``` when an ```EOFError``` is encountered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1,\n",
       " 1: 'hello',\n",
       " 2: ('hello', 'world', '!'),\n",
       " 3: ['hello', 'world', '!'],\n",
       " 4: {'r': 'red', 'g': 'green', 'b': 'blue'},\n",
       " 5: 0.1}"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unpickled_data = {}\n",
    "\n",
    "with open('./files_output/serialised.pkl', mode='rb') as file:\n",
    "    num = 0\n",
    "    while True:\n",
    "        try:\n",
    "            unpickled_data[num] = pickle.load(file=file)\n",
    "            num += 1\n",
    "        except EOFError:\n",
    "            break\n",
    "\n",
    "unpickled_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is more common to store pickled objects in a database and the ```shelve``` module is used to do this. If the identifiers for the ```shelve``` module are examined, notice there is an ```open``` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'constant': ['DEFAULT_PROTOCOL'],\n",
      " 'module': ['collections'],\n",
      " 'method': ['open'],\n",
      " 'upper_class': ['BsdDbShelf',\n",
      "                 'BytesIO',\n",
      "                 'DbfilenameShelf',\n",
      "                 'Pickler',\n",
      "                 'Shelf',\n",
      "                 'Unpickler'],\n",
      " 'datamodel_attribute': ['__all__',\n",
      "                         '__builtins__',\n",
      "                         '__cached__',\n",
      "                         '__doc__',\n",
      "                         '__file__',\n",
      "                         '__loader__',\n",
      "                         '__name__',\n",
      "                         '__package__',\n",
      "                         '__spec__'],\n",
      " 'internal_method': ['_ClosedDict']}\n"
     ]
    }
   ],
   "source": [
    "dir2(shelve)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This behaves analogously to the ```open``` method found in ```io``` and is typically inplemented via a ```with``` code block. The identifiers for the ```DbfilenameShelf``` instance created can be examined. Notice the consistency with ```dict``` identifiers. The database can essentially be thought of as a set of shelves, each shelf has a ```key``` and a ```value```. Conceptually each pickled object is shelved in the database, which is where the name of the module ```shelve``` comes from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'shelve.DbfilenameShelf'>\n",
      "{'attribute': ['cache', 'dict', 'keyencoding', 'writeback'],\n",
      " 'method': ['clear',\n",
      "            'close',\n",
      "            'get',\n",
      "            'items',\n",
      "            'keys',\n",
      "            'pop',\n",
      "            'popitem',\n",
      "            'setdefault',\n",
      "            'sync',\n",
      "            'update',\n",
      "            'values'],\n",
      " 'datamodel_attribute': ['__abstractmethods__',\n",
      "                         '__dict__',\n",
      "                         '__module__',\n",
      "                         '__reversed__',\n",
      "                         '__slots__',\n",
      "                         '__weakref__'],\n",
      " 'datamodel_method': ['__class_getitem__',\n",
      "                      '__contains__',\n",
      "                      '__del__',\n",
      "                      '__delitem__',\n",
      "                      '__enter__',\n",
      "                      '__exit__',\n",
      "                      '__getitem__',\n",
      "                      '__iter__',\n",
      "                      '__len__',\n",
      "                      '__setitem__'],\n",
      " 'internal_attribute': ['_MutableMapping__marker', '_abc_impl', '_protocol']}\n"
     ]
    }
   ],
   "source": [
    "with shelve.open('./files_output/database.pkl') as database:\n",
    "    print(type(database))\n",
    "    dir2(database, object, unique_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each ```value```, a Python ```object``` to be pickled can be saved on a new shelf using a new shelve ```key```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "with shelve.open('./files_output/database.pkl') as database:\n",
    "    database['unicode_string'] = 'hello'\n",
    "    database['archive'] = ('hello', 'world', '!')\n",
    "    database['active'] = ['hello', 'world', '!']\n",
    "    database['mapping'] = {'r': 'red', 'g': 'green', 'b': 'blue'}\n",
    "    database['num'] = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally the database is read using a ```with``` code block but for convenience it can be read directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "database = shelve.open('./files_output/database.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```keys``` attribute only shows a ```KeysView```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KeysView(<shelve.DbfilenameShelf object at 0x7faa7a7d4f20>)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "database.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be cast into a ```list``` to view the keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['unicode_string', 'archive', 'active', 'mapping', 'num']"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(database.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An item can be selected by indexing using its ```key```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello'"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "database['unicode_string']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the item is accessed it is unpickled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the database wasn't accessed in a ```with``` code block it should be closed using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "database.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Anaconda Tutorial](../readme.md)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}