builtins_module_bytes/notebook.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Builtins Module: The Byte String Class (bytes)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebook the Unicode string class ```str``` was examined and was seen to have a Unicode character as a fundamental unit. The byte string class ```bytes``` on the other hand use a byte as a fundamental unit. The byte string was the foundation for text data in Python 2. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorize_Identifiers Module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will use the following functions ```dir2```, ```variables``` and ```view``` in the custom module ```categorize_identifiers``` which is found in the same directory as this notebook file. ```dir2``` is a variant of ```dir``` that groups identifiers into a ```dict``` under categories and ```variables``` is an IPython based a variable inspector. ```view``` is used to view a ```Collection``` in more detail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from categorize_identifiers import dir2, variables, view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bytes Conception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A computer stores data using a bit. A bit can be conceptualised as a single dip switch and has the values Off and On respectively as shown below. A single switch has the possible values ```0```, ```1``` which is ```2 ** 1``` combinations which is a total of ```2```. \n",
    "\n",
    "A single switch ranges between ```0:2```. Since Python uses zero-order indexing, the lower bound ```0``` is included and the upper bound ```2``` is exclusive. i.e. up to and excluding ```2```:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_001.png' alt='img_001' width='400'/>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More typically ```8``` of these switches are combined into a single logical unit called a byte. A byte has ```2 ** 8``` combinations which is a total of ```256```. i.e. a byte comprises of 8 bits and has ```0:256``` combinations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_002.png' alt='img_002' width='400'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each dipswitch represents a power of ```2```. The first dipswitch on the right hand side \"8\" represents the units (```2 ** 0```), the second dipswitch from the right \"7\" represents the power (```2 ** 1```), the third dipswitch from the right \"6\" represents the power (```2 ** 2```) and so on... The number above can therefore be calculated as a decimal number using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "+ 0 * (2 ** 7) \\\n",
    "+ 1 * (2 ** 6) \\\n",
    "+ 1 * (2 ** 5) \\\n",
    "+ 0 * (2 ** 4) \\\n",
    "+ 1 * (2 ** 3) \\\n",
    "+ 0 * (2 ** 2) \\\n",
    "+ 0 * (2 ** 1) \\\n",
    "+ 0 * (2 ** 0) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```bytes``` above can be expressed as a binary number using the prefix ```0b```, this prefix is used to distinguish the base 2 from the base 10 which is used by default for an ```int```. Notice that syntax highlight highlights the base 2 prefix. The decimal ```int``` will be returned in the cell output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0b01101000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leading zeros are normally omitted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0b1101000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although in this context, it is useful to show the leading zeros, so all 8 bits in the byte can be visualised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance is essentially a collection of individual bytes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_003.png' alt='img_003' width='400'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each byte above can be represented as a binary number and grouped into a ```tuple```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(104, 101, 108, 108, 111)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(0b01101000, 0b01100101, 0b01101100, 0b01101100, 0b01101111)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ```tuple``` collection can be cast into ```bytes``` giving text information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes((0b01101000, 0b01100101, 0b01101100, 0b01101100, 0b01101111))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this ```bytes``` instance is viewed, notice the value for each index is a byte which is represented by an ```int``` in decimal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t int                  \t 1      \t 104                            \t\n",
      "1 \t int                  \t 1      \t 101                            \t\n",
      "2 \t int                  \t 1      \t 108                            \t\n",
      "3 \t int                  \t 1      \t 108                            \t\n",
      "4 \t int                  \t 1      \t 111                            \t\n"
     ]
    }
   ],
   "source": [
    "view(bytes((0b01101000, 0b01100101, 0b01101100, 0b01101100, 0b01101111)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ASCII Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the American Standard Code for Information Interchange (ASCII) maps each byte to a physical command or English character. The physical commands were used to control primitive computers that were essentially typewriter based:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_004.png' alt='img_004' width='600'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 128 commands which span the first half of the byte:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|byte|hex|num|command|\n",
    "|---|---|---|---|\n",
    "|00000000|00|000|null|\n",
    "|00000001|01|001|start of heading|\n",
    "|00000010|02|002|start of text|\n",
    "|00000011|03|003|end of text|\n",
    "|00000100|04|004|end of transmission|\n",
    "|00000101|05|005|enquiry|\n",
    "|00000110|06|006|acknowledge|\n",
    "|00000111|07|007|bell|\n",
    "|00001000|08|008|**backspace**|\n",
    "|00001001|09|009|**horizontal tab**|\n",
    "|00001010|0a|010|**new line**|\n",
    "|00001011|0b|011|**vertical tab**|\n",
    "|00001100|0c|012|**form feed**|\n",
    "|00001101|0d|013|**carriage return**|\n",
    "|00001110|0e|014|shift out|\n",
    "|00001111|0f|015|shift in|\n",
    "|00010000|10|016|data link escape|\n",
    "|00010001|11|017|device control 1|\n",
    "|00010010|12|018|device control 2|\n",
    "|00010011|13|019|device control 3|\n",
    "|00010100|14|020|device control 4|\n",
    "|00010101|15|021|negative acknowledge|\n",
    "|00010110|16|022|synchronous idle|\n",
    "|00010111|17|023|end of transmission block|\n",
    "|00011000|18|024|cancel|\n",
    "|00011001|19|025|end of medium|\n",
    "|00011010|1a|026|substitute|\n",
    "|00011011|1b|027|**escape**|\n",
    "|00011100|1c|028|file separator|\n",
    "|00011101|1d|029|group separator|\n",
    "|00011110|1e|030|record separator|\n",
    "|00011111|1f|031|unit seperator|\n",
    "|00100000|20|032|**space**|\n",
    "\n",
    "The remaining commands spanning up to half a byte contained the characters most commonly used in the English language.\n",
    "\n",
    "|byte|hex|num|character|\n",
    "|---|---|---|---|\n",
    "|00100001|21|033|!|\n",
    "|00100010|22|034|\"|\n",
    "|00100011|23|035|#|\n",
    "|00100100|24|036|$|\n",
    "|00100101|25|037|%|\n",
    "|00100110|26|038|&|\n",
    "|00100111|27|039|'|\n",
    "|00101000|28|040|(|\n",
    "|00101001|29|041|)|\n",
    "|00101010|2a|042|*|\n",
    "|00101011|2b|043|+|\n",
    "|00101100|2c|044|,|\n",
    "|00101101|2d|045|-|\n",
    "|00101110|2e|046|.|\n",
    "|00101111|2f|047|/|\n",
    "|00110000|30|048|0|\n",
    "|00110001|31|049|1|\n",
    "|00110010|32|050|2|\n",
    "|00110011|33|051|3|\n",
    "|00110100|34|052|4|\n",
    "|00110101|35|053|5|\n",
    "|00110110|36|054|6|\n",
    "|00110111|37|055|7|\n",
    "|00111000|38|056|8|\n",
    "|00111001|39|057|9|\n",
    "|00111010|3a|058|:|\n",
    "|00111011|3b|059|;|\n",
    "|00111100|3c|060|<|\n",
    "|00111101|3d|061|=|\n",
    "|00111110|3e|062|>|\n",
    "|00111111|3f|063|?|\n",
    "|01000000|40|064|@|\n",
    "|01000001|41|065|A|\n",
    "|01000010|42|066|B|\n",
    "|01000011|43|067|C|\n",
    "|01000100|44|068|D|\n",
    "|01000101|45|069|E|\n",
    "|01000110|46|070|F|\n",
    "|01000111|47|071|G|\n",
    "|01001000|48|072|H|\n",
    "|01001001|49|073|I|\n",
    "|01001010|4a|074|J|\n",
    "|01001011|4b|075|K|\n",
    "|01001100|4c|076|L|\n",
    "|01001101|4d|077|M|\n",
    "|01001110|4e|078|N|\n",
    "|01001111|4f|079|O|\n",
    "|01010000|50|080|P|\n",
    "|01010001|51|081|Q|\n",
    "|01010010|52|082|R|\n",
    "|01010011|53|083|S|\n",
    "|01010100|54|084|T|\n",
    "|01010101|55|085|U|\n",
    "|01010110|56|086|V|\n",
    "|01010111|57|087|W|\n",
    "|01011000|58|088|X|\n",
    "|01011001|59|089|Y|\n",
    "|01011010|5a|090|Z|\n",
    "|01011011|5b|091|[|\n",
    "|01011100|5c|092|\\|\n",
    "|01011101|5d|093|]|\n",
    "|01011110|5e|094|^|\n",
    "|01011111|5f|095|_|\n",
    "|01100000|60|096|`|\n",
    "|01100001|61|097|a|\n",
    "|01100010|62|098|b|\n",
    "|01100011|63|099|c|\n",
    "|01100100|64|100|d|\n",
    "|01100101|65|101|e|\n",
    "|01100110|66|102|f|\n",
    "|01100111|67|103|g|\n",
    "|01101000|68|104|h|\n",
    "|01101001|69|105|i|\n",
    "|01101010|6a|106|j|\n",
    "|01101011|6b|107|k|\n",
    "|01101100|6c|108|l|\n",
    "|01101101|6d|109|m|\n",
    "|01101110|6e|110|n|\n",
    "|01101111|6f|111|o|\n",
    "|01110000|70|112|p|\n",
    "|01110001|71|113|q|\n",
    "|01110010|72|114|r|\n",
    "|01110011|73|115|s|\n",
    "|01110100|74|116|t|\n",
    "|01110101|75|117|u|\n",
    "|01110110|76|118|v|\n",
    "|01110111|77|119|w|\n",
    "|01111000|78|120|x|\n",
    "|01111001|79|121|y|\n",
    "|01111010|7a|122|z|\n",
    "|01111011|7b|123|{|\n",
    "|01111100|7c|124|||\n",
    "|01111101|7d|125|}|\n",
    "|01111110|7e|126|~|\n",
    "|01111111|7f|127||\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the ```string``` module contains the printable ASCII characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the ```bytes``` class, the translation table for an ASCII character is always the same. Therefore in the formal representation instead of displaying the byte for that character, the ASCII character is shown:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~ \\t\\n\\r\\x0b\\x0c'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.printable"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialisation Signature"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The initialisation signature for the ```bytes``` class can be examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mInit signature:\u001b[0m \u001b[0mbytes\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m/\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
       "\u001b[1;31mDocstring:\u001b[0m     \n",
       "bytes(iterable_of_ints) -> bytes\n",
       "bytes(string, encoding[, errors]) -> bytes\n",
       "bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer\n",
       "bytes(int) -> bytes object of size given by the parameter initialized with null bytes\n",
       "bytes() -> empty bytes object\n",
       "\n",
       "Construct an immutable array of bytes from:\n",
       "  - an iterable yielding integers in range(256)\n",
       "  - a text string encoded using the specified encoding\n",
       "  - any object implementing the buffer API.\n",
       "  - an integer\n",
       "\u001b[1;31mType:\u001b[0m           type\n",
       "\u001b[1;31mSubclasses:\u001b[0m     bytes_"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bytes?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the ```bytes``` string class, the initialisation signature shows 5 alternative ways of supplying instance data:\n",
    "\n",
    "```python\n",
    "bytes(self, /, *args, **kwargs)\n",
    "bytes(iterable_of_ints) -> bytes\n",
    "bytes(string, encoding[, errors]) -> bytes\n",
    "bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer\n",
    "bytes(int) -> bytes object of size given by the parameter initialized with null bytes\n",
    "bytes() -> empty bytes object\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the first way is examined:\n",
    "\n",
    "```python\n",
    "bytes(self, /, *args, **kwargs)\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The parenthesis ```( )``` are used to call a function and supply any necessary input arguments.\n",
    "* The comma ```,``` is used as a delimiter to seperate out any input arguments.\n",
    "* ```self``` is used to denote *this instance*. In other words a byte string can be constructed from an existing byte string instance, this is a special case as a byte string is a fundamental datatype.\n",
    "* Any input argument before a ```/``` must be provided positionally\n",
    "* ```*args``` indicates a variable number of additional positional input arguments. These are typically not used for initialisation of the ```bytes``` string class.\n",
    "* ```**kwargs``` indicates a variable number of additional named input arguments. These are typically not used for initialisation of the ```bytes``` string class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance can be instantiated by supplying an existing ```bytes``` instance ```self``` to the ```bytes``` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Hello World!'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(b'Hello World!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However because the ```bytes``` class is a fundamental datatype it can also be instantiated shorthand using the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Hello World!'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'Hello World!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the characters above in the ```bytes``` instance are ASCII printable characters. Therefore each byte value in the ```bytes``` instance above is represented by its corresponding ASCII character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t int                  \t 1      \t 72                             \t\n",
      "1 \t int                  \t 1      \t 101                            \t\n",
      "2 \t int                  \t 1      \t 108                            \t\n",
      "3 \t int                  \t 1      \t 108                            \t\n",
      "4 \t int                  \t 1      \t 111                            \t\n",
      "5 \t int                  \t 1      \t 32                             \t\n",
      "6 \t int                  \t 1      \t 87                             \t\n",
      "7 \t int                  \t 1      \t 111                            \t\n",
      "8 \t int                  \t 1      \t 114                            \t\n",
      "9 \t int                  \t 1      \t 108                            \t\n",
      "10 \t int                  \t 1      \t 100                            \t\n",
      "11 \t int                  \t 1      \t 33                             \t\n"
     ]
    }
   ],
   "source": [
    "view(b'Hello World!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~ \\t\\n\\r\\x0b\\x0c'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.printable"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance can be initialised using an iterable such as a ```tuple``` of ```int``` instances:\n",
    "\n",
    "```python\n",
    "bytes(iterable_of_ints, /) -> bytes\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this to work each ```int``` must be a valid byte value and recall that a byte looks like the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_002.png' alt='img_002' width='400'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since a byte has ```2 ** 8``` combinations, which is a total of ```256``` the range is ```0:256``` inclusive of the lower bound and exclusive of the maximum bound. Therefore the maximum value for an ```int``` instance is ```255```. Note that a trailing comma is required to distinguish a single element ```tuple``` from a numeric calculation using parenthesis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "num = (97)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "archive = (97, )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Size/Shape</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Instance Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>num</th>\n",
       "      <td>int</td>\n",
       "      <td></td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>archive</th>\n",
       "      <td>tuple</td>\n",
       "      <td>1</td>\n",
       "      <td>(97,)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Type Size/Shape  Value\n",
       "Instance Name                         \n",
       "num              int                97\n",
       "archive        tuple          1  (97,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above ASCII table the ```int``` instance ```97``` corresponds to the character ```a``` and this can be seen when this is cast to a ```tuple```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'a'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes((97, ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When an ```int``` exceeds the upper bound ```256``` (up to and exclusive of ```256``` due to zero-order indexing) a ```ValueError``` will display:\n",
    "\n",
    "```python\n",
    "bytes((256, ))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally the ```tuple``` will contain more than one ```int``` instance and each of these will be a valid byte value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello world!'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the decimal ```int``` instance ```104``` can be represented in binary. The ```bin``` function will return this binary number as an Unicode ```str``` instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b1101000'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(104)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For conception it is helpful to see the leading zeros using the ```str``` instance methods ```removeprefix``` and ```zfill```, alongside ```str``` instance concatenation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b01101000'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'0b' + bin(104).removeprefix('0b').zfill(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Unicode ```str``` instance corresponding to this byte can be retrieved using the ```chr``` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'h'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chr(104)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ```bytes``` instance consists of ```12``` individual byte units. The different representation for each byte unit can be examined below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  104      101      108      108      111       32      119      111      114      108      100       33    \n",
      "01101000 01100101 01101100 01101100 01101111 00100000 01110111 01101111 01110010 01101100 01100100 00100001 \n",
      "   h        e        l        l        o                 w        o        r        l        d        !     "
     ]
    }
   ],
   "source": [
    "for number in integers:\n",
    "    print(str(number).center(8), end=' ')\n",
    "\n",
    "print()\n",
    "for number in integers:\n",
    "    print(bin(number).removeprefix('0b').zfill(8), end=' ')\n",
    "\n",
    "print()\n",
    "for number in integers:\n",
    "    print(chr(number).center(8), end=' ')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When every byte is a printable ASCII character it will be displayed instead of the byte sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello world!'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the byte is not printable for example, the ```int``` instance ```0``` which corresponds to the ASCII non-printable command NULL, is represented using a hexadecimal escape sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes((0,))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the whitespace characters with exception to the space are represented using an escape character. For the tab, newline and carriage return characters that are commonly used, there are the escape characters ```\\t``` (decimal 9), ```\\n``` (decimal 10) and ```\\r``` (decimal 13). The vertical tab and form feed are less commonly used and represented by their hexadecimal escape sequences ```\\x0b``` (decimal 11) and ```\\x0c``` (decimal 12):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' \\t\\n\\r\\x0b\\x0c'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.whitespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (9, 10, 11, 12, 13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\t\\n\\x0b\\x0c\\r'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binary ```0b00001100``` is not very human-readable and therefore it is easy for a human to make transcription errors when dealing with binary. To make a byte more human readable the hexadecimal number system is introduced. In hexadecimal the byte is essentially split into 2 halves and each half byte is represented as a hexadecimal character:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_005.png' alt='img_005' width='400'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall **b**inary has ```2``` digits and the prefix ```0b```, decimal has ```10``` digits and no prefix because it is most commonly used numbering system. He**x**adecimal has ```16``` digits and the prefix ```0x```. \n",
    "\n",
    "Hexadecimal takes the first ```10``` digits from decimal and supplements them with the first ```6``` letters in the alphabet. The number of combinations in half a byte is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 ** 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a consequence each hexadecimal character perfectly maps to a 4 bit (half a byte) binary sequence:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|(0b) binary|(0x) hexadecimal character|decimal character|\n",
    "|---|---|---|\n",
    "|0000|0|0|\n",
    "|0001|1|1|\n",
    "|0010|2|2|\n",
    "|0011|3|3|\n",
    "|0100|4|4|\n",
    "|0101|5|5|\n",
    "|0110|6|6|\n",
    "|0111|7|7|\n",
    "|1000|8|8|\n",
    "|1001|9|9|\n",
    "|1010|a|10|\n",
    "|1011|b|11|\n",
    "|1100|c|12|\n",
    "|1101|d|13|\n",
    "|1110|e|14|\n",
    "|1111|f|15|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although uppercase and lowercase can be used to represent a hexadecimal character, notice that the Python interpreter prefers lowercase:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (11, 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x0b\\x0c'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A human is more likely to make a transcription error when reading hexadecimal sequences. For example:\n",
    "\n",
    "```python\n",
    "'ABB4AB8A'\n",
    "```\n",
    "\n",
    "when reading the above quickly, notice the similarity between A and 4 and B and 8. The lowercase characters are more clearly distinguished:\n",
    "\n",
    "```python\n",
    "'abb4ab8a'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following ```bytes``` instance:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/img_005.png' alt='img_005' width='400'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the binary number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0b00001100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value returned in the cell output displays the decimal integer. The ```hex``` function can be used to cast a decimal ```int``` into a Unicode ```str``` of a hexadecimal character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0xc'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hex(0b00001100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hexadecimal value is displayed without the trailing zero, so the first half byte is not shown. This can be added for clarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0x0c'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'0x' + hex(12).removeprefix('0x').zfill(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b00001100'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'0b' + bin(12).removeprefix('0b').zfill(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for clarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0000 1100\n",
      " 0    c  \n"
     ]
    }
   ],
   "source": [
    "print(bin(12).removeprefix('0b').zfill(8)[:4], bin(12).removeprefix('0b').zfill(8)[4:])\n",
    "print(hex(12).removeprefix('0x').zfill(2)[:1].center(4), hex(12).removeprefix('0x').zfill(2)[1:].center(4))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the byte sequence contains a byte that maps to a whitespace character, a non-printable command or is unmapped to an ASCII character there is no corresponding character to display and an escape sequence instead displays. This can be seen when using the first ```32``` integers and integers above ```127```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (0, 1, 2, 29, 30, 31, 128, 129, 130, 253, 254, 255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00\\x01\\x02\\x1d\\x1e\\x1f\\x80\\x81\\x82\\xfd\\xfe\\xff'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that each character is inserted using its own hexadecimal escape sequence prefix ```\\x``` and this instruction expects two hexadecimal characters so a leading zero must be included where applicable. \n",
    "\n",
    "When a ```bytes``` instance contains byte sequences that map to characters these characters will be displayed instead of the hexadecimal escape sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (32, 65, 80, 120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b' APx'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tab ```\\t```, newline ```\\t```, carriage return ```\\r``` and backslash ```\\\\``` itself all have single escape characters (the ```\\``` is shown as ```\\\\``` as the ```\\``` is used to insert an escape character):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (9, 10, 13, 92)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\t\\n\\r\\\\'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(integers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance containing all of these initially appear confusing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "integers = (0, 1, 2, 9, 10, 13, 29, 30, 31, 32, 65, 92, 97, 128, 129, 130, 253, 254, 255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "bytes_string = bytes(integers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Size/Shape</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Instance Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>num</th>\n",
       "      <td>int</td>\n",
       "      <td></td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>archive</th>\n",
       "      <td>tuple</td>\n",
       "      <td>1</td>\n",
       "      <td>(97,)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>integers</th>\n",
       "      <td>tuple</td>\n",
       "      <td>19</td>\n",
       "      <td>(0, 1, 2, 9, 10, 13, 29, 30, 31, 32, 65, 92, 97, 128, 129, 130, 253, 254, 255)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>number</th>\n",
       "      <td>int</td>\n",
       "      <td></td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bytes_string</th>\n",
       "      <td>bytes</td>\n",
       "      <td>19</td>\n",
       "      <td>b'\\x00\\x01\\x02\\t\\n\\r\\x1d\\x1e\\x1f A\\\\a\\x80\\x81\\x82\\xfd\\xfe\\xff'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Type Size/Shape  \\\n",
       "Instance Name                     \n",
       "num              int              \n",
       "archive        tuple          1   \n",
       "integers       tuple         19   \n",
       "number           int              \n",
       "bytes_string   bytes         19   \n",
       "\n",
       "                                                                                        Value  \n",
       "Instance Name                                                                                  \n",
       "num                                                                                        97  \n",
       "archive                                                                                 (97,)  \n",
       "integers       (0, 1, 2, 9, 10, 13, 29, 30, 31, 32, 65, 92, 97, 128, 129, 130, 253, 254, 255)  \n",
       "number                                                                                     33  \n",
       "bytes_string                   b'\\x00\\x01\\x02\\t\\n\\r\\x1d\\x1e\\x1f A\\\\a\\x80\\x81\\x82\\xfd\\xfe\\xff'  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variables()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t int                  \t 1      \t 0                              \t\n",
      "1 \t int                  \t 1      \t 1                              \t\n",
      "2 \t int                  \t 1      \t 2                              \t\n",
      "3 \t int                  \t 1      \t 9                              \t\n",
      "4 \t int                  \t 1      \t 10                             \t\n",
      "5 \t int                  \t 1      \t 13                             \t\n",
      "6 \t int                  \t 1      \t 29                             \t\n",
      "7 \t int                  \t 1      \t 30                             \t\n",
      "8 \t int                  \t 1      \t 31                             \t\n",
      "9 \t int                  \t 1      \t 32                             \t\n",
      "10 \t int                  \t 1      \t 65                             \t\n",
      "11 \t int                  \t 1      \t 92                             \t\n",
      "12 \t int                  \t 1      \t 97                             \t\n",
      "13 \t int                  \t 1      \t 128                            \t\n",
      "14 \t int                  \t 1      \t 129                            \t\n",
      "15 \t int                  \t 1      \t 130                            \t\n",
      "16 \t int                  \t 1      \t 253                            \t\n",
      "17 \t int                  \t 1      \t 254                            \t\n",
      "18 \t int                  \t 1      \t 255                            \t\n"
     ]
    }
   ],
   "source": [
    "view(bytes_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this reason the ```bytes``` class has the method ```hex``` which returns a Unicode ```str``` instance of the hexadecimal values without any of the escape sequences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'000102090a0d1d1e1f20415c61808182fdfeff'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes_string.hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the ```byte``` classes ```hex``` method differs from ```builtins``` function ```hex``` which provides a ```0x``` prefix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0c'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes((12, )).hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0xc'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hex(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```builtins``` function ```hex``` can process a **single** large integer that exceeds 1 byte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0x100'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hex(256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whereas the ```byte``` classes ```hex``` method processes **multiple** integers that are within the constrains of a byte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0c22'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes((12, 34)).hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following ```bytes``` instance can be represented as a ```str``` of hexadecimal characters with 2 hexadecimal characters for each byte using the ```bytes``` class method ```hex```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'68656c6c6f'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'hello'.hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice when each of the ASCII characters is supplied using a hexadecimal escape sequence, the default representation simplifies the output displaying the ASCII character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x68\\x65\\x6c\\x6c\\x6f'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```bytes``` class has the class method ```fromhex``` which is an alternative constructor to create a ```bytes``` instance from a Unicode ```str``` instance of hexadecimal characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes.fromhex('68656c6c6f')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The different ways of representing each byte in the ```bytes``` instance can be examined using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   h        e        l        l        o                 w        o        r        l        d        !     \n",
      "  104      101      108      108      111       32      119      111      114      108      100       33    \n",
      "01101000 01100101 01101100 01101100 01101111 00100000 01110111 01101111 01110010 01101100 01100100 00100001 \n",
      "   68       65       6c       6c       6f       20       77       6f       72       6c       64       21    \n",
      "  0x68     0x65     0x6c     0x6c     0x6f     0x20     0x77     0x6f     0x72     0x6c     0x64     0x21   \n",
      "  \\x68     \\x65     \\x6c     \\x6c     \\x6f     \\x20     \\x77     \\x6f     \\x72     \\x6c     \\x64     \\x21   "
     ]
    }
   ],
   "source": [
    "integers = (104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33)\n",
    "bytes(integers)\n",
    "\n",
    "for number in integers:\n",
    "    print(chr(number).center(8), end=' ')\n",
    "    \n",
    "print()\n",
    "for number in integers:\n",
    "    print(str(number).center(8), end=' ')\n",
    "\n",
    "print()\n",
    "for number in integers:\n",
    "    print(bin(number).removeprefix('0b').zfill(8), end=' ')\n",
    "\n",
    "print()\n",
    "for number in integers:\n",
    "    print((hex(number).removeprefix('0x')).center(8), end=' ')    \n",
    "\n",
    "print()\n",
    "for number in integers:\n",
    "    print((r'0x' + hex(number).removeprefix('0x')).center(8), end=' ')\n",
    "\n",
    "print()\n",
    "for number in integers:\n",
    "    print((r'\\x' + hex(number).removeprefix('0x')).center(8), end=' ')\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance can be instantiated from a Unicode ```str``` however the named parameter ```encoding``` needs to be supplied, which gives the instructions to decode a Unicode character outwith the ASCII range. When the simplest encoding ```'ascii'``` is supplied all characters in the Unicode ```str``` must be within the ASCII range. Generally the current standard ```'utf-8'``` is used which is adaptable and encodes each Unicode character in the Unicode ```str``` to 1-4 bytes in the returned ```bytes``` instance:\n",
    "\n",
    "```python\n",
    "bytes(string, /, encoding[, errors]) -> bytes\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('hello', encoding='ascii')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('hello', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not supplying ```encoding``` gives a ```TypeError```:\n",
    "\n",
    "```python\n",
    "bytes('hello')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supplying a non-Unicode character that is not ASCII and specifying ASCII encoding will also give a ```UnicodeDecodeError```:\n",
    "\n",
    "```python\n",
    "bytes('α', encoding='ascii')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance can be cast from an existing ```bytes``` instance:\n",
    "\n",
    "```python\n",
    "bytes(bytes_or_buffer, /) -> immutable copy of bytes_or_buffer\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A ```bytes``` instance can be instantiated by casting a ```bytearray```. The ```bytearray``` is the mutable counterpart to the ```bytes``` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "bytearray_string = bytearray(b'hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "bytes_string = bytes(bytearray_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Size/Shape</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Instance Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>num</th>\n",
       "      <td>int</td>\n",
       "      <td></td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>archive</th>\n",
       "      <td>tuple</td>\n",
       "      <td>1</td>\n",
       "      <td>(97,)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>integers</th>\n",
       "      <td>tuple</td>\n",
       "      <td>12</td>\n",
       "      <td>(104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>number</th>\n",
       "      <td>int</td>\n",
       "      <td></td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bytes_string</th>\n",
       "      <td>bytes</td>\n",
       "      <td>5</td>\n",
       "      <td>b'hello'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bytearray_string</th>\n",
       "      <td>bytearray</td>\n",
       "      <td>5</td>\n",
       "      <td>bytearray(b'hello')</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Type Size/Shape  \\\n",
       "Instance Name                            \n",
       "num                     int              \n",
       "archive               tuple          1   \n",
       "integers              tuple         12   \n",
       "number                  int              \n",
       "bytes_string          bytes          5   \n",
       "bytearray_string  bytearray          5   \n",
       "\n",
       "                                                                       Value  \n",
       "Instance Name                                                                 \n",
       "num                                                                       97  \n",
       "archive                                                                (97,)  \n",
       "integers          (104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33)  \n",
       "number                                                                    33  \n",
       "bytes_string                                                        b'hello'  \n",
       "bytearray_string                                         bytearray(b'hello')  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " A NULL ```bytes``` instance can also be initialised from an ```int```. The ```int``` is used to specify the number of NULL ```bytes``` and is not cast into an individual byte as seen when an ```int``` is provided via a ```tuple```:\n",
    "\n",
    " ```python\n",
    " bytes(int, /) -> bytes object of size given by the parameter initialized with null bytes\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example a ```bytes``` instance occupying 1 bytes can be instantiated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And another one occupying 4 bytes can be instantiated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00\\x00\\x00\\x00'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the ```bytes``` class with out providing any instantiation data will create a single NULL ```bytes``` instance:\n",
    "\n",
    "```python\n",
    "bytes() -> empty bytes object\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b''"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialising data and then populating is more commonly used for mutable datatypes. For an immutable datatype, the instance cannot be modified and the instance name instead gets reassigned to a new instance."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding and Decoding"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a ```bytes``` instance was instantiated from a Unicode ```str``` instance, an encoding translation table was selected; that is a table that maps the bytes sequence to a specific character. There have been many encoding standards developed throughout the years and by default the current standard ```'utf-8'``` should be used. The Unicode ```str``` class is always ```'utf-8'``` and as a consequence is far easier to use than the ```bytes``` class for most text applications:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|encoding|bytes per character|bits per character|byte order|byte order marker BOM|\n",
    "|---|---|---|---|---|\n",
    "|'utf-8'|1, 2, 3, 4|8, 16, 24, 32|big endian| |\n",
    "|'utf-8-sig'|1, 2, 3, 4|8, 16, 24, 32|big endian|efbbbf|\n",
    "|'utf-32'|4|32|little endian|fffe0000|\n",
    "|'utf-32-le'|4|32|little endian| |\n",
    "|'utf-32-be'|4|32|big endian| |\n",
    "|'utf-16'|2|16|little endian|fffe|\n",
    "|'utf-16-le'|2|16|little endian| |\n",
    "|'utf-16-be'|2|16|big endian| |\n",
    "|'latin1'|1|8| ||\n",
    "|'ascii'|1|8| ||"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ASCII\n",
    "\n",
    "```'ascii'``` is the most basic translation table and each character is encoded over ```1``` byte using only half of the possible values. ASCII is restricted to a small subset of English characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_ascii = b'a'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'a'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_ascii"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0x61'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hex(ord('a'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```'ascii'``` encoding scheme was originally developed using 7 bits with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 ** 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for this reason the commands span over the range ```0:128``` (up to and exclusing the upper bound of ```128```). This covers half the possible values of a byte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "256"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 ** 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extended ASCII Variants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the 1990s there were numerous regional translation tables which mapped the second half of the bytes to regional characters. \n",
    "\n",
    "In the UK, ```'latin1'``` was used which includes the ```£``` sign:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb = bytes('£123.45', encoding='latin1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xa3123.45'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "163"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int('0xa3', base=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'£123.45'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb.decode(encoding='latin1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This regional encoding scheme spanned over the full byte allowing the commonly used regional characters. \n",
    "\n",
    "The problem with early regional encoding was that operating systems and browsers were often configured to use a regional encoding scheme that often differed to the encoding scheme the content itself was written in and as a result non-ASCII characters were often incorrectly substituted. This can be seen for example by decoding the ```bytes``` instance above which was originally encoded in ```'latin1'``` with ```'latin2'```, ```'latin3'```, ```'greek'``` and ```'cyrillic'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Ł123.45'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb.decode(encoding='latin2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'£123.45'"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb.decode(encoding='latin3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'£123.45'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb.decode(encoding='greek')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Ѓ123.45'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb.decode(encoding='cyrillic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these formats should be considered as legacy formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UTF-16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Unicode Transformation Format ```'utf-16'``` was a previous standard where each character occupied ```2``` bytes which is ```2 * 8``` bits and is where the name ```16``` comes from. Using 2 bytes instead of 1 bytes per character increases the number of possible combinations to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65536"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 ** 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we count using numbers we use big endian, for example the number twelve is represented using two decimal digits:\n",
    "\n",
    "```python\n",
    "12\n",
    "```\n",
    "\n",
    "This is big endian and the most significant digit 1 which corresponds to 10 is stated first followed by the digit 2 which corresponds to 2 units.\n",
    "\n",
    "This number twelve could also be represented in little endian:\n",
    "\n",
    "```python\n",
    "21\n",
    "```\n",
    "\n",
    "In little endian, the least significant digit, the digit 2 which corresponds to 2 units is stated first followed by the most significant digit 1 which corresponds to 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the ASCII character ```a``` is encoded using the ```'ascii'``` translation table it occupies a single byte, which recall is represented using two hexadecimal characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'a'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('\\x61', encoding='ascii')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ```'utf-16'```, each character must occupy two bytes. For an ASCII character the single byte that was encoded in ```'ascii'``` is taken as the least significant byte and is accompanied by the NULL byte which acts as the most significant byte. In big endian the most significant byte is placed first followed by the least significant byte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0061'"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x00\\x61'.hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In little endian the least significant byte is instead displayed first, followed by the least significant byte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'6100'"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x61\\x00'.hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the two byte instances are examined, the default representation assumes the ```bytes``` instance is using ```'ascii'``` encoding and so the NULL byte displays as an escape character and the ```'ascii'``` character displays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00a'"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x00\\x61' # big endian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'a\\x00'"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x61\\x00' # little endian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the Unicode ```str``` instance ```'abc'``` is examined and encoded in ```'utf-16-be'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00a\\x00b\\x00c'"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('abc', encoding='utf-16-be')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which looks like the following when the bytes corresponding to ASCII characters aren't processed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x00a\\x00b\\x00c'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x00\\x61\\x00\\x62\\x00\\x63' # big endian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the Unicode ```str``` instance ```'abc'``` is instead encoded in ```'utf-16-le'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'a\\x00b\\x00c\\x00'"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('abc', encoding='utf-16-le')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'a\\x00b\\x00c\\x00'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x61\\x00\\x62\\x00\\x63\\x00' # little endian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When ```'utf-16'``` was introduced there was a deviation in the way processors handled characters that spanned over multiple bytes. Some processors used big endian and others used little endian. Intel, the most dominant processor manufacturer at the time favoured little endian. As there was confusion between the two variants of ```'utf-16'```, Microsoft favoured addition of a Byte Order Marker (BOM). The BOM is at the start of the ```bytes``` instance and like every character in ```'utf-16'``` will span over two bytes (4 hexadecimal characters):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'610062006300'"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('abc', encoding='utf-16-le').hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fffe610062006300'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('abc', encoding='utf-16').hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The BOM can be examined by casting an empty ```str``` instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xff\\xfe'"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('', encoding='utf-16')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fffe'"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('', encoding='utf-16').hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```str``` instance corresponding to the Greek letter alpha can be encoded in ```'utf-16-le'```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha_be = bytes('α', encoding='utf-16-le')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hexadecimal values can be examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'b103'"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alpha_be.hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xb1\\x03'"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alpha_be"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ```bytes``` instance can be decoded back to the original ```str``` instance using the correct encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'α'"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alpha_be.decode(encoding='utf-16-le')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the incorrect decoding is used the wrong character is selected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'넃'"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alpha_be.decode(encoding='utf-16-be')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is equivalent to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'넃'"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b'\\x03\\xb1'.decode(encoding='utf-16-le')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a single byte encoding is used, each byte will be represented as a different character. One of the characters is a non-printable ASCII character so displays as ```\\x03```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'±\\x03'"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alpha_be.decode(encoding='latin1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With 16 bytes there are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65536"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 ** 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "combinations. These are not enough to cover characters from all the languages in the world."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UTF-32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore ```'utf-32'``` was developed which spans over 32 bits which is 4 bytes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4294967296"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 ** 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like ```'utf-16'``` there are BOM variations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(b'\\x00\\x00\\x00a', '00000061')"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('\\x61', encoding='utf-32-be'), bytes('\\x61', encoding='utf-32-be').hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(b'a\\x00\\x00\\x00', '61000000')"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('\\x61', encoding='utf-32-le'), bytes('\\x61', encoding='utf-32-le').hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(b'\\x00\\x00\\x00a', 'fffe000061000000')"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('\\x61', encoding='utf-32-be'), bytes('\\x61', encoding='utf-32').hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives groupings of 4 bytes, which is 8 hexadecimal characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "char:               a        b        α        β        悤        悥        🦒        🦓        \n",
      "utf-32-be:          00000061 00000062 000003b1 000003b2 000060a4 000060a5 0001f992 0001f993 \n",
      "utf-32-le:          61000000 62000000 b1030000 b2030000 a4600000 a5600000 92f90100 93f90100 \n",
      "utf-32:    fffe0000 61000000 62000000 b1030000 b2030000 a4600000 a5600000 92f90100 93f90100 "
     ]
    }
   ],
   "source": [
    "word = 'abαβ悤悥🦒🦓'\n",
    "\n",
    "be = bytes(word, encoding='utf-32-be').hex()\n",
    "le = bytes(word, encoding='utf-32-le').hex()\n",
    "bom_le = bytes(word, encoding='utf-32').hex()\n",
    "\n",
    "\n",
    "print('char', end=':               ')\n",
    "for i in word:\n",
    "    print(i, end='        ')\n",
    "\n",
    "print()\n",
    "print('utf-32-be', end=':          ')\n",
    "\n",
    "for i in range(0, len(be), 8):\n",
    "    print(be [i:i+8], end=' ')\n",
    "\n",
    "print()    \n",
    "print('utf-32-le', end=':          ')\n",
    "\n",
    "for i in range(0, len(le), 8):\n",
    "    print(le [i:i+8], end=' ')  \n",
    "    \n",
    "print()    \n",
    "print('utf-32', end=':    ')\n",
    "\n",
    "for i in range(0, len(bom_le), 8):\n",
    "    print(bom_le [i:i+8], end=' ')    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UTF-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main drawback of ```'utf-32'``` is that it requires a lot more memory per character, had byte order issues and each ASCII character now needs to be accompanied by 3 NULL bytes. The current standard ```'utf-8'``` was developed as an adaptable format and characters span over 1-4 bytes and is always big endian:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 byte: 61 62 63 64 65 \n",
      "2 bytes: ceb1 ceb2 ceb3 ceb4 ceb5 \n",
      "3 bytes: e682a4 e682a5 e682a6 e682a7 e682a8 \n",
      "4 bytes: f09fa692 f09fa693 f09fa694 f09fa695 f09fa696 "
     ]
    }
   ],
   "source": [
    "print('1 byte:', end=' ')\n",
    "for unicode_char in 'abcde':\n",
    "    print(bytes(unicode_char, encoding='utf-8').hex(), end=' ')\n",
    "\n",
    "print()\n",
    "print('2 bytes:', end=' ')    \n",
    "for unicode_char in 'αβγδε':\n",
    "    print(bytes(unicode_char, encoding='utf-8').hex(), end=' ')\n",
    "    \n",
    "print()\n",
    "print('3 bytes:', end=' ')  \n",
    "for unicode_char in '悤悥悦悧您':\n",
    "    print(bytes(unicode_char, encoding='utf-8').hex(), end=' ')\n",
    "\n",
    "print()\n",
    "print('4 bytes:', end=' ')  \n",
    "for unicode_char in '🦒🦓🦔🦕🦖':\n",
    "    print(bytes(unicode_char, encoding='utf-8').hex(), end=' ')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally:\n",
    "\n",
    "* 1 byte is the ```'ascii'``` subset.\n",
    "* 2 bytes is extended European characters. ```'utf-16``` is not a subset as ```'utf-8'``` uses a byte pattern that differs from ```'utf-16'```.\n",
    "* 3 bytes are used for additional languages.\n",
    "* 4 bytes are used for emojis. ```'utf-32``` is not a subset as ```'utf-8'``` uses a byte pattern that differs from ```'utf-32'```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the hood the start of the first byte is used to identify if 1, 2, 3 or 4 bytes is used to identify a character:\n",
    "\n",
    "1 byte: XXXXXXXX (ASCII)\n",
    "\n",
    "2 bytes: **110**XXXXX **10**XXXXXX\n",
    "\n",
    "3 bytes: **1110**XXXX **10**XXXXXX **10**XXXXXX\n",
    "\n",
    "4 bytes: **11110**XXX **10**XXXXXX **10**XXXXXX **10**XXXXXX\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4 of the example characters above can be cast into binary and seen to follow the above pattern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('1 byte', '01100001')"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "('1 byte', bin(0x61).removeprefix('0b').zfill(1 * 8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('2 bytes', '1100111010110001')"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "('2 bytes', bin(0xceb1).removeprefix('0b').zfill(2 * 8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('3 bytes', '111001101000001010100100')"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "('3 bytes', bin(0xe682a4).removeprefix('0b').zfill(3 * 8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('4 bytes', '11110000100111111010011010010010')"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "('4 bytes', bin(0xf09fa692).removeprefix('0b').zfill(4 * 8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UTF-8-Sig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since characters ```'utf-8'``` is always big endian and characters encoded over multiple bytes have a byte pattern, there is generally no need for a BOM. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite ```'utf-8'``` not requiring a BOM, Microsoft often include one in their products using a variation ```utf-8-sig``` and therefore may be seen in data exported from popular Microsoft applications such as Notepad or Excel. The BOM can be seen by compared the casting of an empty Unicode string to a bytes string using these different ```'utf-8'``` and ```'utf-8-sig'``` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('', encoding='utf-8').hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'efbbbf'"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes('', encoding='utf-8-sig').hex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```'utf-8'``` is the current standard and should be used by default. The Unicode string ```str``` class is locked to ```'utf-8'``` and is much easier to work with as there is no worry about encoding. \n",
    "\n",
    "When decoding data from another source that is in byte form, ```'utf-8'``` should be used by default to decode it.  If an unwanted BOM appears at the start when decoding data, then the data probably was processed using a Microsoft product with ```'utf-8-sig'```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hardware"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```bytes``` class was the main text class for Python 2. \n",
    "\n",
    "As ```'utf-8'``` became widely adopted as an encoding standard, the text datatype was redeveloped to use ```'utf-8'``` as the only translation table. This simplified class allowed a Unicode character to be considered as a fundamental unit opposed to a byte. Python 3 introduced major changes over Python 2, in particular changes to the default text class. The ```str``` class is the default text class for Python 3 and should be used in most applications, the older text datatype is the ```bytes``` class and is sometimes used when communicating directly with hardware on the hardware level. The ```bytes``` instance below can be transmitted over a serial port:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "011010000110010101101100011011000110111100100000011101110110111101110010011011000110010000100001"
     ]
    }
   ],
   "source": [
    "for number in b'hello world!':\n",
    "    print(bin(number).removeprefix('0b').zfill(8), end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A serial port for example has a signal pin which is configured to transmit or receive a digital time trace. The baud rate, ```9600``` for example means ```9600``` bits are processed in a second. When the bit is ```0```, the voltage of the signal pin is LOW and when the bit is ```1``` the voltage of the signal pin is HIGH."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bytes``` instances are therefore still used when directly interfacing with hardware for example an Arduino. In such applications it is recommended to decode a ```bytes``` instance to a Unicode ```str``` instance as early as possible in a Python program and only cast the Unicode ```str``` instance back to a ```bytes``` instance as late as possible before transmitting it to reduce the possibility of encoding issues."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bytes Identifiers"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the identifiers of the ```bytes``` class and the ```str``` class are examined, and are seen to be largely consistent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'method': ['capitalize',\n",
      "            'center',\n",
      "            'count',\n",
      "            'decode',\n",
      "            'endswith',\n",
      "            'expandtabs',\n",
      "            'find',\n",
      "            'fromhex',\n",
      "            'hex',\n",
      "            'index',\n",
      "            'isalnum',\n",
      "            'isalpha',\n",
      "            'isascii',\n",
      "            'isdigit',\n",
      "            'islower',\n",
      "            'isspace',\n",
      "            'istitle',\n",
      "            'isupper',\n",
      "            'join',\n",
      "            'ljust',\n",
      "            'lower',\n",
      "            'lstrip',\n",
      "            'maketrans',\n",
      "            'partition',\n",
      "            'removeprefix',\n",
      "            'removesuffix',\n",
      "            'replace',\n",
      "            'rfind',\n",
      "            'rindex',\n",
      "            'rjust',\n",
      "            'rpartition',\n",
      "            'rsplit',\n",
      "            'rstrip',\n",
      "            'split',\n",
      "            'splitlines',\n",
      "            'startswith',\n",
      "            'strip',\n",
      "            'swapcase',\n",
      "            'title',\n",
      "            'translate',\n",
      "            'upper',\n",
      "            'zfill'],\n",
      " 'datamodel_method': ['__add__',\n",
      "                      '__buffer__',\n",
      "                      '__bytes__',\n",
      "                      '__contains__',\n",
      "                      '__getitem__',\n",
      "                      '__getnewargs__',\n",
      "                      '__iter__',\n",
      "                      '__len__',\n",
      "                      '__mod__',\n",
      "                      '__mul__',\n",
      "                      '__rmod__',\n",
      "                      '__rmul__']}\n"
     ]
    }
   ],
   "source": [
    "dir2(bytes, object, unique_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'method': ['capitalize',\n",
      "            'casefold',\n",
      "            'center',\n",
      "            'count',\n",
      "            'encode',\n",
      "            'endswith',\n",
      "            'expandtabs',\n",
      "            'find',\n",
      "            'format',\n",
      "            'format_map',\n",
      "            'index',\n",
      "            'isalnum',\n",
      "            'isalpha',\n",
      "            'isascii',\n",
      "            'isdecimal',\n",
      "            'isdigit',\n",
      "            'isidentifier',\n",
      "            'islower',\n",
      "            'isnumeric',\n",
      "            'isprintable',\n",
      "            'isspace',\n",
      "            'istitle',\n",
      "            'isupper',\n",
      "            'join',\n",
      "            'ljust',\n",
      "            'lower',\n",
      "            'lstrip',\n",
      "            'maketrans',\n",
      "            'partition',\n",
      "            'removeprefix',\n",
      "            'removesuffix',\n",
      "            'replace',\n",
      "            'rfind',\n",
      "            'rindex',\n",
      "            'rjust',\n",
      "            'rpartition',\n",
      "            'rsplit',\n",
      "            'rstrip',\n",
      "            'split',\n",
      "            'splitlines',\n",
      "            'startswith',\n",
      "            'strip',\n",
      "            'swapcase',\n",
      "            'title',\n",
      "            'translate',\n",
      "            'upper',\n",
      "            'zfill'],\n",
      " 'datamodel_method': ['__add__',\n",
      "                      '__contains__',\n",
      "                      '__getitem__',\n",
      "                      '__getnewargs__',\n",
      "                      '__iter__',\n",
      "                      '__len__',\n",
      "                      '__mod__',\n",
      "                      '__mul__',\n",
      "                      '__rmod__',\n",
      "                      '__rmul__']}\n"
     ]
    }
   ],
   "source": [
    "dir2(str, object, unique_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unique identifiers can be examined for each class. The ```str``` method ```'encode'``` casts a ```str``` instance to a ```bytes``` instance using a specified translation table to encode with. The ```bytes``` method ```'decode'``` casts a ```bytes``` instance to a ```str``` instance using a specified translation table to decode with. The class method ```fromhex``` and instance ```hex``` can be used to cast from a Unicode ```str``` instance of hexadecimal characters and to ```return``` a Unicode ```str``` instance of hexadecimal characters respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'method': ['decode', 'fromhex', 'hex'],\n",
      " 'datamodel_method': ['__buffer__', '__bytes__']}\n"
     ]
    }
   ],
   "source": [
    "dir2(bytes, str, unique_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'method': ['casefold',\n",
      "            'encode',\n",
      "            'format',\n",
      "            'format_map',\n",
      "            'isdecimal',\n",
      "            'isidentifier',\n",
      "            'isnumeric',\n",
      "            'isprintable']}\n"
     ]
    }
   ],
   "source": [
    "dir2(str, bytes, unique_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some ```str``` methods related to formatting and methods which check groupings of Unicode characters don't have a counterpart available in the ```bytes``` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The consistent identifiers behave consistently between the two classes. The counterpart to ```str``` methods that return a ```str``` instance will instead return a ```bytes``` instance or single byte which recall is an ```int``` between ```0:256```. Supposing the four instances are instantiated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_b = b'abcde'\n",
    "greek_b = bytes('αβγδε', encoding='UTF-8')\n",
    "english_s = 'abcde'\n",
    "greek_s = 'αβγδε'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables can be viewed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Size/Shape</th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Instance Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>english_b</th>\n",
       "      <td>bytes</td>\n",
       "      <td>5</td>\n",
       "      <td>b'abcde'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>english_s</th>\n",
       "      <td>str</td>\n",
       "      <td>5</td>\n",
       "      <td>abcde</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>greek_b</th>\n",
       "      <td>bytes</td>\n",
       "      <td>10</td>\n",
       "      <td>b'\\xce\\xb1\\xce\\xb2\\xce\\xb3\\xce\\xb4\\xce\\xb5'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>greek_s</th>\n",
       "      <td>str</td>\n",
       "      <td>5</td>\n",
       "      <td>αβγδε</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Type Size/Shape                                        Value\n",
       "Instance Name                                                               \n",
       "english_b      bytes          5                                     b'abcde'\n",
       "english_s        str          5                                        abcde\n",
       "greek_b        bytes         10  b'\\xce\\xb1\\xce\\xb2\\xce\\xb3\\xce\\xb4\\xce\\xb5'\n",
       "greek_s          str          5                                        αβγδε"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variables(['english_b', 'english_s', 'greek_b', 'greek_s'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And examined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t int                  \t 1      \t 97                             \t\n",
      "1 \t int                  \t 1      \t 98                             \t\n",
      "2 \t int                  \t 1      \t 99                             \t\n",
      "3 \t int                  \t 1      \t 100                            \t\n",
      "4 \t int                  \t 1      \t 101                            \t\n"
     ]
    }
   ],
   "source": [
    "view(english_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t str                  \t 1      \t a                              \t\n",
      "1 \t str                  \t 1      \t b                              \t\n",
      "2 \t str                  \t 1      \t c                              \t\n",
      "3 \t str                  \t 1      \t d                              \t\n",
      "4 \t str                  \t 1      \t e                              \t\n"
     ]
    }
   ],
   "source": [
    "view(english_s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t int                  \t 1      \t 206                            \t\n",
      "1 \t int                  \t 1      \t 177                            \t\n",
      "2 \t int                  \t 1      \t 206                            \t\n",
      "3 \t int                  \t 1      \t 178                            \t\n",
      "4 \t int                  \t 1      \t 206                            \t\n",
      "5 \t int                  \t 1      \t 179                            \t\n",
      "6 \t int                  \t 1      \t 206                            \t\n",
      "7 \t int                  \t 1      \t 180                            \t\n",
      "8 \t int                  \t 1      \t 206                            \t\n",
      "9 \t int                  \t 1      \t 181                            \t\n"
     ]
    }
   ],
   "source": [
    "view(greek_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index \t Type                 \t Size   \t Value                         \n",
      "0 \t str                  \t 1      \t α                              \t\n",
      "1 \t str                  \t 1      \t β                              \t\n",
      "2 \t str                  \t 1      \t γ                              \t\n",
      "3 \t str                  \t 1      \t δ                              \t\n",
      "4 \t str                  \t 1      \t ε                              \t\n"
     ]
    }
   ],
   "source": [
    "view(greek_s)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the ```len``` of ```english_s``` and ```greek_s``` are consistently ```5``` as there are ```5``` Unicode characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(english_s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(greek_s)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the ```len``` of ```english_b``` is also ```5``` as each ASCII character spans ```1``` byte, however ```greek_s``` is a length of ```10``` as each character spans ```2``` bytes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(english_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(greek_b)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a ```str``` instance, the Unicode character corresponding to that index is returned. For a ```bytes``` instance on the other hand, the byte in the form of an ```int``` is returned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'α'"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_s[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_b[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ceb1ceb2ceb3ceb4ceb5'"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_b.hex()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0xce"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slicing will instead return a ```bytes``` instance. The difference can be seen when ```1``` byte is selected from a slice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xce'"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_b[:1]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The syntax otherwise is consistent to slicing when used in the ```str``` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xce\\xb1'"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_b[:2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xce\\xce\\xce\\xce\\xce'"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_b[::2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xb1\\xb2\\xb3\\xb4\\xb5'"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "greek_b[1::2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Python Tutorials](../readme.md)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  },
  "vscode": {
   "interpreter": {
    "hash": "4648373dafda34926c036cc9e466ef2f8d360789320441bb704a9be21d427d2a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}