Seminar5/Seminar05.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Text, bytes and fun.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions!\n",
    "\n",
    "Why oh why not again please . . ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose you want to write an open-source library in Python. \n",
    "\n",
    "Consider a piece of very important core code of your library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# this is my first open-source library\n",
    "\n",
    "def some_really_complex_function(arg, another_arg, *here_go_more_args, **and_some_kwargs_too):\n",
    "    print('Here I do some really complicated code')\n",
    "    return 42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here I do some really complicated code\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "some_really_complex_function(1, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How to make this function self-explanatory?\n",
    "\n",
    "You should use docstings!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def some_really_complex_function_with_docstring(arg, another_arg, *here_go_more_args, **and_some_kwargs_too):\n",
    "    '''\n",
    "    This is a really complex function.\n",
    "    \n",
    "    Args:\n",
    "        arg: some argument\n",
    "        another_arg: some other argument\n",
    "        \n",
    "    (c) Лягушонок Пеп <------- you see that?\n",
    "    '''\n",
    "    print('Here I do some really complicated code')\n",
    "    # Yeah, and returns 42\n",
    "    return 42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function some_really_complex_function_with_docstring in module __main__:\n",
      "\n",
      "some_really_complex_function_with_docstring(arg, another_arg, *here_go_more_args, **and_some_kwargs_too)\n",
      "    This is a really complex function.\n",
      "    \n",
      "    Args:\n",
      "        arg: some argument\n",
      "        another_arg: some other argument\n",
      "        \n",
      "    (c) Лягушонок Пеп <------- you see that?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(some_really_complex_function_with_docstring)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Wait a minute, does Python allow Unicode?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84\n"
     ]
    }
   ],
   "source": [
    "def дважды_принт(аргумент):\n",
    "    print(аргумент * 2)\n",
    "    \n",
    "дважды_принт(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'дважды_принт'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "дважды_принт.__name__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0_o with great power comes great responsibility\n",
    "\n",
    "Let's harness it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Today's agenda\n",
    "* Strings in Python 3.x\n",
    "* `str` and `bytes`\n",
    "* < Intermission > : Python's builtin functions\n",
    "* Unicode support: encoding & decoding\n",
    "* Encoding detection\n",
    "* Regular expressions 101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rule #0: you shouldn't use cyrillic letters in code. `def дважды_принт():` is unacceptable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some simple string methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Strings support slices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m dolor si\n",
      "mori\n"
     ]
    }
   ],
   "source": [
    "s = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit'\n",
    "print(s[10 : 20])\n",
    "print(s[10 : 20 : 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Immutable, support basic maths (addition, multiplication)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lorem ipsum dolor sit amet, consectetur adipiscing elit!!\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "'str' object does not support item assignment",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-403ba463b4b5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0ms\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m'!'\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0ms\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'?'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: 'str' object does not support item assignment"
     ]
    }
   ],
   "source": [
    "s += '!' * 2\n",
    "print(s)\n",
    "s[-1] = '?'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* String join / split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Lorem', 'ipsum', 'dolor', 'sit', 'amet,', 'consectetur', 'adipiscing', 'elit!!']\n",
      "['Lorem ipsum dolor ', ' amet, consectetur adipiscing elit!!']\n"
     ]
    }
   ],
   "source": [
    "print(s.split())\n",
    "print(s.split('sit'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = s.split(' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loremipsumdolorsitamet,consecteturadipiscingelit!!\n",
      "Lorem ipsum dolor sit amet, consectetur adipiscing elit!!\n",
      "Lorem ^_^ ipsum ^_^ dolor ^_^ sit ^_^ amet, ^_^ consectetur ^_^ adipiscing ^_^ elit!!\n"
     ]
    }
   ],
   "source": [
    "print(''.join(words))\n",
    "print(' '.join(words))\n",
    "print(' ^_^ '.join(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* Basic transformations: upper, lower"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'LOREM IPSUM DOLOR SIT AMET, CONSECTETUR ADIPISCING ELIT!!'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.upper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'lorem ipsum dolor sit amet, consectetur adipiscing elit!!'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Lorem ipsum dolor sit amet, consectetur adipiscing elit!!'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.lower().capitalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Lorem Ipsum Dolor Sit Amet, Consectetur Adipiscing Elit!!'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.title()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* Substring search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'lorem' in s, 'lorem' in s.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.find('ipsum') # returns first index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.find('nonexistent') # or -1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.index('ipsum')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "substring not found",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-15-e483f12804f9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'nonexistent'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: substring not found"
     ]
    }
   ],
   "source": [
    "s.index('nonexistent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* String examination: isalpha, isdigit etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "string\t\tisalpha\t\tisdigit\t\tisspace\n",
      "\"abc\"\t\tTrue\t\tFalse\t\tFalse\n",
      "\"2\"\t\tFalse\t\tTrue\t\tFalse\n",
      "\"   \"\t\tFalse\t\tFalse\t\tTrue\n"
     ]
    }
   ],
   "source": [
    "strings = ['abc', '2', '   ']\n",
    "\n",
    "print('\\t\\t'.join('string isalpha isdigit isspace'.split()))\n",
    "for s in strings:\n",
    "    print(\"\\\"{}\\\"\".format(s), s.isalpha(), s.isdigit(), s.isspace(), sep='\\t\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* Misc: startswith, endswith, strip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Hello, world!'.startswith('Hel')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Hello, world!'.endswith('world')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello world'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'    Hello world    '.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Модуль string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\n",
      "abcdefghijklmnopqrstuvwxyz\n",
      "0123456789\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "' \\t\\n\\r\\x0b\\x0c'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string\n",
    "\n",
    "print(string.ascii_letters)\n",
    "print(string.ascii_lowercase)\n",
    "print(string.digits)\n",
    "string.whitespace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### String formatting\n",
    "\n",
    "https://pyformat.info/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello, Bob'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Old style\n",
    "\n",
    "name = 'Bob'\n",
    "'Hello, %s' % name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello, Bob'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "name = 'Bob'\n",
    "'Hello, %(name)s' % {'name':name} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello, Bob'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# New style\n",
    "name = 'Bob'\n",
    "'Hello, {}'.format(name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, Bob! Hello, Alice! Hello, Bob again!\n"
     ]
    }
   ],
   "source": [
    "names = ['Bob','Alice']\n",
    "print('Hello, {0}! Hello, {1}! Hello, {0} again!'.format(*names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, Bob! Hello, Alice! Hello, Bob again!\n"
     ]
    }
   ],
   "source": [
    "names = {'name1':'Bob','name2':'Alice'}\n",
    "print('Hello, {name1}! Hello, {name2}! Hello, {name1} again!'.format(**names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hey Bob, I have a decimal number 50159747054!\n",
      "Sammy is a happy blue shark!\n",
      "Sammy is shark blue happy a!\n",
      "Sammy is happy happy happy shark!\n",
      "Coordinates: 37.24, -115.81\n"
     ]
    }
   ],
   "source": [
    "number = 50159747054\n",
    "name = 'Bob'\n",
    "\n",
    "print('Hey {}, I have a decimal number {}!'.format(name, number))\n",
    "\n",
    "# Supports indexes\n",
    "print('Sammy is {} {} {} {}!'.format('a', 'happy', 'blue', 'shark'))\n",
    "print('Sammy is {3} {2} {1} {0}!'.format('a', 'happy', 'blue', 'shark'))\n",
    "print('Sammy is {1} {1} {1} {3}!'.format('a', 'happy', 'blue', 'shark'))\n",
    "\n",
    "# and named args\n",
    "print('Coordinates: {latitude}, {longitude}'.format(latitude=37.24, longitude=-115.81))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "В 2020-м году состоится 4-й сезон курса Python 3\n"
     ]
    }
   ],
   "source": [
    "# f-строки\n",
    "\n",
    "year = 2020\n",
    "season = 4\n",
    "\n",
    "print(f'В {year}-м году состоится {season}-й сезон курса Python 3') # Обратите внимание на символ f перед строкой"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "В 2020-м году состоится  4.00-й сезон курса Python 3\n"
     ]
    }
   ],
   "source": [
    "# f-строки поддерживают форматирование чисел\n",
    "\n",
    "year = 2020\n",
    "season = 4\n",
    "\n",
    "# .2f - вещественное число с двумя знаками после запятой\n",
    "print(f'В {year}-м году состоится {season: .2f}-й сезон курса Python 3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "В 2020-м году состоится 4-й сезон курса Python 3\n"
     ]
    }
   ],
   "source": [
    "# внутри f-строк можно выполнять различные операции\n",
    "\n",
    "year = 2020\n",
    "\n",
    "print(f'В {year}-м году состоится {year-2017+1}-й сезон курса Python 3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "В 2020-м году состоится 4-й сезон курса Python 3\n"
     ]
    }
   ],
   "source": [
    "# можно обращаться к элементам списков по индексу\n",
    "\n",
    "years = [2016,2017,2018,2019,2020]\n",
    "season = 4\n",
    "\n",
    "print(f'В {years[season]}-м году состоится {season}-й сезон курса Python 3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "В 2020-м году состоится 4-й сезон PYTHON 3\n"
     ]
    }
   ],
   "source": [
    "# И даже использовать функции и методы\n",
    "\n",
    "year = 2020\n",
    "season = 4\n",
    "name = 'Python 3'\n",
    "\n",
    "print(f'В {year}-м году состоится {season}-й сезон {name.upper()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Symbols & bytes\n",
    "\n",
    "*Символ* (`韩`) - это абстракция.  \n",
    "*Charset* - это набор \"допустимых\" символов.  \n",
    "*Кодировка* - это правила, как записывать символы в компьютере.  \n",
    "\n",
    "**ASCII** - это кодировка для 128 символов. Каждый символ занимает 7 бит.  \n",
    "**КОИ-8** - это кодировка для 256 символов (расширение ASCII для русского языка). Символ занимает 8 бит.  \n",
    "\n",
    "**Unicode** - это стандарт, включающий в себя charset + несколько кодировок (UTF-8, UTF-16, UTF-32).\n",
    "- В UTF-8 - от 1 до 4 байт\n",
    "- UTF-16 - 2 или 4 байта\n",
    "- UTF-32 - 4 байта  \n",
    "\n",
    "#### string - последовательность символов | bytes - последовательность байтов\n",
    "\n",
    "\n",
    "\n",
    "Unicode: symbol identifier (such as `r`, `Я` or `韩`) != byte representation.\n",
    "\n",
    "You can switch between the Unicode symbol (e.g. \"U+1D11E\") and its integer identifier with `ord` and `chr`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "114"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ord('r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(119070, '𝄞', '𝄞')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ord('\\U0001D11E'), chr(ord('\\U0001D11E')), chr(119070)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38889"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ord('韩')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chr(ord('韩')) == '韩'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How can we store it in memory?\n",
    "\n",
    "```\n",
    "Encoding: symbols (human-readable) -> bytes \n",
    "Decoding: bytes -> symbols\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = 'café'\n",
    "len(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 <class 'bytes'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "b'caf\\xc3\\xa9'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = s.encode('utf8')\n",
    "print(len(b), type(b))\n",
    "b # binary representation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'café'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.decode('utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'caf\\xc3\\xa9'\n",
      "0x63 99 c\n",
      "0x61 97 a\n",
      "0x66 102 f\n",
      "0xc3 195 Ã\n",
      "0xa9 169 ©\n"
     ]
    }
   ],
   "source": [
    "print(b)\n",
    "for byte in b:\n",
    "    print(hex(byte), byte, chr(byte))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## < Fun >\n",
    "Let's print the full ASCII table with one print statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First, some Python built-in functions\n",
    "\n",
    "* `map(function, *iterables)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<map at 0x7f88270aa940>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(chr, range(50, 70))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E']\n"
     ]
    }
   ],
   "source": [
    "print(list(map(chr, range(50, 70))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2', '3', '5', '7', '11', '13']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2, 3, 5, 7, 11, 13]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string = '2 3 5 7 11 13' # input()\n",
    "print(string.split())\n",
    "integers = list(map(int, string.split()))\n",
    "integers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* `zip(*iterables)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<zip object at 0x7f8827071b88>\n",
      "[('Bob', 'Bronx'), ('Sam', 'Jackson')]\n"
     ]
    }
   ],
   "source": [
    "names = ['Bob', 'Sam']\n",
    "surnames = ['Bronx', 'Jackson']\n",
    "print(zip(names, surnames))\n",
    "print(list(zip(names, surnames)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### ASCII table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\t||\t0x0\t||\t\u0000\n",
      "1\t||\t0x1\t||\t\u0001\n",
      "2\t||\t0x2\t||\t\u0002\n",
      "3\t||\t0x3\t||\t\u0003\n",
      "4\t||\t0x4\t||\t\u0004\n",
      "5\t||\t0x5\t||\t\u0005\n",
      "6\t||\t0x6\t||\t\u0006\n",
      "7\t||\t0x7\t||\t\u0007\n",
      "8\t||\t0x8\t||\t\b\n",
      "9\t||\t0x9\t||\t\t\n",
      "10\t||\t0xa\t||\t\n",
      "\n",
      "11\t||\t0xb\t||\t\u000b",
      "\n",
      "12\t||\t0xc\t||\t\f",
      "\n",
      "13\t||\t0xd\t||\t\r\n",
      "14\t||\t0xe\t||\t\u000e\n",
      "15\t||\t0xf\t||\t\u000f\n",
      "16\t||\t0x10\t||\t\u0010\n",
      "17\t||\t0x11\t||\t\u0011\n",
      "18\t||\t0x12\t||\t\u0012\n",
      "19\t||\t0x13\t||\t\u0013\n",
      "20\t||\t0x14\t||\t\u0014\n",
      "21\t||\t0x15\t||\t\u0015\n",
      "22\t||\t0x16\t||\t\u0016\n",
      "23\t||\t0x17\t||\t\u0017\n",
      "24\t||\t0x18\t||\t\u0018\n",
      "25\t||\t0x19\t||\t\u0019\n",
      "26\t||\t0x1a\t||\t\u001a\n",
      "27\t||\t0x1b\t||\t\u001b\n",
      "28\t||\t0x1c\t||\t\u001c",
      "\n",
      "29\t||\t0x1d\t||\t\u001d",
      "\n",
      "30\t||\t0x1e\t||\t\u001e",
      "\n",
      "31\t||\t0x1f\t||\t\u001f\n",
      "32\t||\t0x20\t||\t \n",
      "33\t||\t0x21\t||\t!\n",
      "34\t||\t0x22\t||\t\"\n",
      "35\t||\t0x23\t||\t#\n",
      "36\t||\t0x24\t||\t$\n",
      "37\t||\t0x25\t||\t%\n",
      "38\t||\t0x26\t||\t&\n",
      "39\t||\t0x27\t||\t'\n",
      "40\t||\t0x28\t||\t(\n",
      "41\t||\t0x29\t||\t)\n",
      "42\t||\t0x2a\t||\t*\n",
      "43\t||\t0x2b\t||\t+\n",
      "44\t||\t0x2c\t||\t,\n",
      "45\t||\t0x2d\t||\t-\n",
      "46\t||\t0x2e\t||\t.\n",
      "47\t||\t0x2f\t||\t/\n",
      "48\t||\t0x30\t||\t0\n",
      "49\t||\t0x31\t||\t1\n",
      "50\t||\t0x32\t||\t2\n",
      "51\t||\t0x33\t||\t3\n",
      "52\t||\t0x34\t||\t4\n",
      "53\t||\t0x35\t||\t5\n",
      "54\t||\t0x36\t||\t6\n",
      "55\t||\t0x37\t||\t7\n",
      "56\t||\t0x38\t||\t8\n",
      "57\t||\t0x39\t||\t9\n",
      "58\t||\t0x3a\t||\t:\n",
      "59\t||\t0x3b\t||\t;\n",
      "60\t||\t0x3c\t||\t<\n",
      "61\t||\t0x3d\t||\t=\n",
      "62\t||\t0x3e\t||\t>\n",
      "63\t||\t0x3f\t||\t?\n",
      "64\t||\t0x40\t||\t@\n",
      "65\t||\t0x41\t||\tA\n",
      "66\t||\t0x42\t||\tB\n",
      "67\t||\t0x43\t||\tC\n",
      "68\t||\t0x44\t||\tD\n",
      "69\t||\t0x45\t||\tE\n",
      "70\t||\t0x46\t||\tF\n",
      "71\t||\t0x47\t||\tG\n",
      "72\t||\t0x48\t||\tH\n",
      "73\t||\t0x49\t||\tI\n",
      "74\t||\t0x4a\t||\tJ\n",
      "75\t||\t0x4b\t||\tK\n",
      "76\t||\t0x4c\t||\tL\n",
      "77\t||\t0x4d\t||\tM\n",
      "78\t||\t0x4e\t||\tN\n",
      "79\t||\t0x4f\t||\tO\n",
      "80\t||\t0x50\t||\tP\n",
      "81\t||\t0x51\t||\tQ\n",
      "82\t||\t0x52\t||\tR\n",
      "83\t||\t0x53\t||\tS\n",
      "84\t||\t0x54\t||\tT\n",
      "85\t||\t0x55\t||\tU\n",
      "86\t||\t0x56\t||\tV\n",
      "87\t||\t0x57\t||\tW\n",
      "88\t||\t0x58\t||\tX\n",
      "89\t||\t0x59\t||\tY\n",
      "90\t||\t0x5a\t||\tZ\n",
      "91\t||\t0x5b\t||\t[\n",
      "92\t||\t0x5c\t||\t\\\n",
      "93\t||\t0x5d\t||\t]\n",
      "94\t||\t0x5e\t||\t^\n",
      "95\t||\t0x5f\t||\t_\n",
      "96\t||\t0x60\t||\t`\n",
      "97\t||\t0x61\t||\ta\n",
      "98\t||\t0x62\t||\tb\n",
      "99\t||\t0x63\t||\tc\n",
      "100\t||\t0x64\t||\td\n",
      "101\t||\t0x65\t||\te\n",
      "102\t||\t0x66\t||\tf\n",
      "103\t||\t0x67\t||\tg\n",
      "104\t||\t0x68\t||\th\n",
      "105\t||\t0x69\t||\ti\n",
      "106\t||\t0x6a\t||\tj\n",
      "107\t||\t0x6b\t||\tk\n",
      "108\t||\t0x6c\t||\tl\n",
      "109\t||\t0x6d\t||\tm\n",
      "110\t||\t0x6e\t||\tn\n",
      "111\t||\t0x6f\t||\to\n",
      "112\t||\t0x70\t||\tp\n",
      "113\t||\t0x71\t||\tq\n",
      "114\t||\t0x72\t||\tr\n",
      "115\t||\t0x73\t||\ts\n",
      "116\t||\t0x74\t||\tt\n",
      "117\t||\t0x75\t||\tu\n",
      "118\t||\t0x76\t||\tv\n",
      "119\t||\t0x77\t||\tw\n",
      "120\t||\t0x78\t||\tx\n",
      "121\t||\t0x79\t||\ty\n",
      "122\t||\t0x7a\t||\tz\n",
      "123\t||\t0x7b\t||\t{\n",
      "124\t||\t0x7c\t||\t|\n",
      "125\t||\t0x7d\t||\t}\n",
      "126\t||\t0x7e\t||\t~\n",
      "127\t||\t0x7f\t||\t\n"
     ]
    }
   ],
   "source": [
    "for char_index in range(128):\n",
    "    print('{}\\t||\\t{}\\t||\\t{}'.format(char_index, hex(char_index), chr(char_index)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Can you do this in one line?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\t||\t0x0\t||\t\u0000\n",
      "1\t||\t0x1\t||\t\u0001\n",
      "2\t||\t0x2\t||\t\u0002\n",
      "3\t||\t0x3\t||\t\u0003\n",
      "4\t||\t0x4\t||\t\u0004\n",
      "5\t||\t0x5\t||\t\u0005\n",
      "6\t||\t0x6\t||\t\u0006\n",
      "7\t||\t0x7\t||\t\u0007\n",
      "8\t||\t0x8\t||\t\b\n",
      "9\t||\t0x9\t||\t\t\n",
      "10\t||\t0xa\t||\t\n",
      "\n",
      "11\t||\t0xb\t||\t\u000b",
      "\n",
      "12\t||\t0xc\t||\t\f",
      "\n",
      "13\t||\t0xd\t||\t\r\n",
      "14\t||\t0xe\t||\t\u000e\n",
      "15\t||\t0xf\t||\t\u000f\n",
      "16\t||\t0x10\t||\t\u0010\n",
      "17\t||\t0x11\t||\t\u0011\n",
      "18\t||\t0x12\t||\t\u0012\n",
      "19\t||\t0x13\t||\t\u0013\n",
      "20\t||\t0x14\t||\t\u0014\n",
      "21\t||\t0x15\t||\t\u0015\n",
      "22\t||\t0x16\t||\t\u0016\n",
      "23\t||\t0x17\t||\t\u0017\n",
      "24\t||\t0x18\t||\t\u0018\n",
      "25\t||\t0x19\t||\t\u0019\n",
      "26\t||\t0x1a\t||\t\u001a\n",
      "27\t||\t0x1b\t||\t\u001b\n",
      "28\t||\t0x1c\t||\t\u001c",
      "\n",
      "29\t||\t0x1d\t||\t\u001d",
      "\n",
      "30\t||\t0x1e\t||\t\u001e",
      "\n",
      "31\t||\t0x1f\t||\t\u001f\n",
      "32\t||\t0x20\t||\t \n",
      "33\t||\t0x21\t||\t!\n",
      "34\t||\t0x22\t||\t\"\n",
      "35\t||\t0x23\t||\t#\n",
      "36\t||\t0x24\t||\t$\n",
      "37\t||\t0x25\t||\t%\n",
      "38\t||\t0x26\t||\t&\n",
      "39\t||\t0x27\t||\t'\n",
      "40\t||\t0x28\t||\t(\n",
      "41\t||\t0x29\t||\t)\n",
      "42\t||\t0x2a\t||\t*\n",
      "43\t||\t0x2b\t||\t+\n",
      "44\t||\t0x2c\t||\t,\n",
      "45\t||\t0x2d\t||\t-\n",
      "46\t||\t0x2e\t||\t.\n",
      "47\t||\t0x2f\t||\t/\n",
      "48\t||\t0x30\t||\t0\n",
      "49\t||\t0x31\t||\t1\n",
      "50\t||\t0x32\t||\t2\n",
      "51\t||\t0x33\t||\t3\n",
      "52\t||\t0x34\t||\t4\n",
      "53\t||\t0x35\t||\t5\n",
      "54\t||\t0x36\t||\t6\n",
      "55\t||\t0x37\t||\t7\n",
      "56\t||\t0x38\t||\t8\n",
      "57\t||\t0x39\t||\t9\n",
      "58\t||\t0x3a\t||\t:\n",
      "59\t||\t0x3b\t||\t;\n",
      "60\t||\t0x3c\t||\t<\n",
      "61\t||\t0x3d\t||\t=\n",
      "62\t||\t0x3e\t||\t>\n",
      "63\t||\t0x3f\t||\t?\n",
      "64\t||\t0x40\t||\t@\n",
      "65\t||\t0x41\t||\tA\n",
      "66\t||\t0x42\t||\tB\n",
      "67\t||\t0x43\t||\tC\n",
      "68\t||\t0x44\t||\tD\n",
      "69\t||\t0x45\t||\tE\n",
      "70\t||\t0x46\t||\tF\n",
      "71\t||\t0x47\t||\tG\n",
      "72\t||\t0x48\t||\tH\n",
      "73\t||\t0x49\t||\tI\n",
      "74\t||\t0x4a\t||\tJ\n",
      "75\t||\t0x4b\t||\tK\n",
      "76\t||\t0x4c\t||\tL\n",
      "77\t||\t0x4d\t||\tM\n",
      "78\t||\t0x4e\t||\tN\n",
      "79\t||\t0x4f\t||\tO\n",
      "80\t||\t0x50\t||\tP\n",
      "81\t||\t0x51\t||\tQ\n",
      "82\t||\t0x52\t||\tR\n",
      "83\t||\t0x53\t||\tS\n",
      "84\t||\t0x54\t||\tT\n",
      "85\t||\t0x55\t||\tU\n",
      "86\t||\t0x56\t||\tV\n",
      "87\t||\t0x57\t||\tW\n",
      "88\t||\t0x58\t||\tX\n",
      "89\t||\t0x59\t||\tY\n",
      "90\t||\t0x5a\t||\tZ\n",
      "91\t||\t0x5b\t||\t[\n",
      "92\t||\t0x5c\t||\t\\\n",
      "93\t||\t0x5d\t||\t]\n",
      "94\t||\t0x5e\t||\t^\n",
      "95\t||\t0x5f\t||\t_\n",
      "96\t||\t0x60\t||\t`\n",
      "97\t||\t0x61\t||\ta\n",
      "98\t||\t0x62\t||\tb\n",
      "99\t||\t0x63\t||\tc\n",
      "100\t||\t0x64\t||\td\n",
      "101\t||\t0x65\t||\te\n",
      "102\t||\t0x66\t||\tf\n",
      "103\t||\t0x67\t||\tg\n",
      "104\t||\t0x68\t||\th\n",
      "105\t||\t0x69\t||\ti\n",
      "106\t||\t0x6a\t||\tj\n",
      "107\t||\t0x6b\t||\tk\n",
      "108\t||\t0x6c\t||\tl\n",
      "109\t||\t0x6d\t||\tm\n",
      "110\t||\t0x6e\t||\tn\n",
      "111\t||\t0x6f\t||\to\n",
      "112\t||\t0x70\t||\tp\n",
      "113\t||\t0x71\t||\tq\n",
      "114\t||\t0x72\t||\tr\n",
      "115\t||\t0x73\t||\ts\n",
      "116\t||\t0x74\t||\tt\n",
      "117\t||\t0x75\t||\tu\n",
      "118\t||\t0x76\t||\tv\n",
      "119\t||\t0x77\t||\tw\n",
      "120\t||\t0x78\t||\tx\n",
      "121\t||\t0x79\t||\ty\n",
      "122\t||\t0x7a\t||\tz\n",
      "123\t||\t0x7b\t||\t{\n",
      "124\t||\t0x7c\t||\t|\n",
      "125\t||\t0x7d\t||\t}\n",
      "126\t||\t0x7e\t||\t~\n",
      "127\t||\t0x7f\t||\t\n"
     ]
    }
   ],
   "source": [
    "# ascii_range = range(128)\n",
    "\n",
    "print(*['\\t||\\t'.join(map(str, row)) for row in zip(range(128), map(hex, range(128)), map(chr, range(128)))], sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## < /Fun >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which encodings can we use? Lots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "latin_1\t\tEl Niño\t\tb'El Ni\\xf1o'\n",
      "utf_8\t\tEl Niño\t\tb'El Ni\\xc3\\xb1o'\n",
      "utf_16\t\tEl Niño\t\tb'\\xff\\xfeE\\x00l\\x00 \\x00N\\x00i\\x00\\xf1\\x00o\\x00'\n",
      "cp437\t\tEl Niño\t\tb'El Ni\\xa4o'\n"
     ]
    }
   ],
   "source": [
    "string = 'El Niño'\n",
    "\n",
    "for codec in ['latin_1', 'utf_8', 'utf_16', 'cp437']:\n",
    "    encoded = string.encode(codec)\n",
    "    print(codec, encoded.decode(codec), encoded, sep='\\t\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Some of them won't work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'S\\xc3\\xa3o Paulo'\n",
      "b'\\xff\\xfeS\\x00\\xe3\\x00o\\x00 \\x00P\\x00a\\x00u\\x00l\\x00o\\x00'\n",
      "b'S\\xe3o Paulo'\n"
     ]
    },
    {
     "ename": "UnicodeEncodeError",
     "evalue": "'charmap' codec can't encode character '\\xe3' in position 1: character maps to <undefined>",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnicodeEncodeError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-36-c0871d73a1d9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf_16'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'iso8859_1'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'cp437'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/workspace/shad/lib/python3.5/encodings/cp437.py\u001b[0m in \u001b[0;36mencode\u001b[0;34m(self, input, errors)\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'strict'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcharmap_encode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mencoding_map\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'strict'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mUnicodeEncodeError\u001b[0m: 'charmap' codec can't encode character '\\xe3' in position 1: character maps to <undefined>"
     ]
    }
   ],
   "source": [
    "city = 'São Paulo'\n",
    "print(city.encode('utf_8'))\n",
    "print(city.encode('utf_16'))\n",
    "print(city.encode('iso8859_1'))\n",
    "print(city.encode('cp437'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'So Paulo'\n",
      "b'S?o Paulo'\n",
      "b'S&#227;o Paulo'\n"
     ]
    }
   ],
   "source": [
    "print(city.encode('cp437', errors='ignore')) # bad\n",
    "print(city.encode('cp437', errors='replace')) # better\n",
    "print(city.encode('cp437', errors='xmlcharrefreplace')) # still not perfect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also look at the `codecs` module, it was widely used in Python 2 but is obsolete now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Files support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('unicode_file.txt', 'w', encoding='utf-16le') as f:\n",
    "    f.write('韩国烧酒')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "UnicodeDecodeError",
     "evalue": "'utf-8' codec can't decode bytes in position 0-1: invalid continuation byte",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnicodeDecodeError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-36-35fe901e1e13>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'unicode_file.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/codecs.py\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(self, input, final)\u001b[0m\n\u001b[1;32m    320\u001b[0m         \u001b[0;31m# decode input (taking the buffer into account)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    321\u001b[0m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuffer\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 322\u001b[0;31m         \u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconsumed\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfinal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    323\u001b[0m         \u001b[0;31m# keep undecoded input until the next call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    324\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuffer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mconsumed\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode bytes in position 0-1: invalid continuation byte"
     ]
    }
   ],
   "source": [
    "with open('unicode_file.txt', 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "UnicodeError",
     "evalue": "UTF-16 stream does not start with BOM",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnicodeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-40-339b00e3c746>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'unicode_file.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf-16'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/workspace/shad/lib/python3.5/codecs.py\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(self, input, final)\u001b[0m\n\u001b[1;32m    319\u001b[0m         \u001b[0;31m# decode input (taking the buffer into account)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    320\u001b[0m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuffer\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 321\u001b[0;31m         \u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconsumed\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfinal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    322\u001b[0m         \u001b[0;31m# keep undecoded input until the next call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    323\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuffer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mconsumed\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/workspace/shad/lib/python3.5/encodings/utf_16.py\u001b[0m in \u001b[0;36m_buffer_decode\u001b[0;34m(self, input, errors, final)\u001b[0m\n\u001b[1;32m     65\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecoder\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutf_16_be_decode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m             \u001b[0;32melif\u001b[0m \u001b[0mconsumed\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mUnicodeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"UTF-16 stream does not start with BOM\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconsumed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfinal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mUnicodeError\u001b[0m: UTF-16 stream does not start with BOM"
     ]
    }
   ],
   "source": [
    "with open('unicode_file.txt', 'r', encoding='utf-16') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "韩国烧酒\n"
     ]
    }
   ],
   "source": [
    "with open('unicode_file.txt', 'r', encoding='utf-16le') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So far we didn't encounter any **tough** cases.\n",
    "\n",
    "We can handle them with `chardet` and `UnicodeDammit`, which is part of BeautifulSoup package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### chardet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# !pip install chardet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib, chardet\n",
    "\n",
    "rawdata = urllib.request.urlopen('http://yahoo.co.jp/').read()\n",
    "chardet.detect(rawdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### UnicodeDammit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# !pip install bs4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import UnicodeDammit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a really weird string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "☃☃☃\n",
      "“I like snowmen!”\n"
     ]
    }
   ],
   "source": [
    "snowmen = (u'\\N{SNOWMAN}' * 3)\n",
    "print(snowmen)\n",
    "quote = (u'\\N{LEFT DOUBLE QUOTATION MARK}I like snowmen!\\N{RIGHT DOUBLE QUOTATION MARK}')\n",
    "print(quote)\n",
    "doc = snowmen.encode('utf8') + quote.encode('windows_1252')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\xe2\\x98\\x83\\xe2\\x98\\x83\\xe2\\x98\\x83\\x93I like snowmen!\\x94'\n",
      "â˜ƒâ˜ƒâ˜ƒ“I like snowmen!”\n"
     ]
    },
    {
     "ename": "UnicodeDecodeError",
     "evalue": "'utf-8' codec can't decode byte 0x93 in position 9: invalid start byte",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnicodeDecodeError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-42-f5fee7916f51>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# â˜ƒâ˜ƒâ˜ƒ“I like snowmen!”\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdoc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m \u001b[0;31m# ☃☃☃�I like snowmen!�\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode byte 0x93 in position 9: invalid start byte"
     ]
    }
   ],
   "source": [
    "print(doc)\n",
    "# ☃☃☃�I like snowmen!�\n",
    "\n",
    "print(doc.decode('windows-1252'))\n",
    "# â˜ƒâ˜ƒâ˜ƒ“I like snowmen!”\n",
    "\n",
    "print(doc.decode('utf8'))\n",
    "# ☃☃☃�I like snowmen!�"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "UnicodeDammit CAN handle it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\xe2\\x98\\x83\\xe2\\x98\\x83\\xe2\\x98\\x83\\xe2\\x80\\x9cI like snowmen!\\xe2\\x80\\x9d'\n",
      "☃☃☃“I like snowmen!”\n"
     ]
    }
   ],
   "source": [
    "new_doc = UnicodeDammit.detwingle(doc)\n",
    "print(new_doc)\n",
    "print(new_doc.decode('utf8'))\n",
    "# ☃☃☃“I like snowmen!”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regular expressions 101\n",
    "\n",
    "Regular expressions debuggins is easy to do with https://regex101.com/\n",
    "\n",
    "Metasymbols: ```. ˆ $ * + ? { } [ ] | ( )```\n",
    "\n",
    "Documentation: https://docs.python.org/3/library/re.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['abc', 'abc']\n",
      "['a6c']\n",
      "['abc', 'acc', 'abc', 'a6c']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "string = '__abc__acc__abc__a6c__'\n",
    "print(re.findall(r'abc', string))\n",
    "print(re.findall(r'a\\dc', string))\n",
    "print(re.findall(r'a\\wc', string))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__***__***__***__***__\n"
     ]
    }
   ],
   "source": [
    "print(re.sub(r'a\\wc', '***', string))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text    : Français złoty Österreich\n",
      "Pattern : \\w+\n",
      "ASCII   : ['Fran', 'ais', 'z', 'oty', 'sterreich']\n",
      "Unicode : ['Français', 'złoty', 'Österreich']\n"
     ]
    }
   ],
   "source": [
    "text = u'Français złoty Österreich'\n",
    "pattern = r'\\w+'\n",
    "ascii_pattern = re.compile(pattern, re.ASCII)\n",
    "unicode_pattern = re.compile(pattern)\n",
    "\n",
    "print('Text    :', text)\n",
    "print('Pattern :', pattern)\n",
    "print('ASCII   :', list(ascii_pattern.findall(text)))\n",
    "print('Unicode :', list(unicode_pattern.findall(text)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## So you want to write an open-source library, remember?\n",
    "### Or just pass your 1st Python review\n",
    "\n",
    "How will you do that? Using Version Control Systems!\n",
    "\n",
    "Git especially. And Github. \n",
    "\n",
    "http://rogerdudler.github.io/git-guide/\n",
    "\n",
    "Don't worry, you'll need only `init`, `clone`, `add`, `commit` and `push` commands. Maybe some more.\n",
    "\n",
    "If you're stuck, just ask for help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}