Task08.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:50:04.186203Z",
     "start_time": "2021-01-05T14:50:04.170202Z"
    }
   },
   "source": [
    "<center><h1>第八章 文本数据</h1></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:23:19.318802Z",
     "start_time": "2021-01-06T11:23:17.204086Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、str对象\n",
    "### 1. str对象的设计意图\n",
    "\n",
    "`str`对象是定义在`Index`或`Series`上的属性，专门用于逐元素处理文本内容，其内部定义了大量方法，因此对一个序列进行文本处理，首先需要获取其`str`对象。在Python标准库中也有`str`模块，为了使用上的便利，有许多函数的用法`pandas`照搬了它的设计，例如字母转为大写的操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:51:09.835044Z",
     "start_time": "2021-01-05T14:51:09.790045Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ABCD'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var = 'abcd'\n",
    "str.upper(var) # Python内置str模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:51:20.573279Z",
     "start_time": "2021-01-05T14:51:20.550282Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.strings.accessor.StringMethods at 0x2995700e580>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['abcd', 'efg', 'hi'])\n",
    "s.str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:51:30.063725Z",
     "start_time": "2021-01-05T14:51:30.033729Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    ABCD\n",
       "1     EFG\n",
       "2      HI\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.upper() # pandas中str对象上的upper方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据文档`API`材料，在`pandas`的50个`str`对象方法中，有31个是和标准库中的`str`模块方法同名且功能一致，这为批量处理序列提供了有力的工具。\n",
    "\n",
    "### 2. []索引器\n",
    "\n",
    "对于`str`对象而言，可理解为其对字符串进行了序列化的操作，例如在一般的字符串中，通过`[]`可以取出某个位置的元素："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:51:57.722437Z",
     "start_time": "2021-01-05T14:51:57.693979Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同时也能通过切片得到子串："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:52:30.246012Z",
     "start_time": "2021-01-05T14:52:30.228013Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'db'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var[-1: 0 : -2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过对`str`对象使用`[]`索引器，可以完成完全一致的功能，并且如果超出范围则返回缺失值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:52:57.900919Z",
     "start_time": "2021-01-05T14:52:57.875921Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "1    e\n",
       "2    h\n",
       "dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:53:42.605981Z",
     "start_time": "2021-01-05T14:53:42.583986Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    db\n",
       "1     g\n",
       "2     i\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str[-1 : 0: -2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:54:26.133313Z",
     "start_time": "2021-01-05T14:54:26.113322Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    c\n",
       "1    f\n",
       "2     \n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str[-2 : 0: -2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:56:36.974945Z",
     "start_time": "2021-01-05T14:56:36.952938Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      c\n",
       "1      g\n",
       "2    NaN\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. string类型\n",
    "\n",
    "在上一章提到，从`pandas`的`1.0.0`版本开始，引入了`string`类型，其引入的动机在于：原来所有的字符串类型都会以`object`类型的`Series`进行存储，但`object`类型只应当存储混合类型，例如同时存储浮点、字符串、字典、列表、自定义类型等，因此字符串有必要同数值型或`category`一样，具有自己的数据存放类型，从而引入了`string`类型。\n",
    "\n",
    "总体上说，绝大多数对于`object`和`string`类型的序列使用`str`对象方法产生的结果是一致，但是在下面提到的两点上有较大差异：\n",
    "\n",
    "首先，应当尽量保证每一个序列中的值都是字符串的情况下才使用`str`属性，但这并不是必须的，其必要条件是序列中至少有一个可迭代（Iterable）对象，包括但不限于字符串、字典、列表。对于一个可迭代对象，`string`类型的`str`对象和`object`类型的`str`对象返回结果可能是不同的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:07:05.296115Z",
     "start_time": "2021-01-05T15:07:05.267129Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    temp_1\n",
       "1         b\n",
       "2       NaN\n",
       "3         y\n",
       "dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([{1: 'temp_1', 2: 'temp_2'}, ['a', 'b'], 0.5, 'my_string'])\n",
    "s.str[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T14:59:38.406713Z",
     "start_time": "2021-01-05T14:59:38.362710Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    '\n",
       "2    .\n",
       "3    y\n",
       "dtype: string"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string').str[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:01:00.653746Z",
     "start_time": "2021-01-05T15:01:00.631750Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    {1: 'temp_1', 2: 'temp_2'}\n",
       "1                    ['a', 'b']\n",
       "2                           0.5\n",
       "3                     my_string\n",
       "dtype: string"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:07:08.281266Z",
     "start_time": "2021-01-05T15:07:08.261269Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    {\n",
       "1    [\n",
       "2    0\n",
       "3    m\n",
       "dtype: string"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string').str[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了最后一个字符串元素，前三个元素返回的值都不同，其原因在于当序列类型为`object`时，是对于每一个元素进行`[]`索引，因此对于字典而言，返回temp_1字符串，对于列表则返回第二个值，而第三个为不可迭代对象，返回缺失值，第四个是对字符串进行`[]`索引。而`string`类型的`str`对象先把整个元素转为字面意义的字符串，例如对于列表而言，第一个元素即 \"{\"，而对于最后一个字符串元素而言，恰好转化前后的表示方法一致，因此结果和`object`类型一致。\n",
    "\n",
    "除了对于某些对象的`str`序列化方法不同之外，两者另外的一个差别在于，`string`类型是`Nullable`类型，但`object`不是。这意味着`string`类型的序列，如果调用的`str`方法返回值为整数`Series`和布尔`Series`时，其分别对应的`dtype`是`Int`和`boolean`的`Nullable`类型，而`object`类型则会分别返回`int/float`和`bool/object`，取决于缺失值的存在与否。同时，字符串的比较操作，也具有相似的特性，`string`返回`Nullable`类型，但`object`不会。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:07:18.412979Z",
     "start_time": "2021-01-05T15:07:18.386013Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['a'])\n",
    "s.str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:04:27.299118Z",
     "start_time": "2021-01-05T15:04:27.271113Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string').str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:04:34.315335Z",
     "start_time": "2021-01-05T15:04:34.293337Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    True\n",
       "dtype: bool"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s == 'a'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:05:02.724774Z",
     "start_time": "2021-01-05T15:05:02.706774Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a\n",
       "dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:05:03.483592Z",
     "start_time": "2021-01-05T15:05:03.463598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    True\n",
       "dtype: boolean"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string') == 'a'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:07:27.751187Z",
     "start_time": "2021-01-05T15:07:27.738184Z"
    }
   },
   "outputs": [],
   "source": [
    "s = pd.Series(['a', np.nan]) # 带有缺失值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:07:35.186904Z",
     "start_time": "2021-01-05T15:07:35.164906Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:08:08.678421Z",
     "start_time": "2021-01-05T15:08:08.662422Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.strings.accessor.StringMethods at 0x2995707eeb0>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:08:28.001925Z",
     "start_time": "2021-01-05T15:08:27.979926Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       1\n",
       "1    <NA>\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string').str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:08:34.277122Z",
     "start_time": "2021-01-05T15:08:34.256118Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s == 'a'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:08:53.203948Z",
     "start_time": "2021-01-05T15:08:53.181952Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    True\n",
       "1    <NA>\n",
       "dtype: boolean"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.astype('string') == 'a'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后需要注意的是，对于全体元素为数值类型的序列，即使其类型为`object`或者`category`也不允许直接使用`str`属性。如果需要把数字当成`string`类型处理，可以使用`astype`强制转换为`string`类型的`Series`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:09:21.987565Z",
     "start_time": "2021-01-05T15:09:21.961567Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    4\n",
       "2    7\n",
       "dtype: string"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([12, 345, 6789])\n",
    "s.astype('string').str[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、正则表达式基础\n",
    "\n",
    "这一节的两个表格来自于[learn-regex-zh](https://github.com/cdoco/learn-regex-zh)这个关于正则表达式项目，其使用`MIT`开源许可协议。这里只是介绍正则表达式的基本用法，需要系统学习的读者可参考[正则表达式必知必会](https://book.douban.com/subject/26285406/)一书。\n",
    "\n",
    "### 1. 一般字符的匹配\n",
    "\n",
    "正则表达式是一种按照某种正则模式，从左到右匹配字符串中内容的一种工具。对于一般的字符而言，它可以找到其所在的位置，这里为了演示便利，使用了`python`中`re`模块的`findall`函数来匹配所有出现过但不重叠的模式，第一个参数是正则表达式，第二个参数是待匹配的字符串。例如，在下面的字符串中找出`apple`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:11:50.303731Z",
     "start_time": "2021-01-05T15:11:50.286764Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Apple', 'Apple']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "re.findall('Apple', 'Apple! This Is an Apple!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|元字符 |   描述 |\n",
    "| :-----| ----: |\n",
    "|.       |    匹配除换行符以外的任意字符|\n",
    "|\\[ \\]     |      字符类，匹配方括号中包含的任意字符|\n",
    "|\\[^ \\]     |      否定字符类，匹配方括号中不包含的任意字符|\n",
    "|\\*       |    匹配前面的子表达式零次或多次|\n",
    "|\\+       |    匹配前面的子表达式一次或多次|\n",
    "|?        |   匹配前面的子表达式零次或一次|\n",
    "|{n,m}    |       花括号，匹配前面字符至少 n 次，但是不超过 m 次|\n",
    "|(xyz)   |        字符组，按照确切的顺序匹配字符xyz|\n",
    "|\\|     |      分支结构，匹配符号之前的字符或后面的字符|\n",
    "|\\\\    |       转义符，它可以还原元字符原来的含义|\n",
    "|^    |       匹配行的开始|\n",
    "|$   |        匹配行的结束|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:18:58.595624Z",
     "start_time": "2021-01-05T15:18:58.577629Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('.', 'abc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:19:31.693470Z",
     "start_time": "2021-01-05T15:19:31.675473Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'c']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('[ac]', 'abc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:20:04.136717Z",
     "start_time": "2021-01-05T15:20:04.119724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['b']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('[^ac]', 'abc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:20:24.088318Z",
     "start_time": "2021-01-05T15:20:24.071319Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aa', 'aa', 'bb', 'bb']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('[ab]{2}', 'aaaabbbb') # {n}指匹配n次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:20:30.600719Z",
     "start_time": "2021-01-05T15:20:30.581111Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aaa', 'abb']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('[ab]{3}', 'aaaabbbb') # {n}指匹配n次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:20:52.558127Z",
     "start_time": "2021-01-05T15:20:52.538137Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aaa', 'bbb']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('aaa|bbb', 'aaaabbbb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:21:42.430302Z",
     "start_time": "2021-01-05T15:21:42.411310Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a?', 'a*']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('a\\\\?|a\\*', 'aa?a*a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:23:30.676148Z",
     "start_time": "2021-01-05T15:23:30.660149Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ab', 'aa', 'c', 'ad', 'aa', 'e']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('a?.', 'abaacadaae') # 匹配a 1次或0次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:23:37.245622Z",
     "start_time": "2021-01-05T15:23:37.229375Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aba', 'ac', 'ad', 'aa']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('ab?.', 'abaacadaae') # 匹配b 1次或0次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:24:53.944451Z",
     "start_time": "2021-01-05T15:24:53.924455Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aba', 'aa']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('ab?a', 'abaacadaae') # b 可以匹配 1次或 0次，前后两个 a 是固定的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:25:14.596303Z",
     "start_time": "2021-01-05T15:25:14.580312Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ab', 'aa', 'c', 'ad', 'aa', 'e']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('[ab]?.', 'abaacadaae') # 效果与三行前相似"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:29:36.066984Z",
     "start_time": "2021-01-05T15:29:36.049985Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ab', '', '', '', '', '', '', '']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('(ab)?.', 'abaacadaae') # 就是按照 ab 整体来匹配，但是没有能实现 类似 aba 的匹配"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:29:44.827930Z",
     "start_time": "2021-01-05T15:29:44.811932Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ab']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('(ab).', 'abaacadaae')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 简写字符集\n",
    "此外，正则表达式中还有一类简写字符集，其等价于一组字符的集合：\n",
    "\n",
    "|简写    |  描述 |\n",
    "| :-----| :---- |\n",
    "|\\\\w     |   匹配所有字母、数字、下划线: \\[a-zA-Z0-9\\_\\] |\n",
    "|\\\\W     |   匹配非字母和数字的字符: \\[^\\\\w\\]|\n",
    "|\\\\d     |   匹配数字: \\[0-9\\]|\n",
    "|\\\\D   |     匹配非数字: \\[^\\\\d\\]|\n",
    "|\\\\s    |    匹配空格符: \\[\\\\t\\\\n\\\\f\\\\r\\\\p{Z}\\]|\n",
    "|\\\\S    |    匹配非空格符: \\[^\\\\s\\]|\n",
    "|\\\\B  |      匹配一组非空字符开头或结尾的位置，不代表具体字符|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:32:14.587758Z",
     "start_time": "2021-01-05T15:32:14.569759Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['is', 'Is']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('.s', 'Apple! This Is an Apple!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:32:27.467116Z",
     "start_time": "2021-01-05T15:32:27.450123Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['09', '7w', 'c_', '9q']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('\\w{2}', '09 8? 7w c_ 9q p@')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:34:05.615463Z",
     "start_time": "2021-01-05T15:34:05.596470Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['8?', 'p@']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('\\w\\W\\B', '09 8? 7w c_ 9q p@')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:34:31.499233Z",
     "start_time": "2021-01-05T15:34:31.483234Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['t d', 'g w', 's t', 'e s']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('.\\s.', 'Constant dropping wears the stone.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:36:20.473302Z",
     "start_time": "2021-01-05T15:36:20.455304Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('黄浦区', '方浜中路', '249号'), ('宝山区', '密山路', '5号')]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('上海市(.{2,3}区)(.{2,3}路)(\\d+号)', '上海市黄浦区方浜中路249号 上海市宝山区密山路5号')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、文本处理的五类操作\n",
    "### 1. 拆分\n",
    "\n",
    "`str.split`能够把字符串的列进行拆分，其中第一个参数为正则表达式，可选参数包括从左到右的最大拆分次数`n`，是否展开为多个列`expand`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:36:49.489484Z",
     "start_time": "2021-01-05T15:36:49.454482Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [上海, 黄浦, 方浜中, 249号]\n",
       "1       [上海, 宝山, 密山, 5号]\n",
       "dtype: object"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['上海市黄浦区方浜中路249号', '上海市宝山区密山路5号'])\n",
    "s.str.split('[市区路]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:37:14.417027Z",
     "start_time": "2021-01-05T15:37:14.322047Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>上海</td>\n",
       "      <td>黄浦</td>\n",
       "      <td>方浜中路249号</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>上海</td>\n",
       "      <td>宝山</td>\n",
       "      <td>密山路5号</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0   1         2\n",
       "0  上海  黄浦  方浜中路249号\n",
       "1  上海  宝山     密山路5号"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.split('[市区路]', n=2, expand=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与其类似的函数是`str.rsplit`，其区别在于使用`n`参数的时候是从右到左限制最大拆分次数。但是当前版本下`rsplit`因为`bug`而无法使用正则表达式进行分割："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:38:53.742439Z",
     "start_time": "2021-01-05T15:38:53.709444Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>上海市黄浦区方浜中路249号</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>上海市宝山区密山路5号</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                0\n",
       "0  上海市黄浦区方浜中路249号\n",
       "1     上海市宝山区密山路5号"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.rsplit('[市区路]', n=2, expand=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 合并\n",
    "\n",
    "关于合并一共有两个函数，分别是`str.join`和`str.cat`。`str.join`表示用某个连接符把`Series`中的字符串列表连接起来，如果列表中出现了字符串元素则返回缺失值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:39:09.541749Z",
     "start_time": "2021-01-05T15:39:09.514760Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a-b\n",
       "1    NaN\n",
       "2    NaN\n",
       "dtype: object"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([['a','b'], [1, 'a'], [['a', 'b'], 'c']])\n",
    "s.str.join('-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`str.cat`用于合并两个序列，主要参数为连接符`sep`、连接形式`join`以及缺失值替代符号`na_rep`，其中连接形式默认为以索引为键的左连接。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:39:56.009255Z",
     "start_time": "2021-01-05T15:39:55.981258Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a-cat\n",
       "1    b-dog\n",
       "dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s1 = pd.Series(['a','b'])\n",
    "s2 = pd.Series(['cat','dog'])\n",
    "s1.str.cat(s2,sep='-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:40:03.885124Z",
     "start_time": "2021-01-05T15:40:03.820125Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      a-?\n",
       "1    b-cat\n",
       "2    ?-dog\n",
       "dtype: object"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s2.index = [1, 2] # 索引上导致错位\n",
    "s1.str.cat(s2, sep='-', na_rep='?', join='outer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 匹配\n",
    "\n",
    "`str.contains`返回了每个字符串是否包含正则模式的布尔序列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:24:29.783395Z",
     "start_time": "2021-01-06T11:24:29.704785Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['my cat', 'he is fat', 'railway station'])\n",
    "s.str.contains('\\s\\wat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:44:10.056697Z",
     "start_time": "2021-01-05T15:44:10.040705Z"
    }
   },
   "source": [
    "`str.startswith`和`str.endswith`返回了每个字符串以给定模式为开始和结束的布尔序列，它们都不支持正则表达式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:44:49.301764Z",
     "start_time": "2021-01-05T15:44:49.277775Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1    False\n",
       "2    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.startswith('my')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:44:55.421950Z",
     "start_time": "2021-01-05T15:44:55.400958Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.endswith('t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果需要用正则表达式来检测开始或结束字符串的模式，可以使用`str.match`，其返回了每个字符串起始处是否符合给定正则模式的布尔序列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:45:37.969421Z",
     "start_time": "2021-01-05T15:45:37.949432Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.match('m|h')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:45:58.799080Z",
     "start_time": "2021-01-05T15:45:58.773083Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "dtype: bool"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str[::-1].str.match('ta[f|g]|n') # 反转后匹配"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-05T15:47:57.015713Z",
     "start_time": "2021-01-05T15:47:56.987724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str[::-1].str.match('ta[f|g]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，这些也能通过在`str.contains`的正则中使用`^`和`$`来实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:24:33.653303Z",
     "start_time": "2021-01-06T11:24:33.631305Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.contains('^[m|h]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:26:07.657024Z",
     "start_time": "2021-01-06T11:26:07.633033Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "dtype: bool"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.contains('[f|g]at|n$')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了上述返回值为布尔的匹配之外，还有一种返回索引的匹配函数，即`str.find`与`str.rfind`，其分别返回从左到右和从右到左第一次匹配的位置的索引，未找到则返回-1。需要注意的是这两个函数不支持正则匹配，只能用于字符子串的匹配："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:28:44.898469Z",
     "start_time": "2021-01-06T11:28:44.874472Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    11\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['This is an apple. That is not an apple.'])\n",
    "s.str.find('apple')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:28:54.859950Z",
     "start_time": "2021-01-06T11:28:54.837952Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    33\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.rfind('apple')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 替换\n",
    "\n",
    "`str.replace`和`replace`并不是一个函数，在使用字符串替换时应当使用前者。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:30:21.961713Z",
     "start_time": "2021-01-06T11:30:21.927749Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-10-8a508ece9bef>:2: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  s.str.replace('\\d|\\?', 'new')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    a_new_b\n",
       "1      c_new\n",
       "dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['a_1_b', 'c_?'])\n",
    "s.str.replace('\\d|\\?', 'new')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当需要对不同部分进行有差别的替换时，可以利用`子组`的方法，并且此时可以通过传入自定义的替换函数来分别进行处理，注意`group(k)`代表匹配到的第`k`个子组（圆括号之间的内容）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:31:02.070302Z",
     "start_time": "2021-01-06T11:31:02.051303Z"
    }
   },
   "outputs": [],
   "source": [
    "s = pd.Series(['上海市黄浦区方浜中路249号',\n",
    "                '上海市宝山区密山路5号',\n",
    "                '北京市昌平区北农路2号'])\n",
    "pat = '(\\w+市)(\\w+区)(\\w+路)(\\d+号)'\n",
    "city = {'上海市': 'Shanghai', '北京市': 'Beijing'}\n",
    "district = {'昌平区': 'CP District',\n",
    "            '黄浦区': 'HP District',\n",
    "            '宝山区': 'BS District'}\n",
    "road = {'方浜中路': 'Mid Fangbin Road',\n",
    "        '密山路': 'Mishan Road',\n",
    "        '北农路': 'Beinong Road'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:34:51.518284Z",
     "start_time": "2021-01-06T11:34:51.476257Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-12-27b661c6826a>:10: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  s.str.replace(pat, my_func)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    Shanghai HP District Mid Fangbin Road No. 249\n",
       "1           Shanghai BS District Mishan Road No. 5\n",
       "2           Beijing CP District Beinong Road No. 2\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def my_func(m):\n",
    "    str_city = city[m.group(1)]\n",
    "    str_district = district[m.group(2)]\n",
    "    str_road = road[m.group(3)]\n",
    "    str_no = 'No. ' + m.group(4)[:-1]\n",
    "    return ' '.join([str_city,\n",
    "                    str_district,\n",
    "                    str_road,\n",
    "                    str_no])\n",
    "s.str.replace(pat, my_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:37:01.177431Z",
     "start_time": "2021-01-06T11:37:01.141433Z"
    }
   },
   "source": [
    "这里的数字标识并不直观，可以使用`命名子组`更加清晰地写出子组代表的含义："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:40:11.741074Z",
     "start_time": "2021-01-06T11:40:11.671084Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上海市\n",
      "上海市\n",
      "北京市\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-14-aae9f5b10cbe>:12: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  s.str.replace(pat, my_func)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    Shanghai HP District Mid Fangbin Road No. 249\n",
       "1           Shanghai BS District Mishan Road No. 5\n",
       "2           Beijing CP District Beinong Road No. 2\n",
       "dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pat = '(?P<市名>\\w+市)(?P<区名>\\w+区)(?P<路名>\\w+路)(?P<编号>\\d+号)'\n",
    "def my_func(m):\n",
    "    str_city = city[m.group('市名')]\n",
    "    print(m.group('市名'))\n",
    "    str_district = district[m.group('区名')]\n",
    "    str_road = road[m.group('路名')]\n",
    "    str_no = 'No. ' + m.group('编号')[:-1]\n",
    "    return ' '.join([str_city,\n",
    "                     str_district,\n",
    "                     str_road,\n",
    "                     str_no])\n",
    "s.str.replace(pat, my_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里虽然看起来有些繁杂，但是实际数据处理中对应的替换，一般都会通过代码来获取数据从而构造字典映射，在具体写法上会简洁的多。\n",
    "\n",
    "### 5. 提取\n",
    "\n",
    "提取既可以认为是一种返回具体元素（而不是布尔值或元素对应的索引位置）的匹配操作，也可以认为是一种特殊的拆分操作。前面提到的`str.split`例子中会把分隔符去除，这并不是用户想要的效果，这时候就可以用`str.extract`进行提取："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:42:10.445709Z",
     "start_time": "2021-01-06T11:42:10.235742Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>上海市</td>\n",
       "      <td>黄浦区</td>\n",
       "      <td>方浜中路</td>\n",
       "      <td>249号</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>上海市</td>\n",
       "      <td>宝山区</td>\n",
       "      <td>密山路</td>\n",
       "      <td>5号</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>北京市</td>\n",
       "      <td>昌平区</td>\n",
       "      <td>北农路</td>\n",
       "      <td>2号</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1     2     3\n",
       "0  上海市  黄浦区  方浜中路  249号\n",
       "1  上海市  宝山区   密山路    5号\n",
       "2  北京市  昌平区   北农路    2号"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pat = '(\\w+市)(\\w+区)(\\w+路)(\\d+号)'\n",
    "s.str.extract(pat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过子组的命名，可以直接对新生成`DataFrame`的列命名："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:43:04.321661Z",
     "start_time": "2021-01-06T11:43:04.241675Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>市名</th>\n",
       "      <th>区名</th>\n",
       "      <th>路名</th>\n",
       "      <th>编号</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>上海市</td>\n",
       "      <td>黄浦区</td>\n",
       "      <td>方浜中路</td>\n",
       "      <td>249号</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>上海市</td>\n",
       "      <td>宝山区</td>\n",
       "      <td>密山路</td>\n",
       "      <td>5号</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>北京市</td>\n",
       "      <td>昌平区</td>\n",
       "      <td>北农路</td>\n",
       "      <td>2号</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    市名   区名    路名    编号\n",
       "0  上海市  黄浦区  方浜中路  249号\n",
       "1  上海市  宝山区   密山路    5号\n",
       "2  北京市  昌平区   北农路    2号"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pat = '(?P<市名>\\w+市)(?P<区名>\\w+区)(?P<路名>\\w+路)(?P<编号>\\d+号)'\n",
    "s.str.extract(pat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`str.extractall`不同于`str.extract`只匹配一次，它会把所有符合条件的模式全部匹配出来，如果存在多个结果，则以多级索引的方式存储："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:44:29.398732Z",
     "start_time": "2021-01-06T11:44:29.311744Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>match</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">my_A</th>\n",
       "      <th>0</th>\n",
       "      <td>135</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>26</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">my_B</th>\n",
       "      <th>0</th>\n",
       "      <td>674</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0   1\n",
       "     match         \n",
       "my_A 0      135  15\n",
       "     1       26   5\n",
       "my_B 0      674   2\n",
       "     1       25   6"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['A135T15,A26S5','B674S2,B25T6'], index = ['my_A','my_B'])\n",
    "pat = '[A|B](\\d+)[T|S](\\d+)'\n",
    "s.str.extractall(pat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:44:39.107320Z",
     "start_time": "2021-01-06T11:44:39.049362Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name1</th>\n",
       "      <th>name2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>match</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">my_A</th>\n",
       "      <th>0</th>\n",
       "      <td>135</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>26</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">my_B</th>\n",
       "      <th>0</th>\n",
       "      <td>674</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name1 name2\n",
       "     match            \n",
       "my_A 0       135    15\n",
       "     1        26     5\n",
       "my_B 0       674     2\n",
       "     1        25     6"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pat_with_name = '[A|B](?P<name1>\\d+)[T|S](?P<name2>\\d+)'\n",
    "s.str.extractall(pat_with_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`str.findall`的功能类似于`str.extractall`，区别在于前者把结果存入列表中，而后者处理为多级索引，每个行只对应一组匹配，而不是把所有匹配组合构成列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:48:56.767879Z",
     "start_time": "2021-01-06T11:48:56.741881Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "my_A    [(135, 15), (26, 5)]\n",
       "my_B     [(674, 2), (25, 6)]\n",
       "dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.findall(pat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 四、常用字符串函数\n",
    "\n",
    "除了上述介绍的五类字符串操作有关的函数之外，`str`对象上还定义了一些实用的其他方法，在此进行介绍：\n",
    "\n",
    "### 1. 字母型函数\n",
    "\n",
    "`upper, lower, title, capitalize, swapcase`这五个函数主要用于字母的大小写转化，从下面的例子中就容易领会其功能："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:49:35.097281Z",
     "start_time": "2021-01-06T11:49:35.069294Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                 LOWER\n",
       "1              CAPITALS\n",
       "2    THIS IS A SENTENCE\n",
       "3              SWAPCASE\n",
       "dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['lower', 'CAPITALS', 'this is a sentence', 'SwApCaSe'])\n",
    "s.str.upper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:49:44.637225Z",
     "start_time": "2021-01-06T11:49:44.614227Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                 lower\n",
       "1              capitals\n",
       "2    this is a sentence\n",
       "3              swapcase\n",
       "dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:49:53.319059Z",
     "start_time": "2021-01-06T11:49:53.294061Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                 Lower\n",
       "1              Capitals\n",
       "2    This Is A Sentence\n",
       "3              Swapcase\n",
       "dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.title()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:50:48.986554Z",
     "start_time": "2021-01-06T11:50:48.965557Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                 Lower\n",
       "1              Capitals\n",
       "2    This is a sentence\n",
       "3              Swapcase\n",
       "dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.capitalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:51:03.154812Z",
     "start_time": "2021-01-06T11:51:03.128842Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                 LOWER\n",
       "1              capitals\n",
       "2    THIS IS A SENTENCE\n",
       "3              sWaPcAsE\n",
       "dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.swapcase()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 数值型函数\n",
    "\n",
    "这里着重需要介绍的是`pd.to_numeric`方法，它虽然不是`str`对象上的方法，但是能够对字符格式的数值进行快速转换和筛选。其主要参数包括`errors`和`downcast`分别代表了非数值的处理模式和转换类型。其中，对于不能转换为数值的有三种`errors`选项，`raise, coerce, ignore`分别表示直接报错、设为缺失以及保持原来的字符串。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:51:36.413545Z",
     "start_time": "2021-01-06T11:51:36.384554Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       1\n",
       "1     2.2\n",
       "2      2e\n",
       "3      ??\n",
       "4    -2.1\n",
       "5       0\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['1', '2.2', '2e', '??', '-2.1', '0'])\n",
    "pd.to_numeric(s, errors='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:52:10.267933Z",
     "start_time": "2021-01-06T11:52:10.242937Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    2.2\n",
       "2    NaN\n",
       "3    NaN\n",
       "4   -2.1\n",
       "5    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_numeric(s, errors='coerce')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在数据清洗时，可以利用`coerce`的设定，快速查看非数值型的行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:54:15.275041Z",
     "start_time": "2021-01-06T11:54:15.240039Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    2e\n",
       "3    ??\n",
       "dtype: object"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s[pd.to_numeric(s, errors='coerce').isna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 统计型函数\n",
    "\n",
    "`count`和`len`的作用分别是返回出现正则模式的次数和字符串的长度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:55:56.992249Z",
     "start_time": "2021-01-06T11:55:56.965260Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['cat rat fat at', 'get feed sheet heat'])\n",
    "s.str.count('[r|f]at|ee')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:58:19.684771Z",
     "start_time": "2021-01-06T11:58:19.658772Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3\n",
       "1    2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.count('[c|r|f]at|ee') # 后面是匹配 ee 的意思"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T11:59:40.808694Z",
     "start_time": "2021-01-06T11:59:40.783693Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    14\n",
       "1    19\n",
       "dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.len()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 格式型函数\n",
    "格式型函数主要分为两类，第一种是除空型，第二种时填充型。其中，第一类函数一共有三种，它们分别是`strip, rstrip, lstrip`，分别代表去除两侧空格、右侧空格和左侧空格。这些函数在数据清洗时是有用的，特别是列名含有非法空格的时候。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:03:11.224562Z",
     "start_time": "2021-01-06T12:03:11.197584Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([4, 4, 4], dtype='int64')"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_index = pd.Index([' col1', 'col2 ', ' col3 '])\n",
    "my_index.str.strip().str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:05:11.170813Z",
     "start_time": "2021-01-06T12:05:11.151812Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['col1', 'col2', 'col3'], dtype='object')"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_index.str.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:03:53.706693Z",
     "start_time": "2021-01-06T12:03:53.686696Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([5, 4, 5], dtype='int64')"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_index.str.rstrip().str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:04:22.904383Z",
     "start_time": "2021-01-06T12:04:22.868390Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([4, 5, 5], dtype='int64')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_index.str.lstrip().str.len()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于填充型函数而言，`pad`是最灵活的，它可以选定字符串长度、填充的方向和填充内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:06:43.484964Z",
     "start_time": "2021-01-06T12:06:43.456974Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    ****a\n",
       "1    ****b\n",
       "2    ****c\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series(['a', 'b', 'c'])\n",
    "s.str.pad(5, 'left', '*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:07:02.341782Z",
     "start_time": "2021-01-06T12:07:02.316817Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a****\n",
       "1    b****\n",
       "2    c****\n",
       "dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.pad(5,'right','*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:07:08.393789Z",
     "start_time": "2021-01-06T12:07:08.372792Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    **a**\n",
       "1    **b**\n",
       "2    **c**\n",
       "dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.pad(5,'both','*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:08:02.978437Z",
     "start_time": "2021-01-06T12:08:02.942441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    ****a\n",
       "1    ***bd\n",
       "2    ****c\n",
       "dtype: object"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 这个原来是确定到固定的长度，可以用于第一道测试题的添加 股票编号 的地方\n",
    "s1 = pd.Series(['a', 'bd', 'c'])\n",
    "s1.str.pad(5, 'left', '*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述的三种情况可以分别用`rjust, ljust, center`来等效完成，需要注意`ljust`是指右侧填充而不是左侧填充："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:10:56.782152Z",
     "start_time": "2021-01-06T12:10:56.753157Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    ****a\n",
       "1    ****b\n",
       "2    ****c\n",
       "dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.rjust(5, '*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:11:43.905766Z",
     "start_time": "2021-01-06T12:11:43.879761Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    a****\n",
       "1    b****\n",
       "2    c****\n",
       "dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.ljust(5, '*')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:11:50.043674Z",
     "start_time": "2021-01-06T12:11:50.012224Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    **a**\n",
       "1    **b**\n",
       "2    **c**\n",
       "dtype: object"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.center(5, '*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在读取`excel`文件时，经常会出现数字前补0的需求，例如证券代码读入的时候会把\"000007\"作为数值7来处理，`pandas`中除了可以使用上面的左侧填充函数进行操作之外，还可用`zfill`来实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:12:55.513847Z",
     "start_time": "2021-01-06T12:12:55.473852Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    000007\n",
       "1    000155\n",
       "2    303000\n",
       "dtype: string"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([7, 155, 303000]).astype('string')\n",
    "s.str.pad(6,'left','0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:13:06.082669Z",
     "start_time": "2021-01-06T12:13:06.061680Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    000007\n",
       "1    000155\n",
       "2    303000\n",
       "dtype: string"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.rjust(6,'0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:13:11.759693Z",
     "start_time": "2021-01-06T12:13:11.733695Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    000007\n",
       "1    000155\n",
       "2    303000\n",
       "dtype: string"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.zfill(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 五、练习\n",
    "### Ex1：房屋信息数据集\n",
    "现有一份房屋信息数据集如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:26:24.287132Z",
     "start_time": "2021-01-06T12:26:19.442705Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>floor</th>\n",
       "      <th>year</th>\n",
       "      <th>area</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>高层（共6层）</td>\n",
       "      <td>1986年建</td>\n",
       "      <td>58.23㎡</td>\n",
       "      <td>155万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>中层（共20层）</td>\n",
       "      <td>2020年建</td>\n",
       "      <td>88㎡</td>\n",
       "      <td>155万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>低层（共28层）</td>\n",
       "      <td>2010年建</td>\n",
       "      <td>89.33㎡</td>\n",
       "      <td>365万</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      floor    year    area price\n",
       "0   高层（共6层）  1986年建  58.23㎡  155万\n",
       "1  中层（共20层）  2020年建     88㎡  155万\n",
       "2  低层（共28层）  2010年建  89.33㎡  365万"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_excel('data/house_info.xls', usecols=['floor','year','area','price'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:26:24.895047Z",
     "start_time": "2021-01-06T12:26:24.684068Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>floor</th>\n",
       "      <th>year</th>\n",
       "      <th>area</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>高层（共6层）</td>\n",
       "      <td>1986.0</td>\n",
       "      <td>58.23㎡</td>\n",
       "      <td>155万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>中层（共20层）</td>\n",
       "      <td>2020.0</td>\n",
       "      <td>88㎡</td>\n",
       "      <td>155万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>低层（共28层）</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>89.33㎡</td>\n",
       "      <td>365万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>低层（共20层）</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>82㎡</td>\n",
       "      <td>308万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>高层（共1层）</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>98㎡</td>\n",
       "      <td>117万</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      floor    year    area price\n",
       "0   高层（共6层）  1986.0  58.23㎡  155万\n",
       "1  中层（共20层）  2020.0     88㎡  155万\n",
       "2  低层（共28层）  2010.0  89.33㎡  365万\n",
       "3  低层（共20层）  2014.0     82㎡  308万\n",
       "4   高层（共1层）  2015.0     98㎡  117万"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1.将year列改为整数年份存储。\n",
    "df['year'] = pd.to_numeric(df.year.str[:-2])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:26:28.982990Z",
     "start_time": "2021-01-06T12:26:27.391473Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Level</th>\n",
       "      <th>Highest</th>\n",
       "      <th>year</th>\n",
       "      <th>area</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>高层</td>\n",
       "      <td>6</td>\n",
       "      <td>1986.0</td>\n",
       "      <td>58.23㎡</td>\n",
       "      <td>155万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>中层</td>\n",
       "      <td>20</td>\n",
       "      <td>2020.0</td>\n",
       "      <td>88㎡</td>\n",
       "      <td>155万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>低层</td>\n",
       "      <td>28</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>89.33㎡</td>\n",
       "      <td>365万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>低层</td>\n",
       "      <td>20</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>82㎡</td>\n",
       "      <td>308万</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>高层</td>\n",
       "      <td>1</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>98㎡</td>\n",
       "      <td>117万</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Level Highest    year    area price\n",
       "0    高层       6  1986.0  58.23㎡  155万\n",
       "1    中层      20  2020.0     88㎡  155万\n",
       "2    低层      28  2010.0  89.33㎡  365万\n",
       "3    低层      20  2014.0     82㎡  308万\n",
       "4    高层       1  2015.0     98㎡  117万"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2.将floor列替换为Level, Highest两列，其中的元素分别为string类型的层类别（高层、中层、低层）与整数类型的最高层数。\n",
    "pat = '(?P<Level>^\\w+)\\D+(?P<Highest>\\d+)\\D+$'\n",
    "df_temp = df.floor.str.extract(pat)\n",
    "df = pd.concat([df_temp, df.drop(columns='floor')], 1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:28:43.391392Z",
     "start_time": "2021-01-06T12:28:42.372241Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Level</th>\n",
       "      <th>Highest</th>\n",
       "      <th>year</th>\n",
       "      <th>area</th>\n",
       "      <th>price</th>\n",
       "      <th>avg_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>高层</td>\n",
       "      <td>6</td>\n",
       "      <td>1986.0</td>\n",
       "      <td>58.23㎡</td>\n",
       "      <td>155万</td>\n",
       "      <td>26618.58148720591元/平米</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>中层</td>\n",
       "      <td>20</td>\n",
       "      <td>2020.0</td>\n",
       "      <td>88㎡</td>\n",
       "      <td>155万</td>\n",
       "      <td>17613.636363636364元/平米</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>低层</td>\n",
       "      <td>28</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>89.33㎡</td>\n",
       "      <td>365万</td>\n",
       "      <td>40859.733572148216元/平米</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>低层</td>\n",
       "      <td>20</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>82㎡</td>\n",
       "      <td>308万</td>\n",
       "      <td>37560.9756097561元/平米</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>高层</td>\n",
       "      <td>1</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>98㎡</td>\n",
       "      <td>117万</td>\n",
       "      <td>11938.775510204083元/平米</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Level Highest    year    area price               avg_price\n",
       "0    高层       6  1986.0  58.23㎡  155万   26618.58148720591元/平米\n",
       "1    中层      20  2020.0     88㎡  155万  17613.636363636364元/平米\n",
       "2    低层      28  2010.0  89.33㎡  365万  40859.733572148216元/平米\n",
       "3    低层      20  2014.0     82㎡  308万    37560.9756097561元/平米\n",
       "4    高层       1  2015.0     98㎡  117万  11938.775510204083元/平米"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3.计算房屋每平米的均价avg_price，以***元/平米的格式存储到表中，其中***为整数。\n",
    "s_area = pd.to_numeric(df.area.str[:-1])\n",
    "s_price = pd.to_numeric(df.price.str[:-1])\n",
    "df['avg_price'] = ((s_price/s_area)*10000).astype('string') + '元/平米'\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ex2：《权力的游戏》剧本数据集\n",
    "现有一份权力的游戏剧本数据集如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:29:09.871685Z",
     "start_time": "2021-01-06T12:29:09.402758Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Release Date</th>\n",
       "      <th>Season</th>\n",
       "      <th>Episode</th>\n",
       "      <th>Episode Title</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>waymar royce</td>\n",
       "      <td>What do you expect? They're savages. One lot s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>will</td>\n",
       "      <td>I've never seen wildlings do a thing like this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>waymar royce</td>\n",
       "      <td>How close did you get?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Release Date    Season   Episode      Episode Title          Name  \\\n",
       "0   2011-04-17  Season 1  Episode 1  Winter is Coming  waymar royce   \n",
       "1   2011-04-17  Season 1  Episode 1  Winter is Coming          will   \n",
       "2   2011-04-17  Season 1  Episode 1  Winter is Coming  waymar royce   \n",
       "\n",
       "                                            Sentence  \n",
       "0  What do you expect? They're savages. One lot s...  \n",
       "1  I've never seen wildlings do a thing like this...  \n",
       "2                             How close did you get?  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/script.csv')\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:37:18.487998Z",
     "start_time": "2021-01-06T12:37:18.404008Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Season    Episode   \n",
       "Season 1  Episode 1     327\n",
       "          Episode 10    266\n",
       "          Episode 2     283\n",
       "          Episode 3     353\n",
       "          Episode 4     404\n",
       "Name: Sentence, dtype: int64"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1.计算每一个Episode的台词条数。\n",
    "# 报错：df.groupby('Episode').count()\n",
    "# 参考答案\n",
    "df.columns = df.columns.str.strip()\n",
    "df.groupby(['Season', 'Episode'])['Sentence'].count().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:37:42.661118Z",
     "start_time": "2021-01-06T12:37:42.551969Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Release Date</th>\n",
       "      <th>Season</th>\n",
       "      <th>Episode</th>\n",
       "      <th>Episode Title</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>waymar royce</td>\n",
       "      <td>What do you expect? They're savages. One lot s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>will</td>\n",
       "      <td>I've never seen wildlings do a thing like this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>waymar royce</td>\n",
       "      <td>How close did you get?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>will</td>\n",
       "      <td>Close as any man would.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-04-17</td>\n",
       "      <td>Season 1</td>\n",
       "      <td>Episode 1</td>\n",
       "      <td>Winter is Coming</td>\n",
       "      <td>gared</td>\n",
       "      <td>We should head back to the wall.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Release Date    Season    Episode     Episode Title          Name  \\\n",
       "0   2011-04-17  Season 1  Episode 1  Winter is Coming  waymar royce   \n",
       "1   2011-04-17  Season 1  Episode 1  Winter is Coming          will   \n",
       "2   2011-04-17  Season 1  Episode 1  Winter is Coming  waymar royce   \n",
       "3   2011-04-17  Season 1  Episode 1  Winter is Coming          will   \n",
       "4   2011-04-17  Season 1  Episode 1  Winter is Coming         gared   \n",
       "\n",
       "                                            Sentence  \n",
       "0  What do you expect? They're savages. One lot s...  \n",
       "1  I've never seen wildlings do a thing like this...  \n",
       "2                             How close did you get?  \n",
       "3                            Close as any man would.  \n",
       "4                   We should head back to the wall.  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()\n",
    "# df.columns.str.strip() 好像是去除空格？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:41:13.792032Z",
     "start_time": "2021-01-06T12:41:13.339096Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name\n",
       "male singer          109.000000\n",
       "slave owner           77.000000\n",
       "manderly              62.000000\n",
       "lollys stokeworth     62.000000\n",
       "dothraki matron       56.666667\n",
       "Name: Sentence, dtype: float64"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 以空格为单词的分割符号，请求出单句台词平均单词量最多的前五个人。\n",
    "# 参考答案\n",
    "df.set_index('Name').Sentence.str.split().str.len().groupby('Name').mean().sort_values(ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:41:40.858143Z",
     "start_time": "2021-01-06T12:41:40.780154Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name\n",
       "waymar royce        What do you expect? They're savages. One lot s...\n",
       "will                I've never seen wildlings do a thing like this...\n",
       "waymar royce                                   How close did you get?\n",
       "will                                          Close as any man would.\n",
       "gared                                We should head back to the wall.\n",
       "                                          ...                        \n",
       "brienne             I think we can all agree that ships take prece...\n",
       "bronn                   I think that's a very presumptuous statement.\n",
       "tyrion lannister    I once brought a jackass and a honeycomb into ...\n",
       "man                                           The Queen in the North!\n",
       "all                 The Queen in the North! The Queen in the North...\n",
       "Name: Sentence, Length: 23911, dtype: object"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('Name').Sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:41:55.550245Z",
     "start_time": "2021-01-06T12:41:55.194297Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name\n",
       "waymar royce        [What, do, you, expect?, They're, savages., On...\n",
       "will                [I've, never, seen, wildlings, do, a, thing, l...\n",
       "waymar royce                             [How, close, did, you, get?]\n",
       "will                                    [Close, as, any, man, would.]\n",
       "gared                        [We, should, head, back, to, the, wall.]\n",
       "                                          ...                        \n",
       "brienne             [I, think, we, can, all, agree, that, ships, t...\n",
       "bronn               [I, think, that's, a, very, presumptuous, stat...\n",
       "tyrion lannister    [I, once, brought, a, jackass, and, a, honeyco...\n",
       "man                                     [The, Queen, in, the, North!]\n",
       "all                 [The, Queen, in, the, North!, The, Queen, in, ...\n",
       "Name: Sentence, Length: 23911, dtype: object"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('Name').Sentence.str.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:44:46.841226Z",
     "start_time": "2021-01-06T12:44:46.774238Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name\n",
       "will            What do you expect? They're savages. One lot s...\n",
       "waymar royce    I've never seen wildlings do a thing like this...\n",
       "will                                       How close did you get?\n",
       "gared                                     Close as any man would.\n",
       "royce                            We should head back to the wall.\n",
       "dtype: object"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 若某人的台词中含有问号，那么下一个说台词的人即为回答者。若上一人台词中含有 𝑛 个问号，\n",
    "# 则认为回答者回答了 𝑛 个问题，请求出回答最多问题的前五个人。\n",
    "# 参考答案\n",
    "s = pd.Series(df.Sentence.values, index=df.Name.shift(-1))\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-06T12:45:09.454604Z",
     "start_time": "2021-01-06T12:45:09.236638Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Name\n",
       "tyrion lannister    527\n",
       "jon snow            374\n",
       "jaime lannister     283\n",
       "arya stark          265\n",
       "cersei lannister    246\n",
       "dtype: int64"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.str.count('\\?').groupby('Name').sum().sort_values(ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 由于备考，练习题没有过多尝试，答案给人很多启发"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}