04-text-byte.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 文本和字节序列\n",
    "\n",
    "> 人类使用文本，计算机使用字节序列  \n",
    "> —— Esther Nam 和 Travis Fischer  \"Character Encoding and Unicode in Python\"\n",
    "\n",
    "Python 3 明确区分了人类可读的文本字符串和原始的字节序列。  \n",
    "隐式地把字节序列转换成 Unicode 文本（的行为）已成过去。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 字符与编码\n",
    "字符的标识，及**码位**，是 0~1114111 的数字，在 Unicode 标准中用 4-6 个十六进制数字表示，如 A 为 U+0041, 高音谱号为 U+1D11E，😂 为 U+1F602.  \n",
    "字符的具体表述取决于所用的**编码**。编码时在码位与字节序列自减转换时使用的算法。  \n",
    "把码位转换成字节序列的过程是**编码**，把字节序列转成码位的过程是**解码**。\n",
    "\n",
    "### 序列类型\n",
    "Python 内置了两种基本的二进制序列类型：不可变的 `bytes` 和可变的 `bytearray`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "utf_8 b'S\\xc3\\xa3o Paulo'\n",
      "utf_16 b'\\xff\\xfeS\\x00\\xe3\\x00o\\x00 \\x00P\\x00a\\x00u\\x00l\\x00o\\x00'\n",
      "'charmap' codec can't encode character '\\xe3' in position 1: character maps to <undefined>\n",
      "b'So Paulo'\n",
      "b'S?o Paulo'\n",
      "b'S&#227;o Paulo'\n"
     ]
    }
   ],
   "source": [
    "# 基本的编码\n",
    "content = \"São Paulo\"\n",
    "for codec in [\"utf_8\", \"utf_16\"]:\n",
    "    print(codec, content.encode(codec))\n",
    "\n",
    "# UnicodeEncodeError\n",
    "try:\n",
    "    content.encode('cp437')\n",
    "except UnicodeEncodeError as e:\n",
    "    print(e)\n",
    "\n",
    "# 忽略无法编码的字符\n",
    "print(content.encode('cp437', errors='ignore'))\n",
    "# 把无法编码的字符替换成 ?\n",
    "print(content.encode('cp437', errors='replace'))\n",
    "# 把无法编码的字符替换成 xml 实体\n",
    "print(content.encode('cp437', errors='xmlcharrefreplace'))\n",
    "\n",
    "# 还可以自己设置错误处理方式\n",
    "# https://docs.python.org/3/library/codecs.html#codecs.register_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 基本的解码\n",
    "# 处理 UnicodeDecodeError\n",
    "octets = b'Montr\\xe9al'\n",
    "print(octets.decode('cp1252'))\n",
    "print(octets.decode('iso8859_7'))\n",
    "print(octets.decode('koi8_r'))\n",
    "try:\n",
    "    print(octets.decode('utf-8'))\n",
    "except UnicodeDecodeError as e:\n",
    "    print(e)\n",
    "\n",
    "# 将错误字符替换成 � (U+FFFD)\n",
    "octets.decode('utf-8', errors='replace')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python3 可以使用非 ASCII 名称\n",
    "São = 'Paulo'\n",
    "# 但是不能用 Emoji…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以用 `chardet` 检测字符所使用的编码\n",
    "\n",
    "BOM：字节序标记 (byte-order mark)：  \n",
    "`\\ufffe` 为字节序标记，放在文件开头，UTF-16 用它来表示文本以大端表示(`\\xfe\\xff`)还是小端表示(`\\xff\\xfe`)。  \n",
    "UTF-8 编码并不需要 BOM，但是微软还是给它加了 BOM，非常烦人。\n",
    "\n",
    "### 处理文本文件\n",
    "处理文本文件的最佳实践是“三明治”：要尽早地把输入的字节序列解码成字符串，尽量晚地对字符串进行编码输出；在处理逻辑中只处理字符串对象，不应该去编码或解码。  \n",
    "除非想判断编码，否则不要再二进制模式中打开文本文件；即便如此，也应该使用 `Chardet`，而不是重新发明轮子。  \n",
    "常规代码只应该使用二进制模式打开二进制文件，比如图像。\n",
    "\n",
    "### 默认编码\n",
    "可以使用 `sys.getdefaultincoding()` 获取系统默认编码；  \n",
    "Linux 的默认编码为 `UTF-8`，Windows 系统中不同语言设置使用的编码也不同，这导致了更多的问题。  \n",
    "`locale.getpreferredencoding()` 返回的编码是最重要的：这是打开文件的默认编码，也是重定向到文件的 `sys.stdout/stdin/stderr` 的默认编码。不过这个编码在某些系统中是可以改的…  \n",
    "所以，关于编码默认值的最佳建议是：别依赖默认值。\n",
    "\n",
    "### Unicode 编码方案\n",
    "```python\n",
    "a = 'café'\n",
    "b = 'cafe\\u0301'\n",
    "print(a, b)                       # café café\n",
    "print(ascii(a), ascii(b))         # 'caf\\xe9' 'cafe\\u0301'\n",
    "print(len(a), len(b), a == b)     # 4 5 False\n",
    "```\n",
    "\n",
    "在 Unicode 标准中，é 和 e\\u0301 这样的序列叫“标准等价物”，应用程序应将它视为相同的字符。但 Python 看到的是不同的码位序列，因此判断两者不相同。  \n",
    "我们可以用 `unicodedata.normalize` 将 Unicode 字符串规范化。有四种规范方式：NFC, NFD, NFKC, NFKD\n",
    "\n",
    "NFC 使用最少的码位构成等价的字符串，而 NFD 会把组合字符分解成基字符和单独的组合字符。  \n",
    "NFKC 和 NFKD 是出于兼容性考虑，在分解时会将字符替换成“兼容字符”，这种情况下会有格式损失。  \n",
    "兼容性方案可能会损失或曲解信息（如 \"4²\" 会被转换成 \"42\"），但可以为搜索和索引提供便利的中间表述。\n",
    "\n",
    "> 使用 NFKC 和 NFKC 规范化形式时要小心，而且只能在特殊情况中使用，例如搜索和索引，而不能用户持久存储，因为这两种转换会导致数据损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unicodedata import normalize, name\n",
    "# Unicode 码位\n",
    "a = 'café'\n",
    "b = 'cafe\\u0301'\n",
    "print(a, b)\n",
    "print(ascii(a), ascii(b))\n",
    "print(len(a), len(b), a == b)\n",
    "\n",
    "## NFC 和 NFD\n",
    "print(len(normalize('NFC', a)), len(normalize('NFC', b)))\n",
    "print(len(normalize('NFD', a)), len(normalize('NFD', b)))\n",
    "print(len(normalize('NFC', a)) == len(normalize('NFC', b)))\n",
    "\n",
    "print('-' * 15)\n",
    "# NFKC & NFKD\n",
    "s = '\\u00bd'\n",
    "l = [s, normalize('NFKC', s),  normalize('NFKD', s)]\n",
    "print(*l)\n",
    "print(*map(ascii, l))\n",
    "micro = 'μ'\n",
    "l = [s, normalize('NFKC', micro)]\n",
    "print(*l)\n",
    "print(*map(ascii, l))\n",
    "print(*map(name, l), sep='; ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unicode 数据库\n",
    "`unicodedata` 库中提供了很多关于 Unicode 的操作及判断功能，比如查看字符名称的 `name`，判断数字大小的 `numric` 等。  \n",
    "文档见 <https://docs.python.org/3.7/library/unicodedata.html>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VULGAR FRACTION ONE HALF\n",
      "0.5 30.0\n"
     ]
    }
   ],
   "source": [
    "import unicodedata\n",
    "print(unicodedata.name('½'))\n",
    "print(unicodedata.numeric('½'), unicodedata.numeric('卅'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 处理鬼符：按字节序将无法处理的字节序列依序替换成 \\udc00 - \\udcff 之间的码位\n",
    "x = 'digits-of-π'\n",
    "s = x.encode('gb2312')\n",
    "print(s)                                              # b'digits-of-\\xa6\\xd0'\n",
    "ascii_err = s.decode('ascii', 'surrogateescape')\n",
    "print(ascii_err)                                      # 'digits-of-\\udca6\\udcd0'\n",
    "print(ascii_err.encode('ascii', 'surrogateescape'))   # b'digits-of-\\xa6\\xd0'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}