site/zh-cn/guide/data.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jxv6goXm7oGF"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "llMNufAK7nfK"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\"); { display-mode: \"form\" }\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Byow2J6LaPl"
      },
      "source": [
        "# tf.data：构建 TensorFlow 输入流水线"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGXS3UWBBNoc"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td><a target=\"_blank\" href=\"https://tensorflow.google.cn/guide/data\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\">View 在 TensorFlow.org 上查看</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/guide/data.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">在 Google Colab 中运行 </a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/guide/data.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">在 GitHub 中查看源代码</a></td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/guide/data.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">下载笔记本</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Qo3HgDjbDcI"
      },
      "source": [
        "借助 `tf.data` API，您可以根据简单的可重用片段构建复杂的输入流水线。例如，图像模型的流水线可以聚合来自分布式文件系统中文件的数据，对每个图像应用随机扰动，并将随机选中的图像合并成一个批次进行训练。文本模型的流水线可能涉及从原始文本数据中提取符号，将提取的符号转换为带有查找表的嵌入向量标识符，并将不同长度的序列组合成一个批次。`tf.data` API 可以处理大量数据、从不同的数据格式读取数据和执行复杂的转换。\n",
        "\n",
        "`tf.data` API 引入了一个 `tf.data.Dataset` 抽象，它表示一个元素序列，其中每个元素都由一个或多个组件组成。例如，在一个图像流水线中，一个元素可能是一个训练样本，有一对表示图像及其标签的张量组件。\n",
        "\n",
        "您可以通过两种不同的方式创建数据集：\n",
        "\n",
        "- 数据**源**从存储在内存中或存储在一个或多个文件中的数据构造 `Dataset`。\n",
        "\n",
        "- 数据**转换**从一个或多个 `tf.data.Dataset` 对象构造数据集。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UJIEjEIBdf-h"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7Y0JtWBNR9E5"
      },
      "outputs": [],
      "source": [
        "import pathlib\n",
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "np.set_printoptions(precision=4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0l4a0ALxdaWF"
      },
      "source": [
        "## 基本机制\n",
        "\n",
        "<a id=\"basic-mechanics\"></a>\n",
        "\n",
        "要创建输入流水线，您必须从数据*源*开始。例如，要从内存中的数据构造一个 `Dataset`，您可以使用 `tf.data.Dataset.from_tensors()` 或 `tf.data.Dataset.from_tensor_slices()`。或者，如果您的输入数据以推荐的 TFRecord 格式存储在文件中，则您可以使用 `tf.data.TFRecordDataset()`。\n",
        "\n",
        "有了一个 `Dataset` 对象之后，您可以通过链接 <code>tf.data.Dataset</code> 对象上的方法调用将其<em>转换</em>成一个新的 `Dataset`。例如，您可以应用逐元素转换（例如 `Dataset.map`）和多元素转换（例如 `Dataset.batch`）。有关完整的转换列表，请参阅 `tf.data.Dataset` 文档。\n",
        "\n",
        "`Dataset` 对象是一个 Python 可迭代对象。这使得利用 for 循环使用它的元素成为可能："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0F-FDnjB6t6J"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1])\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pwJsRJ-FbDcJ"
      },
      "outputs": [],
      "source": [
        "for elem in dataset:\n",
        "  print(elem.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m0yy80MobDcM"
      },
      "source": [
        "或者使用 `iter` 显式创建一个 Python 迭代器，并利用 `next` 来使用它的元素："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "03w9oxFfbDcM"
      },
      "outputs": [],
      "source": [
        "it = iter(dataset)\n",
        "\n",
        "print(next(it).numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q4CgCL8qbDcO"
      },
      "source": [
        "或者，还可以利用 `reduce` 转换来使用数据集元素，从而减少所有元素以生成单个结果。以下示例演示了如何使用 `reduce` 转换来计算整数数据集的和。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C2bHAeNxbDcO"
      },
      "outputs": [],
      "source": [
        "print(dataset.reduce(0, lambda state, value: state + value).numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2Fzwt2nbDcR"
      },
      "source": [
        "<!-- TODO(jsimsa): Talk about `tf.function` support. -->\n",
        "\n",
        "<a id=\"dataset_structure\"></a>\n",
        "\n",
        "### 数据集结构\n",
        "\n",
        "数据集会生成一系列*元素*，其中每个元素都是*组件*的相同（嵌套）结构。结构的各个组件可以是可由 `tf.TypeSpec` 表示的任何类型，包括 `tf.Tensor`、`tf.sparse.SparseTensor`、`tf.RaggedTensor`、`tf.TensorArray` 或 `tf.data.Dataset`。\n",
        "\n",
        "可用于表达元素的（嵌套）结构的 Python 构造包括 `tuple`、`dict`、`NamedTuple` 和 `OrderedDict`。特别要指出的是，`list` 不是用于表达数据集元素结构的有效构造。这是因为早期的 `tf.data` 用户坚决要求将 `list` 输入（例如，传递到 `tf.data.Dataset.from_tensors` 时）自动打包为张量，并将 `list` 输出（例如，用户定义函数的返回值）强制转换为 `tuple`。因此，如果您希望将 `list` 输入视为结构，则需要将其转换为 `tuple`，而如果要将 `list` 输出作为单个组件，则需要使用 `tf.stack` 将其显式打包。\n",
        "\n",
        "`Dataset.element_spec` 属性允许您检查每个元素组件的类型。该属性会返回 <code>tf.TypeSpec</code> 对象与元素结构相匹配的<em>嵌套结构</em>，可以是单个组件、组件元组，或者组件的嵌套元组。例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mg0m1beIhXGn"
      },
      "outputs": [],
      "source": [
        "dataset1 = tf.data.Dataset.from_tensor_slices(tf.random.uniform([4, 10]))\n",
        "\n",
        "dataset1.element_spec"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cwyemaghhXaG"
      },
      "outputs": [],
      "source": [
        "dataset2 = tf.data.Dataset.from_tensor_slices(\n",
        "   (tf.random.uniform([4]),\n",
        "    tf.random.uniform([4, 100], maxval=100, dtype=tf.int32)))\n",
        "\n",
        "dataset2.element_spec"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1CL7aB0ahXn_"
      },
      "outputs": [],
      "source": [
        "dataset3 = tf.data.Dataset.zip((dataset1, dataset2))\n",
        "\n",
        "dataset3.element_spec"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m5bz7R1xhX1f"
      },
      "outputs": [],
      "source": [
        "# Dataset containing a sparse tensor.\n",
        "dataset4 = tf.data.Dataset.from_tensors(tf.SparseTensor(indices=[[0, 0], [1, 2]], values=[1, 2], dense_shape=[3, 4]))\n",
        "\n",
        "dataset4.element_spec"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lVOPHur_hYQv"
      },
      "outputs": [],
      "source": [
        "# Use value_type to see the type of value represented by the element spec\n",
        "dataset4.element_spec.value_type"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r5xNsFFvhUnr"
      },
      "source": [
        "`Dataset` 转换支持任何结构的数据集。使用 `Dataset.map` 和 `Dataset.filter` 转换时会将函数应用于每个元素，而元素结构会决定函数的参数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2myAr3Pxd-zF"
      },
      "outputs": [],
      "source": [
        "dataset1 = tf.data.Dataset.from_tensor_slices(\n",
        "    tf.random.uniform([4, 10], minval=1, maxval=10, dtype=tf.int32))\n",
        "\n",
        "dataset1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "woPXMP14gUTg"
      },
      "outputs": [],
      "source": [
        "for z in dataset1:\n",
        "  print(z.numpy())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "53PA4x6XgLar"
      },
      "outputs": [],
      "source": [
        "dataset2 = tf.data.Dataset.from_tensor_slices(\n",
        "   (tf.random.uniform([4]),\n",
        "    tf.random.uniform([4, 100], maxval=100, dtype=tf.int32)))\n",
        "\n",
        "dataset2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2ju4sNSebDcR"
      },
      "outputs": [],
      "source": [
        "dataset3 = tf.data.Dataset.zip((dataset1, dataset2))\n",
        "\n",
        "dataset3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BgxsfAS2g6gk"
      },
      "outputs": [],
      "source": [
        "for a, (b,c) in dataset3:\n",
        "  print('shapes: {a.shape}, {b.shape}, {c.shape}'.format(a=a, b=b, c=c))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M1s2K0g-bDcT"
      },
      "source": [
        "## 读取输入数据\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3JG2f0h2683"
      },
      "source": [
        "### 使用 NumPy 数组\n",
        "\n",
        "请参阅[加载 NumPy 数组](../tutorials/load_data/numpy.ipynb)教程了解更多示例。\n",
        "\n",
        "如果您所有的输入数据都适合装入内存，那么从这些数据创建 `Dataset` 的最简单方式是将它们转换为 `tf.Tensor` 对象并使用 `Dataset.from_tensor_slices`。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NmaE6PjjhQ47"
      },
      "outputs": [],
      "source": [
        "train, test = tf.keras.datasets.fashion_mnist.load_data()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J6cNiuDBbDcU"
      },
      "outputs": [],
      "source": [
        "images, labels = train\n",
        "images = images/255\n",
        "\n",
        "dataset = tf.data.Dataset.from_tensor_slices((images, labels))\n",
        "dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkwrDHN5bDcW"
      },
      "source": [
        "注：上面的代码段会将 `features` 和 `labels` 数组作为 `tf.constant()` 运算嵌入到 TensorFlow 计算图中。这对于小数据集来说效果很好，但是会浪费内存（因为数组的内容会被多次复制），并且可能会达到 `tf.GraphDef` 协议缓冲区的 2GB 上限。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pO4ua2gEmIhR"
      },
      "source": [
        "### 使用 Python 生成器\n",
        "\n",
        "另一个可被轻松整合为 `tf.data.Dataset` 的常用数据源是 Python 生成器。\n",
        "\n",
        "小心：虽然这种方式比较简便，但它的可移植性和可扩缩性有限。它必须在创建生成器的同一 Python 进程中运行，且仍受 Python [GIL](https://en.wikipedia.org/wiki/Global_interpreter_lock) 约束。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9njpME-jmDza"
      },
      "outputs": [],
      "source": [
        "def count(stop):\n",
        "  i = 0\n",
        "  while i<stop:\n",
        "    yield i\n",
        "    i += 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xwqLrjnTpD8Y"
      },
      "outputs": [],
      "source": [
        "for n in count(5):\n",
        "  print(n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_BB_PhxnVVx"
      },
      "source": [
        "`Dataset.from_generator` 构造函数会将 Python 生成器转换为具有完整功能的 `tf.data.Dataset`。\n",
        "\n",
        "构造函数会获取可调用对象作为输入，而非迭代器。这样，构造函数结束后便可重启生成器。构造函数会获取一个可选的 `args` 参数，作为可调用对象的参数。\n",
        "\n",
        "`output_types` 参数是必需的，因为 `tf.data` 会在内部构建 `tf.Graph`，而计算图边缘需要 `tf.dtype`。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GFga_OTwm0Je"
      },
      "outputs": [],
      "source": [
        "ds_counter = tf.data.Dataset.from_generator(count, args=[25], output_types=tf.int32, output_shapes = (), )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fel1SUuBnDUE"
      },
      "outputs": [],
      "source": [
        "for count_batch in ds_counter.repeat().batch(10).take(10):\n",
        "  print(count_batch.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wxy9hDMTq1zD"
      },
      "source": [
        "`output_shapes` 参数虽然不是*必需的*，但强烈建议添加，因为许多 TensorFlow 运算不支持秩未知的张量。如果特定轴的长度未知或可变，请在 `output_shapes` 中将其设置为 `None`。\n",
        "\n",
        "还需要注意的是，`output_shapes` 和 `output_types` 与其他数据集方法遵循相同的嵌套规则。\n",
        "\n",
        "下面的示例生成器对两方面进行了演示，它会返回由数组组成的元组，其中第二个数组是长度未知的向量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "allFX1g8rGKe"
      },
      "outputs": [],
      "source": [
        "def gen_series():\n",
        "  i = 0\n",
        "  while True:\n",
        "    size = np.random.randint(0, 10)\n",
        "    yield i, np.random.normal(size=(size,))\n",
        "    i += 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Ku26Yb9rcJX"
      },
      "outputs": [],
      "source": [
        "for i, series in gen_series():\n",
        "  print(i, \":\", str(series))\n",
        "  if i > 5:\n",
        "    break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LmkynGilx0qf"
      },
      "source": [
        "第一个输出是 `int32`，第二个输出是 `float32`。\n",
        "\n",
        "第一个条目是标量，形状为 `()`，第二个条目是长度未知的向量，形状为 `(None,)` "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zDTfhEzhsliM"
      },
      "outputs": [],
      "source": [
        "ds_series = tf.data.Dataset.from_generator(\n",
        "    gen_series, \n",
        "    output_types=(tf.int32, tf.float32), \n",
        "    output_shapes=((), (None,)))\n",
        "\n",
        "ds_series"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WWxvSyQiyN0o"
      },
      "source": [
        "现在，可以将它当作常规 `tf.data.Dataset` 使用。请注意，在批处理形状可变的数据集时，需要使用 `Dataset.padded_batch`。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A7jEpj3As1lO"
      },
      "outputs": [],
      "source": [
        "ds_series_batch = ds_series.shuffle(20).padded_batch(10)\n",
        "\n",
        "ids, sequence_batch = next(iter(ds_series_batch))\n",
        "print(ids.numpy())\n",
        "print()\n",
        "print(sequence_batch.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hcqOccJ1CxG"
      },
      "source": [
        "举个更加实际的例子，请尝试将 `preprocessing.image.ImageDataGenerator` 封装为 `tf.data.Dataset`。\n",
        "\n",
        "先下载数据："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g-_JCFRQ1CXM"
      },
      "outputs": [],
      "source": [
        "flowers = tf.keras.utils.get_file(\n",
        "    'flower_photos',\n",
        "    'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',\n",
        "    untar=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UIjPhvQ87jUT"
      },
      "source": [
        "创建 `image.ImageDataGenerator`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vPCZeBQE5DfH"
      },
      "outputs": [],
      "source": [
        "img_gen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, rotation_range=20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "my4PxqfH26p6"
      },
      "outputs": [],
      "source": [
        "images, labels = next(img_gen.flow_from_directory(flowers))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hd96nH1w3eKH"
      },
      "outputs": [],
      "source": [
        "print(images.dtype, images.shape)\n",
        "print(labels.dtype, labels.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KvRwvt5E2rTH"
      },
      "outputs": [],
      "source": [
        "ds = tf.data.Dataset.from_generator(\n",
        "    lambda: img_gen.flow_from_directory(flowers), \n",
        "    output_types=(tf.float32, tf.float32), \n",
        "    output_shapes=([32,256,256,3], [32,5])\n",
        ")\n",
        "\n",
        "ds.element_spec"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LcaULBCXj_2_"
      },
      "outputs": [],
      "source": [
        "for images, labels in ds.take(1):\n",
        "  print('images.shape: ', images.shape)\n",
        "  print('labels.shape: ', labels.shape)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ma4XoYzih2f4"
      },
      "source": [
        "### 处理 TFRecord 数据\n",
        "\n",
        "请参阅[加载 TFRecord](../tutorials/load_data/tfrecord.ipynb) 教程了解端到端示例。\n",
        "\n",
        "`tf.data` API 支持多种文件格式，因此可以处理不适合存储在内存中的大型数据集。例如，TFRecord 文件格式是一种简单的、面向记录的二进制格式，许多 TensorFlow 应用都将其用于训练数据。您可以利用 `tf.data.TFRecordDataset` 类将一个或多个 TFRecord 文件的内容作为输入流水线的一部分进行流式传输。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LiatWUloRJc4"
      },
      "source": [
        "下面的示例使用了来自 French Street Name Signs (FSNS) 的测试文件。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jZo_4fzdbDcW"
      },
      "outputs": [],
      "source": [
        "# Creates a dataset that reads all of the examples from two files.\n",
        "fsns_test_file = tf.keras.utils.get_file(\"fsns.tfrec\", \"https://storage.googleapis.com/download.tensorflow.org/data/fsns-20160927/testdata/fsns-00000-of-00001\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "seD5bOH3RhBP"
      },
      "source": [
        "`TFRecordDataset` 初始值设定项的 `filenames` 参数可以是字符串、字符串列表，或由字符串组成的 `tf.Tensor`。因此，如果您有两组分别用于训练和验证的文件，则可以创建一个工厂方法来生成数据集，并将文件名作为输入参数：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e2WV5d7DRUA-"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.TFRecordDataset(filenames = [fsns_test_file])\n",
        "dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "62NC3vz9U8ww"
      },
      "source": [
        "许多 TensorFlow 项目在其 TFRecord 文件中使用序列化的 `tf.train.Example` 记录。这些记录需要在检查前进行解码："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3tk29nlMl5P3"
      },
      "outputs": [],
      "source": [
        "raw_example = next(iter(dataset))\n",
        "parsed = tf.train.Example.FromString(raw_example.numpy())\n",
        "\n",
        "parsed.features.feature['image/text']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJAUib10bDcb"
      },
      "source": [
        "### 使用文本数据\n",
        "\n",
        "请参阅[加载文本](../tutorials/load_data/text.ipynb)教程了解端到端示例。\n",
        "\n",
        "许多数据集会作为一个或多个文本文件进行分发。`tf.data.TextLineDataset` 提供了一种从一个或多个文本文件中提取行的简便方式。如果给定一个或多个文件名，`TextLineDataset` 会为这些文件的每一行生成一个字符串元素。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hQMoFu2TbDcc"
      },
      "outputs": [],
      "source": [
        "directory_url = 'https://storage.googleapis.com/download.tensorflow.org/data/illiad/'\n",
        "file_names = ['cowper.txt', 'derby.txt', 'butler.txt']\n",
        "\n",
        "file_paths = [\n",
        "    tf.keras.utils.get_file(file_name, directory_url + file_name)\n",
        "    for file_name in file_names\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "il4cOjiVwj95"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.TextLineDataset(file_paths)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MevIbDiwy4MC"
      },
      "source": [
        "这是第一个文件的前几行："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vpEHKyvHxu8A"
      },
      "outputs": [],
      "source": [
        "for line in dataset.take(5):\n",
        "  print(line.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lJyVw8ro7fey"
      },
      "source": [
        "要交错不同文件中的行，请使用 `Dataset.interleave`。这样可以更轻松地重排文件。以下是来自每个转换的第一、第二和第三行："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1UCveWOt7fDE"
      },
      "outputs": [],
      "source": [
        "files_ds = tf.data.Dataset.from_tensor_slices(file_paths)\n",
        "lines_ds = files_ds.interleave(tf.data.TextLineDataset, cycle_length=3)\n",
        "\n",
        "for i, line in enumerate(lines_ds.take(9)):\n",
        "  if i % 3 == 0:\n",
        "    print()\n",
        "  print(line.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2F_pOIDubDce"
      },
      "source": [
        "默认情况下，`TextLineDataset` 会生成每个文件的*每一*行，但这可能并不理想（例如，有时文件会以标题行开始，或者包含注释）。可以使用 `Dataset.skip()` 或 `Dataset.filter` 转换移除这些行。如下所示，跳过第一行，然后过滤出剩余内容。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X6b20Gua2jPO"
      },
      "outputs": [],
      "source": [
        "titanic_file = tf.keras.utils.get_file(\"train.csv\", \"https://storage.googleapis.com/tf-datasets/titanic/train.csv\")\n",
        "titanic_lines = tf.data.TextLineDataset(titanic_file)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5M1pauNT68B2"
      },
      "outputs": [],
      "source": [
        "for line in titanic_lines.take(10):\n",
        "  print(line.numpy())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dEIP95cibDcf"
      },
      "outputs": [],
      "source": [
        "def survived(line):\n",
        "  return tf.not_equal(tf.strings.substr(line, 0, 1), \"0\")\n",
        "\n",
        "survivors = titanic_lines.skip(1).filter(survived)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "odQ4618h1XqD"
      },
      "outputs": [],
      "source": [
        "for line in survivors.take(10):\n",
        "  print(line.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5z5B11UjDTd"
      },
      "source": [
        "### 使用 CSV 数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ChDHNi3qbDch"
      },
      "source": [
        "请参阅[加载 CSV 文件](../tutorials/load_data/csv.ipynb)和[加载 Pandas DataFrame](../tutorials/load_data/pandas_dataframe.ipynb) 教程了解更多示例。\n",
        "\n",
        "CSV 文件格式是一种以纯文本形式存储表格数据的流行格式。\n",
        "\n",
        "例如："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kj28j5u49Bjm"
      },
      "outputs": [],
      "source": [
        "titanic_file = tf.keras.utils.get_file(\"train.csv\", \"https://storage.googleapis.com/tf-datasets/titanic/train.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ghvtmW40LM0B"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv(titanic_file)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J9uBqt5oGsR-"
      },
      "source": [
        "如果您的数据适合存储在内存中，那么 `Dataset.from_tensor_slices` 方法对字典同样有效，使这些数据可以被轻松导入："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JmAMCiPJA0qO"
      },
      "outputs": [],
      "source": [
        "titanic_slices = tf.data.Dataset.from_tensor_slices(dict(df))\n",
        "\n",
        "for feature_batch in titanic_slices.take(1):\n",
        "  for key, value in feature_batch.items():\n",
        "    print(\"  {!r:20s}: {}\".format(key, value))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47yippqaHFk6"
      },
      "source": [
        "更具可扩展性的方式是根据需要从磁盘加载。\n",
        "\n",
        "`tf.data` 模块提供了从一个或多个符合 [RFC 4180](https://tools.ietf.org/html/rfc4180) 的 CSV 文件提取记录的方法。\n",
        "\n",
        "`experimental.make_csv_dataset` 函数是用来读取 CSV 文件集的高级接口。它支持列类型推断和许多其他功能，如批处理和重排，以简化使用。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zHUDrM_s_brq"
      },
      "outputs": [],
      "source": [
        "titanic_batches = tf.data.experimental.make_csv_dataset(\n",
        "    titanic_file, batch_size=4,\n",
        "    label_name=\"survived\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TsZfhz79_Wlg"
      },
      "outputs": [],
      "source": [
        "for feature_batch, label_batch in titanic_batches.take(1):\n",
        "  print(\"'survived': {}\".format(label_batch))\n",
        "  print(\"features:\")\n",
        "  for key, value in feature_batch.items():\n",
        "    print(\"  {!r:20s}: {}\".format(key, value))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k_5N7CdNGYAa"
      },
      "source": [
        "如果只需要列的一个子集，您可以使用 `select_columns` 参数。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H9KNHyDwF2Sc"
      },
      "outputs": [],
      "source": [
        "titanic_batches = tf.data.experimental.make_csv_dataset(\n",
        "    titanic_file, batch_size=4,\n",
        "    label_name=\"survived\", select_columns=['class', 'fare', 'survived'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7C2uosFnGIT8"
      },
      "outputs": [],
      "source": [
        "for feature_batch, label_batch in titanic_batches.take(1):\n",
        "  print(\"'survived': {}\".format(label_batch))\n",
        "  for key, value in feature_batch.items():\n",
        "    print(\"  {!r:20s}: {}\".format(key, value))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TSVgJJ1HJD6M"
      },
      "source": [
        "还有一个级别更低的 `experimental.CsvDataset` 类，该类可以提供粒度更细的控制，但它不支持列类型推断，您必须指定每个列的类型。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wP1Y_NXA8bYl"
      },
      "outputs": [],
      "source": [
        "titanic_types  = [tf.int32, tf.string, tf.float32, tf.int32, tf.int32, tf.float32, tf.string, tf.string, tf.string, tf.string]\n",
        "dataset = tf.data.experimental.CsvDataset(titanic_file, titanic_types , header=True)\n",
        "\n",
        "for line in dataset.take(10):\n",
        "  print([item.numpy() for item in line])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oZSuLVsTbDcj"
      },
      "source": [
        "如果某些列为空，则此低级接口允许您提供默认值，而非列类型。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qry-g90FMo2I"
      },
      "outputs": [],
      "source": [
        "%%writefile missing.csv\n",
        "1,2,3,4\n",
        ",2,3,4\n",
        "1,,3,4\n",
        "1,2,,4\n",
        "1,2,3,\n",
        ",,,"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d5_hbiE9bDck"
      },
      "outputs": [],
      "source": [
        "# Creates a dataset that reads all of the records from two CSV files, each with\n",
        "# four float columns which may have missing values.\n",
        "\n",
        "record_defaults = [999,999,999,999]\n",
        "dataset = tf.data.experimental.CsvDataset(\"missing.csv\", record_defaults)\n",
        "dataset = dataset.map(lambda *items: tf.stack(items))\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "__jc7iD9M9FC"
      },
      "outputs": [],
      "source": [
        "for line in dataset:\n",
        "  print(line.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_4g0cIvbDcl"
      },
      "source": [
        "默认情况下，`CsvDataset` 会生成文件*每一*行的*每一*列，但这可能并不理想（例如，有时文件会以一个应该忽略的标题行开始，或者输入中不需要某些列）。可以分别使用 `header` 和 `select_cols` 参数移除这些行和字段。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p2IF_K0obDcm"
      },
      "outputs": [],
      "source": [
        "# Creates a dataset that reads all of the records from two CSV files with\n",
        "# headers, extracting float data from columns 2 and 4.\n",
        "record_defaults = [999, 999] # Only provide defaults for the selected columns\n",
        "dataset = tf.data.experimental.CsvDataset(\"missing.csv\", record_defaults, select_cols=[1, 3])\n",
        "dataset = dataset.map(lambda *items: tf.stack(items))\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-5aLprDeRNb0"
      },
      "outputs": [],
      "source": [
        "for line in dataset:\n",
        "  print(line.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-CJfhb03koVN"
      },
      "source": [
        "### 使用文件集"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yAO7SZDSk57_"
      },
      "source": [
        "许多数据集会作为文件集进行分发，其中，每个文件都是一个样本。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1dZwN3CS-jV2"
      },
      "outputs": [],
      "source": [
        "flowers_root = tf.keras.utils.get_file(\n",
        "    'flower_photos',\n",
        "    'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',\n",
        "    untar=True)\n",
        "flowers_root = pathlib.Path(flowers_root)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4099UU8n-jHP"
      },
      "source": [
        "注：这些图像已获得 CC-BY 许可，请参阅 LICENSE.txt 以了解详情。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FCyTYpmDs_jE"
      },
      "source": [
        "根目录包含每个类的路径："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_2iCXsHu6jJH"
      },
      "outputs": [],
      "source": [
        "for item in flowers_root.glob(\"*\"):\n",
        "  print(item.name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ylj9fgkamgWZ"
      },
      "source": [
        "每个类目录中的文件是样本："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lAkQp5uxoINu"
      },
      "outputs": [],
      "source": [
        "list_ds = tf.data.Dataset.list_files(str(flowers_root/'*/*'))\n",
        "\n",
        "for f in list_ds.take(5):\n",
        "  print(f.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91CPfUUJ_8SZ"
      },
      "source": [
        "使用 `tf.io.read_file` 函数读取数据，并从路径提取标签，返回 `(image, label)` 对："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-xhBRgvNqRRe"
      },
      "outputs": [],
      "source": [
        "def process_path(file_path):\n",
        "  label = tf.strings.split(file_path, os.sep)[-2]\n",
        "  return tf.io.read_file(file_path), label\n",
        "\n",
        "labeled_ds = list_ds.map(process_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kxrl0lGdnpRz"
      },
      "outputs": [],
      "source": [
        "for image_raw, label_text in labeled_ds.take(1):\n",
        "  print(repr(image_raw.numpy()[:100]))\n",
        "  print()\n",
        "  print(label_text.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEh46Ee0oSH5"
      },
      "source": [
        "<!--\n",
        "TODO(mrry): Add this section.\n",
        "\n",
        "### Handling text data with unusual sizes\n",
        "-->\n",
        "\n",
        "## 批处理数据集元素\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gR-2xY-8oSH4"
      },
      "source": [
        "### 简单批处理\n",
        "\n",
        "最简单的批处理方式是将数据集的 `n` 个连续元素堆叠成单个元素。`Dataset.batch()` 转换就负责执行此操作，它有和 `tf.stack()` 算子相同的约束，应用于元素的每个组件：也就是说，对于每个组件 *i*，所有元素都必须有一个形状完全相同的张量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xB7KeceLoSH0"
      },
      "outputs": [],
      "source": [
        "inc_dataset = tf.data.Dataset.range(100)\n",
        "dec_dataset = tf.data.Dataset.range(0, -100, -1)\n",
        "dataset = tf.data.Dataset.zip((inc_dataset, dec_dataset))\n",
        "batched_dataset = dataset.batch(4)\n",
        "\n",
        "for batch in batched_dataset.take(4):\n",
        "  print([arr.numpy() for arr in batch])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LlV1tpFdoSH0"
      },
      "source": [
        "当 `tf.data` 试图传播形状信息时，`Dataset.batch` 的默认设置会导致未知的批次大小，因为最后一个批次可能不完整。请注意形状中的 `None`："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yN7hn7OBoSHx"
      },
      "outputs": [],
      "source": [
        "batched_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "It1fPA3NoSHw"
      },
      "source": [
        "使用 `drop_remainder` 参数忽略最后一个批次，以获得完整的形状传播："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BycWC7WCoSHt"
      },
      "outputs": [],
      "source": [
        "batched_dataset = dataset.batch(7, drop_remainder=True)\n",
        "batched_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mj9nRxFZoSHs"
      },
      "source": [
        "### 批处理带填充的张量\n",
        "\n",
        "上述方式适用于所有具有相同大小的张量。然而，许多模型（包括序列模型）处理的输入数据可能具有不同的大小（例如，长度不同的序列）。为了处理这种情况，可以通过 `Dataset.padded_batch` 转换指定一个或多个可能被填充的维度，从而批处理不同形状的张量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kycwO0JooSHn"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.range(100)\n",
        "dataset = dataset.map(lambda x: tf.fill([tf.cast(x, tf.int32)], x))\n",
        "dataset = dataset.padded_batch(4, padded_shapes=(None,))\n",
        "\n",
        "for batch in dataset.take(2):\n",
        "  print(batch.numpy())\n",
        "  print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wl3yhth1oSHm"
      },
      "source": [
        "您可以通过 `Dataset.padded_batch` 转换为每个组件的每个维度设置不同的填充，它可以是可变长度（在上面的示例中由 `None` 表示）或是固定长度。它还可以重写填充值，填充值默认为 0。\n",
        "\n",
        "<!--\n",
        "TODO(mrry): Add this section.\n",
        "\n",
        "### Dense ragged -> tf.SparseTensor\n",
        "-->\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G8zbAxMwoSHl"
      },
      "source": [
        "## 训练工作流\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UnlhzF_AoSHk"
      },
      "source": [
        "### 处理多个周期\n",
        "\n",
        "`tf.data` API 提供了两种主要方式来处理同一数据的多个周期。\n",
        "\n",
        "要在多个周期内迭代数据集，最简单的方式是使用 `Dataset.repeat()` 转换。首先，创建一个由 Titanic 数据组成的数据集："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0tODHZzRoSHg"
      },
      "outputs": [],
      "source": [
        "titanic_file = tf.keras.utils.get_file(\"train.csv\", \"https://storage.googleapis.com/tf-datasets/titanic/train.csv\")\n",
        "titanic_lines = tf.data.TextLineDataset(titanic_file)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LMO6mlXxoSHc"
      },
      "outputs": [],
      "source": [
        "def plot_batch_sizes(ds):\n",
        "  batch_sizes = [batch.shape[0] for batch in ds]\n",
        "  plt.bar(range(len(batch_sizes)), batch_sizes)\n",
        "  plt.xlabel('Batch number')\n",
        "  plt.ylabel('Batch size')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WfVzmqL7oSHa"
      },
      "source": [
        "如果应用不带参数的 `Dataset.repeat()` 转换，将无限次地重复输入。\n",
        "\n",
        "`Dataset.repeat` 转换会连接其参数，而不会在一个周期结束和下一个周期开始时发出信号。因此，在 `Dataset.repeat` 之后应用的 `Dataset.batch` 将生成跨越周期边界的批次："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nZ0G1cztoSHX"
      },
      "outputs": [],
      "source": [
        "titanic_batches = titanic_lines.repeat(3).batch(128)\n",
        "plot_batch_sizes(titanic_batches)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "moH-4gBEoSHW"
      },
      "source": [
        "如果您需要明确的周期分隔，请将 `Dataset.batch` 置于重复前："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wmbmdK1qoSHS"
      },
      "outputs": [],
      "source": [
        "titanic_batches = titanic_lines.batch(128).repeat(3)\n",
        "\n",
        "plot_batch_sizes(titanic_batches)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DlEM5f9loSHR"
      },
      "source": [
        "如果您想在每个周期结束时执行自定义计算（例如，收集统计信息），最简单的方式是在每个周期上重新启动数据集迭代："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YyekyeY7oSHO"
      },
      "outputs": [],
      "source": [
        "epochs = 3\n",
        "dataset = titanic_lines.batch(128)\n",
        "\n",
        "for epoch in range(epochs):\n",
        "  for batch in dataset:\n",
        "    print(batch.shape)\n",
        "  print(\"End of epoch: \", epoch)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Bci79WCoSHN"
      },
      "source": [
        "### 随机重排输入数据\n",
        "\n",
        "`Dataset.shuffle()` 转换会维持一个固定大小的缓冲区，并从该缓冲区均匀地随机选择下一个元素：\n",
        "\n",
        "注：虽然较大的 buffer_size 可以更彻底地重排，但可能会占用大量的内存和时间来填充。如果这成为问题，请考虑跨文件使用 `Dataset.interleave`。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YvXr-qeoSHL"
      },
      "source": [
        "向数据集添加索引，便能看到效果："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Io4iJH1toSHI"
      },
      "outputs": [],
      "source": [
        "lines = tf.data.TextLineDataset(titanic_file)\n",
        "counter = tf.data.experimental.Counter()\n",
        "\n",
        "dataset = tf.data.Dataset.zip((counter, lines))\n",
        "dataset = dataset.shuffle(buffer_size=100)\n",
        "dataset = dataset.batch(20)\n",
        "dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T6tNYRcsoSHH"
      },
      "source": [
        "由于 `buffer_size` 是 100，而批次大小是 20，第一个批次不包含索引大于 120 的元素。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ayM3FFFAoSHC"
      },
      "outputs": [],
      "source": [
        "n,line_batch = next(iter(dataset))\n",
        "print(n.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLrfIjTHoSHB"
      },
      "source": [
        "对于 `Dataset.batch`，与 `Dataset.repeat` 的相对顺序很重要。\n",
        "\n",
        "在重排缓冲区为空之前，`Dataset.shuffle` 不会发出周期结束的信号。因此，置于重复之前的重排会先显示一个周期内的每个元素，然后移至下一个周期："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YX3pe7zZoSG6"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.zip((counter, lines))\n",
        "shuffled = dataset.shuffle(buffer_size=100).batch(10).repeat(2)\n",
        "\n",
        "print(\"Here are the item ID's near the epoch boundary:\\n\")\n",
        "for n, line_batch in shuffled.skip(60).take(5):\n",
        "  print(n.numpy())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H9hlE-lGoSGz"
      },
      "outputs": [],
      "source": [
        "shuffle_repeat = [n.numpy().mean() for n, line_batch in shuffled]\n",
        "plt.plot(shuffle_repeat, label=\"shuffle().repeat()\")\n",
        "plt.ylabel(\"Mean item ID\")\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UucIgCxWoSGx"
      },
      "source": [
        "但在重排之前的重复会将周期边界混合在一起："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bhxb5YGZoSGm"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.zip((counter, lines))\n",
        "shuffled = dataset.repeat(2).shuffle(buffer_size=100).batch(10)\n",
        "\n",
        "print(\"Here are the item ID's near the epoch boundary:\\n\")\n",
        "for n, line_batch in shuffled.skip(55).take(15):\n",
        "  print(n.numpy())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VAM4cbpZoSGL"
      },
      "outputs": [],
      "source": [
        "repeat_shuffle = [n.numpy().mean() for n, line_batch in shuffled]\n",
        "\n",
        "plt.plot(shuffle_repeat, label=\"shuffle().repeat()\")\n",
        "plt.plot(repeat_shuffle, label=\"repeat().shuffle()\")\n",
        "plt.ylabel(\"Mean item ID\")\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ianlfbrxbDco"
      },
      "source": [
        "## 预处理数据\n",
        "\n",
        "`Dataset.map(f)` 转换会通过对输入数据集的每个元素应用一个给定函数 `f` 来生成一个新的数据集。它基于 [`map()`](https://en.wikipedia.org/wiki/Map_(higher-order_function)) 函数，该函数通常应用于函数式编程语言中的列表（和其他结构）。函数 `f` 会获取在输入中表示单个元素的 `tf.Tensor` 对象，并返回在新数据集中表示单个元素的 `tf.Tensor` 对象。它的实现使用标准的 TensorFlow 运算来将一个元素转换为另一个元素。\n",
        "\n",
        "本部分介绍了关于 `Dataset.map()` 使用方法的常见示例。\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXw1IZVdbDcq"
      },
      "source": [
        "### 解码图像数据并调整大小\n",
        "\n",
        "<!-- TODO(markdaoust): link to image augmentation when it exists -->\n",
        "\n",
        "使用真实的图像数据训练神经网络时，常常需要将不同大小的图像转换为统一大小，以便将其批处理成某一固定大小。\n",
        "\n",
        "重建花卉文件名数据集："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rMGlj8V-u-NH"
      },
      "outputs": [],
      "source": [
        "list_ds = tf.data.Dataset.list_files(str(flowers_root/'*/*'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GyhZLB8N5jBm"
      },
      "source": [
        "编写一个操作数据集元素的函数。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fZObC0debDcr"
      },
      "outputs": [],
      "source": [
        "# Reads an image from a file, decodes it into a dense tensor, and resizes it\n",
        "# to a fixed shape.\n",
        "def parse_image(filename):\n",
        "  parts = tf.strings.split(filename, os.sep)\n",
        "  label = parts[-2]\n",
        "\n",
        "  image = tf.io.read_file(filename)\n",
        "  image = tf.io.decode_jpeg(image)\n",
        "  image = tf.image.convert_image_dtype(image, tf.float32)\n",
        "  image = tf.image.resize(image, [128, 128])\n",
        "  return image, label"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e0dVJlCA5qHA"
      },
      "source": [
        "测试它的有效性。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y8xuN_HBzGup"
      },
      "outputs": [],
      "source": [
        "file_path = next(iter(list_ds))\n",
        "image, label = parse_image(file_path)\n",
        "\n",
        "def show(image, label):\n",
        "  plt.figure()\n",
        "  plt.imshow(image)\n",
        "  plt.title(label.numpy().decode('utf-8'))\n",
        "  plt.axis('off')\n",
        "\n",
        "show(image, label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3P8N-S55vDu"
      },
      "source": [
        "将它映射到数据集。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SzO8LI_H5Sk_"
      },
      "outputs": [],
      "source": [
        "images_ds = list_ds.map(parse_image)\n",
        "\n",
        "for image, label in images_ds.take(2):\n",
        "  show(image, label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Ff7IqB9bDcs"
      },
      "source": [
        "### 应用任意 Python 逻辑\n",
        "\n",
        "出于性能考虑，请尽可能使用 TensorFlow 运算预处理数据。不过，在解析输入数据时，调用外部 Python 库有时会很有帮助。您可以在 `Dataset.map` 转换中使用 `tf.py_function` 运算。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R2u7CeA67DU8"
      },
      "source": [
        "例如，如果您想应用一个随机旋转，而 `tf.image` 模块只有 `tf.image.rot90`，这对图像增强不是很有帮助。\n",
        "\n",
        "注：`tensorflow_addons` 在 `tensorflow_addons.image.rotate` 中有一个与 TensorFlow 兼容的 `rotate`。\n",
        "\n",
        "为了演示 `tf.py_function`，请尝试使用 `scipy.ndimage.rotate` 函数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tBUmbERt7Czz"
      },
      "outputs": [],
      "source": [
        "import scipy.ndimage as ndimage\n",
        "\n",
        "def random_rotate_image(image):\n",
        "  image = ndimage.rotate(image, np.random.uniform(-30, 30), reshape=False)\n",
        "  return image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_wEyL7bS9S6t"
      },
      "outputs": [],
      "source": [
        "image, label = next(iter(images_ds))\n",
        "image = random_rotate_image(image)\n",
        "show(image, label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxVx7z-ABNyq"
      },
      "source": [
        "要将此函数用于 `Dataset.map`，需留意与 `Dataset.from_generator` 相同的注意事项，在应用该函数时需要描述返回的形状和类型："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cn2nIu92BMp0"
      },
      "outputs": [],
      "source": [
        "def tf_random_rotate_image(image, label):\n",
        "  im_shape = image.shape\n",
        "  [image,] = tf.py_function(random_rotate_image, [image], [tf.float32])\n",
        "  image.set_shape(im_shape)\n",
        "  return image, label"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bWPqKbTnbDct"
      },
      "outputs": [],
      "source": [
        "rot_ds = images_ds.map(tf_random_rotate_image)\n",
        "\n",
        "for image, label in rot_ds.take(2):\n",
        "  show(image, label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykx59-cMBwOT"
      },
      "source": [
        "### 解析 `tf.Example` 协议缓冲区消息\n",
        "\n",
        "许多输入流水线从 TFRecord 格式文件中提取 `tf.train.Example` 协议缓冲区消息。每个 `tf.train.Example` 记录包含一个或多个“特征”，而输入流水线通常会将这些特征转换为张量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6wnE134b32KY"
      },
      "outputs": [],
      "source": [
        "fsns_test_file = tf.keras.utils.get_file(\"fsns.tfrec\", \"https://storage.googleapis.com/download.tensorflow.org/data/fsns-20160927/testdata/fsns-00000-of-00001\")\n",
        "dataset = tf.data.TFRecordDataset(filenames = [fsns_test_file])\n",
        "dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HGypdgYOlXZz"
      },
      "source": [
        "您可以在 `tf.data.Dataset` 外部使用 `tf.train.Example` 协议来理解数据："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4znsVNqnF73C"
      },
      "outputs": [],
      "source": [
        "raw_example = next(iter(dataset))\n",
        "parsed = tf.train.Example.FromString(raw_example.numpy())\n",
        "\n",
        "feature = parsed.features.feature\n",
        "raw_img = feature['image/encoded'].bytes_list.value[0]\n",
        "img = tf.image.decode_png(raw_img)\n",
        "plt.imshow(img)\n",
        "plt.axis('off')\n",
        "_ = plt.title(feature[\"image/text\"].bytes_list.value[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cwzqp8IGC_vQ"
      },
      "outputs": [],
      "source": [
        "raw_example = next(iter(dataset))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y2X1dQNfC8Lu"
      },
      "outputs": [],
      "source": [
        "def tf_parse(eg):\n",
        "  example = tf.io.parse_example(\n",
        "      eg[tf.newaxis], {\n",
        "          'image/encoded': tf.io.FixedLenFeature(shape=(), dtype=tf.string),\n",
        "          'image/text': tf.io.FixedLenFeature(shape=(), dtype=tf.string)\n",
        "      })\n",
        "  return example['image/encoded'][0], example['image/text'][0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lGJhKDp_61A_"
      },
      "outputs": [],
      "source": [
        "img, txt = tf_parse(raw_example)\n",
        "print(txt.numpy())\n",
        "print(repr(img.numpy()[:20]), \"...\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8vFIUFzD5qIC"
      },
      "outputs": [],
      "source": [
        "decoded = dataset.map(tf_parse)\n",
        "decoded"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vRYNYkEej7Ix"
      },
      "outputs": [],
      "source": [
        "image_batch, text_batch = next(iter(decoded.batch(10)))\n",
        "image_batch.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ry1n0UBeczit"
      },
      "source": [
        "<a id=\"time_series_windowing\"></a>\n",
        "\n",
        "### 时间序列窗口化"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t0JMgvXEz9y1"
      },
      "source": [
        "有关端到端的时间序列示例，请参阅：[时间序列预测](../../tutorials/structured_data/time_series.ipynb)。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hzBABBkAkkVJ"
      },
      "source": [
        "时间序列数据通常以保持完整时间轴的方式进行组织。\n",
        "\n",
        "用一个简单的 `Dataset.range` 来演示："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kTQgo49skjuY"
      },
      "outputs": [],
      "source": [
        "range_ds = tf.data.Dataset.range(100000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6GLGhxgpazJ"
      },
      "source": [
        "通常，基于此类数据的模型需要一个连续的时间片。\n",
        "\n",
        "最简单的方式是批处理这些数据："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ETqB7QvTCNty"
      },
      "source": [
        "#### 使用 `batch`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pSs9XqwQpvIN"
      },
      "outputs": [],
      "source": [
        "batches = range_ds.batch(10, drop_remainder=True)\n",
        "\n",
        "for batch in batches.take(5):\n",
        "  print(batch.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mgb2qikEtk5W"
      },
      "source": [
        "或者，要对未来进行一步密集预测，您可以将特征和标签相对彼此移动一步："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "47XfwPhetkIN"
      },
      "outputs": [],
      "source": [
        "def dense_1_step(batch):\n",
        "  # Shift features and labels one step relative to each other.\n",
        "  return batch[:-1], batch[1:]\n",
        "\n",
        "predict_dense_1_step = batches.map(dense_1_step)\n",
        "\n",
        "for features, label in predict_dense_1_step.take(3):\n",
        "  print(features.numpy(), \" => \", label.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DjsXuINKqsS_"
      },
      "source": [
        "要预测整个窗口而非一个固定偏移量，您可以将批次分成两部分："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FMmkQB1Gqo6x"
      },
      "outputs": [],
      "source": [
        "batches = range_ds.batch(15, drop_remainder=True)\n",
        "\n",
        "def label_next_5_steps(batch):\n",
        "  return (batch[:-5],   # Inputs: All except the last 5 steps\n",
        "          batch[-5:])   # Labels: The last 5 steps\n",
        "\n",
        "predict_5_steps = batches.map(label_next_5_steps)\n",
        "\n",
        "for features, label in predict_5_steps.take(3):\n",
        "  print(features.numpy(), \" => \", label.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5a611Qr3jlhl"
      },
      "source": [
        "要允许一个批次的特征和另一批次的标签部分重叠，请使用 `Dataset.zip`："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "11dF3wyFjk2J"
      },
      "outputs": [],
      "source": [
        "feature_length = 10\n",
        "label_length = 3\n",
        "\n",
        "features = range_ds.batch(feature_length, drop_remainder=True)\n",
        "labels = range_ds.batch(feature_length).skip(1).map(lambda labels: labels[:label_length])\n",
        "\n",
        "predicted_steps = tf.data.Dataset.zip((features, labels))\n",
        "\n",
        "for features, label in predicted_steps.take(5):\n",
        "  print(features.numpy(), \" => \", label.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adew3o2mCURC"
      },
      "source": [
        "#### 使用 `window`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fF6pEdlduq8E"
      },
      "source": [
        "尽管使用 `Dataset.batch` 奏效，某些情况可能需要更精细的控制。`Dataset.window` 方法可以为您提供完全控制，但需要注意：它返回的是由 `Datasets` 组成的 `Dataset`。请参阅[数据集结构](#dataset_structure)部分以了解详情。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZEI2W_EBw2OX"
      },
      "outputs": [],
      "source": [
        "window_size = 5\n",
        "\n",
        "windows = range_ds.window(window_size, shift=1)\n",
        "for sub_ds in windows.take(5):\n",
        "  print(sub_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r82hWdk4x-46"
      },
      "source": [
        "`Dataset.flat_map` 方法可以获取由数据集组成的数据集，并将其合并为一个数据集："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SB8AI03mnF8u"
      },
      "outputs": [],
      "source": [
        " for x in windows.flat_map(lambda x: x).take(30):\n",
        "   print(x.numpy(), end=' ')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sgLIwq9Anc34"
      },
      "source": [
        "在几乎所有情况下，您都需要先 `Dataset.batch` 数据集："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5j_y84rmyVQa"
      },
      "outputs": [],
      "source": [
        "def sub_to_batch(sub):\n",
        "  return sub.batch(window_size, drop_remainder=True)\n",
        "\n",
        "for example in windows.flat_map(sub_to_batch).take(5):\n",
        "  print(example.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVugrmND3Grp"
      },
      "source": [
        "现在，您可以看到 `shift` 参数控制着每个窗口的移动量。\n",
        "\n",
        "将上述内容整合起来，您就可以编写出以下函数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LdFRv_0D4FqW"
      },
      "outputs": [],
      "source": [
        "def make_window_dataset(ds, window_size=5, shift=1, stride=1):\n",
        "  windows = ds.window(window_size, shift=shift, stride=stride)\n",
        "\n",
        "  def sub_to_batch(sub):\n",
        "    return sub.batch(window_size, drop_remainder=True)\n",
        "\n",
        "  windows = windows.flat_map(sub_to_batch)\n",
        "  return windows\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-iVxcVfEdf5b"
      },
      "outputs": [],
      "source": [
        "ds = make_window_dataset(range_ds, window_size=10, shift = 5, stride=3)\n",
        "\n",
        "for example in ds.take(10):\n",
        "  print(example.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fMGMTPQ4w8pr"
      },
      "source": [
        "然后，可以像之前一样轻松提取标签："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F0fPfZkZw6j_"
      },
      "outputs": [],
      "source": [
        "dense_labels_ds = ds.map(dense_1_step)\n",
        "\n",
        "for inputs,labels in dense_labels_ds.take(3):\n",
        "  print(inputs.numpy(), \"=>\", labels.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyi_-ft0kvy4"
      },
      "source": [
        "### 重采样\n",
        "\n",
        "在处理类非常不平衡的数据集时，您可能需要对数据集重新采样。`tf.data` 为此提供了两种方法。信用卡欺诈数据集就是此类问题一个很好的例子。\n",
        "\n",
        "注：请转到[不平衡数据分类](../tutorials/structured_data/imbalanced_data.ipynb)了解完整教程。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U2e8dxVUlFHO"
      },
      "outputs": [],
      "source": [
        "zip_path = tf.keras.utils.get_file(\n",
        "    origin='https://storage.googleapis.com/download.tensorflow.org/data/creditcard.zip',\n",
        "    fname='creditcard.zip',\n",
        "    extract=True)\n",
        "\n",
        "csv_path = zip_path.replace('.zip', '.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EhkkM4Wx75S_"
      },
      "outputs": [],
      "source": [
        "creditcard_ds = tf.data.experimental.make_csv_dataset(\n",
        "    csv_path, batch_size=1024, label_name=\"Class\",\n",
        "    # Set the column types: 30 floats and an int.\n",
        "    column_defaults=[float()]*30+[int()])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A8O47EmHlxYX"
      },
      "source": [
        "现在，检查类的分布，它是高度倾斜的："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a8-Ss69XlzXD"
      },
      "outputs": [],
      "source": [
        "def count(counts, batch):\n",
        "  features, labels = batch\n",
        "  class_1 = labels == 1\n",
        "  class_1 = tf.cast(class_1, tf.int32)\n",
        "\n",
        "  class_0 = labels == 0\n",
        "  class_0 = tf.cast(class_0, tf.int32)\n",
        "\n",
        "  counts['class_0'] += tf.reduce_sum(class_0)\n",
        "  counts['class_1'] += tf.reduce_sum(class_1)\n",
        "\n",
        "  return counts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O1a3t_B4l_f6"
      },
      "outputs": [],
      "source": [
        "counts = creditcard_ds.take(10).reduce(\n",
        "    initial_state={'class_0': 0, 'class_1': 0},\n",
        "    reduce_func = count)\n",
        "\n",
        "counts = np.array([counts['class_0'].numpy(),\n",
        "                   counts['class_1'].numpy()]).astype(np.float32)\n",
        "\n",
        "fractions = counts/counts.sum()\n",
        "print(fractions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z1b8lFhSnDdv"
      },
      "source": [
        "使用不平衡的数据集进行训练的一种常用方式是使其平衡。`tf.data` 包括一些能够让此工作流变得可行的方法："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y8jQWsgMnjQG"
      },
      "source": [
        "#### 数据集采样"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ov14SRrQyQE3"
      },
      "source": [
        "对数据集重新采样的一种方式是使用 `sample_from_datasets`。当每个类都有单独的 `tf.data.Dataset` 时，这种方式更加适用。\n",
        "\n",
        "下面用过滤器从信用卡欺诈数据中生成一个重采样数据集："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6YKfCPa-nioA"
      },
      "outputs": [],
      "source": [
        "negative_ds = (\n",
        "  creditcard_ds\n",
        "    .unbatch()\n",
        "    .filter(lambda features, label: label==0)\n",
        "    .repeat())\n",
        "positive_ds = (\n",
        "  creditcard_ds\n",
        "    .unbatch()\n",
        "    .filter(lambda features, label: label==1)\n",
        "    .repeat())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8FNd3sQjzl9-"
      },
      "outputs": [],
      "source": [
        "for features, label in positive_ds.batch(10).take(1):\n",
        "  print(label.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GxLAr-7p0ATX"
      },
      "source": [
        "要使用 `tf.data.Dataset.sample_from_datasets` 传递数据集以及每个数据集的权重，请运行以下代码："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vjdPVIFCngOb"
      },
      "outputs": [],
      "source": [
        "balanced_ds = tf.data.Dataset.sample_from_datasets(\n",
        "    [negative_ds, positive_ds], [0.5, 0.5]).batch(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2K4ObOms082B"
      },
      "source": [
        "现在，数据集为每个类生成样本的概率是 50/50："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Myvkw21Rz-fH"
      },
      "outputs": [],
      "source": [
        "for features, labels in balanced_ds.take(10):\n",
        "  print(labels.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OUTE3eb9nckY"
      },
      "source": [
        "#### 拒绝重采样"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kZ9ezkK6irMD"
      },
      "source": [
        "上述 `Dataset.sample_from_datasets` 方式的一个问题是每个类需要一个单独的 `tf.data.Dataset`。您可以使用 `Dataset.filter` 创建这两个数据集，但这会导致所有数据被加载两次。\n",
        "\n",
        "可以将 `tf.data.Dataset.rejection_resample` 方法应用于数据集以使其重新平衡，而且只需加载一次数据。元素将被删除或重复，以实现平衡。\n",
        "\n",
        "`rejection_resample` 需要一个 `class_func` 参数。这个 `class_func` 参数会被应用至每个数据集元素，并且会被用来确定某个样本属于哪一类，以实现平衡的目的。\n",
        "\n",
        "这里的目标是平衡标签分布，而 `creditcard_ds` 的元素已经形成 `(features, label)` 对。因此，`class_func` 只需返回这些标签："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zC_Cuzw8lhI5"
      },
      "outputs": [],
      "source": [
        "def class_func(features, label):\n",
        "  return label"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YxJrOZVToGuE"
      },
      "source": [
        "重采样方法会处理单个样本，因此您必须在应用该方法前 `unbatch` 数据集。\n",
        "\n",
        "该方法需要一个目标分布，以及一个可选的初始分布估计作为输入："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fY6VIhr3oGHG"
      },
      "outputs": [],
      "source": [
        "resample_ds = (\n",
        "    creditcard_ds\n",
        "    .unbatch()\n",
        "    .rejection_resample(class_func, target_dist=[0.5,0.5],\n",
        "                        initial_dist=fractions)\n",
        "    .batch(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L-HnC1s8idqV"
      },
      "source": [
        "`rejection_resample` 方法会返回 `(class, example)` 对，其中的 `class` 为 `class_func` 的输出。在此例中，`example` 已经形成 `(feature, label)` 对，因此请使用 `map` 删除多余的标签副本："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KpfCGU6BiaZq"
      },
      "outputs": [],
      "source": [
        "balanced_ds = resample_ds.map(lambda extra_label, features_and_label: features_and_label)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3d2jyEhx9kD"
      },
      "source": [
        "现在，数据集为每个类生成样本的概率是 50/50："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XGLYChBQwkDV"
      },
      "outputs": [],
      "source": [
        "for features, labels in balanced_ds.take(10):\n",
        "  print(labels.numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vYFKQx3bUBeU"
      },
      "source": [
        "## 迭代器检查点操作"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SOGg1UFhUE4z"
      },
      "source": [
        "TensorFlow 支持[获取检查点 ](./checkpoint.ipynb)，这样当训练过程重启时，可以还原至最新的检查点，从而恢复大部分进度。除了可以对模型变量进行检查点操作外，您还可以为数据集迭代器的进度设置检查点。如果您有一个很大的数据集，并且不想在每次重启后都从头开始，此功能会非常有用。但是请注意，迭代器检查点可能会很大，因为像 `Dataset.shuffle` 和 `Dataset.prefetch` 之类的转换需要在迭代器内缓冲元素。\n",
        "\n",
        "要在检查点中包含迭代器，请将迭代器传递至 `tf.train.Checkpoint` 构造函数。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3Fsm9wvKUsNC"
      },
      "outputs": [],
      "source": [
        "range_ds = tf.data.Dataset.range(20)\n",
        "\n",
        "iterator = iter(range_ds)\n",
        "ckpt = tf.train.Checkpoint(step=tf.Variable(0), iterator=iterator)\n",
        "manager = tf.train.CheckpointManager(ckpt, '/tmp/my_ckpt', max_to_keep=3)\n",
        "\n",
        "print([next(iterator).numpy() for _ in range(5)])\n",
        "\n",
        "save_path = manager.save()\n",
        "\n",
        "print([next(iterator).numpy() for _ in range(5)])\n",
        "\n",
        "ckpt.restore(manager.latest_checkpoint)\n",
        "\n",
        "print([next(iterator).numpy() for _ in range(5)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gxWglTwX9Fex"
      },
      "source": [
        "注：无法为依赖于外部状态的迭代器（例如 `tf.py_function`）设置检查点。尝试这样做将引发外部状态异常。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uLRdedPpbDdD"
      },
      "source": [
        "## 结合使用 tf.data 与 tf.keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JTQe8daMcgFz"
      },
      "source": [
        "`tf.keras` API 在创建和执行机器学习模型的许多方面进行了简化。它的 `Model.fit`、`Model.evaluate` 和 `Model.predict` API 支持将数据集作为输入。下面是一个快速的数据集和模型设置："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-bfjqm0hOfES"
      },
      "outputs": [],
      "source": [
        "train, test = tf.keras.datasets.fashion_mnist.load_data()\n",
        "\n",
        "images, labels = train\n",
        "images = images/255.0\n",
        "labels = labels.astype(np.int32)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wDhF3rGnbDdD"
      },
      "outputs": [],
      "source": [
        "fmnist_train_ds = tf.data.Dataset.from_tensor_slices((images, labels))\n",
        "fmnist_train_ds = fmnist_train_ds.shuffle(5000).batch(32)\n",
        "\n",
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(10)\n",
        "])\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), \n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rdogg8CfHs-G"
      },
      "source": [
        "只需向 `Model.fit` 和 `Model.evaluate` 传递一个由 `(feature, label)` 对组成的数据集："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9cu4kPzOHnlt"
      },
      "outputs": [],
      "source": [
        "model.fit(fmnist_train_ds, epochs=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FzpAQfJMJF41"
      },
      "source": [
        "如果您要传递一个无限大的数据集（比如通过调用 `Dataset.repeat`），您只需要同时传递 `steps_per_epoch` 参数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bp1BpzlyJinb"
      },
      "outputs": [],
      "source": [
        "model.fit(fmnist_train_ds.repeat(), epochs=2, steps_per_epoch=20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iTLsw_nqJpTw"
      },
      "source": [
        "对于评估，可以传递评估步数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TnlRHlaL-XUI"
      },
      "outputs": [],
      "source": [
        "loss, accuracy = model.evaluate(fmnist_train_ds)\n",
        "print(\"Loss :\", loss)\n",
        "print(\"Accuracy :\", accuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8UBU3CJKEA4"
      },
      "source": [
        "对于长数据集，可以设置要评估的步数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uVgamf9HKDon"
      },
      "outputs": [],
      "source": [
        "loss, accuracy = model.evaluate(fmnist_train_ds.repeat(), steps=10)\n",
        "print(\"Loss :\", loss)\n",
        "print(\"Accuracy :\", accuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aZYhJ_YSIl6w"
      },
      "source": [
        "调用 `Model.predict` 时不需要标签。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "343lXJ-pIqWD"
      },
      "outputs": [],
      "source": [
        "predict_ds = tf.data.Dataset.from_tensor_slices(images).batch(32)\n",
        "result = model.predict(predict_ds, steps = 10)\n",
        "print(result.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YfzZORwLI202"
      },
      "source": [
        "但是，如果您传递了包含标签的数据集，则标签会被忽略："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mgQJTPrT-2WF"
      },
      "outputs": [],
      "source": [
        "result = model.predict(fmnist_train_ds, steps = 10)\n",
        "print(result.shape)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "data.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}