courses/machine_learning/deepdive2/tensorflow_extended/solutions/penguin_transform.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Using a TFX Pipeline and TensorFlow Transform with Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "* Prepare example data.\n",
    "* Create a pipeline.\n",
    "* Run the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this notebook, you create and run a TensorFlow Extended (TFX) pipeline\n",
    "to ingest raw input data and preprocess it appropriately for ML training.\n",
    "This notebook is based on the TFX pipeline built in\n",
    "[Data validation using TFX Pipeline and TensorFlow Data Validation Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_tfdv).\n",
    "If you have not read that tutorial yet, you should read it before proceeding with\n",
    "this notebook.\n",
    "\n",
    "You can increase the predictive quality of your data and/or reduce\n",
    "dimensionality with feature engineering. One of the benefits of using TFX is\n",
    "that you write your transformation code once, and the resulting transforms\n",
    "will be consistent between training and serving in\n",
    "order to avoid training/serving skew.\n",
    "\n",
    "You will add a Transform component to the pipeline. The Transform component is\n",
    "implemented using the\n",
    "[tf.transform](https://www.tensorflow.org/tfx/transform/get_started) library.\n",
    "\n",
    "For more information about various concepts in TFX, see\n",
    "[Understanding TFX Pipelines](https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines).\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/penguin_transform.ipynb) -- try to complete that notebook first before reviewing this solution notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fmgi8ZvQkScg"
   },
   "source": [
    "## Set Up\n",
    "You first need to install the TFX Python package and download\n",
    "the dataset which you will use for our model.\n",
    "\n",
    "### Upgrade Pip\n",
    "\n",
    "To avoid upgrading Pip in a system when running locally,\n",
    "check to make sure that you are running in Colab.\n",
    "Local systems can of course be upgraded separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "as4OTe2ukSqm",
    "outputId": "051a4e55-5907-4dae-e67d-db97fa1c14b3"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  import colab\n",
    "  !pip install --upgrade pip\n",
    "except:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MZOYTt1RW4TK"
   },
   "source": [
    "### Install TFX\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iyQtljP-qPHY",
    "outputId": "cc1cd21b-143b-4f87-ee8c-2d4029640d50"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tfx\n",
      "  Downloading tfx-1.7.1-py3-none-any.whl (2.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m37.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: apache-beam[gcp]<3,>=2.36 in /opt/conda/lib/python3.7/site-packages (from tfx) (2.37.0)\n",
      "Collecting pyarrow<6,>=1\n",
      "  Downloading pyarrow-5.0.0-cp37-cp37m-manylinux2014_x86_64.whl (23.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.6/23.6 MB\u001b[0m \u001b[31m35.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-model-analysis<0.39,>=0.38.0\n",
      "  Downloading tensorflow_model_analysis-0.38.0-py3-none-any.whl (1.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m57.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting packaging<21,>=20\n",
      "  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting kubernetes<13,>=10.0.1\n",
      "  Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m64.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyyaml<6,>=3.12\n",
      "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m636.6/636.6 KB\u001b[0m \u001b[31m44.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting portpicker<2,>=1.3.1\n",
      "  Downloading portpicker-1.5.0-py3-none-any.whl (14 kB)\n",
      "Requirement already satisfied: tfx-bsl<1.8.0,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tfx) (1.7.0)\n",
      "Collecting ml-pipelines-sdk==1.7.1\n",
      "  Downloading ml_pipelines_sdk-1.7.1-py3-none-any.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m48.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3,>=1.15 in /opt/conda/lib/python3.7/site-packages (from tfx) (2.8.0)\n",
      "Collecting ml-metadata<1.8.0,>=1.7.0\n",
      "  Downloading ml_metadata-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m54.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting attrs<21,>=19.3.0\n",
      "  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 KB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-api-python-client<2,>=1.8\n",
      "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 KB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5\n",
      "  Downloading tensorflow-2.8.0-cp37-cp37m-manylinux2010_x86_64.whl (497.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m497.5/497.5 MB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: absl-py<2.0.0,>=0.9 in /opt/conda/lib/python3.7/site-packages (from tfx) (0.15.0)\n",
      "Requirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx) (0.12.0)\n",
      "Requirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /opt/conda/lib/python3.7/site-packages (from tfx) (2.34.2)\n",
      "Requirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /opt/conda/lib/python3.7/site-packages (from tfx) (1.11.0)\n",
      "Collecting docker<5,>=4.1\n",
      "  Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.0/147.0 KB\u001b[0m \u001b[31m25.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting click<8,>=7\n",
      "  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.8/82.8 KB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: grpcio<2,>=1.28.1 in /opt/conda/lib/python3.7/site-packages (from tfx) (1.44.0)\n",
      "Collecting tensorflow-data-validation<1.8.0,>=1.7.0\n",
      "  Downloading tensorflow_data_validation-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m79.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: protobuf<4,>=3.13 in /opt/conda/lib/python3.7/site-packages (from tfx) (3.19.4)\n",
      "Requirement already satisfied: jinja2<4,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from tfx) (2.11.3)\n",
      "Requirement already satisfied: tensorflow-transform<1.8.0,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tfx) (1.7.0)\n",
      "Requirement already satisfied: google-apitools<1,>=0.5 in /opt/conda/lib/python3.7/site-packages (from tfx) (0.5.31)\n",
      "Requirement already satisfied: keras-tuner<2,>=1.0.4 in /opt/conda/lib/python3.7/site-packages (from tfx) (1.1.0)\n",
      "Requirement already satisfied: numpy<2,>=1.16 in /opt/conda/lib/python3.7/site-packages (from tfx) (1.19.5)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<2.0.0,>=0.9->tfx) (1.15.0)\n",
      "Requirement already satisfied: cloudpickle<3,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2.0.0)\n",
      "Requirement already satisfied: proto-plus<2,>=1.7.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (1.20.3)\n",
      "Collecting dill<0.3.2,>=0.3.1.1\n",
      "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.0/152.0 KB\u001b[0m \u001b[31m23.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: requests<3.0.0,>=2.24.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2.27.1)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2.8.2)\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (1.4.2)\n",
      "Requirement already satisfied: fastavro<2,>=0.23.6 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (1.4.10)\n",
      "Requirement already satisfied: oauth2client<5,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (4.1.3)\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2.6.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (3.10.0.2)\n",
      "Collecting httplib2<0.20.0,>=0.8\n",
      "  Downloading httplib2-0.19.1-py3-none-any.whl (95 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 KB\u001b[0m \u001b[31m15.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: crcmod<2.0,>=1.7 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (1.7)\n",
      "Requirement already satisfied: orjson<4.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (3.6.7)\n",
      "Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (3.12.3)\n",
      "Requirement already satisfied: pytz>=2018.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2021.3)\n",
      "Collecting google-cloud-videointelligence<2,>=1.8.0\n",
      "  Downloading google_cloud_videointelligence-1.16.2-py2.py3-none-any.whl (183 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.9/183.9 KB\u001b[0m \u001b[31m28.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-recommendations-ai<=0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (0.2.0)\n",
      "Requirement already satisfied: grpcio-gcp<1,>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (0.2.2)\n",
      "Requirement already satisfied: google-cloud-bigquery-storage>=2.6.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2.13.0)\n",
      "Collecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)\n",
      "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (2.11.0)\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.1-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 KB\u001b[0m \u001b[31m31.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-datastore<2,>=1.8.0\n",
      "  Downloading google_cloud_datastore-1.15.4-py2.py3-none-any.whl (134 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.2/134.2 KB\u001b[0m \u001b[31m20.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-vision<2,>=0.38.0\n",
      "  Downloading google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m435.1/435.1 KB\u001b[0m \u001b[31m46.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (1.4.1)\n",
      "Requirement already satisfied: cachetools<5,>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (4.2.4)\n",
      "Requirement already satisfied: google-auth<3,>=1.18.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (1.35.0)\n",
      "Collecting google-cloud-language<2,>=1.3.0\n",
      "  Downloading google_cloud_language-1.3.1-py2.py3-none-any.whl (83 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.6/83.6 KB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx) (3.6.2)\n",
      "Collecting google-cloud-spanner<2,>=1.13.0\n",
      "  Downloading google_cloud_spanner-1.19.2-py2.py3-none-any.whl (255 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m255.5/255.5 KB\u001b[0m \u001b[31m30.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker<5,>=4.1->tfx) (1.3.1)\n",
      "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx) (2.5.0)\n",
      "Collecting uritemplate<4dev,>=3.0.0\n",
      "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx) (0.1.0)\n",
      "Requirement already satisfied: fasteners>=0.14 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx) (0.17.3)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (2.2.1)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx) (2.3.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2<4,>=2.7.3->tfx) (2.0.1)\n",
      "Requirement already satisfied: tensorboard in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx) (2.6.0)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx) (1.7.3)\n",
      "Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx) (7.32.0)\n",
      "Requirement already satisfied: kt-legacy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx) (1.0.4)\n",
      "Requirement already satisfied: urllib3>=1.24.2 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx) (1.26.8)\n",
      "Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx) (2021.10.8)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx) (1.3.1)\n",
      "Requirement already satisfied: setuptools>=21.0.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx) (59.8.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<21,>=20->tfx) (3.0.7)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from portpicker<2,>=1.3.1->tfx) (5.9.0)\n",
      "Collecting tf-estimator-nightly==2.8.0.dev2021122109\n",
      "  Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m462.5/462.5 KB\u001b[0m \u001b[31m47.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (0.2.0)\n",
      "Requirement already satisfied: flatbuffers>=1.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (1.12)\n",
      "Requirement already satisfied: gast>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (0.4.0)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (1.12.1)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (1.1.2)\n",
      "Collecting numpy<2,>=1.16\n",
      "  Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m70.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: h5py>=2.9.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (3.1.0)\n",
      "Collecting libclang>=9.0.1\n",
      "  Downloading libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.5/14.5 MB\u001b[0m \u001b[31m75.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (3.3.0)\n",
      "Collecting tensorboard\n",
      "  Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m103.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-io-gcs-filesystem>=0.23.1\n",
      "  Downloading tensorflow_io_gcs_filesystem-0.25.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m94.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (1.1.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (1.6.3)\n",
      "Collecting keras<2.9,>=2.8.0rc0\n",
      "  Downloading keras-2.8.0-py2.py3-none-any.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m59.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting joblib<0.15,>=0.12\n",
      "  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.9/294.9 KB\u001b[0m \u001b[31m37.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-metadata<1.8,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx) (1.7.0)\n",
      "Collecting pyfarmhash<0.4,>=0.2\n",
      "  Downloading pyfarmhash-0.3.2.tar.gz (99 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.9/99.9 KB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pandas<2,>=1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx) (1.3.5)\n",
      "Requirement already satisfied: ipywidgets<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.39,>=0.38.0->tfx) (7.6.5)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (0.37.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx) (1.54.0)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx) (1.44.0)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx) (0.2.7)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx) (4.8)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigtable<2,>=0.31.1->apache-beam[gcp]<3,>=2.36->tfx) (0.12.3)\n",
      "Collecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.1-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.7.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.6.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.5.0-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.3-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.2-py2.py3-none-any.whl (26 kB)\n",
      "  Downloading google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)\n",
      "INFO: pip is looking at multiple versions of google-cloud-bigtable to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 KB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-cloud-bigquery-storage to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigquery-storage>=2.6.3\n",
      "  Downloading google_cloud_bigquery_storage-2.13.1-py2.py3-none-any.whl (180 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m180.1/180.1 KB\u001b[0m \u001b[31m1.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-auth-httplib2 to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth-httplib2>=0.0.3\n",
      "  Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)\n",
      "INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth<3,>=1.18.0\n",
      "  Downloading google_auth-2.6.6-py2.py3-none-any.whl (156 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m156.7/156.7 KB\u001b[0m \u001b[31m23.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.9/152.9 KB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5\n",
      "  Downloading google_api_core-2.7.3-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.7.2-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m16.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.0-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m20.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m17.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.8/111.8 KB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.7/111.7 KB\u001b[0m \u001b[31m18.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m109.8/109.8 KB\u001b[0m \u001b[31m16.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.3/93.3 KB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: overrides<7.0.0,>=6.0.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx) (6.1.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx) (1.1.2)\n",
      "Requirement already satisfied: cached-property in /opt/conda/lib/python3.7/site-packages (from h5py>=2.9.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx) (1.5.2)\n",
      "Requirement already satisfied: docopt in /opt/conda/lib/python3.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.36->tfx) (0.6.2)\n",
      "Collecting pyparsing>=2.0.2\n",
      "  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.8/67.8 KB\u001b[0m \u001b[31m9.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (5.1.1)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (4.8.0)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (0.7.5)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (0.18.1)\n",
      "Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (0.2.0)\n",
      "Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (0.1.3)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (3.0.27)\n",
      "Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (5.1.1)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx) (2.11.2)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (3.5.2)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (5.2.0)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (6.9.2)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (1.0.2)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.2.0)\n",
      "Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]<3,>=2.36->tfx) (0.4.8)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx) (2.0.12)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx) (3.3)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx) (0.6.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx) (2.0.3)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx) (0.4.6)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx) (3.3.6)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx) (3.2.0)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx) (1.15.0)\n",
      "Requirement already satisfied: debugpy<2.0,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (1.5.1)\n",
      "Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (1.5.4)\n",
      "Requirement already satisfied: jupyter-client<8.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (7.1.2)\n",
      "Requirement already satisfied: tornado<7.0,>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (6.1)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->keras-tuner<2,>=1.0.4->tfx) (0.8.3)\n",
      "Requirement already satisfied: importlib-metadata>=4.4 in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard->keras-tuner<2,>=1.0.4->tfx) (4.11.3)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (4.9.2)\n",
      "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (4.4.0)\n",
      "Requirement already satisfied: typing-utils>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from overrides<7.0.0,>=6.0.1->google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx) (0.1.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->keras-tuner<2,>=1.0.4->tfx) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->keras-tuner<2,>=1.0.4->tfx) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (6.4.10)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx) (2.21)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard->keras-tuner<2,>=1.0.4->tfx) (3.7.0)\n",
      "Requirement already satisfied: importlib-resources>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (5.4.0)\n",
      "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.18.1)\n",
      "Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (22.3.0)\n",
      "Requirement already satisfied: entrypoints in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.4)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.13.3)\n",
      "Requirement already satisfied: nbconvert>=5 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (6.4.4)\n",
      "Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (21.3.0)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.13.1)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (1.8.0)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (1.5.0)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (4.1.0)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.5.13)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.8.4)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.7.1)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.6.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.1.2)\n",
      "Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (4.10.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /opt/conda/lib/python3.7/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (21.2.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.7/site-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (2.3.1)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx) (0.5.1)\n",
      "Building wheels for collected packages: dill, pyfarmhash\n",
      "  Building wheel for dill (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=1ad472f60bdf77b5c656ea7ce72d48d23217985cd1f86f2d592b4800624744be\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
      "  Building wheel for pyfarmhash (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pyfarmhash: filename=pyfarmhash-0.3.2-cp37-cp37m-linux_x86_64.whl size=108607 sha256=6c82d8ba7bb434d44368cae0bde9f0b49521852309ec96bf285116f15de8efcb\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/53/58/7a/3b040f3a2ee31908e3be916e32660db6db53621ce6eba838dc\n",
      "Successfully built dill pyfarmhash\n",
      "Installing collected packages: tf-estimator-nightly, pyfarmhash, libclang, keras, joblib, uritemplate, tensorflow-io-gcs-filesystem, pyyaml, pyparsing, portpicker, numpy, dill, click, attrs, pyarrow, packaging, ml-metadata, httplib2, docker, kubernetes, google-api-core, tensorboard, google-cloud-core, google-api-python-client, tensorflow, ml-pipelines-sdk, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-language, google-cloud-datastore, google-cloud-bigtable, tensorflow-model-analysis, tensorflow-data-validation, tfx\n",
      "  Attempting uninstall: keras\n",
      "    Found existing installation: keras 2.6.0\n",
      "    Uninstalling keras-2.6.0:\n",
      "      Successfully uninstalled keras-2.6.0\n",
      "  Attempting uninstall: joblib\n",
      "    Found existing installation: joblib 1.0.1\n",
      "    Uninstalling joblib-1.0.1:\n",
      "      Successfully uninstalled joblib-1.0.1\n",
      "  Attempting uninstall: uritemplate\n",
      "    Found existing installation: uritemplate 4.1.1\n",
      "    Uninstalling uritemplate-4.1.1:\n",
      "      Successfully uninstalled uritemplate-4.1.1\n",
      "  Attempting uninstall: pyyaml\n",
      "    Found existing installation: PyYAML 6.0\n",
      "    Uninstalling PyYAML-6.0:\n",
      "      Successfully uninstalled PyYAML-6.0\n",
      "  Attempting uninstall: pyparsing\n",
      "    Found existing installation: pyparsing 3.0.7\n",
      "    Uninstalling pyparsing-3.0.7:\n",
      "      Successfully uninstalled pyparsing-3.0.7\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.19.5\n",
      "    Uninstalling numpy-1.19.5:\n",
      "      Successfully uninstalled numpy-1.19.5\n",
      "  Attempting uninstall: dill\n",
      "    Found existing installation: dill 0.3.4\n",
      "    Uninstalling dill-0.3.4:\n",
      "      Successfully uninstalled dill-0.3.4\n",
      "  Attempting uninstall: click\n",
      "    Found existing installation: click 8.0.4\n",
      "    Uninstalling click-8.0.4:\n",
      "      Successfully uninstalled click-8.0.4\n",
      "  Attempting uninstall: attrs\n",
      "    Found existing installation: attrs 21.4.0\n",
      "    Uninstalling attrs-21.4.0:\n",
      "      Successfully uninstalled attrs-21.4.0\n",
      "  Attempting uninstall: pyarrow\n",
      "    Found existing installation: pyarrow 7.0.0\n",
      "    Uninstalling pyarrow-7.0.0:\n",
      "      Successfully uninstalled pyarrow-7.0.0\n",
      "  Attempting uninstall: packaging\n",
      "    Found existing installation: packaging 21.3\n",
      "    Uninstalling packaging-21.3:\n",
      "      Successfully uninstalled packaging-21.3\n",
      "  Attempting uninstall: httplib2\n",
      "    Found existing installation: httplib2 0.20.4\n",
      "    Uninstalling httplib2-0.20.4:\n",
      "      Successfully uninstalled httplib2-0.20.4\n",
      "  Attempting uninstall: docker\n",
      "    Found existing installation: docker 5.0.3\n",
      "    Uninstalling docker-5.0.3:\n",
      "      Successfully uninstalled docker-5.0.3\n",
      "  Attempting uninstall: kubernetes\n",
      "    Found existing installation: kubernetes 23.3.0\n",
      "    Uninstalling kubernetes-23.3.0:\n",
      "      Successfully uninstalled kubernetes-23.3.0\n",
      "  Attempting uninstall: google-api-core\n",
      "    Found existing installation: google-api-core 2.5.0\n",
      "    Uninstalling google-api-core-2.5.0:\n",
      "      Successfully uninstalled google-api-core-2.5.0\n",
      "  Attempting uninstall: tensorboard\n",
      "    Found existing installation: tensorboard 2.6.0\n",
      "    Uninstalling tensorboard-2.6.0:\n",
      "      Successfully uninstalled tensorboard-2.6.0\n",
      "  Attempting uninstall: google-cloud-core\n",
      "    Found existing installation: google-cloud-core 2.2.3\n",
      "    Uninstalling google-cloud-core-2.2.3:\n",
      "      Successfully uninstalled google-cloud-core-2.2.3\n",
      "  Attempting uninstall: google-api-python-client\n",
      "    Found existing installation: google-api-python-client 2.41.0\n",
      "    Uninstalling google-api-python-client-2.41.0:\n",
      "      Successfully uninstalled google-api-python-client-2.41.0\n",
      "  Attempting uninstall: tensorflow\n",
      "    Found existing installation: tensorflow 2.6.3\n",
      "    Uninstalling tensorflow-2.6.3:\n",
      "      Successfully uninstalled tensorflow-2.6.3\n",
      "  Attempting uninstall: google-cloud-vision\n",
      "    Found existing installation: google-cloud-vision 2.7.1\n",
      "    Uninstalling google-cloud-vision-2.7.1:\n",
      "      Successfully uninstalled google-cloud-vision-2.7.1\n",
      "  Attempting uninstall: google-cloud-videointelligence\n",
      "    Found existing installation: google-cloud-videointelligence 2.6.1\n",
      "    Uninstalling google-cloud-videointelligence-2.6.1:\n",
      "      Successfully uninstalled google-cloud-videointelligence-2.6.1\n",
      "  Attempting uninstall: google-cloud-spanner\n",
      "    Found existing installation: google-cloud-spanner 3.13.0\n",
      "    Uninstalling google-cloud-spanner-3.13.0:\n",
      "      Successfully uninstalled google-cloud-spanner-3.13.0\n",
      "  Attempting uninstall: google-cloud-language\n",
      "    Found existing installation: google-cloud-language 2.4.1\n",
      "    Uninstalling google-cloud-language-2.4.1:\n",
      "      Successfully uninstalled google-cloud-language-2.4.1\n",
      "  Attempting uninstall: google-cloud-datastore\n",
      "    Found existing installation: google-cloud-datastore 2.5.1\n",
      "    Uninstalling google-cloud-datastore-2.5.1:\n",
      "      Successfully uninstalled google-cloud-datastore-2.5.1\n",
      "  Attempting uninstall: google-cloud-bigtable\n",
      "    Found existing installation: google-cloud-bigtable 2.7.0\n",
      "    Uninstalling google-cloud-bigtable-2.7.0:\n",
      "      Successfully uninstalled google-cloud-bigtable-2.7.0\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.25.0 which is incompatible.\n",
      "statsmodels 0.13.2 requires packaging>=21.3, but you have packaging 20.9 which is incompatible.\n",
      "pandas-profiling 3.1.0 requires joblib~=1.0.1, but you have joblib 0.14.1 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
      "black 22.1.0 requires click>=8.0.0, but you have click 7.1.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed attrs-20.3.0 click-7.1.2 dill-0.3.1.1 docker-4.4.4 google-api-core-1.31.5 google-api-python-client-1.12.11 google-cloud-bigtable-1.7.1 google-cloud-core-2.2.2 google-cloud-datastore-1.15.4 google-cloud-language-1.3.1 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 httplib2-0.19.1 joblib-0.14.1 keras-2.8.0 kubernetes-12.0.1 libclang-14.0.1 ml-metadata-1.7.0 ml-pipelines-sdk-1.7.1 numpy-1.21.6 packaging-20.9 portpicker-1.5.0 pyarrow-5.0.0 pyfarmhash-0.3.2 pyparsing-2.4.7 pyyaml-5.4.1 tensorboard-2.8.0 tensorflow-2.8.0 tensorflow-data-validation-1.7.0 tensorflow-io-gcs-filesystem-0.25.0 tensorflow-model-analysis-0.38.0 tf-estimator-nightly-2.8.0.dev2021122109 tfx-1.7.1 uritemplate-3.0.1\n"
     ]
    }
   ],
   "source": [
    "!pip install -U tfx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EwT0nov5QO1M"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "Please ignore any incompatibility warnings and errors. **Restart** the kernel to use updated packages. (On the Notebook menu, select Kernel > Restart Kernel > Restart)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BDnPgN8UJtzN"
   },
   "source": [
    "Check the TensorFlow and TFX versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6jh7vKSRqPHb",
    "outputId": "15ad5523-730a-42d8-ee7b-a97e778a1bbb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.8.0\n",
      "TFX version: 1.7.1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print('TensorFlow version: {}'.format(tf.__version__))\n",
    "from tfx import v1 as tfx\n",
    "print('TFX version: {}'.format(tfx.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDtLdSkvqPHe"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "There are some variables used to define a pipeline. You can customize these\n",
    "variables as you want. By default all output from the pipeline will be\n",
    "generated under the current directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EcUseqJaE2XN"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "PIPELINE_NAME = \"penguin-transform\"\n",
    "\n",
    "# Output directory to store artifacts generated from the pipeline.\n",
    "PIPELINE_ROOT = os.path.join('pipelines', PIPELINE_NAME)\n",
    "# Path to a SQLite DB file to use as an MLMD storage.\n",
    "METADATA_PATH = os.path.join('metadata', PIPELINE_NAME, 'metadata.db')\n",
    "# Output directory where created models from the pipeline will be exported.\n",
    "SERVING_MODEL_DIR = os.path.join('serving_model', PIPELINE_NAME)\n",
    "\n",
    "from absl import logging\n",
    "logging.set_verbosity(logging.INFO)  # Set default logging level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qsO0l5F3dzOr"
   },
   "source": [
    "## Prepare example data\n",
    "You will download the example dataset for use in our TFX pipeline. The dataset\n",
    "you are using is\n",
    "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html).\n",
    "\n",
    "However, unlike previous notebooks which used an already preprocessed dataset,\n",
    "you will use the **raw** Palmer Penguins dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "11J7XiCq6AFP"
   },
   "source": [
    "Because the TFX ExampleGen component reads inputs from a directory, you need\n",
    "to create a directory and copy the dataset to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4fxMs6u86acP",
    "outputId": "84c5faab-0987-419b-f8fb-e1bb41fa9d19"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('/tmp/tfx-datax3mqgjxt/data.csv', <http.client.HTTPMessage at 0x7fb93dc9a290>)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "import tempfile\n",
    "\n",
    "DATA_ROOT = tempfile.mkdtemp(prefix='tfx-data')  # TODO 1: Create a temporary directory.\n",
    "_data_path = 'https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins_size.csv'\n",
    "_data_filepath = os.path.join(DATA_ROOT, \"data.csv\")\n",
    "urllib.request.urlretrieve(_data_path, _data_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ASpoNmxKSQjI"
   },
   "source": [
    "Take a quick look at what the raw data looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-eSz28UDSnlG",
    "outputId": "ec0bdc3b-c13a-4e73-c140-d14c4d53103b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species,island,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g,sex\n",
      "Adelie,Torgersen,39.1,18.7,181,3750,MALE\n",
      "Adelie,Torgersen,39.5,17.4,186,3800,FEMALE\n",
      "Adelie,Torgersen,40.3,18,195,3250,FEMALE\n",
      "Adelie,Torgersen,NA,NA,NA,NA,NA\n",
      "Adelie,Torgersen,36.7,19.3,193,3450,FEMALE\n",
      "Adelie,Torgersen,39.3,20.6,190,3650,MALE\n",
      "Adelie,Torgersen,38.9,17.8,181,3625,FEMALE\n",
      "Adelie,Torgersen,39.2,19.6,195,4675,MALE\n",
      "Adelie,Torgersen,34.1,18.1,193,3475,NA\n"
     ]
    }
   ],
   "source": [
    "!head {_data_filepath}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OTtQNq1DdVvG"
   },
   "source": [
    "There are some entries with missing values which are represented as `NA`.\n",
    "You will just delete those entries in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fQhpoaqff9ca",
    "outputId": "86595c70-edd6-4ce1-c3ea-05df85754638"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species,island,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g,sex\n",
      "Adelie,Torgersen,39.1,18.7,181,3750,MALE\n",
      "Adelie,Torgersen,39.5,17.4,186,3800,FEMALE\n",
      "Adelie,Torgersen,40.3,18,195,3250,FEMALE\n",
      "Adelie,Torgersen,36.7,19.3,193,3450,FEMALE\n",
      "Adelie,Torgersen,39.3,20.6,190,3650,MALE\n",
      "Adelie,Torgersen,38.9,17.8,181,3625,FEMALE\n",
      "Adelie,Torgersen,39.2,19.6,195,4675,MALE\n",
      "Adelie,Torgersen,41.1,17.6,182,3200,FEMALE\n",
      "Adelie,Torgersen,38.6,21.2,191,3800,MALE\n"
     ]
    }
   ],
   "source": [
    "!sed -i '/\\bNA\\b/d' {_data_filepath}\n",
    "!head {_data_filepath}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z8EOfCy1dzO2"
   },
   "source": [
    "You should be able to see seven features which describe penguins. You will use\n",
    "the same set of features as the previous notebooks - 'culmen_length_mm',\n",
    "'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g' - and will predict\n",
    "the 'species' of a penguin.\n",
    "\n",
    "**The only difference will be that the input data is not preprocessed.** Note\n",
    "that you will not use other features like 'island' or 'sex' in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jtbrkjjc-IKA"
   },
   "source": [
    "### Prepare a schema file\n",
    "\n",
    "As described in\n",
    "[Data validation using TFX Pipeline and TensorFlow Data Validation Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_tfdv),\n",
    "you need a schema file for the dataset. Because the dataset is different, you need to generate it again. In this notebook, you will skip those steps and just use a prepared schema file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EDoB97m8B9nG",
    "outputId": "e7071269-7156-4d82-cc59-034215c47796"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('schema/schema.pbtxt', <http.client.HTTPMessage at 0x7fb93dc9af50>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import shutil\n",
    "\n",
    "SCHEMA_PATH = 'schema'\n",
    "\n",
    "_schema_uri = 'https://raw.githubusercontent.com/tensorflow/tfx/master/tfx/examples/penguin/schema/raw/schema.pbtxt'\n",
    "_schema_filename = 'schema.pbtxt'\n",
    "_schema_filepath = os.path.join(SCHEMA_PATH, _schema_filename)\n",
    "\n",
    "os.makedirs(SCHEMA_PATH, exist_ok=True)\n",
    "urllib.request.urlretrieve(_schema_uri, _schema_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gKJ_HDJQB94b"
   },
   "source": [
    "This schema file was created with the same pipeline as in the previous notebook\n",
    "without any manual changes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nH6gizcpSwWV"
   },
   "source": [
    "## Create a pipeline\n",
    "\n",
    "TFX pipelines are defined using Python APIs. You will add `Transform`\n",
    "component to the pipeline you created in the\n",
    "[Data Validation tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_tfdv).\n",
    "\n",
    "A Transform component requires input data from an `ExampleGen` component and\n",
    "a schema from a `SchemaGen` component, and produces a \"transform graph\". The\n",
    "output will be used in a `Trainer` component. Transform can optionally\n",
    "produce \"transformed data\" in addition, which is the materialized data after\n",
    "transformation.\n",
    "However, you will transform data during training in this notebook without\n",
    "materialization of the intermediate transformed data.\n",
    "\n",
    "One thing to note is that you need to define a Python function,\n",
    "`preprocessing_fn` to describe how input data should be transformed. This is\n",
    "similar to a Trainer component which also requires user code for model\n",
    "definition.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOjDv93eS5xV"
   },
   "source": [
    "### Write preprocessing and training code\n",
    "\n",
    "You need to define two Python functions. One for Transform and one for Trainer.\n",
    "\n",
    "#### preprocessing_fn\n",
    "The Transform component will find a function named `preprocessing_fn` in the\n",
    "given module file as you did for `Trainer` component. You can also specify a\n",
    "specific function using the\n",
    "`preprocessing_fn` [parameter](https://github.com/tensorflow/tfx/blob/142de6e887f26f4101ded7925f60d7d4fe9d42ed/tfx/components/transform/component.py#L113)\n",
    "of the Transform component.\n",
    "\n",
    "In this example, you will do two kinds of transformation. For continuous numeric\n",
    "features like `culmen_length_mm` and `body_mass_g`, you will normalize these\n",
    "values using the\n",
    "[tft.scale_to_z_score](https://www.tensorflow.org/tfx/transform/api_docs/python/tft/scale_to_z_score)\n",
    "function. For the label feature, you need to convert string labels into numeric\n",
    "index values. You will use\n",
    "[tf.lookup.StaticHashTable](https://www.tensorflow.org/api_docs/python/tf/lookup/StaticHashTable)\n",
    "for conversion.\n",
    "\n",
    "To identify transformed fields easily, you append a `_xf` suffix to the\n",
    "transformed feature names.\n",
    "\n",
    "#### run_fn\n",
    "\n",
    "The model itself is almost the same as in the previous notebooks, but this time\n",
    "you will transform the input data using the transform graph from the Transform\n",
    "component.\n",
    "\n",
    "One more important difference compared to the previous notebook is that now you\n",
    "export a model for serving which includes not only the computation graph of the\n",
    "model, but also the transform graph for preprocessing, which is generated in\n",
    "Transform component. You need to define a separate function which will be used\n",
    "for serving incoming requests. You can see that the same function\n",
    "`_apply_preprocessing` was used for both of the training data and the\n",
    "serving request.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "aES7Hv5QTDK3"
   },
   "outputs": [],
   "source": [
    "_module_file = 'penguin_utils.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gnc67uQNTDfW",
    "outputId": "b2f3fcde-c489-4c2c-ba08-388efcceb43d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing penguin_utils.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {_module_file}\n",
    "\n",
    "\n",
    "from typing import List, Text\n",
    "from absl import logging\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow_metadata.proto.v0 import schema_pb2\n",
    "import tensorflow_transform as tft\n",
    "from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "from tfx import v1 as tfx\n",
    "from tfx_bsl.public import tfxio\n",
    "\n",
    "# Specify features that you will use.\n",
    "_FEATURE_KEYS = [\n",
    "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
    "]\n",
    "_LABEL_KEY = 'species'\n",
    "\n",
    "_TRAIN_BATCH_SIZE = 20\n",
    "_EVAL_BATCH_SIZE = 10\n",
    "\n",
    "\n",
    "# NEW: TFX Transform will call this function.\n",
    "def preprocessing_fn(inputs):\n",
    "  \"\"\"tf.transform's callback function for preprocessing inputs.\n",
    "\n",
    "  Args:\n",
    "    inputs: map from feature keys to raw not-yet-transformed features.\n",
    "\n",
    "  Returns:\n",
    "    Map from string feature key to transformed feature.\n",
    "  \"\"\"\n",
    "  outputs = {}\n",
    "\n",
    "  # Uses features defined in _FEATURE_KEYS only.\n",
    "  for key in _FEATURE_KEYS:\n",
    "    # tft.scale_to_z_score computes the mean and variance of the given feature\n",
    "    # and scales the output based on the result.\n",
    "    outputs[key] = tft.scale_to_z_score(inputs[key])\n",
    "\n",
    "  # For the label column you provide the mapping from string to index.\n",
    "  # You could instead use `tft.compute_and_apply_vocabulary()` in order to\n",
    "  # compute the vocabulary dynamically and perform a lookup.\n",
    "  # Since in this example there are only 3 possible values, you use a hard-coded\n",
    "  # table for simplicity.\n",
    "  table_keys = ['Adelie', 'Chinstrap', 'Gentoo']\n",
    "  initializer = tf.lookup.KeyValueTensorInitializer(\n",
    "      keys=table_keys,\n",
    "      values=tf.cast(tf.range(len(table_keys)), tf.int64),\n",
    "      key_dtype=tf.string,\n",
    "      value_dtype=tf.int64)\n",
    "  table = tf.lookup.StaticHashTable(initializer, default_value=-1)\n",
    "  outputs[_LABEL_KEY] = table.lookup(inputs[_LABEL_KEY])\n",
    "\n",
    "  return outputs\n",
    "\n",
    "\n",
    "# NEW: This function will apply the same transform operation to training data\n",
    "#      and serving requests.\n",
    "def _apply_preprocessing(raw_features, tft_layer):\n",
    "  transformed_features = tft_layer(raw_features)\n",
    "  if _LABEL_KEY in raw_features:\n",
    "    transformed_label = transformed_features.pop(_LABEL_KEY)\n",
    "    return transformed_features, transformed_label\n",
    "  else:\n",
    "    return transformed_features, None\n",
    "\n",
    "\n",
    "# NEW: This function will create a handler function which gets a serialized\n",
    "#      tf.example, preprocess and run an inference with it.\n",
    "def _get_serve_tf_examples_fn(model, tf_transform_output):\n",
    "  # You must save the tft_layer to the model to ensure its assets are kept and\n",
    "  # tracked.\n",
    "  model.tft_layer = tf_transform_output.transform_features_layer()\n",
    "\n",
    "  @tf.function(input_signature=[\n",
    "      tf.TensorSpec(shape=[None], dtype=tf.string, name='examples')\n",
    "  ])\n",
    "  def serve_tf_examples_fn(serialized_tf_examples):\n",
    "    # Expected input is a string which is serialized tf.Example format.\n",
    "    feature_spec = tf_transform_output.raw_feature_spec()\n",
    "    # Because input schema includes unnecessary fields like 'species' and\n",
    "    # 'island', you filter feature_spec to include required keys only.\n",
    "    required_feature_spec = {\n",
    "        k: v for k, v in feature_spec.items() if k in _FEATURE_KEYS\n",
    "    }\n",
    "    parsed_features = tf.io.parse_example(serialized_tf_examples,\n",
    "                                          required_feature_spec)\n",
    "\n",
    "    # Preprocess parsed input with transform operation defined in\n",
    "    # preprocessing_fn().\n",
    "    transformed_features, _ = _apply_preprocessing(parsed_features,\n",
    "                                                   model.tft_layer)\n",
    "    # Run inference with ML model.\n",
    "    return model(transformed_features)\n",
    "\n",
    "  return serve_tf_examples_fn\n",
    "\n",
    "\n",
    "def _input_fn(file_pattern: List[Text],\n",
    "              data_accessor: tfx.components.DataAccessor,\n",
    "              tf_transform_output: tft.TFTransformOutput,\n",
    "              batch_size: int = 200) -> tf.data.Dataset:\n",
    "  \"\"\"Generates features and label for tuning/training.\n",
    "\n",
    "  Args:\n",
    "    file_pattern: List of paths or patterns of input tfrecord files.\n",
    "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
    "    tf_transform_output: A TFTransformOutput.\n",
    "    batch_size: representing the number of consecutive elements of returned\n",
    "      dataset to combine in a single batch\n",
    "\n",
    "  Returns:\n",
    "    A dataset that contains (features, indices) tuple where features is a\n",
    "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
    "  \"\"\"\n",
    "  dataset = data_accessor.tf_dataset_factory(\n",
    "      file_pattern,\n",
    "      tfxio.TensorFlowDatasetOptions(batch_size=batch_size),\n",
    "      schema=tf_transform_output.raw_metadata.schema)\n",
    "\n",
    "  transform_layer = tf_transform_output.transform_features_layer()\n",
    "  def apply_transform(raw_features):\n",
    "    return _apply_preprocessing(raw_features, transform_layer)\n",
    "\n",
    "  return dataset.map(apply_transform).repeat()\n",
    "\n",
    "\n",
    "def _build_keras_model() -> tf.keras.Model:\n",
    "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
    "\n",
    "  Returns:\n",
    "    A Keras Model.\n",
    "  \"\"\"\n",
    "  # The model below is built with Functional API, please refer to\n",
    "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
    "  inputs = [\n",
    "      keras.layers.Input(shape=(1,), name=key)\n",
    "      for key in _FEATURE_KEYS\n",
    "  ]\n",
    "  d = keras.layers.concatenate(inputs)\n",
    "  for _ in range(2):\n",
    "    d = keras.layers.Dense(8, activation='relu')(d)\n",
    "  outputs = keras.layers.Dense(3)(d)\n",
    "\n",
    "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "  model.compile(\n",
    "      optimizer=keras.optimizers.Adam(1e-2),\n",
    "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "  model.summary(print_fn=logging.info)\n",
    "  return model\n",
    "\n",
    "\n",
    "# TFX Trainer will call this function.\n",
    "def run_fn(fn_args: tfx.components.FnArgs):\n",
    "  \"\"\"Train the model based on given args.\n",
    "\n",
    "  Args:\n",
    "    fn_args: Holds args used to train the model as name/value pairs.\n",
    "  \"\"\"\n",
    "  tf_transform_output = tft.TFTransformOutput(fn_args.transform_output)\n",
    "\n",
    "  train_dataset = _input_fn(\n",
    "      fn_args.train_files,\n",
    "      fn_args.data_accessor,\n",
    "      tf_transform_output,\n",
    "      batch_size=_TRAIN_BATCH_SIZE)\n",
    "  eval_dataset = _input_fn(\n",
    "      fn_args.eval_files,\n",
    "      fn_args.data_accessor,\n",
    "      tf_transform_output,\n",
    "      batch_size=_EVAL_BATCH_SIZE)\n",
    "\n",
    "  model = _build_keras_model()\n",
    "  model.fit(\n",
    "      train_dataset,\n",
    "      steps_per_epoch=fn_args.train_steps,\n",
    "      validation_data=eval_dataset,\n",
    "      validation_steps=fn_args.eval_steps)\n",
    "\n",
    "  # NEW: Save a computation graph including transform layer.\n",
    "  signatures = {\n",
    "      'serving_default': _get_serve_tf_examples_fn(model, tf_transform_output),\n",
    "  }\n",
    "  model.save(fn_args.serving_model_dir, save_format='tf', signatures=signatures)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "blaw0rs-emEf"
   },
   "source": [
    "Now you have completed all of the preparation steps to build a TFX pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w3OkNz3gTLwM"
   },
   "source": [
    "### Write a pipeline definition\n",
    "\n",
    "You define a function to create a TFX pipeline. A `Pipeline` object\n",
    "represents a TFX pipeline, which can be run using one of the pipeline\n",
    "orchestration systems that TFX supports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "M49yYVNBTPd4"
   },
   "outputs": [],
   "source": [
    "def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,\n",
    "                     schema_path: str, module_file: str, serving_model_dir: str,\n",
    "                     metadata_path: str) -> tfx.dsl.Pipeline:\n",
    "  \"\"\"Implements the penguin pipeline with TFX.\"\"\"\n",
    "  # Brings data into the pipeline or otherwise joins/converts training data.\n",
    "  example_gen = tfx.components.CsvExampleGen(input_base=data_root)\n",
    "\n",
    "  # TODO 2: Computes statistics over data for visualization and example validation.\n",
    "  statistics_gen = tfx.components.StatisticsGen(\n",
    "      examples=example_gen.outputs['examples'])\n",
    "\n",
    "  # Import the schema.\n",
    "  schema_importer = tfx.dsl.Importer(\n",
    "      source_uri=schema_path,\n",
    "      artifact_type=tfx.types.standard_artifacts.Schema).with_id(\n",
    "          'schema_importer')\n",
    "\n",
    "  # Performs anomaly detection based on statistics and data schema.\n",
    "  example_validator = tfx.components.ExampleValidator(\n",
    "      statistics=statistics_gen.outputs['statistics'],\n",
    "      schema=schema_importer.outputs['result'])\n",
    "\n",
    "  # NEW: Transforms input data using preprocessing_fn in the 'module_file'.\n",
    "  transform = tfx.components.Transform(\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      schema=schema_importer.outputs['result'],\n",
    "      materialize=False,\n",
    "      module_file=module_file)\n",
    "\n",
    "  # Uses user-provided Python function that trains a model.\n",
    "  trainer = tfx.components.Trainer(\n",
    "      module_file=module_file,\n",
    "      examples=example_gen.outputs['examples'],\n",
    "\n",
    "      # NEW: Pass transform_graph to the trainer.\n",
    "      transform_graph=transform.outputs['transform_graph'],\n",
    "\n",
    "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
    "      eval_args=tfx.proto.EvalArgs(num_steps=5))\n",
    "\n",
    "  # Pushes the model to a filesystem destination.\n",
    "  pusher = tfx.components.Pusher(\n",
    "      model=trainer.outputs['model'],\n",
    "      push_destination=tfx.proto.PushDestination(\n",
    "          filesystem=tfx.proto.PushDestination.Filesystem(\n",
    "              base_directory=serving_model_dir)))\n",
    "\n",
    "  components = [\n",
    "      example_gen,\n",
    "      statistics_gen,\n",
    "      schema_importer,\n",
    "      example_validator,\n",
    "\n",
    "      transform,  # NEW: Transform component was added to the pipeline.\n",
    "\n",
    "      trainer,\n",
    "      pusher,\n",
    "  ]\n",
    "\n",
    "  return tfx.dsl.Pipeline(\n",
    "      pipeline_name=pipeline_name,\n",
    "      pipeline_root=pipeline_root,\n",
    "      metadata_connection_config=tfx.orchestration.metadata\n",
    "      .sqlite_metadata_connection_config(metadata_path),\n",
    "      components=components)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJbq07THU2GV"
   },
   "source": [
    "## Run the pipeline\n",
    "\n",
    "You will use `LocalDagRunner` as in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "fAtfOZTYWJu-",
    "outputId": "2704a32f-d347-4046-a093-760744abd6a6"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Excluding no splits because exclude_splits is not set.\n",
      "INFO:absl:Excluding no splits because exclude_splits is not set.\n",
      "INFO:absl:Generating ephemeral wheel package for '/home/jupyter/penguin_utils.py' (including modules: ['penguin_utils']).\n",
      "INFO:absl:User module package has hash fingerprint version a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.\n",
      "INFO:absl:Executing: ['/opt/conda/bin/python', '/tmp/tmpnx5tttz7/_tfx_generated_setup.py', 'bdist_wheel', '--bdist-dir', '/tmp/tmpt71l5dzh', '--dist-dir', '/tmp/tmpwek7i2ro']\n",
      "/opt/conda/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n",
      "  setuptools.SetuptoolsDeprecationWarning,\n",
      "INFO:absl:Successfully built user code wheel distribution at 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'; target user module is 'penguin_utils'.\n",
      "INFO:absl:Full user module path is 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'\n",
      "INFO:absl:Generating ephemeral wheel package for '/home/jupyter/penguin_utils.py' (including modules: ['penguin_utils']).\n",
      "INFO:absl:User module package has hash fingerprint version a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.\n",
      "INFO:absl:Executing: ['/opt/conda/bin/python', '/tmp/tmp0p1x0o_v/_tfx_generated_setup.py', 'bdist_wheel', '--bdist-dir', '/tmp/tmp91n31aas', '--dist-dir', '/tmp/tmpdkooczqd']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running bdist_wheel\n",
      "running build\n",
      "running build_py\n",
      "creating build\n",
      "creating build/lib\n",
      "copying penguin_utils.py -> build/lib\n",
      "installing to /tmp/tmpt71l5dzh\n",
      "running install\n",
      "running install_lib\n",
      "copying build/lib/penguin_utils.py -> /tmp/tmpt71l5dzh\n",
      "running install_egg_info\n",
      "running egg_info\n",
      "creating tfx_user_code_Transform.egg-info\n",
      "writing tfx_user_code_Transform.egg-info/PKG-INFO\n",
      "writing dependency_links to tfx_user_code_Transform.egg-info/dependency_links.txt\n",
      "writing top-level names to tfx_user_code_Transform.egg-info/top_level.txt\n",
      "writing manifest file 'tfx_user_code_Transform.egg-info/SOURCES.txt'\n",
      "reading manifest file 'tfx_user_code_Transform.egg-info/SOURCES.txt'\n",
      "writing manifest file 'tfx_user_code_Transform.egg-info/SOURCES.txt'\n",
      "Copying tfx_user_code_Transform.egg-info to /tmp/tmpt71l5dzh/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3.7.egg-info\n",
      "running install_scripts\n",
      "creating /tmp/tmpt71l5dzh/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/WHEEL\n",
      "creating '/tmp/tmpwek7i2ro/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl' and adding '/tmp/tmpt71l5dzh' to it\n",
      "adding 'penguin_utils.py'\n",
      "adding 'tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/METADATA'\n",
      "adding 'tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/WHEEL'\n",
      "adding 'tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/top_level.txt'\n",
      "adding 'tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/RECORD'\n",
      "removing /tmp/tmpt71l5dzh\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n",
      "  setuptools.SetuptoolsDeprecationWarning,\n",
      "INFO:absl:Successfully built user code wheel distribution at 'pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'; target user module is 'penguin_utils'.\n",
      "INFO:absl:Full user module path is 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'\n",
      "INFO:absl:Using deployment config:\n",
      " executor_specs {\n",
      "  key: \"CsvExampleGen\"\n",
      "  value {\n",
      "    beam_executable_spec {\n",
      "      python_executor_spec {\n",
      "        class_path: \"tfx.components.example_gen.csv_example_gen.executor.Executor\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"ExampleValidator\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.example_validator.executor.Executor\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"Pusher\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.pusher.executor.Executor\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"StatisticsGen\"\n",
      "  value {\n",
      "    beam_executable_spec {\n",
      "      python_executor_spec {\n",
      "        class_path: \"tfx.components.statistics_gen.executor.Executor\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"Trainer\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.trainer.executor.GenericExecutor\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "executor_specs {\n",
      "  key: \"Transform\"\n",
      "  value {\n",
      "    beam_executable_spec {\n",
      "      python_executor_spec {\n",
      "        class_path: \"tfx.components.transform.executor.Executor\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "custom_driver_specs {\n",
      "  key: \"CsvExampleGen\"\n",
      "  value {\n",
      "    python_class_executable_spec {\n",
      "      class_path: \"tfx.components.example_gen.driver.FileBasedDriver\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "metadata_connection_config {\n",
      "  database_connection_config {\n",
      "    sqlite {\n",
      "      filename_uri: \"metadata/penguin-transform/metadata.db\"\n",
      "      connection_mode: READWRITE_OPENCREATE\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:Using connection config:\n",
      " sqlite {\n",
      "  filename_uri: \"metadata/penguin-transform/metadata.db\"\n",
      "  connection_mode: READWRITE_OPENCREATE\n",
      "}\n",
      "\n",
      "INFO:absl:Component CsvExampleGen is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.example_gen.csv_example_gen.component.CsvExampleGen\"\n",
      "  }\n",
      "  id: \"CsvExampleGen\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.CsvExampleGen\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Examples\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          properties {\n",
      "            key: \"version\"\n",
      "            value: INT\n",
      "          }\n",
      "          base_type: DATASET\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"input_base\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"/tmp/tfx-datax3mqgjxt\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"input_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_data_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 6\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_file_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 5\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "downstream_nodes: \"StatisticsGen\"\n",
      "downstream_nodes: \"Trainer\"\n",
      "downstream_nodes: \"Transform\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:select span and version = (0, None)\n",
      "INFO:absl:latest span and version = (0, None)\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Going to run a new execution 1\n",
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=1, input_dict={}, output_dict=defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-transform/CsvExampleGen/examples/1\"\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:13161,xor_checksum:1652267862,sum_checksum:1652267862\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:CsvExampleGen:examples:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]}), exec_properties={'input_base': '/tmp/tfx-datax3mqgjxt', 'output_config': '{\\n  \"split_config\": {\\n    \"splits\": [\\n      {\\n        \"hash_buckets\": 2,\\n        \"name\": \"train\"\\n      },\\n      {\\n        \"hash_buckets\": 1,\\n        \"name\": \"eval\"\\n      }\\n    ]\\n  }\\n}', 'output_data_format': 6, 'output_file_format': 5, 'input_config': '{\\n  \"splits\": [\\n    {\\n      \"name\": \"single_split\",\\n      \"pattern\": \"*\"\\n    }\\n  ]\\n}', 'span': 0, 'version': None, 'input_fingerprint': 'split:single_split,num_files:1,total_bytes:13161,xor_checksum:1652267862,sum_checksum:1652267862'}, execution_output_uri='pipelines/penguin-transform/CsvExampleGen/.system/executor_execution/1/executor_output.pb', stateful_working_dir='pipelines/penguin-transform/CsvExampleGen/.system/stateful_working_dir/2022-05-11T11:18:03.288294', tmp_dir='pipelines/penguin-transform/CsvExampleGen/.system/executor_execution/1/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.example_gen.csv_example_gen.component.CsvExampleGen\"\n",
      "  }\n",
      "  id: \"CsvExampleGen\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.CsvExampleGen\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Examples\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          properties {\n",
      "            key: \"version\"\n",
      "            value: INT\n",
      "          }\n",
      "          base_type: DATASET\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"input_base\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"/tmp/tfx-datax3mqgjxt\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"input_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_data_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 6\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"output_file_format\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 5\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "downstream_nodes: \"StatisticsGen\"\n",
      "downstream_nodes: \"Trainer\"\n",
      "downstream_nodes: \"Transform\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-transform\"\n",
      ", pipeline_run_id='2022-05-11T11:18:03.288294')\n",
      "INFO:absl:Generating examples.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running bdist_wheel\n",
      "running build\n",
      "running build_py\n",
      "creating build\n",
      "creating build/lib\n",
      "copying penguin_utils.py -> build/lib\n",
      "installing to /tmp/tmp91n31aas\n",
      "running install\n",
      "running install_lib\n",
      "copying build/lib/penguin_utils.py -> /tmp/tmp91n31aas\n",
      "running install_egg_info\n",
      "running egg_info\n",
      "creating tfx_user_code_Trainer.egg-info\n",
      "writing tfx_user_code_Trainer.egg-info/PKG-INFO\n",
      "writing dependency_links to tfx_user_code_Trainer.egg-info/dependency_links.txt\n",
      "writing top-level names to tfx_user_code_Trainer.egg-info/top_level.txt\n",
      "writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "reading manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "Copying tfx_user_code_Trainer.egg-info to /tmp/tmp91n31aas/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3.7.egg-info\n",
      "running install_scripts\n",
      "creating /tmp/tmp91n31aas/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/WHEEL\n",
      "creating '/tmp/tmpdkooczqd/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl' and adding '/tmp/tmp91n31aas' to it\n",
      "adding 'penguin_utils.py'\n",
      "adding 'tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/METADATA'\n",
      "adding 'tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/WHEEL'\n",
      "adding 'tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/top_level.txt'\n",
      "adding 'tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9.dist-info/RECORD'\n",
      "removing /tmp/tmp91n31aas\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (typeof window.interactive_beam_jquery == 'undefined') {\n",
       "          var jqueryScript = document.createElement('script');\n",
       "          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
       "          jqueryScript.type = 'text/javascript';\n",
       "          jqueryScript.onload = function() {\n",
       "            var datatableScript = document.createElement('script');\n",
       "            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
       "            datatableScript.type = 'text/javascript';\n",
       "            datatableScript.onload = function() {\n",
       "              window.interactive_beam_jquery = jQuery.noConflict(true);\n",
       "              window.interactive_beam_jquery(document).ready(function($){\n",
       "                \n",
       "              });\n",
       "            }\n",
       "            document.head.appendChild(datatableScript);\n",
       "          };\n",
       "          document.head.appendChild(jqueryScript);\n",
       "        } else {\n",
       "          window.interactive_beam_jquery(document).ready(function($){\n",
       "            \n",
       "          });\n",
       "        }"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Processing input csv data /tmp/tfx-datax3mqgjxt/* to TFExample.\n",
      "WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.\n",
      "E0511 11:18:04.216559042   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n",
      "WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.\n",
      "INFO:absl:Examples generated.\n",
      "INFO:absl:Value type <class 'NoneType'> of key version in exec_properties is not supported, going to drop it\n",
      "INFO:absl:Value type <class 'list'> of key _beam_pipeline_args in exec_properties is not supported, going to drop it\n",
      "INFO:absl:Cleaning up stateless execution info.\n",
      "INFO:absl:Execution 1 succeeded.\n",
      "INFO:absl:Cleaning up stateful execution info.\n",
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-transform/CsvExampleGen/examples/1\"\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:13161,xor_checksum:1652267862,sum_checksum:1652267862\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:CsvExampleGen:examples:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]}) for execution 1\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Component CsvExampleGen is finished.\n",
      "INFO:absl:Component schema_importer is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.dsl.components.common.importer.Importer\"\n",
      "  }\n",
      "  id: \"schema_importer\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.schema_importer\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"result\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Schema\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"artifact_uri\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"schema\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"reimport\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 0\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "downstream_nodes: \"ExampleValidator\"\n",
      "downstream_nodes: \"Transform\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:Running as an importer node.\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Processing source uri: schema, properties: {}, custom_properties: {}\n",
      "INFO:absl:Component schema_importer is finished.\n",
      "INFO:absl:Component StatisticsGen is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.statistics_gen.component.StatisticsGen\"\n",
      "    base_type: PROCESS\n",
      "  }\n",
      "  id: \"StatisticsGen\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.StatisticsGen\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"statistics\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleStatistics\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          base_type: STATISTICS\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"exclude_splits\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"[]\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "downstream_nodes: \"ExampleValidator\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Going to run a new execution 3\n",
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=3, input_dict={'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-transform/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:13161,xor_checksum:1652267862,sum_checksum:1652267862\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:CsvExampleGen:examples:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267884894\n",
      "last_update_time_since_epoch: 1652267884894\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'statistics': [Artifact(artifact: uri: \"pipelines/penguin-transform/StatisticsGen/statistics/3\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:StatisticsGen:statistics:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")]}), exec_properties={'exclude_splits': '[]'}, execution_output_uri='pipelines/penguin-transform/StatisticsGen/.system/executor_execution/3/executor_output.pb', stateful_working_dir='pipelines/penguin-transform/StatisticsGen/.system/stateful_working_dir/2022-05-11T11:18:03.288294', tmp_dir='pipelines/penguin-transform/StatisticsGen/.system/executor_execution/3/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.statistics_gen.component.StatisticsGen\"\n",
      "    base_type: PROCESS\n",
      "  }\n",
      "  id: \"StatisticsGen\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.StatisticsGen\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"statistics\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleStatistics\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          base_type: STATISTICS\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"exclude_splits\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"[]\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "downstream_nodes: \"ExampleValidator\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-transform\"\n",
      ", pipeline_run_id='2022-05-11T11:18:03.288294')\n",
      "INFO:absl:Generating statistics for split train.\n",
      "INFO:absl:Statistics for split train written to pipelines/penguin-transform/StatisticsGen/statistics/3/Split-train.\n",
      "INFO:absl:Generating statistics for split eval.\n",
      "INFO:absl:Statistics for split eval written to pipelines/penguin-transform/StatisticsGen/statistics/3/Split-eval.\n",
      "WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.\n",
      "INFO:absl:Cleaning up stateless execution info.\n",
      "INFO:absl:Execution 3 succeeded.\n",
      "INFO:absl:Cleaning up stateful execution info.\n",
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'statistics': [Artifact(artifact: uri: \"pipelines/penguin-transform/StatisticsGen/statistics/3\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:StatisticsGen:statistics:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")]}) for execution 3\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Component StatisticsGen is finished.\n",
      "INFO:absl:Component Transform is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.transform.component.Transform\"\n",
      "    base_type: TRANSFORM\n",
      "  }\n",
      "  id: \"Transform\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.Transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"schema\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"schema_importer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.schema_importer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Schema\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"result\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"post_transform_anomalies\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleAnomalies\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"post_transform_schema\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Schema\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"post_transform_stats\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleStatistics\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          base_type: STATISTICS\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"pre_transform_schema\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Schema\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"pre_transform_stats\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleStatistics\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          base_type: STATISTICS\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"transform_graph\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"TransformGraph\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"updated_analyzer_cache\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"TransformCache\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"disable_statistics\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 0\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"force_tf_compat_v1\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 0\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"module_path\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "upstream_nodes: \"schema_importer\"\n",
      "downstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Going to run a new execution 4\n",
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=4, input_dict={'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-transform/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:13161,xor_checksum:1652267862,sum_checksum:1652267862\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:CsvExampleGen:examples:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267884894\n",
      "last_update_time_since_epoch: 1652267884894\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")], 'schema': [Artifact(artifact: id: 2\n",
      "type_id: 17\n",
      "uri: \"schema\"\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267884931\n",
      "last_update_time_since_epoch: 1652267884931\n",
      ", artifact_type: id: 17\n",
      "name: \"Schema\"\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'pre_transform_schema': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/pre_transform_schema/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:pre_transform_schema:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Schema\"\n",
      ")], 'updated_analyzer_cache': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/updated_analyzer_cache/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:updated_analyzer_cache:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"TransformCache\"\n",
      ")], 'pre_transform_stats': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/pre_transform_stats/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:pre_transform_stats:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")], 'post_transform_schema': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/post_transform_schema/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:post_transform_schema:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Schema\"\n",
      ")], 'post_transform_stats': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/post_transform_stats/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:post_transform_stats:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")], 'transform_graph': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/transform_graph/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:transform_graph:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"TransformGraph\"\n",
      ")], 'post_transform_anomalies': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/post_transform_anomalies/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:post_transform_anomalies:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleAnomalies\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      ")]}), exec_properties={'custom_config': 'null', 'force_tf_compat_v1': 0, 'module_path': 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl', 'disable_statistics': 0}, execution_output_uri='pipelines/penguin-transform/Transform/.system/executor_execution/4/executor_output.pb', stateful_working_dir='pipelines/penguin-transform/Transform/.system/stateful_working_dir/2022-05-11T11:18:03.288294', tmp_dir='pipelines/penguin-transform/Transform/.system/executor_execution/4/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.transform.component.Transform\"\n",
      "    base_type: TRANSFORM\n",
      "  }\n",
      "  id: \"Transform\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.Transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"schema\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"schema_importer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.schema_importer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Schema\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"result\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"post_transform_anomalies\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleAnomalies\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"post_transform_schema\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Schema\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"post_transform_stats\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleStatistics\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          base_type: STATISTICS\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"pre_transform_schema\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Schema\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"pre_transform_stats\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleStatistics\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "          base_type: STATISTICS\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"transform_graph\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"TransformGraph\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"updated_analyzer_cache\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"TransformCache\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"disable_statistics\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 0\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"force_tf_compat_v1\"\n",
      "    value {\n",
      "      field_value {\n",
      "        int_value: 0\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"module_path\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "upstream_nodes: \"schema_importer\"\n",
      "downstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-transform\"\n",
      ", pipeline_run_id='2022-05-11T11:18:03.288294')\n",
      "INFO:absl:Analyze the 'train' split and transform all splits when splits_config is not set.\n",
      "INFO:absl:udf_utils.get_fn {'module_file': None, 'module_path': 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl', 'preprocessing_fn': None} 'preprocessing_fn'\n",
      "INFO:absl:Installing 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl' to a temporary directory.\n",
      "INFO:absl:Executing: ['/opt/conda/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmp5noy1t0k', 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl']\n",
      "E0511 11:18:08.413466051   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing ./pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Successfully installed 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'.\n",
      "INFO:absl:udf_utils.get_fn {'module_file': None, 'module_path': 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl', 'stats_options_updater_fn': None} 'stats_options_updater_fn'\n",
      "INFO:absl:Installing 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl' to a temporary directory.\n",
      "INFO:absl:Executing: ['/opt/conda/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmpl0yh21hg', 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: tfx-user-code-Transform\n",
      "Successfully installed tfx-user-code-Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "E0511 11:18:11.319338519   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing ./pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Successfully installed 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'.\n",
      "INFO:absl:Installing 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl' to a temporary directory.\n",
      "INFO:absl:Executing: ['/opt/conda/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmp0vrw84pl', 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: tfx-user-code-Transform\n",
      "Successfully installed tfx-user-code-Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "E0511 11:18:14.194614319   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing ./pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Successfully installed 'pipelines/penguin-transform/_wheels/tfx_user_code_Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: tfx-user-code-Transform\n",
      "Successfully installed tfx-user-code-Transform-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9\n",
      "WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow_transform/tf_utils.py:325: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use ref() instead.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-05-11 11:18:17.210843: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64\n",
      "2022-05-11 11:18:17.210895: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-05-11 11:18:17.210921: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (tensorflow-2-6-20220511-163056): /proc/driver/nvidia/version does not exist\n",
      "2022-05-11 11:18:17.211232: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow_transform/tf_utils.py:325: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use ref() instead.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Tuple[Dict[str, Union[NoneType, _Dataset]], Union[Dict[str, Dict[str, PCollection]], NoneType], int] instead.\n",
      "WARNING:absl:Tables initialized inside a tf.function  will be re-initialized on every invocation of the function. This  re-initialization can have significant impact on performance. Consider lifting  them out of the graph context using  `tf.init_scope`.: key_value_init/LookupTableImportV2\n",
      "WARNING:absl:Tables initialized inside a tf.function  will be re-initialized on every invocation of the function. This  re-initialization can have significant impact on performance. Consider lifting  them out of the graph context using  `tf.init_scope`.: key_value_init/LookupTableImportV2\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Tuple[Dict[str, Union[NoneType, _Dataset]], Union[Dict[str, Dict[str, PCollection]], NoneType], int] instead.\n",
      "WARNING:absl:Tables initialized inside a tf.function  will be re-initialized on every invocation of the function. This  re-initialization can have significant impact on performance. Consider lifting  them out of the graph context using  `tf.init_scope`.: key_value_init/LookupTableImportV2\n",
      "WARNING:absl:Tables initialized inside a tf.function  will be re-initialized on every invocation of the function. This  re-initialization can have significant impact on performance. Consider lifting  them out of the graph context using  `tf.init_scope`.: key_value_init/LookupTableImportV2\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: pipelines/penguin-transform/Transform/transform_graph/4/.temp_path/tftransform_tmp/c9a14a39da934454b23e883ac0a128c4/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-05-11 11:18:22.514820: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n",
      "INFO:tensorflow:Assets written to: pipelines/penguin-transform/Transform/transform_graph/4/.temp_path/tftransform_tmp/c9a14a39da934454b23e883ac0a128c4/assets\n",
      "WARNING:absl:Tables initialized inside a tf.function  will be re-initialized on every invocation of the function. This  re-initialization can have significant impact on performance. Consider lifting  them out of the graph context using  `tf.init_scope`.: key_value_init/LookupTableImportV2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: pipelines/penguin-transform/Transform/transform_graph/4/.temp_path/tftransform_tmp/46c46bb910d54d6d96de51a5da7e96bd/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: pipelines/penguin-transform/Transform/transform_graph/4/.temp_path/tftransform_tmp/46c46bb910d54d6d96de51a5da7e96bd/assets\n",
      "WARNING:absl:Tables initialized inside a tf.function  will be re-initialized on every invocation of the function. This  re-initialization can have significant impact on performance. Consider lifting  them out of the graph context using  `tf.init_scope`.: key_value_init/LookupTableImportV2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n",
      "INFO:absl:Cleaning up stateless execution info.\n",
      "INFO:absl:Execution 4 succeeded.\n",
      "INFO:absl:Cleaning up stateful execution info.\n",
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'pre_transform_schema': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/pre_transform_schema/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:pre_transform_schema:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Schema\"\n",
      ")], 'updated_analyzer_cache': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/updated_analyzer_cache/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:updated_analyzer_cache:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"TransformCache\"\n",
      ")], 'pre_transform_stats': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/pre_transform_stats/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:pre_transform_stats:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")], 'post_transform_schema': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/post_transform_schema/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:post_transform_schema:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Schema\"\n",
      ")], 'post_transform_stats': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/post_transform_stats/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:post_transform_stats:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")], 'transform_graph': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/transform_graph/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:transform_graph:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"TransformGraph\"\n",
      ")], 'post_transform_anomalies': [Artifact(artifact: uri: \"pipelines/penguin-transform/Transform/post_transform_anomalies/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:post_transform_anomalies:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleAnomalies\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      ")]}) for execution 4\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Component Transform is finished.\n",
      "INFO:absl:Component ExampleValidator is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.example_validator.component.ExampleValidator\"\n",
      "  }\n",
      "  id: \"ExampleValidator\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.ExampleValidator\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"schema\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"schema_importer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.schema_importer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Schema\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"result\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"statistics\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"StatisticsGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.StatisticsGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"ExampleStatistics\"\n",
      "            base_type: STATISTICS\n",
      "          }\n",
      "        }\n",
      "        output_key: \"statistics\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"anomalies\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleAnomalies\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"exclude_splits\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"[]\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"StatisticsGen\"\n",
      "upstream_nodes: \"schema_importer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Going to run a new execution 5\n",
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=5, input_dict={'schema': [Artifact(artifact: id: 2\n",
      "type_id: 17\n",
      "uri: \"schema\"\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267884931\n",
      "last_update_time_since_epoch: 1652267884931\n",
      ", artifact_type: id: 17\n",
      "name: \"Schema\"\n",
      ")], 'statistics': [Artifact(artifact: id: 3\n",
      "type_id: 19\n",
      "uri: \"pipelines/penguin-transform/StatisticsGen/statistics/3\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:StatisticsGen:statistics:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267888353\n",
      "last_update_time_since_epoch: 1652267888353\n",
      ", artifact_type: id: 19\n",
      "name: \"ExampleStatistics\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "base_type: STATISTICS\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'anomalies': [Artifact(artifact: uri: \"pipelines/penguin-transform/ExampleValidator/anomalies/5\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:ExampleValidator:anomalies:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleAnomalies\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      ")]}), exec_properties={'exclude_splits': '[]'}, execution_output_uri='pipelines/penguin-transform/ExampleValidator/.system/executor_execution/5/executor_output.pb', stateful_working_dir='pipelines/penguin-transform/ExampleValidator/.system/stateful_working_dir/2022-05-11T11:18:03.288294', tmp_dir='pipelines/penguin-transform/ExampleValidator/.system/executor_execution/5/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.example_validator.component.ExampleValidator\"\n",
      "  }\n",
      "  id: \"ExampleValidator\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.ExampleValidator\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"schema\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"schema_importer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.schema_importer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Schema\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"result\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"statistics\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"StatisticsGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.StatisticsGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"ExampleStatistics\"\n",
      "            base_type: STATISTICS\n",
      "          }\n",
      "        }\n",
      "        output_key: \"statistics\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"anomalies\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ExampleAnomalies\"\n",
      "          properties {\n",
      "            key: \"span\"\n",
      "            value: INT\n",
      "          }\n",
      "          properties {\n",
      "            key: \"split_names\"\n",
      "            value: STRING\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"exclude_splits\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"[]\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"StatisticsGen\"\n",
      "upstream_nodes: \"schema_importer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-transform\"\n",
      ", pipeline_run_id='2022-05-11T11:18:03.288294')\n",
      "INFO:absl:Validating schema against the computed statistics for split train.\n",
      "INFO:absl:Validation complete for split train. Anomalies written to pipelines/penguin-transform/ExampleValidator/anomalies/5/Split-train.\n",
      "INFO:absl:Validating schema against the computed statistics for split eval.\n",
      "INFO:absl:Validation complete for split eval. Anomalies written to pipelines/penguin-transform/ExampleValidator/anomalies/5/Split-eval.\n",
      "INFO:absl:Cleaning up stateless execution info.\n",
      "INFO:absl:Execution 5 succeeded.\n",
      "INFO:absl:Cleaning up stateful execution info.\n",
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'anomalies': [Artifact(artifact: uri: \"pipelines/penguin-transform/ExampleValidator/anomalies/5\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:ExampleValidator:anomalies:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ExampleAnomalies\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      ")]}) for execution 5\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Component ExampleValidator is finished.\n",
      "INFO:absl:Component Trainer is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.trainer.component.Trainer\"\n",
      "    base_type: TRAIN\n",
      "  }\n",
      "  id: \"Trainer\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.Trainer\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"transform_graph\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Transform\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.Transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"TransformGraph\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"transform_graph\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Model\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"model_run\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelRun\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"eval_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 5\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"module_path\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"train_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 100\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "upstream_nodes: \"Transform\"\n",
      "downstream_nodes: \"Pusher\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Going to run a new execution 6\n",
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=6, input_dict={'examples': [Artifact(artifact: id: 1\n",
      "type_id: 15\n",
      "uri: \"pipelines/penguin-transform/CsvExampleGen/examples/1\"\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value {\n",
      "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"file_format\"\n",
      "  value {\n",
      "    string_value: \"tfrecords_gzip\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"input_fingerprint\"\n",
      "  value {\n",
      "    string_value: \"split:single_split,num_files:1,total_bytes:13161,xor_checksum:1652267862,sum_checksum:1652267862\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:CsvExampleGen:examples:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"payload_format\"\n",
      "  value {\n",
      "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"span\"\n",
      "  value {\n",
      "    int_value: 0\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267884894\n",
      "last_update_time_since_epoch: 1652267884894\n",
      ", artifact_type: id: 15\n",
      "name: \"Examples\"\n",
      "properties {\n",
      "  key: \"span\"\n",
      "  value: INT\n",
      "}\n",
      "properties {\n",
      "  key: \"split_names\"\n",
      "  value: STRING\n",
      "}\n",
      "properties {\n",
      "  key: \"version\"\n",
      "  value: INT\n",
      "}\n",
      "base_type: DATASET\n",
      ")], 'transform_graph': [Artifact(artifact: id: 9\n",
      "type_id: 22\n",
      "uri: \"pipelines/penguin-transform/Transform/transform_graph/4\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Transform:transform_graph:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267908233\n",
      "last_update_time_since_epoch: 1652267908233\n",
      ", artifact_type: id: 22\n",
      "name: \"TransformGraph\"\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'model': [Artifact(artifact: uri: \"pipelines/penguin-transform/Trainer/model/6\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Trainer:model:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Model\"\n",
      "base_type: MODEL\n",
      ")], 'model_run': [Artifact(artifact: uri: \"pipelines/penguin-transform/Trainer/model_run/6\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Trainer:model_run:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ModelRun\"\n",
      ")]}), exec_properties={'eval_args': '{\\n  \"num_steps\": 5\\n}', 'train_args': '{\\n  \"num_steps\": 100\\n}', 'custom_config': 'null', 'module_path': 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'}, execution_output_uri='pipelines/penguin-transform/Trainer/.system/executor_execution/6/executor_output.pb', stateful_working_dir='pipelines/penguin-transform/Trainer/.system/stateful_working_dir/2022-05-11T11:18:03.288294', tmp_dir='pipelines/penguin-transform/Trainer/.system/executor_execution/6/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.trainer.component.Trainer\"\n",
      "    base_type: TRAIN\n",
      "  }\n",
      "  id: \"Trainer\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.Trainer\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"examples\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"CsvExampleGen\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.CsvExampleGen\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Examples\"\n",
      "            base_type: DATASET\n",
      "          }\n",
      "        }\n",
      "        output_key: \"examples\"\n",
      "      }\n",
      "      min_count: 1\n",
      "    }\n",
      "  }\n",
      "  inputs {\n",
      "    key: \"transform_graph\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Transform\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.Transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"TransformGraph\"\n",
      "          }\n",
      "        }\n",
      "        output_key: \"transform_graph\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"Model\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  outputs {\n",
      "    key: \"model_run\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"ModelRun\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"eval_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 5\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"module_path\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"train_args\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"num_steps\\\": 100\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"CsvExampleGen\"\n",
      "upstream_nodes: \"Transform\"\n",
      "downstream_nodes: \"Pusher\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-transform\"\n",
      ", pipeline_run_id='2022-05-11T11:18:03.288294')\n",
      "INFO:absl:Train on the 'train' split when train_args.splits is not set.\n",
      "INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.\n",
      "INFO:absl:udf_utils.get_fn {'eval_args': '{\\n  \"num_steps\": 5\\n}', 'train_args': '{\\n  \"num_steps\": 100\\n}', 'custom_config': 'null', 'module_path': 'penguin_utils@pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'} 'run_fn'\n",
      "INFO:absl:Installing 'pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl' to a temporary directory.\n",
      "INFO:absl:Executing: ['/opt/conda/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmpul1u0rjs', 'pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl']\n",
      "E0511 11:18:28.371198963   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing ./pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:Successfully installed 'pipelines/penguin-transform/_wheels/tfx_user_code_Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9-py3-none-any.whl'.\n",
      "INFO:absl:Training model.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: tfx-user-code-Trainer\n",
      "Successfully installed tfx-user-code-Trainer-0.0+a5e9139bd7facf5026b5306a6aea534f89db0dea58ebe1bb1fb5ebb9df5fdea9\n",
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:struct2tensor is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_decision_forests is not available.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tensorflow_text is not available.\n",
      "INFO:absl:Feature body_mass_g has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature island has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature sex has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Feature species has a shape dim {\n",
      "  size: 1\n",
      "}\n",
      ". Setting to DenseTensor.\n",
      "INFO:absl:Model: \"model\"\n",
      "INFO:absl:__________________________________________________________________________________________________\n",
      "INFO:absl: Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "INFO:absl:==================================================================================================\n",
      "INFO:absl: culmen_length_mm (InputLayer)  [(None, 1)]          0           []                               \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: culmen_depth_mm (InputLayer)   [(None, 1)]          0           []                               \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: flipper_length_mm (InputLayer)  [(None, 1)]         0           []                               \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: body_mass_g (InputLayer)       [(None, 1)]          0           []                               \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: concatenate (Concatenate)      (None, 4)            0           ['culmen_length_mm[0][0]',       \n",
      "INFO:absl:                                                                  'culmen_depth_mm[0][0]',        \n",
      "INFO:absl:                                                                  'flipper_length_mm[0][0]',      \n",
      "INFO:absl:                                                                  'body_mass_g[0][0]']            \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: dense (Dense)                  (None, 8)            40          ['concatenate[0][0]']            \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: dense_1 (Dense)                (None, 8)            72          ['dense[0][0]']                  \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl: dense_2 (Dense)                (None, 3)            27          ['dense_1[0][0]']                \n",
      "INFO:absl:                                                                                                  \n",
      "INFO:absl:==================================================================================================\n",
      "INFO:absl:Total params: 139\n",
      "INFO:absl:Trainable params: 139\n",
      "INFO:absl:Non-trainable params: 0\n",
      "INFO:absl:__________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100/100 [==============================] - 1s 4ms/step - loss: 0.2782 - sparse_categorical_accuracy: 0.8905 - val_loss: 0.0218 - val_sparse_categorical_accuracy: 1.0000\n",
      "INFO:tensorflow:Assets written to: pipelines/penguin-transform/Trainer/model/6/Format-Serving/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: pipelines/penguin-transform/Trainer/model/6/Format-Serving/assets\n",
      "INFO:absl:Training complete. Model written to pipelines/penguin-transform/Trainer/model/6/Format-Serving. ModelRun written to pipelines/penguin-transform/Trainer/model_run/6\n",
      "INFO:absl:Cleaning up stateless execution info.\n",
      "INFO:absl:Execution 6 succeeded.\n",
      "INFO:absl:Cleaning up stateful execution info.\n",
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'model': [Artifact(artifact: uri: \"pipelines/penguin-transform/Trainer/model/6\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Trainer:model:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"Model\"\n",
      "base_type: MODEL\n",
      ")], 'model_run': [Artifact(artifact: uri: \"pipelines/penguin-transform/Trainer/model_run/6\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Trainer:model_run:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"ModelRun\"\n",
      ")]}) for execution 6\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Component Trainer is finished.\n",
      "INFO:absl:Component Pusher is running.\n",
      "INFO:absl:Running launcher for node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.pusher.component.Pusher\"\n",
      "    base_type: DEPLOY\n",
      "  }\n",
      "  id: \"Pusher\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.Pusher\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Trainer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.Trainer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"pushed_model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"PushedModel\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"push_destination\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"filesystem\\\": {\\n    \\\"base_directory\\\": \\\"serving_model/penguin-transform\\\"\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      "\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Going to run a new execution 7\n",
      "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=7, input_dict={'model': [Artifact(artifact: id: 12\n",
      "type_id: 26\n",
      "uri: \"pipelines/penguin-transform/Trainer/model/6\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Trainer:model:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      "state: LIVE\n",
      "create_time_since_epoch: 1652267916190\n",
      "last_update_time_since_epoch: 1652267916190\n",
      ", artifact_type: id: 26\n",
      "name: \"Model\"\n",
      "base_type: MODEL\n",
      ")]}, output_dict=defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: \"pipelines/penguin-transform/Pusher/pushed_model/7\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Pusher:pushed_model:0\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"PushedModel\"\n",
      "base_type: MODEL\n",
      ")]}), exec_properties={'custom_config': 'null', 'push_destination': '{\\n  \"filesystem\": {\\n    \"base_directory\": \"serving_model/penguin-transform\"\\n  }\\n}'}, execution_output_uri='pipelines/penguin-transform/Pusher/.system/executor_execution/7/executor_output.pb', stateful_working_dir='pipelines/penguin-transform/Pusher/.system/stateful_working_dir/2022-05-11T11:18:03.288294', tmp_dir='pipelines/penguin-transform/Pusher/.system/executor_execution/7/.temp/', pipeline_node=node_info {\n",
      "  type {\n",
      "    name: \"tfx.components.pusher.component.Pusher\"\n",
      "    base_type: DEPLOY\n",
      "  }\n",
      "  id: \"Pusher\"\n",
      "}\n",
      "contexts {\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"pipeline_run\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"2022-05-11T11:18:03.288294\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  contexts {\n",
      "    type {\n",
      "      name: \"node\"\n",
      "    }\n",
      "    name {\n",
      "      field_value {\n",
      "        string_value: \"penguin-transform.Pusher\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "inputs {\n",
      "  inputs {\n",
      "    key: \"model\"\n",
      "    value {\n",
      "      channels {\n",
      "        producer_node_query {\n",
      "          id: \"Trainer\"\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"pipeline_run\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"2022-05-11T11:18:03.288294\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        context_queries {\n",
      "          type {\n",
      "            name: \"node\"\n",
      "          }\n",
      "          name {\n",
      "            field_value {\n",
      "              string_value: \"penguin-transform.Trainer\"\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "        artifact_query {\n",
      "          type {\n",
      "            name: \"Model\"\n",
      "            base_type: MODEL\n",
      "          }\n",
      "        }\n",
      "        output_key: \"model\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "outputs {\n",
      "  outputs {\n",
      "    key: \"pushed_model\"\n",
      "    value {\n",
      "      artifact_spec {\n",
      "        type {\n",
      "          name: \"PushedModel\"\n",
      "          base_type: MODEL\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "parameters {\n",
      "  parameters {\n",
      "    key: \"custom_config\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"null\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "  parameters {\n",
      "    key: \"push_destination\"\n",
      "    value {\n",
      "      field_value {\n",
      "        string_value: \"{\\n  \\\"filesystem\\\": {\\n    \\\"base_directory\\\": \\\"serving_model/penguin-transform\\\"\\n  }\\n}\"\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "upstream_nodes: \"Trainer\"\n",
      "execution_options {\n",
      "  caching_options {\n",
      "  }\n",
      "}\n",
      ", pipeline_info=id: \"penguin-transform\"\n",
      ", pipeline_run_id='2022-05-11T11:18:03.288294')\n",
      "WARNING:absl:Pusher is going to push the model without validation. Consider using Evaluator or InfraValidator in your pipeline.\n",
      "INFO:absl:Model version: 1652267916\n",
      "INFO:absl:Model written to serving path serving_model/penguin-transform/1652267916.\n",
      "INFO:absl:Model pushed to pipelines/penguin-transform/Pusher/pushed_model/7.\n",
      "INFO:absl:Cleaning up stateless execution info.\n",
      "INFO:absl:Execution 7 succeeded.\n",
      "INFO:absl:Cleaning up stateful execution info.\n",
      "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: \"pipelines/penguin-transform/Pusher/pushed_model/7\"\n",
      "custom_properties {\n",
      "  key: \"name\"\n",
      "  value {\n",
      "    string_value: \"penguin-transform:2022-05-11T11:18:03.288294:Pusher:pushed_model:0\"\n",
      "  }\n",
      "}\n",
      "custom_properties {\n",
      "  key: \"tfx_version\"\n",
      "  value {\n",
      "    string_value: \"1.7.1\"\n",
      "  }\n",
      "}\n",
      ", artifact_type: name: \"PushedModel\"\n",
      "base_type: MODEL\n",
      ")]}) for execution 7\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Component Pusher is finished.\n"
     ]
    }
   ],
   "source": [
    "tfx.orchestration.LocalDagRunner().run(\n",
    "  _create_pipeline(\n",
    "      pipeline_name=PIPELINE_NAME,\n",
    "      pipeline_root=PIPELINE_ROOT,\n",
    "      data_root=DATA_ROOT,\n",
    "      schema_path=SCHEMA_PATH,\n",
    "      module_file=_module_file,\n",
    "      serving_model_dir=SERVING_MODEL_DIR,\n",
    "      metadata_path=METADATA_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ppERq0Mj6xvW"
   },
   "source": [
    "You should see \"INFO:absl:Component Pusher is finished.\" if the pipeline\n",
    "finished successfully.\n",
    "\n",
    "The pusher component pushes the trained model to the `SERVING_MODEL_DIR` which\n",
    "is the `serving_model/penguin-transform` directory if you did not change\n",
    "the variables in the previous steps. You can see the result from the file\n",
    "browser in the left-side panel in Colab, or using the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NTHROkqX6yHx",
    "outputId": "61188cc4-b346-4eb1-fe88-f20208651ed4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serving_model/penguin-transform\n",
      "serving_model/penguin-transform/1652267916\n",
      "serving_model/penguin-transform/1652267916/variables\n",
      "serving_model/penguin-transform/1652267916/variables/variables.index\n",
      "serving_model/penguin-transform/1652267916/variables/variables.data-00000-of-00001\n",
      "serving_model/penguin-transform/1652267916/saved_model.pb\n",
      "serving_model/penguin-transform/1652267916/assets\n",
      "serving_model/penguin-transform/1652267916/keras_metadata.pb\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "E0511 11:23:03.128278732   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n"
     ]
    }
   ],
   "source": [
    "# List files in created model directory.\n",
    "!find {SERVING_MODEL_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VTqM-WiZkPbt"
   },
   "source": [
    "You can also check the signature of the generated model using the\n",
    "[`saved_model_cli` tool](https://www.tensorflow.org/guide/saved_model#show_command)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YBfUzD_OkOq_",
    "outputId": "1f350b74-8a0e-4995-dc73-d23386267f99"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "E0511 11:26:56.976836031   13850 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The given SavedModel SignatureDef contains the following input(s):\n",
      "  inputs['examples'] tensor_info:\n",
      "      dtype: DT_STRING\n",
      "      shape: (-1)\n",
      "      name: serving_default_examples:0\n",
      "The given SavedModel SignatureDef contains the following output(s):\n",
      "  outputs['output_0'] tensor_info:\n",
      "      dtype: DT_FLOAT\n",
      "      shape: (-1, 3)\n",
      "      name: StatefulPartitionedCall_2:0\n",
      "Method name is: tensorflow/serving/predict\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --dir {SERVING_MODEL_DIR}/$(ls -1 {SERVING_MODEL_DIR} | sort -nr | head -1) --tag_set serve --signature_def serving_default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DkAxFs_QszoZ"
   },
   "source": [
    "Because you defined `serving_default` with our own `serve_tf_examples_fn`\n",
    "function, the signature shows that it takes a single string.\n",
    "This string is a serialized string of tf.Examples and will be parsed with the\n",
    "[tf.io.parse_example()](https://www.tensorflow.org/api_docs/python/tf/io/parse_example)\n",
    "function as you defined earlier (learn more about tf.Examples [here](https://www.tensorflow.org/tutorials/load_data/tfrecord)).\n",
    "\n",
    "You can load the exported model and try some inferences with a few examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z1Yw5yYdvqKf",
    "outputId": "ff428494-f9fd-4904-8d3b-ac13e9532129"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Inconsistent references when loading the checkpoint into this object graph. For example, in the saved checkpoint object, `model.layer.weight` and `model.layer_copy.weight` reference the same variable, while in the current object these are two different variables. The referenced variables are:(<keras.saving.saved_model.load.TensorFlowTransform>TransformFeaturesLayer object at 0x7fb965050090> and <keras.engine.input_layer.InputLayer object at 0x7fb9648798d0>).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Inconsistent references when loading the checkpoint into this object graph. For example, in the saved checkpoint object, `model.layer.weight` and `model.layer_copy.weight` reference the same variable, while in the current object these are two different variables. The referenced variables are:(<keras.saving.saved_model.load.TensorFlowTransform>TransformFeaturesLayer object at 0x7fb965050090> and <keras.engine.input_layer.InputLayer object at 0x7fb9648798d0>).\n"
     ]
    }
   ],
   "source": [
    "# Find a model with the latest timestamp.\n",
    "model_dirs = (item for item in os.scandir(SERVING_MODEL_DIR) if item.is_dir())\n",
    "model_path = max(model_dirs, key=lambda i: int(i.name)).path\n",
    "\n",
    "# TODO 3: Load a model saved via model.save()\n",
    "loaded_model = tf.keras.models.load_model(model_path)\n",
    "inference_fn = loaded_model.signatures['serving_default']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xrOHIvnIv0-4",
    "outputId": "5bf7542a-7878-46b3-aa84-005e9545e5ff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-6.449994  -5.3602533  2.3187149]]\n"
     ]
    }
   ],
   "source": [
    "# Prepare an example and run inference.\n",
    "features = {\n",
    "  'culmen_length_mm': tf.train.Feature(float_list=tf.train.FloatList(value=[49.9])),\n",
    "  'culmen_depth_mm': tf.train.Feature(float_list=tf.train.FloatList(value=[16.1])),\n",
    "  'flipper_length_mm': tf.train.Feature(int64_list=tf.train.Int64List(value=[213])),\n",
    "  'body_mass_g': tf.train.Feature(int64_list=tf.train.Int64List(value=[5400])),\n",
    "}\n",
    "example_proto = tf.train.Example(features=tf.train.Features(feature=features))\n",
    "examples = example_proto.SerializeToString()\n",
    "\n",
    "result = inference_fn(examples=tf.constant([examples]))\n",
    "print(result['output_0'].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cri3mTgZ0SQ2"
   },
   "source": [
    "The third element, which corresponds to 'Gentoo' species, is expected to be the\n",
    "largest among three."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "08R8qvweThRf"
   },
   "source": [
    "## Next steps\n",
    "\n",
    "If you want to learn more about Transform component, see\n",
    "[Transform Component guide](https://www.tensorflow.org/tfx/guide/transform).\n",
    "You can find more resources on https://www.tensorflow.org/tfx/tutorials.\n",
    "\n",
    "Please see\n",
    "[Understanding TFX Pipelines](https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines)\n",
    "to learn more about various concepts in TFX.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "DjUA6S30k52h"
   ],
   "name": "penguin_transform.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m91",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m91"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}