rsv2-hudi-table-pii-reader-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "945395f1-4e95-4f97-aca9-cfdba9150201",
   "metadata": {},
   "source": [
    "# 0.Notebook Setup With Hudi Table PII Reader EMR Runtime Role\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd07d484-93bd-439d-b0f2-7380471e7383",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%configure -f\n",
    "{ \"conf\": {\n",
    "    \"spark.jars\":\"hdfs:///apps/hudi/lib/hudi-spark-bundle.jar\",\n",
    "    \"spark.serializer\":\"org.apache.spark.serializer.KryoSerializer\",\n",
    "    \"spark.sql.catalog.spark_catalog\": \"org.apache.spark.sql.hudi.catalog.HoodieCatalog\",\n",
    "    \"spark.sql.extensions\":\"org.apache.spark.sql.hudi.HoodieSparkSessionExtension,com.amazonaws.emr.recordserver.connector.spark.sql.RecordServerSQLExtension\",\n",
    "    \"spark.sql.catalog.spark_catalog.lf.managed\":\"true\"\n",
    "}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91b53231-51b5-457d-8603-7e47cbe196af",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "\n",
    "import os\n",
    "from datetime import datetime\n",
    "\n",
    "from pyspark.sql.functions import col,lit, current_timestamp,unix_timestamp, min, when, desc, split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a133f99b-7d39-428e-bb90-344d7262ea41",
   "metadata": {},
   "source": [
    "## 0.1 Global variables setup\n",
    "\n",
    "Go to 'CloudFormation'. Select the blog stack, and select 'Outputs' tab. Copy 'S3BucketName' value, and replace `<STACK-OUTPUTS-S3-BUCKET-NAME>` in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e27deb12-ad45-45b1-8d09-44d567089246",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "S3_BUCKET_NAME = <\"STACK-OUTPUTS-S3-BUCKET-NAME\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eee01985-82e5-4cc7-8f15-f51a4e8af093",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "VERSION = 1\n",
    "\n",
    "HUDI_CATALOG = \"spark_catalog\"\n",
    "TABLE_NAME = \"dl_tpc_customer\"\n",
    "HUDI_DATABASE = f\"rsv2_blog_hudi_db_{VERSION}\"\n",
    "HUDI_DATABASE_LOCATION = os.path.join(\n",
    "    \"s3://\",\n",
    "    S3_BUCKET_NAME,\n",
    "    HUDI_DATABASE\n",
    ")\n",
    "\n",
    "COW_TABLE_NAME_SQL = f\"rsv2_blog_hudi_cow_sql_{TABLE_NAME}_{VERSION}\"\n",
    "COW_TABLE_LOCATION_SQL = os.path.join(\n",
    "    HUDI_DATABASE_LOCATION,\n",
    "    COW_TABLE_NAME_SQL\n",
    ")\n",
    "\n",
    "MOR_TABLE_NAME_SQL = f\"rsv2_blog_hudi_mor_sql_{TABLE_NAME}_{VERSION}\"\n",
    "MOR_TABLE_LOCATION_SQL = os.path.join(\n",
    "    HUDI_DATABASE_LOCATION,\n",
    "    MOR_TABLE_NAME_SQL\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad80232-747a-4838-9bc8-a45e8352a10b",
   "metadata": {},
   "source": [
    "## 0.2 Spark variables setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ecb4308-92d2-4d75-8404-c81612b91684",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# sparkmagic SQL configs\n",
    "\n",
    "spark.conf.set('hudi_catalog', HUDI_CATALOG)\n",
    "spark.conf.set('hudi_db', HUDI_DATABASE)\n",
    "spark.conf.set('hudi_db_location', HUDI_DATABASE_LOCATION)\n",
    "\n",
    "spark.conf.set('cow_table_name_sql', COW_TABLE_NAME_SQL)\n",
    "spark.conf.set('cow_table_location_sql', COW_TABLE_LOCATION_SQL)\n",
    "\n",
    "spark.conf.set('mor_table_name_sql', MOR_TABLE_NAME_SQL)\n",
    "spark.conf.set('mor_table_location_sql', MOR_TABLE_LOCATION_SQL)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c098b812-e8d6-4e1e-8970-ec12b7822bfb",
   "metadata": {},
   "source": [
    "# 1. Hudi Table PII Reader Lake Formation Configuration\n",
    "\n",
    "***Please do the following steps in Blog \"Query Hudi tables having PII columns\" Section before runing the following cells***\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97ef305b-e864-47bc-b9ef-0beaccd885b3",
   "metadata": {},
   "source": [
    "# 2. Hudi CoW Table Queries\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8afc41c1-f518-414a-92f9-af5bd38549c2",
   "metadata": {},
   "source": [
    "## 2.1 CoW snapshot query\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b1c47cc-5c4e-4c07-b3fe-6f868ab05ab2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT \n",
    "    c_birth_country, \n",
    "    count(*) \n",
    "FROM ${hudi_catalog}.${hudi_db}.${cow_table_name_sql} \n",
    "GROUP BY c_birth_country;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260c9557-7e84-41c4-b2ed-9807bf7ea3f2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT\n",
    "    a.is_masked, count(*)\n",
    "FROM (\n",
    "    SELECT \n",
    "        CASE \n",
    "            WHEN c_first_name = 'MASKED' THEN 'MASKED'\n",
    "            WHEN c_first_name != 'MASKED' THEN 'NON_MASKED'\n",
    "        END AS is_masked       \n",
    "    FROM ${hudi_catalog}.${hudi_db}.${cow_table_name_sql}\n",
    "    WHERE c_birth_country = 'CHINA'\n",
    ") a\n",
    "GROUP BY a.is_masked;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfe0b407-772e-4fa7-a6d1-33c1042c0b0a",
   "metadata": {},
   "source": [
    "### 2.1.1 CoW snapshot query observations\n",
    "\n",
    "Comparing with the the same query run in previous **rsv2-hudi-db-creator-notebook:Section 2.2** when first inserting the data to CoW table:\n",
    "```\n",
    "HONG KONG   514\n",
    "CHINA\t    486\n",
    "```\n",
    "* Partition `HONG KONG` has no change.\n",
    "* Partition `INDIA` has been added by `INSERT INTO` DML in **Section 2.3**.\n",
    "* Partition `CHINA` has 3 records been added by `INSERT INTO` DML in **Section 2.3**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe03c96-0fee-44ba-bd95-959091827f07",
   "metadata": {},
   "source": [
    "## 2.2 CoW incremental query\n",
    "\n",
    "Note: Spark SQL using `hudi_table_changes` TVF mentioned on [Hudi website](https://hudi.apache.org/docs/sql_queries#incremental-query) is not supported yet. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "876a3f5b-0b74-4a8b-8030-b29bffac2be7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get Hudi commit time\n",
    "\n",
    "df = spark.sql(f\"\"\"\n",
    "SELECT DISTINCT _hoodie_commit_time FROM {HUDI_CATALOG}.{HUDI_DATABASE}.{COW_TABLE_NAME_SQL} ORDER BY _hoodie_commit_time\n",
    "\"\"\")\n",
    "\n",
    "commit_ts = [row.asDict()[\"_hoodie_commit_time\"] for row in df.collect()]\n",
    "commit_ts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b75b05aa-6cc1-4ff1-80c1-ee33ae4b2739",
   "metadata": {},
   "source": [
    "We set `begin.instanttime` to the most recent `commit_ts` to see the most recent update of Hudi table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d95544a-61ad-4625-ac1b-1fa21873d105",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "incremental_df = spark.sql(f\"\"\"\n",
    "SELECT * FROM {HUDI_CATALOG}.{HUDI_DATABASE}.{COW_TABLE_NAME_SQL} WHERE _hoodie_commit_time >= {commit_ts[-1]}\n",
    "\"\"\")\n",
    "\n",
    "incremental_df.createOrReplaceTempView(\"incremental_view\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9388415-3128-4794-ac48-04403cb3215e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT \n",
    "    c_birth_country, \n",
    "    count(*) \n",
    "FROM incremental_view\n",
    "GROUP BY c_birth_country;\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6cd1878-ac28-4856-9e58-d0e32545e996",
   "metadata": {},
   "source": [
    "### 2.2.1 CoW incremental query observations\n",
    "\n",
    "The incremental query shows latest state of Hudi table, where 3 records from Partition `CHINA` and 3 records from Partition `INDIA`. These 6 records are recnetly committed in **rsv2-hudi-db-creator-notebook:Section 2.3**. We cannot see 3 updated records from Partition `HONG KONG` because they have been deleted in **rsv2-hudi-db-creator-notebook:Secton 2.4**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b763d0f-d714-4a2d-92a5-7c89012f2b11",
   "metadata": {},
   "source": [
    "## 2.3 CoW time travel query\n",
    "\n",
    "Time travel back to the very beginning. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1755f921-d08d-4591-8bef-92f203eb48ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get Hudi commit time\n",
    "\n",
    "df = spark.sql(f\"\"\"\n",
    "SELECT DISTINCT _hoodie_commit_time FROM {HUDI_CATALOG}.{HUDI_DATABASE}.{COW_TABLE_NAME_SQL} ORDER BY _hoodie_commit_time\n",
    "\"\"\")\n",
    "\n",
    "commit_ts = [row.asDict()[\"_hoodie_commit_time\"] for row in df.collect()]\n",
    "commit_ts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f1176be-6f9b-4b66-b53f-73388fda5cf7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT\n",
    "    c_birth_country, COUNT(*) as count\n",
    "FROM ${hudi_catalog}.${hudi_db}.${cow_table_name_sql}\n",
    "WHERE _hoodie_commit_time IN\n",
    "(\n",
    "    SELECT DISTINCT _hoodie_commit_time FROM ${hudi_catalog}.${hudi_db}.${cow_table_name_sql} ORDER BY _hoodie_commit_time LIMIT 1 \n",
    ")\n",
    "GROUP BY c_birth_country"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c6f4863-684f-447c-84bf-67a58d0d13c1",
   "metadata": {},
   "source": [
    "### 2.3.1 CoW time travel query observations\n",
    "\n",
    "The count statistics for each partition match the one in **rsv2-hudi-db-creator-notebook::Section 2.2** when the data has been inserted to Hudi table at the first time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "963f9193-73ee-43a0-9535-27e0a142045e",
   "metadata": {},
   "source": [
    "# 3. Hudi MoR Table Queries\n",
    "\n",
    "In the context of MoR tables, Apache Hudi creates three logical views of the data, which are represented as three different table names in the Hive metastore\n",
    "\n",
    "* The first table (e.g. `rsv2_blog_hudi_db_{VERSION}.rsv2_blog_hudi_mor_sql_dl_tpc_customer_{VERSION}`) is the one with the name we specified when creating the table. This is the *base table* and it stores the original data.\n",
    "* The second table has the same name as the base table, but with a `_ro` suffix. This is the *Read Optimized (RO)* table. The `_ro` suffix stands for \"read-optimized\", and this table provides a view of the data that is optimized for read-heavy workloads. It shows the latest compacted data, which means it may not include the most recent updates or changes.\n",
    "* The third table also has the same name as the base table, but with a `_rt` suffix. This is the *Real-Time (RT)* table. The `_rt` suffix stands for \"real-time\", and this table provides a view of the data that includes both the compacted data and the most recent updates or changes. This allows for real-time querying of the data\n",
    "\n",
    "Let's grant the following permissions in Lake Formation\n",
    "\n",
    "* Grant `Select` and `Describe` permission to Hudi table PII reader role (e.g. ARN: `arn:aws:iam::<ACCOUNT_ID>:role/<STACK_NAME>-hudi-table-pii-role`) for the following tables:\n",
    "  * `rsv2_blog_hudi_db_{VERSION}.rsv2_blog_hudi_mor_sql_dl_tpc_customer_{VERSION}` \n",
    "  * `rsv2_blog_hudi_db_{VERSION}.rsv2_blog_hudi_mor_sql_dl_tpc_customer_{VERSION}_ro`\n",
    "  * `rsv2_blog_hudi_db_{VERSION}.rsv2_blog_hudi_mor_sql_dl_tpc_customer_{VERSION}_rt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81c72d5c-e355-41bb-8364-19ab366fef38",
   "metadata": {},
   "source": [
    "## 3.1 MoR query on read optimized (RO) table vs. real-time (RT) table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30409026-7253-47e6-8033-02c0bd1e4fc7",
   "metadata": {},
   "source": [
    "### 3.1.1 After first `INSERT` DML\n",
    "\n",
    "In **rsv2-hudi-db-creator-notebook:Section 3.3**, we update `rsv2_blog_hudi_db_{VERSION}.rsv2_blog_hudi_mor_sql_dl_tpc_customer_{VERSION}` table by setting `c_email_address` to `UNKNOWN` for a certain percentage of people born in Hong Kong. \n",
    "\n",
    "Let's compare the RO and RT tables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97858405-a295-4b26-a4d6-eac87e7fa3cf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT\n",
    "    a.email_label,\n",
    "    count(*)\n",
    "FROM (\n",
    "    SELECT\n",
    "        CASE\n",
    "            WHEN c_email_address = 'UNKNOWN' THEN 'UNKNOWN'\n",
    "            ELSE 'NOT_UNKNOWN'\n",
    "        END AS email_label\n",
    "    FROM ${hudi_catalog}.${hudi_db}.${mor_table_name_sql}_ro\n",
    "    WHERE c_birth_country = 'HONG KONG'\n",
    ") a\n",
    "GROUP BY a.email_label;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c79b9665-f0df-47b7-95c8-3e96129eaa4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT\n",
    "    a.email_label,\n",
    "    count(*)\n",
    "FROM (\n",
    "    SELECT\n",
    "        CASE\n",
    "            WHEN c_email_address = 'UNKNOWN' THEN 'UNKNOWN'\n",
    "            ELSE 'NOT_UNKNOWN'\n",
    "        END AS email_label\n",
    "    FROM ${hudi_catalog}.${hudi_db}.${mor_table_name_sql}_rt\n",
    "    WHERE c_birth_country = 'HONG KONG'\n",
    ") a\n",
    "GROUP BY a.email_label;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "486b6a27-c70c-4edb-930d-89ad4761ff6c",
   "metadata": {},
   "source": [
    "#### Observation\n",
    "\n",
    "The MoR RO table has not captured the `UPDATE` changes like set `c_email_address` to UNKOWN, while RT table has captured the changes where 66 records' `c_email_address` has been set to `UNKNOWN`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9b9714-de79-42f7-b763-606af1b486d4",
   "metadata": {},
   "source": [
    "### 3.1.2 After second `INSERT INTO` DML\n",
    "\n",
    "In **rsv2-hudi-db-creator-notebook:Section 3.4**, we append 3 records from INDIA and update 3 records' `c_first_name` to `MASK` from CHINA in `rsv2_blog_hudi_db_{VERSION}.rsv2_blog_hudi_mor_sql_dl_tpc_customer_{VERSION}` table. \n",
    "\n",
    "Let's compare RO and RT tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99dae5cd-7c25-426b-8be9-29cb2726d8bb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT *  \n",
    "FROM ${hudi_catalog}.${hudi_db}.${mor_table_name_sql}_ro\n",
    "WHERE \n",
    "    c_birth_country = 'INDIA' OR c_first_name = 'MASKED'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebff6248-00fd-427e-bd1d-b0cba83129db",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "\n",
    "SELECT *  \n",
    "FROM ${hudi_catalog}.${hudi_db}.${mor_table_name_sql}_rt\n",
    "WHERE \n",
    "    c_birth_country = 'INDIA' OR c_first_name = 'MASKED'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c840676d-e70f-46fb-b8c4-233ac30a8c0f",
   "metadata": {},
   "source": [
    "#### Observation\n",
    "\n",
    "* Both RO and RT tables have append records from INDIA\n",
    "* RT tables captures the updates from CHINA, but RO tables does not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04e20907-aece-4821-82dd-b49ba39e5da0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "PySpark (SparkMagic)",
   "language": "python",
   "name": "pysparkkernel__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-sparkmagic"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "pyspark",
   "pygments_lexer": "python3"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}