 archived/pyspark-etl-training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ff2d442",
   "metadata": {},
   "source": [
    "# Perform ETL and train a model using PySpark\n",
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a1828f9-efdc-4d12-a676-a2f3432e9ab0",
   "metadata": {},
   "source": [
    "To perform extract transform load (ETL) operations on multiple files, we recommend opening a Jupyter notebook within Amazon SageMaker Studio and using the `Glue PySpark and Ray` kernel. The kernel is connected to an AWS Glue Interactive Session. The session connects your notebook to a cluster that automatically scales up the storage and compute to meet your data processing needs. When you shut down the kernel, the session stops and you're no longer charged for the compute on the cluster.\n",
    "\n",
    "Within the notebook you can use Spark commands to join and transform your data. Writing Spark commands is both faster and easier than writing SQL queries. For example, you can use the join command to join two tables. Instead of writing a query that can sometimes take minutes to complete, you can join a table within seconds.\n",
    "\n",
    "To show the utility of using the PySpark kernel for your ETL and model training worklows, we're predicting the fare amount of the NYC taxi dataset. It imports data from 47 files across 2 different Amazon Simple Storage Service (Amazon S3) locations. Amazon S3 is an object storage service that you can use to save and access data and machine learning artifacts for your models. For more information about Amazon S3, see [What is Amazon S3?](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html).\n",
    "\n",
    "The notebook is not meant to be a comprehensive analysis. Instead, it's meant to be a proof of concept to help you quickly get started.\n",
    "\n",
    "__Prerequisites:__\n",
    "\n",
    "This tutorial assumes that you've in the us-east-1 AWS Region. It also assumes that you've provided the IAM role you're using to run the notebook with permissions to use Glue. For more information, see [Providing AWS Glue permissions\n",
    "](docs.aws.amazon.com/sagemaker/latest/dg/perform-etl-and-train-model-pyspark.html#providing-aws-glue-permissions)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dffc1f72-88d2-442d-97ee-0d1c4e095ffb",
   "metadata": {},
   "source": [
    "## Solution overview \n",
    "\n",
    "To perform ETL on the NYC taxi data and train a model, we do the following\n",
    "\n",
    "1. Start a Glue Session and load the SageMaker Python SDK\n",
    "2. Set up the utilities needed to work with AWS Glue.\n",
    "3. Load the data from the Amazon S3 into Spark dataframes.\n",
    "4. Verify that we've loaded the data successfully.\n",
    "5. Save a 20000 row sample of the Spark dataframe as a pandas dataframe.\n",
    "6. Create a correlation matrix as an example of the types of analyses we can perform.\n",
    "7. Split the Spark dataframe into training, validation, and test datasets.\n",
    "8. Write the datasets to Amazon S3 locations that can be accessed by an Amazon SageMaker training job.\n",
    "9. Use the training and validation datasets to train a model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e472c953-1625-49df-8df9-9529344783ab",
   "metadata": {},
   "source": [
    "### Start a Glue Session and load the SageMaker Python SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94172c75-f8a9-4590-a443-c872fb5c5d6e",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "%additional_python_modules sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "725bd4b6-82a0-4f02-95b9-261ce62c71b0",
   "metadata": {},
   "source": [
    "### Set up the utilities needed to work with AWS Glue\n",
    "\n",
    "We're importing `Join` to join our Spark dataframes.  `GlueContext` provides methods for transforming our dataframes. In the context of the notebook, it reads the data from the Amazon S3 locations and uses the Spark cluster to transform the data. `SparkContext` represents the connection to the Spark cluster. `GlueContext` uses `SparkContext` to transform the data. `getResolvedOptions` lets you resolve configuration options within the Glue interactive session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ea1c3a4-8881-48b0-8888-9319812750e7",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from awsglue.transforms import Join\n",
    "from awsglue.utils import getResolvedOptions\n",
    "from pyspark.context import SparkContext\n",
    "from awsglue.context import GlueContext\n",
    "from awsglue.job import Job\n",
    "\n",
    "glueContext = GlueContext(SparkContext.getOrCreate())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e03664e5-89a2-4296-ba83-3518df4a58f0",
   "metadata": {},
   "source": [
    "### Create the `df_ride_info` dataframe\n",
    "\n",
    "Create a single dataframe from all the ride_info Parquet files for 2019."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba577de7-9ffe-4bae-b4c0-b225181306d9",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_ride_info = glueContext.create_dynamic_frame_from_options(\n",
    "    connection_type=\"s3\",\n",
    "    format=\"parquet\",\n",
    "    connection_options={\n",
    "        \"paths\": [\n",
    "            \"s3://dsoaws/nyc-taxi-orig-cleaned-split-parquet-per-year-multiple-files/ride-info/year=2019/\"\n",
    "        ],\n",
    "        \"recurse\": True,\n",
    "    },\n",
    ").toDF()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04ce553-bf3d-4922-bbb1-4aa264447276",
   "metadata": {},
   "source": [
    "### Create the `df_ride_info` dataframe\n",
    "\n",
    "Create a single dataframe from all the ride_fare Parquet files for 2019."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6efc3d4a-81d7-40f5-bb62-cd206924a0c9",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_ride_fare = glueContext.create_dynamic_frame_from_options(\n",
    "    connection_type=\"s3\",\n",
    "    format=\"parquet\",\n",
    "    connection_options={\n",
    "        \"paths\": [\n",
    "            \"s3://dsoaws/nyc-taxi-orig-cleaned-split-parquet-per-year-multiple-files/ride-fare/year=2019/\"\n",
    "        ],\n",
    "        \"recurse\": True,\n",
    "    },\n",
    ").toDF()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c8664da-2105-4ada-b480-06d50c59e878",
   "metadata": {},
   "source": [
    "### Show the first five rows of `dr_ride_fare`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d63af3a3-358f-4c6e-97d4-97a1f1a552de",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_ride_fare.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "688a17e8-0c83-485d-a328-e89344a0e8bf",
   "metadata": {},
   "source": [
    "### Join df_ride_fare and df_ride_info on the `ride_id` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a3baab-44b0-416a-b12e-049a270af8bd",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_joined = df_ride_info.join(df_ride_fare, [\"ride_id\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "236c2efc-85f8-43f8-b6d3-7f0e61ccefb0",
   "metadata": {},
   "source": [
    "### Show the first five rows of the joined dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a456733-4533-4688-8174-368e50f4dd66",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_joined.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1396f6ee-c581-4274-baf8-243d38ec000b",
   "metadata": {},
   "source": [
    "### Show the data types of the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a52a903-f394-4d00-a216-6af8c2132d83",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_joined.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18bb75a2-eba5-4d06-8a26-f30e31776a02",
   "metadata": {},
   "source": [
    "### Count the number of rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6bcc15f-8d41-4def-ae49-edaef4105343",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_joined.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2daa67c-4b21-433a-b46e-eed518ba9ce7",
   "metadata": {},
   "source": [
    "### Drop duplicates if there are any"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d13d8d9-7eed-4efb-b972-601baf291842",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_no_dups = df_joined.dropDuplicates([\"ride_id\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "657e48dc-1f4a-4550-afe1-d9754e6d0e1e",
   "metadata": {},
   "source": [
    "### Count the number of rows after dropping the duplicates\n",
    "\n",
    "In this case, there were no duplicates in the original dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e3e82a3-e3db-4752-8bab-f42cbbae4928",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_no_dups.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae4c0fc4-7cb5-4b70-8430-965b5fe4506e",
   "metadata": {},
   "source": [
    "### Drop columns\n",
    "Time series data and categorical data is outside of the scope of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dc1d15f-53f6-404d-86fd-5a28f3792db8",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_cleaned = df_joined.drop(\n",
    "    \"pickup_at\", \"dropoff_at\", \"store_and_fwd_flag\", \"vendor_id\", \"payment_type\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081c81f9-f052-4ddb-b769-4d41b6138f6a",
   "metadata": {},
   "source": [
    "### Take a sample from the notebook and convert it to a pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48382726-c767-4b0e-9336-decbf8184938",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_sample = df_cleaned.sample(False, 0.1, seed=0).limit(20000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bf2f181-0096-4044-8210-7d9de299d966",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_sample.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8b2f670-c5f9-4a01-8d9f-6a29a3dae660",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_pandas = df_sample.toPandas()\n",
    "df_pandas.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "246c98e9-64bd-4644-a163-b86a943d6a09",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "print(\"Dataset shape: \", df_pandas.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5b2727c-de75-4cc0-94e9-d254e235d003",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_pandas.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d69b48b6-98c2-4851-9c7a-f24f092bae41",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_pandas.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34222bea-8864-4934-8c93-a71a7e72325b",
   "metadata": {},
   "source": [
    "### Create a correlation matrix of the features\n",
    "\n",
    "We're creating a correlation matrix to see which features are the most predictive. This is an example of an analysis that you can use for your own use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7f3e4f7-e04e-41e1-b94b-b32eb3bc3bbf",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.stat import Correlation\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd  # not sure how the kernel runs, but it looks like I have import pandas again after going back to the notebook after a while\n",
    "\n",
    "vector_col = \"corr_features\"\n",
    "assembler = VectorAssembler(inputCols=df_sample.columns, outputCol=vector_col)\n",
    "df_vector = assembler.transform(df_sample).select(vector_col)\n",
    "\n",
    "matrix = Correlation.corr(df_vector, vector_col).collect()[0][0]\n",
    "corr_matrix = matrix.toArray().tolist()\n",
    "corr_matrix_df = pd.DataFrame(data=corr_matrix, columns=df_sample.columns, index=df_sample.columns)\n",
    "\n",
    "plt.figure(figsize=(16, 10))\n",
    "sns.heatmap(\n",
    "    corr_matrix_df,\n",
    "    xticklabels=corr_matrix_df.columns.values,\n",
    "    yticklabels=corr_matrix_df.columns.values,\n",
    "    cmap=\"Greens\",\n",
    "    annot=True,\n",
    ")\n",
    "\n",
    "%matplot plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbde3b29-d37d-485a-a114-5313c5a702c7",
   "metadata": {},
   "source": [
    "### Split the dataset into train, validation, and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e207c64-2e22-468f-a0c7-948090bcfce2",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_train, df_val, df_test = df_cleaned.randomSplit([0.7, 0.15, 0.15])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01a4d181-e2f0-4743-ab35-dd1f68b0fd31",
   "metadata": {},
   "source": [
    "### Define the Amazon S3 locations that store the datasets\n",
    "\n",
    "If you're getting a module not found error, restart the kernel and run all the cells again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f16ea3a1-6d6d-4755-94ad-c743298bd130",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "# Define the S3 locations to store the datasets\n",
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "s3_bucket = sagemaker_session.default_bucket()\n",
    "train_data_prefix = \"sandbox/glue-demo/train\"\n",
    "validation_data_prefix = \"sandbox/glue-demo/validation\"\n",
    "test_data_prefix = \"sandbox/glue-demo/test\"\n",
    "region = boto3.Session().region_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8899a159-700c-403a-b4f5-a00c62b06e5a",
   "metadata": {},
   "source": [
    "### Write the files to the locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64d7ae48-6158-4273-8bb3-2f00abb1c20c",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_train.write.parquet(f\"s3://{s3_bucket}/{train_data_prefix}\", mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de3d1190-4717-4944-846d-0169c093cb90",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_val.write.parquet(f\"s3://{s3_bucket}/{validation_data_prefix}\", mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d18ef1c-fc2f-4e34-a692-4a6c48be7cba",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "df_test.write.parquet(f\"s3://{s3_bucket}/{test_data_prefix}\", mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73c947e4-b4a9-4cc4-aefe-755aa0a713c8",
   "metadata": {},
   "source": [
    "### Train a model\n",
    "\n",
    "The following code uses the `df_train` and `df_val` datasets to train an XGBoost model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a31b7742-93df-44c5-8674-b6355032c508",
   "metadata": {
    "vscode": {
     "languageId": "python_glue_session"
    }
   },
   "outputs": [],
   "source": [
    "from sagemaker import image_uris\n",
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "hyperparameters = {\n",
    "    \"max_depth\": \"5\",\n",
    "    \"eta\": \"0.2\",\n",
    "    \"gamma\": \"4\",\n",
    "    \"min_child_weight\": \"6\",\n",
    "    \"subsample\": \"0.7\",\n",
    "    \"objective\": \"reg:squarederror\",\n",
    "    \"num_round\": \"50\",\n",
    "}\n",
    "\n",
    "# Set an output path to save the trained model.\n",
    "prefix = \"sandbox/glue-demo\"\n",
    "output_path = f\"s3://{s3_bucket}/{prefix}/xgb-built-in-algo/output\"\n",
    "\n",
    "# The following line looks for the XGBoost image URI and builds an XGBoost container.\n",
    "# We use version 1.7-1 of the image URI, you can specify a version that you prefer.\n",
    "xgboost_container = sagemaker.image_uris.retrieve(\"xgboost\", region, \"1.7-1\")\n",
    "\n",
    "# Construct a SageMaker estimator that calls the xgboost-container\n",
    "estimator = sagemaker.estimator.Estimator(\n",
    "    image_uri=xgboost_container,\n",
    "    hyperparameters=hyperparameters,\n",
    "    role=sagemaker.get_execution_role(),\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.4xlarge\",\n",
    "    output_path=output_path,\n",
    ")\n",
    "\n",
    "content_type = \"application/x-parquet\"\n",
    "train_input = TrainingInput(f\"s3://{s3_bucket}/{prefix}/train/\", content_type=content_type)\n",
    "validation_input = TrainingInput(\n",
    "    f\"s3://{s3_bucket}/{prefix}/validation/\", content_type=content_type\n",
    ")\n",
    "\n",
    "# Run the XGBoost training job\n",
    "estimator.fit({\"train\": train_input, \"validation\": validation_input})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1b1d546-1c7e-48f5-9262-939289ada936",
   "metadata": {},
   "source": [
    "### Clean up\n",
    "\n",
    "To clean up, shut down the kernel. Shutting down the kernel, stops the Glue cluster. You won't be charged for any more compute other than what you used to run the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99668011",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "    \n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/use-cases|pyspark_etl_and_training|pyspark-etl-training.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Glue PySpark and Ray",
   "language": "python",
   "name": "glue_pyspark"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "Python_Glue_Session",
   "pygments_lexer": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}