Why would I use sagemaker-spark if I have to use anyways my Spark cluster?

[Manelmc]

In the line of "Building your own algorithm container", is it possible to use Spark code entirely (and distributively) on Sagemaker?. What I get from the documentation, is that I'm supposed to do ETL in my Spark Cluster. And then, when fitting the data to the model, use a sagemaker_pyspark algorithm that will create a Sagemaker training job. Moving the dataframe into S3 with protobuf format, to then train with a new Sagemaker instance cluster.

The question is: If I already have my dataframe loaded into my distributed cluster, why would I want to use Sagemaker? I might as well use Spark ML for it, which has a better algorithm support, and avoids creating an additional cluster. Maybe I got the whole thing wrong...

[Manelmc]

I got a response from Julien Simon on youtube about this topic, I paste it here:

Hi Manel, thank you for your question. Spark and Spark ML are great tools, more power to both :) Still, I see a few reasons why combining them with Sagemaker would make sense:

1. Decoupling ETL infrastructure from training infrastructure. They could have different sizing requirements (compute and/or storage) and you wouldn't want to oversize your Spark cluster just because one part of the overall process requires more capacity. Sure, you can resize an EMR cluster dynamically, but it's extra work and potentially troublesome if you have a lot of data. SageMaker will start and terminate instances automatically, which I find cleaner and simpler :)

2. In the same vein, imagine you need GPUs for training. It would be costly and sub-optimal to run your EMR cluster on GPU instances just for this.

3. Spark ML is great and has indeed a lot of algos. SageMaker is just getting started and we'll keep adding scalable implementions of state-of-the-art algos, so maybe one of them will actually catch your eye, such as DeepAR (released a couple of days ago) https://aws.amazon.com/blogs/machine-learning/now-available-in-amazon-sagemaker-deepar-algorithm-for-more-accurate-time-series-forecasting/

4. Deploying models in production on managed infrastructure. To me, this is the single most important feature in SageMaker: call a single API and deploy an endpoint.

You've certainly seen this repo, it has a number of examples: https://github.com/aws/sagemaker-spark

Hope this helps. Have fun :)

[andremoeller]

Very well said by Julien Simon. A couple of additional points to expand a bit on point 4:

1. SageMaker also helps to decouple model hosting and inference from the rest of the ML pipeline. If you just used Spark ML, you'd generally need to load your data into a DataFrame to make predictions by transforming your DataFrame after loading your model and your data in your cluster. While SageMaker Spark lets you transform a DataFrame using a SageMaker-hosted model, you don't need to use SageMaker Spark to make inferences after your model is trained and hosted -- you can get predictions using the InvokeEndpoint API, as SageMaker Spark does internally.

2. Latency for small-batch predictions. It can be difficult to achieve low-latency predictions with Spark ML models (though some projects like MLeap make this achievable). Spark is really well-suited for inferences on large DataFrames, but due to various overhead costs Spark incurs, not for small batches of data. You can get real-time predictions with models hosted in SageMaker.