Parametrising spot instance use in a Sagemaker Pipeline is not possible

[lorenzwalthert]

Describe the feature you'd like

Outside a pipeline context, I used a variable to set use_spot_instances to either True or False.

sagemaker.sklearn.SKLearn(
        use_spot_instances=params["use_spot_instances"],
        max_run=3 * 3600, 
        max_wait=1800 * 48,  
    )

Where params["use_spot_instances"] is boolean. I note that setting max_wait without enabling spot training returns in an error.

ClientError: Failed to invoke sagemaker:CreateTrainingJob. Error Details: Invalid MaxWaitTimeInSeconds. It is only supported when EnableManagedSpotTraining is set to true

So I improved to

sagemaker.sklearn.SKLearn(
        use_spot_instances=params["use_spot_instances"],
        max_run=3 * 3600, 
        max_wait=1800 * 48 if params["use_spot_instances"] else None,  
    )

However, if use_spot_instances is a Pipeline variable instead of a literal boolean value (which is supported according to typing annotations), I have to set the argument max_wait as well if at run time, use_spot_instancse evaluates to True. If I don't want spot training, max_wait has to be None. Problem: I can't use native Python if else at built time (=when I define the pipeline) to only set the argument if params["use_spot_instances"], because the expression should only be evaluated at runtime.

The simple solution would be to simply ignore max_wait whenever use_spot_instances is set, and emit an info / warning instead of failing. Then, there would be no need for delayed ifelse evaluation logic similar to sagemaker.workflow.functions.Join and friends.

How would this feature be used? Please describe.

Allow choosing between spot instances or regular instances for training when starting a pipeline execution, not just at built time (i.e. when creating the pipeline).

Describe alternatives you've considered

• Always use spot or always use regular.
• Upsert the pipeline definition (and loose caching) to switch between the two.

[qidewenwhen]

Hi @lorenzwalthert, thanks for reaching out!

ClientError: Failed to invoke sagemaker:CreateTrainingJob. Error Details: Invalid MaxWaitTimeInSeconds. It is only supported when EnableManagedSpotTraining is set to true

This error was thrown from the TrainingJob service API rather than the SageMaker Python SDK, so I doubt we can change it to be a warning in the SDK level.

Allow choosing between spot instances or regular instances for training when starting a pipeline execution, not just at built time (i.e. when creating the pipeline).

For this I think our support engineer has provided a solution with LambdaStep, which is the best approach I can think about. Without an extra step, e.g. LambdaStep, currently the Pipeline is not able to resolve the dependency among pipeline variables (e.g. the max_wait's runtime value depends on the use_spot_instances's).

I'll keep this github issue open as a feature request in our queue and we may evaluate if we can improve on this.

[ShwetaSingh801]

Hi @lorenzwalthert,

Thanks for using SageMaker and taking the time to suggest ways to improve SageMaker Python SDK. We have added your feature request it to our backlog of feature requests and may consider putting it into future SDK versions. I will go ahead and close the issue now, please let me know if you have any more feedback. Let me know if you have any other questions.

Best,
Shweta

[lorenzwalthert]

I just overwrite the pipeline definition and loose caching but a lambda step seems too complicated.