SageMaker pipeline Caching Pipeline Steps not working for Transform step

[mouhannadali]

Describe the bug
Caching Pipeline Steps will not work for the Transform step if a ModelStep step is before

To reproduce
I have created a Sagemaker pipeline consisting of preprocessing step -> ModelStep -> transfer step and then -> postprocessing step
I have activated the Caching Pipeline and it is working for the preprocessing step but for the transfer step, it is not working. I believe this because inside the model step we need to call model.create() every time we execute the pipeline which means the input parameters for the transfer step are changing. however, I couldn't find any way to have the model step fixed (without model.create())

Expected behavior
Caching Pipeline Steps should be working for the Transform step if ModelStep is executed before

Screenshots or logs
If applicable, add screenshots or logs to help explain your problem.

System information
A description of your system. Please provide:

• SageMaker Python SDK version:
• Framework name (eg. PyTorch) or algorithm (eg. KMeans):
• Framework version:
• Python version:
• CPU or GPU:
• Custom Docker image (Y/N):

Additional context
Add any other context about the problem here.

[mouhannadali]

any update here

[jerrypeng7773]

@brockwade633

[brockwade633]

Hi @mouhannadali, thanks for the info. Would you mind sharing the python / notebook code that is producing this behavior? That will help us to reproduce the issue. Also, what version of the sagemaker python sdk are you using?

[brockwade633]

Closing as this has been abandoned for over a month. Please feel free to re-open

[francisco-camargo]

I have this exact issue and have had it across multiple SageMaker Pipelines.

Since I have a TransformStep after a ModelStep, the first I wanted to try was to run TransformStep without the need to proceed it with ModelStep, and see if that allowed me to cache the step. However I have not been able to get this to work, nor have a found an example online that tries to do this.

Thinking about how caching could be going wrong I thought it could be the pathing used to save the output artifacts. That is I have been using the output_path argument of Transformer(). This is an Optional argument so I thought to not use it, however this caused an error:

Here my code:

model_name = step_create_model.properties.ModelName

# Transform Step
transformer_step_name = 'Batch-Transformer'
transformer_s3_path = define_step_s3_path(default_bucket, base_job_prefix, step_name=transformer_step_name)
step_transform = define_transformer_step(
    step_name=transformer_step_name,
    instance_type=training_instance_type,
    instance_count=training_instance_count,
    data=s3_test_data_input_path,
    step_process=step_process,
    model_name=model_name,
    output_path=transformer_s3_path,
    sagemaker_session=sagemaker_session,
    cache_config=transformer_cache_config,
    )

def define_step_s3_path(default_bucket, base_job_prefix, step_name):
    return f's3://{default_bucket}/{base_job_prefix}/{step_name}'

def define_transformer_step(
    step_name,
    instance_type,
    instance_count,
    data,
    model_name,
    output_path,
    sagemaker_session,
    cache_config,
    ):

    transformer = Transformer(
        model_name=model_name,
        instance_type=instance_type,
        instance_count=instance_count,
        accept="text/csv",
        assemble_with="Line",
        output_path=output_path,
        sagemaker_session=sagemaker_session,
        )

    step_transform = TransformStep(
        name=step_name,
        transformer=transformer,
        inputs=TransformInput(
            data=data,
            input_filter="$[1:]",
            join_source="Input",
            output_filter='$',
            content_type="text/csv",
            split_type="Line",
            ),
        cache_config=cache_config,
        )

    return step_transform