Error running a pipeline with a Processing Job using a LocalSession

[svpino]

Describe the bug
Trying to run a pipeline with a Processing Job using a LocalSession fails with the following error:

Pipeline step 'preprocess-data' FAILED. Failure message is: TypeError: create_processing_job() got multiple values for argument 

This is happening using sagemaker version `2.199.0.

To reproduce
A clear, step-by-step set of instructions to reproduce the bug.

processor = SKLearnProcessor(
    base_job_name="preprocess-data",
    framework_version="1.2-1",
    instance_type="ml.m5.xlarge",
    instance_count=1,
    role=role,
    sagemaker_session=local_sagemaker_session,
)

preprocessing_step = ProcessingStep(
    name="preprocess-data",
    step_args=processor.run(
        code="preprocessor.py",
        inputs=[
            ProcessingInput(source=dataset_location, destination="/opt/ml/processing/input"),
        ],
        outputs=[
             ...
        ],
    )
)

pipeline = Pipeline(
    name="sample-pipeline",
    parameters=[dataset_location],
    steps=[preprocessing_step],
    sagemaker_session=local_sagemaker_session,
)

pipeline.upsert(role_arn=role)

Expected behavior
The pipeline should run locally like it does in version 2.192.1.

Screenshots or logs
Here are the full logs when running the sample code:

Starting execution for pipeline sample-pipeline. Execution ID is b3417a61-042d-4174-84ec-7d39cd529451
Starting pipeline step: 'preprocess-data'
Pipeline step 'preprocess-data' FAILED. Failure message is: TypeError: create_processing_job() got multiple values for argument 'ProcessingJobName'
Pipeline execution b3417a61-042d-4174-84ec-7d39cd529451 FAILED because step 'preprocess-data' failed.

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.199.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A
• Framework version: N/A
• Python version: 3.9
• CPU or GPU: Apple M1
• Custom Docker image (Y/N): N

[coderfon]

Same issue here. Similar system information but Intel i7 1165G7 processor

[lorenzwalthert]

I think in remote execution, the base job name argument is ignored (#3971). I think if you don't supply one (and let the SDK take care of that), your code may run locally.

[svpino]

I think in remote execution, the base job name argument is ignored (#3971). I think if you don't supply one (and let the SDK take care of that), your code may run locally.

Tried that, but it doesn't fix the problem.

But even if that worked, it's not a solution because we need the same code to run locally and in SageMaker. We don't want to have to change the code depending on where we are running it.

[lorenzwalthert]

Another thing I noticed in comparison of your code snippet to my functioning pipelines is that I use a sagemaker.workflow.pipeline_context.PipelineSession in SKLearnProcessor() for both local and remote execution and a sagemaker.workflow.pipeline_context.LocalPipelineSession only goes into sagemaker.workflow.pipeline.Pipeline(). Strange this works, since it's not the way it should be apparently.

[muammerkaan]

Was there any resolution this bug?

[muammerkaan]

@svpino

[svpino]

Was there any resolution this bug?

No, I haven't heard of any solution. I had to downgrade the library to version 2.192.1 for it to work.

[muammerkaan]

thank you

[qidewenwhen]

Hi @svpino, thanks for reaching out!
It's interesting that I'm not able to reproduce this issue with the supplied code snippet, py3.9 and the latest SDK version (v2.203.1 as of writing time).

Could you try upgrading the SDK and see if it resolves the issue?
If no, could you help to provide us a notebook with the entire sample code to reproduce the issue as well as the error logs printed in the notebook?
Asking because when I used the sample code at the top, I got another error saying that source of ProcessingInput should not be a pipeline Parameter. Thus I made changes accordingly to the code snippet. In case I missed anything else, a complete notebook would help a lot on issue reproducing. Thanks!

...
        inputs=[
            ProcessingInput(source=dataset_location, destination="/opt/ml/processing/input"),
        ],
...
pipeline = Pipeline(
    name="sample-pipeline",
    parameters=[dataset_location],  # <<<<<<<<<<<<<<<<<<
    steps=[preprocessing_step],
    sagemaker_session=local_sagemaker_session,
)

[svpino]

Hey @qidewenwhen,

Thanks for taking a look! I upgraded the library to 2.203.0, and I still get the same issue.

I'm attaching a notebook that fails. Here is the output I see when I run it:

INFO:sagemaker.local.entities:Starting execution for pipeline pipeline. Execution ID is 447fd26e-72e8-4af0-93ee-d38072113b16
INFO:sagemaker.local.entities:Starting pipeline step: 'preprocess-data'
INFO:sagemaker.local.entities:Pipeline step 'preprocess-data' FAILED. Failure message is: TypeError: create_processing_job() got multiple values for argument 'ProcessingJobName'
INFO:sagemaker.local.entities:Pipeline execution 447fd26e-72e8-4af0-93ee-d38072113b16 FAILED because step 'preprocess-data' failed.

failing-notebook.ipynb.zip

[qidewenwhen]

Hi @svpino, thanks for the notebook! I can reproduce the issue with it.
It turns out it's the pipeline_definition_config causing the issue. I commented out the pipeline_definition_config and the local pipeline went well.

pipeline_definition_config = PipelineDefinitionConfig(use_custom_job_prefix=True)

pipeline = Pipeline(
    name="pipeline",
    parameters=[dataset_location],
    steps=[
        preprocessing_step,
    ],
    # pipeline_definition_config=pipeline_definition_config,  <<<<<<<<<<<<<<<<<<<
    sagemaker_session=local_pipeline_session,
)

Seems we're missing tests for PipelineDefinitionConfig in local mode.
I'm working on a fix and will update once the fix is ready. Meanwhile, you can bypass the issue via commenting out the pipeline_definition_config at this moment.