SageMaker Processing Step not working with ScriptProcessor()

[mccartni-aws]

Describe the bug
The error message is ValueError: either step_args or processor need to be given, but not both. This indicates that within the ProcessingStep of your SageMaker pipeline, you are not supposed to pass both the processor (which you've initialized as script_processor) and the step_args (which contains the run method from your script_processor) to the ProcessingStep constructor. Only one of these should be provided to define the processing step.

To reproduce

To reproduce this bug, you would create a SageMaker ScriptProcessor and use it to run a script with some arguments. Then, create a ProcessingStep passing in both step_args generated from the script_processor.run() method and the processor itself. The following pseudo-code demonstrates the essence of the steps that would lead to such an error:

script_processor = ScriptProcessor(...)
step_args = script_processor.run(code="script.py", arguments=[...])

finetune_step = ProcessingStep(
    name="FinetuneStep",
    processor=script_processor,  # This should not be provided along with step_args 
    step_args=step_args
)

To fix the issue, one should in theory remove the direct reference to the processor when you are providing step_args since step_args already contains the execution context needed for the processing job, however I am not including the processor. See code snippet for my implementation.

Expected behavior
The ProcesingStep should be created without fail, as only the step_args at explicilty passed into the ProcessingStep(). When the code runs correctly, you would expect the ProcessingStep to be created without raising a ValueError. The processing step would then use the provided arguments to execute the specified Python script (finetune.py) within the configured SageMaker processing environment. The job would process input data as specified, perform the finetuning operation, and output the results to the defined output path.

Screenshots or logs

System information
A description of your system. Please provide:

• SageMaker Python SDK version:
• Framework name (eg. PyTorch) or algorithm (eg. KMeans):
• Framework version:
• Python version:
• CPU or GPU:
• Custom Docker image (Y/N):

Additional context
Add any other context about the problem here.

[mccartni-aws]

This has been resolved. I was not passing in a pipeline session as the sagmaker session.