Specifying a code location in ProcessingStep should be optional

[system123]

Describe the bug
When defining a ProcessingStep using the Python SDK the pipeline compiler complains if the code= argument is not specified. However, the SDK documentation and code have code=None as a default (which is invalid) and the AWS documentation for processing steps states that the code parameter may be None if the code already exists in the container. In this case the ScriptProcessor already contains the code, and defines how to execute it through command= parameter.

To reproduce
Defining a processing step without a code argument will cause an error.

evaluation_step = ProcessingStep(
    name="EvaluateModel",
    processor=script_processor,
    inputs=[
        sagemaker.processing.ProcessingInput(
            source=train_step.properties.ModelArtifacts.S3ModelArtifacts,
            destination="/opt/ml/processing/model",
        ),
        sagemaker.processing.ProcessingInput(
            source=input_data_uri,
            destination="/opt/ml/processing/data",
        ),
    ],
    outputs=[
        sagemaker.processing.ProcessingOutput(
            output_name="evaluation",
            source="/opt/ml/processing/evaluation",
            destination="s3://my-bucket/models/"
        ),
    ],
    property_files=[evaluation_report],
)

Expected behavior
If a ScriptProcessor is used which is based upon a custom image, the command should just be run directly. No specific code needs to be uploaded or pulled into the container. The expected behaviour can be obtained using the SDK currently by pointing code to any dummy file on S3 or the local machine. This is then pushed to the container, but the command specified by the Script Processor is still executed.

Screenshots or logs

ValueError: code None url scheme b'' is not recognized. Please pass a file path or S3 url

[fjpa121197]

@system123 can you try creating the step this way:

preprocessing_job = ScriptProcessor(
        image_uri=processing_image_uri,
        command=["python3"],
        role=role_pipeline,
        instance_type=instance_type,
        instance_count=instance_count,
        sagemaker_session=pipeline_session,
    )

    step_args_preprocessing = preprocessing_job.run(
        code=os.path.join(BASE_DIR, "preprocess.py"),
        inputs=[
            ProcessingInput(...)
        ],
        outputs=[
            ProcessingOutput(...)
        ],
    )

    step_preprocessing = ProcessingStep(
        name="PreprocessingStep",
        step_args=step_args_preprocessing,
    )

hope that solves the issue

[qidewenwhen]

Hi @system123, thanks for reaching out!

I have received an internal customer ticket on the same topic and responded to that. Not sure if that was from you, so replying here as well.

The ScriptProcessor, as its name suggested, is for the use case of supplying custom script or code. That's why it has the code argument as required. In other words, ScriptProcessor is not for the use case of Bring Your Own Processor Container.

However, there is still another more general class to use, i.e. Processor,

sagemaker-python-sdk/src/sagemaker/processing.py

Line 70 in 1e679b4

class Processor(object):

for which, you don't need to supply the code. And instead, you'll need to supply an image uri, which can be your custom image. This class works with ProcessingStep as well. See the example below:

processor = Processor(
    image_uri=IMAGE_URI,
    role=ROLE,
    instance_count=1,
    instance_type=INSTANCE_TYPE,
    sagemaker_session=pipeline_session,
)

step_args = processor.run(
    inputs=processing_input, 
)

step = ProcessingStep(
            name="MyProcessingStep",
            step_args=step_args,
)

Hope this can help.

[mollyheamazon]

Hi @system123, please kindly let us know if this issue is resolved.

[system123]

@mollyheamazon sorry for the delay, yes the solution provided by @qidewenwhen solved the issue.