_repack_model.py does not work with S3 URI from processing job

[isunli]

Describe the bug
I noticed that in this change, the model_path logic in the _repack_model.py has changed from
model_path = os.path.join(data_directory, model_archive)
to
model_path = os.path.join(data_directory, model_archive.split("/")[-1]) . This change break the model artifacts reference from a processing job in the sagemaker pipeline. For example when I reference model data in the create model step in the sagemaker pipeline, it use:

model_data=step_processing.properties.ProcessingOutputConfig.Outputs[
            "model"
        ].S3Output.S3Uri

And model_archive.split("/")[-1] will take whatever in the last after string split to the model_path, this result in a path not exists error. For example:

data_directory = "/opt/ml/input/data/training"
model_archive = "s3://a/b"
model_path -> "/opt/ml/input/data/training/b" (not exist)

To reproduce
Code I use

model_path_param = ParameterString(
    name="model_path", default_value=f"{s3_prefix}/models/sklearn/processing")

step_processing = ProcessingStep(
    name="ProcessInputData",
    processor=sklearn_processor,
    outputs=[
        ProcessingOutput(
            output_name="model", source="/opt/ml/processing/model", destination=model_path_param
           ),
        ],
    )

preprocessing_model = SKLearnModel(
    name="CreatePreprocessingModel",
    model_data=step_processing.properties.ProcessingOutputConfig.Outputs[
        "model"
    ].S3Output.S3Uri,
    entry_point="preprocessing.py",
    source_dir=script_dir,
    code_location=f"{s3_prefix}/inference/sklearn/processing",
    role=role,
   )

When execute pipeline following error will be thrown:

ClientError: Failed to invoke sagemaker:CreateTrainingJob. Error Details: No S3 objects found under S3 URL "s3://.../models/sklearn/processing/model.tar.gz" given in input data source. Please ensure that the bucket exists in the selected region (us-east-1), that objects exist under that S3 prefix, and that the role...

Expected behavior
Model passed by s3 path can be correctly referred in the sagemaker pipeline

Screenshots or logs
See above

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 1.102.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): SKLearnProcessor
• Framework version: 1.0-1
• Python version: 3.8
• CPU or GPU: CPU
• Custom Docker image (Y/N): N

Additional context
Add any other context about the problem here.

[qidewenwhen]

Hi @isunli, thanks for reaching out!

As you've pointed out offline, you've made this change to bypass the issue, which breaks the dependent relation between processing step and the register model step:

preprocessing_model = SKLearnModel(
  name="CreatePreprocessingModel",
  model_data=model_path_param.default_value + "/model.tar.gz",  <<<<<<<<<<
...

Can you update it to use the Join method? It should persist the dependent relation and join the path and the model file name as well.

The updated code sample would be:

preprocessing_model = SKLearnModel(
  name="CreatePreprocessingModel",
  model_data=Join(
     on='/', 
     values=[
         step_processing.properties.ProcessingOutputConfig.Outputs["model"].S3Output.S3Uri,
         "/model.tar.gz"
     ]
  ),
...

Let me know if it works.

[isunli]

Sure, looks like a more reasonable solution

[qidewenwhen]

Here're some more context if you're interested.

As you've pointed out, this change changed the behavior in _repack_model.py

As indicated in the previous doc string in _repack_model.py before the change

model_archive (str): The name of the model TAR archive.

It was expecting a model TAR file name without the path. In other words, there was a split operation placed somewhere else (see here).
We moved the split into the _repact_model.py as without this change, if the model_data is a Parameter/Join... (a placeholder variable in pipeline compile time), the split will fail immediately during the pipeline generation/compile time. On the other hand, the _repact_model.py is executed during Pipeline execution/run time, and all its input values have been interpreted by the backend so we're safe to do the split there.

For this specific issue, as you've already figured out yourself, is caused due to model_data = step_processing.properties.ProcessingOutputConfig.Outputs["model"].S3Output.S3Uri - a path without file name is assigned to model_data.
As we need to join a Properties and a string file name, the Join method defined in the SDK should be used.

[qidewenwhen]

Closing this issue. Feel free to reopen if you have any other questions.