This one missing feature is ruining SageMaker Pipelines: Pipeline step reference + Str variable

[jonathanhillwebsite]

To be able to append a string to a pipeline step value
I understand the value of the pipeline steps are not created in advance of running a pipeline. But the ability to append or apply regex expressions to future pipeline step values is crucial in all my pipelines.

How would this feature be used? Please describe.

I'd be able to append a string to the values provided by the pipeline steps, such as selecting on of the files from the output of one of my processing jobs as opposed to creating a separate folder for each output (which doesn't work in the case of manifest files, another bug)

Describe alternatives you've considered
Uploading and downloading data manually in each processing or training step so I know where the locations are. but this is messy and I want to avoid doing this at the end of my scripts.

Additional context
It's by far the most debilitating missing feature there is and it makes SageMaker Pipelines an absolute nightmare to work with. This feature alone would also allow everyone to make any short-term patches while other issues are resolved - such as the passing of manifest file locations from a processing job output (e.g. that image above, the output path can only be a folder and manifest files require an exact s3 uri path to the manifest file)

[jonathanhillwebsite]

Actually, sorry the example above was not a good example because that example could be fixed easily with the existing helper workflow functions.

Adding more workflow functions would significantly help with adopting sagemaker pipelines though, treating the sagemaker steps parameter values as strings would make it so much easier to integrate and faster development

[navaj0]

Adding more workflow functions would significantly help with adopting sagemaker pipelines though, treating the sagemaker steps parameter values as strings would make it so much easier to integrate and faster development

This is a quite reasonable ask and we should design an overall improvement .

[ShwetaSingh801]

Hi @jonathanhillwebsite,

Thanks for using SageMaker and taking the time to suggest ways to improve SageMaker Python SDK. We have added your feature request it to our backlog of feature requests and may consider putting it into future SDK versions. I will go ahead and close the issue now, please let me know if you have any more feedback. Let me know if you have any other questions.

Best,
Shweta