Hyperparameters not provided when using a `.sh` script as Estimator entrypoint

[Guillem96]

Describe the bug
When setting a .sh script as an Estimator entrypoint I do not get the hyperparameters as CLI options.

To reproduce

# train.sh
echo "python train.py $@"  # For some reason $@ is empty

# run_train_job.py
from sagemaker.pytorch.estimator import PyTorch

input_path = ...
instance_type = "ml.m5.large"

estimator = PyTorch(
        entry_point="train.sh",
        source_dir=".",
        instance_type=instance_type,
        instance_count=1,
        framework_version="1.10",
        py_version="py38",
        hyperparameters={"test": "test-value"},
)
estimator.fit({"train": input_path}, job_name=f"jk-training-job-{job_suffix}")

Expected behavior
The train.sh files echos: python tain.py --test test-value. But actually the $@ is empty.

Additional context

Checking the cloudwatch logs, I see that sagemaker tries to run my script like so:

$ /bin/sh -c train.sh  --test test-value

If I replicate this command locally, I can easily reproduce the error.
I've realized that's the -c option that "empties" the $@ (and other special variables). Is there any particular reason to use that -c flag?

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.88.3
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
• Framework version: 1.10
• Python version: py38
• CPU or GPU: CPU
• Custom Docker image (Y/N): N

[jponf]

Right now we are dealing with this issue too using the following snipped.

# Handle user arguments
if [ -n "${SM_USER_ARGS}" ]; then
    user_args=`echo "
import os
print(
    \" \".join(
        map(str, eval(os.environ[\"SM_USER_ARGS\"]))
    )
)" | python3`
else
    user_args="${@}"
fi

It works but just looking at it feels bad ...

[Guillem96]

Thanks! Good workaround. 👌🏼
But I am still waiting for a fix or at least an explanation from SageMaker team

[davidbrochart]

According to @jponf's workaround, it seems that the hyper-parameters are provided to the entry point script through the SM_USER_ARGS environment variable.
@Guillem96 you seem to have a bash script that launches Python, but you could directly pass the path to your Python file, right? And from there read the SM_USER_ARGS environment variable more easily.
Would that solve your issue?

[sage-maker]

Is this issue still persisting? I have tried to replicate but unable to replicate.

3 years outdated

[sage-maker]

Official guidance is to move towards using the new ModelTrainer functionality as ModelTrainer can intake and pass through hyperparameters.

The Estimator's primary limitation is its forced conversion of all hyperparameters to strings (ref). This means that when developers try to pass complex data types like lists, dictionaries, or boolean values, the data structure is lost and requires complicated custom parsing logic in training scripts to reconstruct. For example, a simple list [1, 2, 3] becomes a string that needs careful handling to be usable again.

ModelTrainer ensures consistent behavior across Python and shell scripts, properly passing hyperparameters in both cases. It also provides better support for complex data types in hyperparameters.

We recommend transitioning to the ModelTrainer for improved reliability and consistency in handling hyperparameters. If you encounter any issues or have questions about using the ModelTrainer, please don't hesitate to reach out.

Thank you for your patience and for helping us improve the SageMaker Python SDK.