Sagemaker with Dockerfile

[celsofranssa]

I am trying to automatize AWS Sagemaker as a base for a large series of deep-learning experiments.

My current approach uses the Estimators as the code snippet shows:

import hydra
from omegaconf import omegaconf
from sagemaker.pytorch.estimator import PyTorch


@hydra.main(config_path="setting/", config_name="setting.yaml", version_base=None)
def run_on_sagemaker(params):
    role = "<your sagemaker arn role>"
    estimator = PyTorch(
        entry_point=params.sagemaker.entry_point,
        role=role,
        framework_version=params.sagemaker.framework_version,
        py_version=params.sagemaker.py_version,
        instance_type=params.sagemaker.instance_type,
        instance_count=params.sagemaker.instance_count,
        volume_size=params.sagemaker.volume_size,
        hyperparameters=omegaconf.OmegaConf.to_container(params, resolve=True, throw_on_missing=True)
    )
    estimator.fit()


if __name__ == '__main__':
    run_on_sagemaker()

However, when building the indicated Pytorch docker image, it seems that Sagemaker is not able to install all the dependencies correctly.

Therefore, would it be possible to explicitly declare (through a Dockerfile, for instance) how to create the Docker image?

[j-adamczyk]

You can use ECR to run anything in SageMaker with your own Docker image. You have to create images and push to ECR yourself, however.

[celsofranssa]

Awesome. Once the container is started, is it possible to run some kind of script before the Python entry point (bash commands, for example)?

[celsofranssa]

You can use ECR to run anything in SageMaker with your own Docker image. You have to create images and push to ECR yourself, however.

What should be in the Docker image? Just the configured environment (based on requirements.txt, for instance), or should it contain all the source code, training, prediction, and evaluation data?

[j-adamczyk]

Just the environment, the source code and data will be copied by SageMaker. Entry point will be run immediately, you need to run everything from Python e.g. with os.system() to call bash scripts. See "Bring Your Own Container (BYOC)" sections in SageMaker docs and examples for more.