Poor error reporting of docker failures in local mode

[leopd]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Local mode with BYOC Tensorflow example
• Framework Version: n/a
• Python Version: 3
• CPU or GPU: CPU
• Python SDK Version: AttributeError: module 'sagemaker' has no attribute '__version__'
• Are you using a custom image: Yes, I'm trying to

Describe the problem

I'm working through the BYOC TF example https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/tensorflow_bring_your_own/tensorflow_bring_your_own.ipynb but modifying it and I see that the error message from running docker locally isn't getting reported properly.

Specifically, if you just run just this code in a notebook:

from sagemaker.estimator import Estimator

hyperparameters = {'train-steps': 100}

instance_type = 'local'

estimator = Estimator(role=role,
                      train_instance_count=1,
                      train_instance_type=instance_type,
                      image_name='tensorflow-cifar10-example:latest',
                      hyperparameters=hyperparameters)

estimator.fit('file:///tmp/cifar-10-data')

it fails (because I don't have a docker image called tensorflow-cifar10-example:latest). I get a big stack trace, but the key problem is that the actual error message has been swallowed. The exception says RuntimeError: Failed to run: ['docker-compose', '-f', '/private/var/folders/kw/8b59cw0s1c74qm8vc3xnzx50bj1cn5/T/tmpjl2m87y5/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1 which doesn't tell me what's wrong. But when I run the docker-compose command myself from the commandline I get:

$ docker-compose -f /private/var/folders/kw/8b59cw0s1c74qm8vc3xnzx50bj1cn5/T/tmpjl2m87y5/docker-compose.yaml up --build --abort-on-container-exit
Pulling algo-1-HNZFC (tensorflow-cifar10-example:latest)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y
Pulling algo-1-HNZFC (tensorflow-cifar10-example:latest)...
ERROR: pull access denied for tensorflow-cifar10-example, repository does not exist or may require 'docker login'

It looks like when docker-compose is invoked in local-mode, stdout gets printed, but stderr is swallowed. Because in the notebook I see what looks like the stdout part of docker-compose, but not stderr:

INFO:sagemaker:Creating training-job with name: tensorflow-cifar10-example-2018-09-26-01-13-11-189
Continue with the new image? [yN]Pulling algo-1-NX951 (tensorflow-cifar10-example:latest)...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

The key thing that's missing in the notebook output is the two error messages

ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.
ERROR: pull access denied for tensorflow-cifar10-example, repository does not exist or may require 'docker login'

Minimal repro / logs

See above.

[ChoiByungWook]

Hello @leopd,

Thanks for pointing this out to us. This is definitely a miss on our part and we will make sure that the proper error is reported properly. I am not too sure when a fix will be placed, but we will try our best.

[FedericoV]

This is still ongoing, just experienced it with the latest scikit-learn image. I caused this error by deleting the /tmp/ directory because I had filled it up with a few experiments.

[w601sxs]

Is this fixed? Got this with a custom sklearn image that I tried passing through the "image=" parameter in the SKLearn estimator from a sagemaker notebook instance.

I first initialize the estimator:

sklearn_model2 = SKLearnModel(model_data="file://model.tar.gz", role=get_execution_role(), entry_point="falcontransform.py", image= image)
where "image" is a custom image

and then do ..
predictor = sklearn_model.deploy(endpoint_name='local-test-endpoint',instance_type='local',initial_instance_count=1)

produces:

`Exception in thread Thread-5:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 614, in run
_stream_output(self.process)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 673, in _stream_output
raise RuntimeError("Process exited with code: %s" % exit_code)
RuntimeError: Process exited with code: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 619, in run
raise RuntimeError(msg)
RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmpdl9ws_eq/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1`

[cfregly]

@ChoiByungWook this extremely confusing - and happens quite a bit with my own code - as well as the SageMaker examples. would be really good to fix this.

my root cause is the following:

WARNING: Host is already in use by another container
ting tmpe4yrjkhf_algo-1-r5vno_1 ... error
ERROR: for tmpe4yrjkhf_algo-1-r5vno_1  Cannot start service algo-1-r5vno: driver failed programming external connectivity on endpoint tmpe4yrjkhf_algo-1-r5vno_1 (4f86fc022583bdc40d8a4c5887a8e957a92e17386dc1784fb1f4ba07ff65703a): Bind for 0.0.0.0:8080 failed: port is already allocated

ERROR: for algo-1-r5vno  Cannot start service algo-1-r5vno: driver failed programming external connectivity on endpoint tmpe4yrjkhf_algo-1-r5vno_1 (4f86fc022583bdc40d8a4c5887a8e957a92e17386dc1784fb1f4ba07ff65703a): Bind for 0.0.0.0:8080 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

[Reshma2412]

is this issue resolved? i am still facing this. I am using a custom scikit learn image.

Training with local mode happening properly. But during deployment getting this error message

**Exception in thread Thread-12:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 627, in run
_stream_output(self.process)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 687, in _stream_output
raise RuntimeError("Process exited with code: %s" % exit_code)
RuntimeError: Process exited with code: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 632, in run
raise RuntimeError(msg)
RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmp26kopljo/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1**

Please help e to resolve this error.

[sridharr24]

Is this resolved.. how to resolve? Or is it a docker permission issue. I am facing the same in scikit-learn on cloud with restrictions in environment. I am not sure what permissions do I need to add. Instance_type set to 'local':
Exception in thread Thread-5:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 627, in run
_stream_output(self.process)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 687, in _stream_output
raise RuntimeError("Process exited with code: %s" % exit_code)
RuntimeError: Process exited with code: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/local/image.py", line 632, in run
raise RuntimeError(msg)
RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmp8o5kx_rd/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1

[ChoiByungWook]

There can be multiple reasons for your local mode container failing with the error you are seeing above.

1. If you have more than one container running that is using the same ports, then I believe your local mode can fail with the corresponding error you are seeing above. This can happen if you attempt to use local mode in conjunction with another local mode run that didn't finish.

To list out currently running containers

docker ps

If you see another container that was started by local mode, most likely you will need to finish that training job or remove that local endpoint.

To stop a running container

docker stop <CONTAINER_ID>

To remove a running container

docker rm <CONTAINER_ID>

2. Local mode fails due to a problem with your container during runtime.

The best way to debug this is to list out your previously ran local mode container and check the logs.

To list out previously ran containers

docker ps -a

To check the log of a container

docker logs <CONTAINER_ID>

[sridharr24]

Thanks, I am using scikit-learn ezmdeploy local model deploy: ez = ezsmdeploy.Deploy(model = 'model.tar.gz', # if you intend to add models later, pass model as list, otherwise str script = 'modelscript_ensemble_sklearn.py', name = 'xyz', requirements = ['scikit-learn=='+sklearn.__version__,'numpy','joblib'], #or pass in the path to requirements.txt instance_type = 'local', bucket = mybucket, prefix = myprefix, wait = True) There are no local containers running in this instance of Sagemaker NB: [image: image.png] Thanks, Sri

…

On Sat, Mar 6, 2021 at 2:47 PM Dan ***@***.***> wrote: There can be multiple reasons for your local mode container failing with the error you are seeing above. 1. If you have more than one container running that is using the same ports, then I believe your local mode can fail with the corresponding error you are seeing above. This can happen if you attempt to use local mode in conjunction with another local mode run that didn't finish. To list out currently running containers <https://docs.docker.com/engine/reference/commandline/ps/> docker ps If you see another container that was started by local mode, most likely you will need to finish that training job or remove that local endpoint. To stop a running container <https://docs.docker.com/engine/reference/commandline/stop/> docker stop <CONTAINER_ID> To remove a running container <https://docs.docker.com/engine/reference/commandline/rm/> docker rm <CONTAINER_ID> 1. Local mode fails due to a problem with your container during runtime. The best way to debug this is to list out your previously ran local mode container and check the logs. To list out previously ran containers <https://docs.docker.com/engine/reference/commandline/ps/#options> docker ps -a To check the log of a container <https://docs.docker.com/engine/reference/commandline/logs/> docker logs <CONTAINER_ID> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#405 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7XA5SRH5XSWS4CZHXKVTTTCKIENANCNFSM4FXG5YAA> .

[ismaelc]

Obviously not a long-term solution, but restarting my Sagemaker instance fixed it.

Edit: Also I'm using a TF Estimator, so probably not the same issue here.

[edgBR]

Hi guys, same issue here:

This is my Dockerfile:

FROM python:3.7 AS build
COPY ./code/requirements.txt .
RUN python3 -m pip install --upgrade pip && pip install -r ./requirements.txt

FROM gcr.io/distroless/python3-debian10

COPY --from=build /usr/local/lib/python3.7/site-packages/  /usr/lib/python3.7/.
COPY . /opt/ml/
WORKDIR /opt/ml/code

ENV SAGEMAKER_PROGRAM app.py

I have in my EC2 instance docker and docker compose.

And I am doing the following to call the local mode:

from sagemaker.local import LocalSession
import boto3
from sagemaker.estimator import Estimator

my_session = boto3.session.Session(region_name=AWS_DEFAULT_REGION)
sagemaker_session = LocalSession(boto_session=my_session)
sagemaker_session.config = {'local': {'local_code': True}}

print("Start training")
local_estimator = Estimator(image_uri='local-docker-image',
                      role = execution_role,
                      sagemaker_session = sagemaker_session, 
                      instance_count=1,
                      hyperparameters=hyperparameters,
                      instance_type="local")

local_train = '../input/data/training/preprocessing.csv'
train_location = 'file://'+local_train

local_estimator.fit({'train':train_location}, logs=True)

This is the error I get:

Failed to run: ['docker-compose', '-f', '/tmp/tmppbuhykzi/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 2

[space4VV]

For me it was because of the running containers in local mode. Restarting kernels did not help but docker stop and then remove commands worked! Thanks @ChoiByungWook

[cceyda]

It would be also nice to have some progress reporting on when an image is being pulled(download progress). When I tried local mode for the first time I kept killing it because it looked like it was doing nothing(no output log message etc) after waiting for 3mins... at one point I just gave up and let it run which then finished downloading after an hour. I don't know if there is a docker command to check ongoing downloads but surely if sagemaker is doing the 'docker pull' command the least it can do is pipe that output so we wont have to wait on belief alone that it is going to work.