Python SDK uses the wrong image for Pytorch

[tanguycdls]

In the latest version of Pytorch all the images changed of names from sagemaker-pytorch to pytorch-training that change has not been taken into account in the SDK and returns an error.

System Information

pip freeze

alabaster==0.7.10
anaconda-client==1.6.14
anaconda-project==0.8.2
asn1crypto==0.24.0
astroid==1.6.3
astropy==3.0.2
attrs==18.1.0
Automat==0.3.0
autovizwidget==0.12.9
awscli==1.16.256
Babel==2.5.3
backcall==0.1.0
backports.shutil-get-terminal-size==1.0.0
bcrypt==3.1.7
beautifulsoup4==4.6.0
bitarray==0.8.1
bkcharts==0.2
blaze==0.11.3
bleach==2.1.3
bokeh==1.0.4
boto==2.48.0
boto3==1.9.246
botocore==1.12.246
Bottleneck==1.2.1
cached-property==1.5.1
certifi==2019.6.16
cffi==1.11.5
characteristic==14.3.0
chardet==3.0.4
click==6.7
cloudpickle==0.5.3
clyent==1.2.2
colorama==0.3.9
contextlib2==0.5.5
cryptography==2.7
cycler==0.10.0
Cython==0.28.4
cytoolz==0.9.0.1
dask==0.17.5
datashape==0.5.4
decorator==4.3.0
defusedxml==0.6.0
distributed==1.21.8
docker==3.7.3
docker-compose==1.24.1
docker-pycreds==0.4.0
dockerpty==0.4.1
docopt==0.6.2
docutils==0.14
entrypoints==0.2.3
environment-kernels==1.1.1
et-xmlfile==1.0.1
fabric==2.5.0
fastcache==1.0.2
filelock==3.0.4
Flask==1.0.2
Flask-Cors==3.0.4
gevent==1.3.0
glob2==0.6
gmpy2==2.0.8
greenlet==0.4.13
h5py==2.8.0
hdijupyterutils==0.12.9
heapdict==1.0.0
html5lib==1.0.1
idna==2.6
imageio==2.3.0
imagesize==1.0.0
invoke==1.3.0
ipykernel==4.8.2
ipyparallel==6.2.2
ipython==6.4.0
ipython-genutils==0.2.0
ipywidgets==7.4.0
isort==4.3.4
itsdangerous==0.24
jdcal==1.4
jedi==0.12.0
Jinja2==2.10
jmespath==0.9.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
jupyterlab==0.32.1
jupyterlab-launcher==0.10.5
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
llvmlite==0.23.1
locket==0.2.0
lxml==4.2.1
MarkupSafe==1.0
matplotlib==3.0.3
mccabe==0.6.1
mistune==0.8.3
mkl-fft==1.0.0
mkl-random==1.0.1
mock==3.0.5
more-itertools==4.1.0
mpmath==1.0.0
msgpack==0.6.0
msgpack-python==0.5.6
multipledispatch==0.5.0
nb-conda==2.2.1
nb-conda-kernels==2.2.2
nbconvert==5.4.1
nbformat==4.4.0
networkx==2.1
nltk==3.3
nose==1.3.7
notebook==5.5.0
numba==0.38.0
numexpr==2.6.5
numpy==1.14.3
numpydoc==0.8.0
odo==0.5.1
olefile==0.45.1
openpyxl==2.5.3
packaging==17.1
pandas==0.24.2
pandocfilters==1.4.2
paramiko==2.6.0
parso==0.2.0
partd==0.3.8
path.py==11.0.1
pathlib2==2.3.2
patsy==0.5.0
pep8==1.7.1
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.2.0
pkginfo==1.4.2
plotly==4.0.0
pluggy==0.6.0
ply==3.11
prompt-toolkit==1.0.15
protobuf==3.6.1
protobuf3-to-dict==0.1.5
psutil==5.4.5
psycopg2==2.7.5
ptyprocess==0.5.2
py==1.5.3
py4j==0.10.7
pyasn1==0.4.7
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0.1
pyflakes==1.6.0
pygal==2.4.0
Pygments==2.2.0
pykerberos==1.2.1
pylint==1.8.4
PyNaCl==1.3.0
pyodbc==4.0.23
pyOpenSSL==18.0.0
pyparsing==2.2.0
PySocks==1.6.8
pyspark==2.3.2
pytest==3.5.1
pytest-arraydiff==0.2
pytest-astropy==0.3.0
pytest-doctestplus==0.1.3
pytest-openfiles==0.3.0
pytest-remotedata==0.2.1
python-dateutil==2.7.3
pytz==2018.4
PyWavelets==0.5.2
PyYAML==3.12
pyzmq==17.0.0
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.4.1
requests==2.20.0
requests-kerberos==0.12.0
retrying==1.3.3
rope==0.10.7
rsa==3.4.2
ruamel-yaml==0.15.35
s3fs==0.1.5
s3transfer==0.2.1
sagemaker==1.43.2
sagemaker-pyspark==1.2.5
scikit-image==0.13.1
scikit-learn==0.20.3
scipy==1.1.0
seaborn==0.8.1
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snowballstemmer==1.2.1
sortedcollections==0.6.1
sortedcontainers==1.5.10
sparkmagic==0.12.5
Sphinx==1.7.4
sphinxcontrib-websupport==1.0.1
spyder==3.2.8
SQLAlchemy==1.2.11
statsmodels==0.9.0
sympy==1.1.1
tables==3.4.3
tblib==1.3.2
terminado==0.8.1
testpath==0.3.1
texttable==0.9.1
toolz==0.9.0
tornado==5.0.2
traitlets==4.3.2
typing==3.6.4
unicodecsv==0.14.1
urllib3==1.23
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.56.0
Werkzeug==0.14.1
widgetsnbextension==3.4.2
wrapt==1.10.11
xlrd==1.1.0
XlsxWriter==1.0.4
xlwt==1.3.0
zict==0.1.3

Minimal repro / logs

If you ask for

from sagemaker.pytorch import PyTorch
estimator = PyTorch(entry_point='entrypoint.py',
                    role=role,
                    framework_version='1.2.0',
                    train_instance_count=1,
                    train_instance_type='local',
                )

it returns :

Command 'docker pull 520713654638.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-pytorch:1.2.0-cpu-py3' returned non-zero exit status 1.

If you run it manually the error is docker not found.

It should look for the docker:

763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.2.0-gpu-py3

[laurenyu]

can you verify that your environment is indeed using sagemaker==1.43.2 (picked the version from your pip freeze - any version 1.43.0+ should work)

I just tried this myself:

>>> import sagemaker
>>> sagemaker.__version__
'1.43.2'

>>> from sagemaker.pytorch import PyTorch
>>> pt = PyTorch(entry_point='foo.py', role='foo', framework_version='1.2.0', train_instance_count=1, train_instance_type='local')
>>> pt.train_image()
'763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:1.2.0-cpu-py3'

[elicutler]

I am also experiencing this issue. Interestingly it only happens when I specify train_instance_type='local' – if I instead specify train_instance_type='ml.p2.xlarge' it runs successfully on the remote GPU. For running locally I tested on two CPU notebook instances: ml.t2.medium and ml.t2.xlarge.

With train_instance_type='local' the same issue occurs whether I specify framework_version='1.1' or framework_version='1.2'.

[tanguycdls]

@laurenyu Yes indeed I ran that command in a Sagemaker notebook and the version there is still 1.42.8... Now that I updated sagemaker sdk it's fine.

[laurenyu]

@elicutler if you're experiencing an issue with PyTorch 1.1, then it sounds like it might be a different issue than not being able to find the PyTorch 1.2 image. Do you mind opening a new issue in this repository, and including the logs/stacktrace for what you're seeing?

[elicutler]

Thanks @laurenyu; I have opened a new issue here: #1105