Sagemaker training smdebug/core/state_store.py FileNotFoundError

[guptaanshul201989]

Hello,

I am trying to train a model using MXnet estimator. As soon as training starts I see following error when Sagemaker tries to upload checkpoints:

File "/opt/ml/code/translation.py", line 224, in __call__
return super(NMTModel, self).__call__(src_seq, tgt_seq, src_valid_length, tgt_valid_length)
File "/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py", line 756, in __call__
hook(self, args)
File "/usr/local/lib/python3.6/site-packages/smdebug/mxnet/hook.py", line 143, in forward_pre_hook
self._increment_step()
File "/usr/local/lib/python3.6/site-packages/smdebug/core/hook.py", line 511, in _increment_step
self._write_state()
File "/usr/local/lib/python3.6/site-packages/smdebug/core/hook.py", line 523, in _write_state
if self.state_store.is_checkpoint_updated():
File "/usr/local/lib/python3.6/site-packages/smdebug/core/state_store.py", line 112, in is_checkpoint_updated
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.6/site-packages/smdebug/core/state_store.py", line 112, in <listcomp>
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.6/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/metadata.json.sagemaker-uploading'
[2020-07-31 05:38:53.405 ip-10-2-236-41.ec2.internal:144 INFO utils.py:25] The end of training job file will not be written for jobs running under SageMaker.
2020-07-31 05:38:54,756 sagemaker-training-toolkit ERROR ExecuteUserScriptError:

Few details about the job:

1. I am using Mxnet estimator with distributed setting
2. Using 4 p3.16xlarge instances

[laurallu]

Hello,

I encountered a similar issue using Tensorflow estimator. I used pre-built container image with framework version = 2.2 and py_version = py37. It has trained 3 epochs and threw the error in the 4th epoch.

• 1 p3.2xlarge instance, no distributed training
• Enabled managed spot training. I haven't got such an error without spot training. But I did test the spot training code to make sure it could recover from prior checkpoints.

Traceback (most recent call last):
File "segmentation.py", line 549, in
verbose = 2)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 67, in _method_wrapper
return method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 868, in fit
callbacks.on_train_batch_begin(step)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 384, in on_train_batch_begin
self._call_batch_hook(ModeKeys.TRAIN, 'begin', batch, logs=logs)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py", line 299, in _call_batch_hook
batch_hook(batch, logs)
File "/usr/local/lib/python3.7/site-packages/smdebug/tensorflow/keras.py", line 734, in on_train_batch_begin
self._on_any_batch_begin(batch, ModeKeys.TRAIN, logs=logs)
File "/usr/local/lib/python3.7/site-packages/smdebug/tensorflow/keras.py", line 701, in _on_any_batch_begin
self._increment_step()
File "/usr/local/lib/python3.7/site-packages/smdebug/core/hook.py", line 533, in _increment_step
self._write_state()
File "/usr/local/lib/python3.7/site-packages/smdebug/core/hook.py", line 552, in _write_state
if self.state_store.is_checkpoint_updated():
File "/usr/local/lib/python3.7/site-packages/smdebug/core/state_store.py", line 99, in is_checkpoint_updated
timestamps = [os.path.getmtime(file) for file in checkpoint_files]
File "/usr/local/lib/python3.7/site-packages/smdebug/core/state_store.py", line 99, in
timestamps = [os.path.getmtime(file) for file in checkpoint_files]
File "/usr/local/lib/python3.7/genericpath.py", line 55, in getmtime
return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/unet-003.h5.sagemaker-uploading'

[guoren66]

Hi
same issue with laurallu's.

• 1 p3.2x instance, no spot requested.
• SageMaker Tensorflow 2.2 , py37

I1121 13:42:35.783539 140148447449344 basic_session_run_hooks.py:260] loss = 0.65470797 (0.405 sec)
Traceback (most recent call last):
File "0-deepfm-local-tf2.py", line 380, in
tf.compat.v1.app.run()
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "0-deepfm-local-tf2.py", line 356, in main
tf.estimator.train_and_evaluate(DeepFM, train_spec, eval_spec)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 472, in train_and_evaluate
return executor.run()
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
return self.run_local()
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 350, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1186, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1219, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1522, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 784, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1289, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1390, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1375, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1456, in run
run_metadata=run_metadata))
File "/usr/local/lib/python3.7/site-packages/smdebug/tensorflow/session.py", line 404, in after_run
self._increment_step()
File "/usr/local/lib/python3.7/site-packages/smdebug/core/hook.py", line 517, in _increment_step
self._write_state()
File "/usr/local/lib/python3.7/site-packages/smdebug/core/hook.py", line 529, in _write_state
if self.state_store.is_checkpoint_updated():
File "/usr/local/lib/python3.7/site-packages/smdebug/core/state_store.py", line 112, in is_checkpoint_updated
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.7/site-packages/smdebug/core/state_store.py", line 112, in
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.7/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/metadata.json.sagemaker-uploaded'