debuggerHook is not saving tensors in s3

[tiru1930]

Describe the bug

Exception during rule evaluation: Customer Error: No debugging data was saved by the training job. Check that the debugger hook was configured correctly before starting the training job. Exception: Training job has ended. All the collection files could not be loaded | Exception during rule evaluation: Customer Error: No debugging data was saved by the training job. Check that the debugger hook was configured correctly before starting the training job. Exception: Training job has ended. All the collection files could not be loaded

To reproduce
Train FrameWork Xgboost with debugger hook as below

from sagemaker.xgboost import XGBoost
from sagemaker.debugger import rule_configs, Rule, DebuggerHookConfig, CollectionConfig

hyperparams = {"max_depth":5,
               "subsample":0.8,
               "num_round":600,
               "eta":0.2,
               "gamma":4,
               "min_child_weight":6,
               "silent":0,
               "objective":'multi:softmax',
               "num_class":len(le.classes_),
               "smdebug_path":f"s3://{bucket}/{prefix}/debug",
               "smdebug_collections":"metrics,feature_importance"
              }
save_interval = 5

entry_point_script = "xgboost_dest_prediction.py"

trial = Trial.create(trial_name="framework-mode-trial-{}".format(time.strftime("%Y-%m-%d-%H-%M-%S", time.gmtime())), 
                     experiment_name=destination_prediction_experiment.experiment_name,
                     sagemaker_boto_client=boto3.client('sagemaker'))

framework_xgb = XGBoost(
                      entry_point=entry_point_script,
                      role=sagemaker.get_execution_role(),
                      framework_version='0.90-2',
                      py_version="py3",
                      hyperparameters=hyperparams,
                      instance_count=1, 
                      instance_type='ml.m4.xlarge',
                      output_path='s3://{}/{}/output'.format(bucket, prefix),
                      base_job_name="demo-xgboost-destination-prediction",
                      sagemaker_session=sm_sess,
#                       rules=debug_rules,
                      use_spot_instances = True,
                      max_run = 3600,
                      max_wait = 3600,
                      input_mode = 'File',
                      debugger_hook_config=DebuggerHookConfig(
                            s3_output_path=f"s3://{bucket}/{prefix}/debug",  # Required
                            collection_configs=[
                                CollectionConfig(
                                    name="metrics",
                                    parameters={
                                        "save_interval": str(save_interval)
                                    }
                                )
                            ],
                        ),

                      rules=[
                            Rule.sagemaker(
                                rule_configs.loss_not_decreasing(),
                                rule_parameters={
                                    "collection_names": "metrics",
                                    "num_steps": str(save_interval * 2),
                                },
                            ),
                        ],

                    )

framework_xgb.fit({'train': s3_input_train,
                   'validation': s3_input_validation}, 
                  experiment_config={
                      "ExperimentName": destination_prediction_experiment.experiment_name, 
                      "TrialName": trial.trial_name,
                      "TrialComponentDisplayName": "Training",
                  })

Expected behavior
I should get tensors saved in s3

Screenshots or logs

[{'RuleConfigurationName': 'LossNotDecreasing',
  'RuleEvaluationJobArn': 'arn:aws:sagemaker:us-west-2:990360540682:processing-job/demo-xgboost-destination-p-lossnotdecreasing-abb2296f',
  'RuleEvaluationStatus': 'Error',
  'StatusDetails': 'ClientError: No debugging data was saved by the training job. Check that the debugger hook was configured correctly before starting the training job. Exception: Training job has ended. All the collection files could not be loaded\nTraceback (most recent call last):\n  File "evaluate.py", line 112, in _create_trials\n    range_steps=(self.start_step, self.end_step))\n  File "/usr/local/lib/python3.7/site-packages/smdebug/trials/utils.py", line 20, in create_trial\n    return LocalTrial(name=name, dirname=path, **kwargs)\n  File "/usr/local/lib/python3.7/site-packages/smdebug/trials/local_trial.py", line 36, in __init__\n    self._load_collections()\n  File "/usr/local/lib/python3.7/site-packages/smdebug/trials/trial.py", line 168, in _load_collections\n    _wait_for_collection_files(1)  # wait for the first collection file\n  File "/usr/local/lib/python3.7/site-packages/smdebug/trials/trial.py", line 165, in _wait_for_collection_files\n    raise MissingCollectionFiles\nsmdebug.exceptions.MissingCollectionFiles: Trainin',
  'LastModifiedTime': datetime.datetime(2020, 9, 18, 11, 6, 27, 290000, tzinfo=tzlocal())}]

System information
SageMaker Python SDK version: 2.6
Framework name (eg. PyTorch) or algorithm (eg. KMeans): xgboost frame work
Framework version: 0.90-2
Python version: 3.8
CPU or GPU: CPU
Custom Docker image (Y/N): N

[icywang86rui]

@tiru1930 Sorry that you run into this problem.

• Is the Iam role you used has the proper permission setup to access the debug S3 bucket s3://{bucket}/{prefix}/debug
• Could you show me the entire training job if possible?
• How long does the training job run? Did it successfully generated a model?

[tiru1930]

Is the Iam role you used has the proper permission setup to access the debug S3 bucket s3://{bucket}/{prefix}/debug
Yes,
Could you show me the entire training job if possible?
do you mean logs?

How long does the training job run? Did it successfully generated a model?
it ran for few mins, and yes, it generated model and stored in s3, i was able to deploy

[icywang86rui]

Could you show me the entire training job if possible?
do you mean logs?

Yes. sorry for the delay. are you still experiencing this problem?

[craigbosco]

Hi, I'm having this issue as well. The model will train successfully and I can deploy it to an endpoint, but the "training_job_end.ts" file is empty.

Here's the estimator object I'm using:

xgb = sagemaker.estimator.Estimator(
    container,
    role,
    instance_count=1,
    instance_type='ml.m4.xlarge',
    output_path=s3_xgb_output_location,
    sagemaker_session=sagemaker_session,
    hyperparameters=hyperparameters,
    debugger_hook_config=DebuggerHookConfig(
        s3_output_path=s3_xgb_output_location,
        collection_configs=[
            CollectionConfig(name="metrics", parameters={"save_interval": str(save_interval)}),
            CollectionConfig(
                name="feature_importance", parameters={"save_interval": str(save_interval)}
            ),
            CollectionConfig(name="full_shap", parameters={"save_interval": str(save_interval)}),
            CollectionConfig(name="average_shap", parameters={"save_interval": str(save_interval)}),
        ],
    ),
    rules=[
        Rule.sagemaker(
            rule_configs.loss_not_decreasing(),
            rule_parameters={
                "collection_names": "metrics",
                "num_steps": str(save_interval * 2),
            },
        ),
    ],                          
)

Then, I try to get the smdebug trial artifacts:

s3_output_path = xgb.latest_job_debugger_artifacts_path()
trial = smd.create_trial(s3_output_path)

Exception: MissingCollectionFiles: Training job has ended. All the collection files could not be loaded