Training job not saved to S3 despite providing S3 output location due to no model artifact saved under path /opt/ml/model

[yshen92]

Describe the bug
After training is done, model is not saved to S3 although an output S3 URI has been specified for the Estimator.
From CloudWatch log:
2022-11-04 03:58:58,042 sagemaker_tensorflow_container.training WARNING No model artifact is saved under path /opt/ml/model. Your training job will not save any model files to S3.

To reproduce
Tested on Amazon_JumpStart_Text_Classification 4. Finetune the pre-trained model on a custom dataset onwards

Expected behavior
After HPO, model is successfully saved to the output S3 URI specified

Screenshots or logs
hpo_text_classifier.log

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.116.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): Jumpstart model
• Framework version: *
• Python version: 3.10
• CPU or GPU: GPU
• Custom Docker image (Y/N): -

[Einengutenmorgen]

Same error. If I open the link in Trainingjobs, it shows a s3 location but the location is empty

[trungleduc]

@xchen909 @Einengutenmorgen hi, could you provide a full reproduction code? The notebook Amazon_JumpStart_Text_Classification does not work anymore with following error

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
Cell In[9], line 42
     32 model = Model(
     33     image_uri=deploy_image_uri,
     34     source_dir=deploy_source_uri,
   (...)
     39     name=endpoint_name,
     40 )
     41 # deploy the Model.
---> 42 base_model_predictor = model.deploy(
     43     initial_instance_count=1,
     44     instance_type=inference_instance_type,
     45     endpoint_name=endpoint_name,
     46 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model.py:1452, in Model.deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, async_inference_config, serverless_inference_config, volume_size, model_data_download_timeout, container_startup_health_check_timeout, inference_recommendation_id, explainer_config, accept_eula, **kwargs)
   1449     if self._base_name is not None:
   1450         self._base_name = "-".join((self._base_name, compiled_model_suffix))
-> 1452 self._create_sagemaker_model(
   1453     instance_type=instance_type,
   1454     accelerator_type=accelerator_type,
   1455     tags=tags,
   1456     serverless_inference_config=serverless_inference_config,
   1457 )
   1459 serverless_inference_config_dict = (
   1460     serverless_inference_config._to_request_dict() if is_serverless else None
   1461 )
   1462 production_variant = sagemaker.production_variant(
   1463     self.name,
   1464     instance_type,
   (...)
   1470     container_startup_health_check_timeout=container_startup_health_check_timeout,
   1471 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model.py:826, in Model._create_sagemaker_model(self, instance_type, accelerator_type, tags, serverless_inference_config)
    824         self.name = model_package.name
    825 else:
--> 826     container_def = self.prepare_container_def(
    827         instance_type,
    828         accelerator_type=accelerator_type,
    829         serverless_inference_config=serverless_inference_config,
    830     )
    832     if not isinstance(self.sagemaker_session, PipelineSession):
    833         # _base_name, model_name are not needed under PipelineSession.
    834         # the model_data may be Pipeline variable
    835         # which may break the _base_name generation
    836         model_uri = None

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model.py:630, in Model.prepare_container_def(self, instance_type, accelerator_type, serverless_inference_config)
    628 deploy_env = copy.deepcopy(self.env)
    629 if self.source_dir or self.dependencies or self.entry_point or self.git_config:
--> 630     self._upload_code(deploy_key_prefix, repack=self.is_repack())
    631     deploy_env.update(self._script_mode_env_vars())
    633 return sagemaker.container_def(
    634     self.image_uri,
    635     self.repacked_model_data or self.model_data,
   (...)
    638     accept_eula=getattr(self, "accept_eula", None),
    639 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model.py:729, in Model._upload_code(self, key_prefix, repack)
    713     self.uploaded_code = fw_utils.UploadedCode(
    714         s3_prefix=repacked_model_data,
    715         script_name=os.path.basename(self.entry_point),
    716     )
    718 LOGGER.info(
    719     "Repacking model artifact (%s), script artifact "
    720     "(%s), and dependencies (%s) "
   (...)
    726     repacked_model_data,
    727 )
--> 729 utils.repack_model(
    730     inference_script=self.entry_point,
    731     source_directory=self.source_dir,
    732     dependencies=self.dependencies,
    733     model_uri=self.model_data,
    734     repacked_model_uri=repacked_model_data,
    735     sagemaker_session=self.sagemaker_session,
    736     kms_key=self.model_kms_key,
    737 )
    739 self.repacked_model_data = repacked_model_data

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/utils.py:517, in repack_model(inference_script, source_directory, dependencies, model_uri, repacked_model_uri, sagemaker_session, kms_key)
    510 local_download_dir = (
    511     None
    512     if sagemaker_session.settings is None
    513     or sagemaker_session.settings.local_download_dir is None
    514     else sagemaker_session.settings.local_download_dir
    515 )
    516 with _tmpdir(directory=local_download_dir) as tmp:
--> 517     model_dir = _extract_model(model_uri, sagemaker_session, tmp)
    519     _create_or_update_code_dir(
    520         model_dir,
    521         inference_script,
   (...)
    525         tmp,
    526     )
    528     tmp_model_path = os.path.join(tmp, "temp-model.tar.gz")

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/utils.py:603, in _extract_model(model_uri, sagemaker_session, tmp)
    601 if model_uri.lower().startswith("s3://"):
    602     local_model_path = os.path.join(tmp, "tar_file")
--> 603     download_file_from_url(model_uri, local_model_path, sagemaker_session)
    604 else:
    605     local_model_path = model_uri.replace("file://", "")

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/utils.py:616, in download_file_from_url(url, dst, sagemaker_session)
    613 url = parse.urlparse(url)
    614 bucket, key = url.netloc, url.path.lstrip("/")
--> 616 download_file(bucket, key, dst, sagemaker_session)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/utils.py:634, in download_file(bucket_name, path, target, sagemaker_session)
    632 s3 = boto_session.resource("s3", region_name=sagemaker_session.boto_region_name)
    633 bucket = s3.Bucket(bucket_name)
--> 634 bucket.download_file(path, target)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/boto3/s3/inject.py:277, in bucket_download_file(self, Key, Filename, ExtraArgs, Callback, Config)
    243 def bucket_download_file(
    244     self, Key, Filename, ExtraArgs=None, Callback=None, Config=None
    245 ):
    246     """Download an S3 object to a file.
    247 
    248     Usage::
   (...)
    275         transfer.
    276     """
--> 277     return self.meta.client.download_file(
    278         Bucket=self.name,
    279         Key=Key,
    280         Filename=Filename,
    281         ExtraArgs=ExtraArgs,
    282         Callback=Callback,
    283         Config=Config,
    284     )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/boto3/s3/inject.py:190, in download_file(self, Bucket, Key, Filename, ExtraArgs, Callback, Config)
    155 """Download an S3 object to a file.
    156 
    157 Usage::
   (...)
    187     transfer.
    188 """
    189 with S3Transfer(self, Config) as transfer:
--> 190     return transfer.download_file(
    191         bucket=Bucket,
    192         key=Key,
    193         filename=Filename,
    194         extra_args=ExtraArgs,
    195         callback=Callback,
    196     )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/boto3/s3/transfer.py:326, in S3Transfer.download_file(self, bucket, key, filename, extra_args, callback)
    322 future = self._manager.download(
    323     bucket, key, filename, extra_args, subscribers
    324 )
    325 try:
--> 326     future.result()
    327 # This is for backwards compatibility where when retries are
    328 # exceeded we need to throw the same error from boto3 instead of
    329 # s3transfer's built in RetriesExceededError as current users are
    330 # catching the boto3 one instead of the s3transfer exception to do
    331 # their own retries.
    332 except S3TransferRetriesExceededError as e:

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/s3transfer/futures.py:103, in TransferFuture.result(self)
     98 def result(self):
     99     try:
    100         # Usually the result() method blocks until the transfer is done,
    101         # however if a KeyboardInterrupt is raised we want want to exit
    102         # out of this and propagate the exception.
--> 103         return self._coordinator.result()
    104     except KeyboardInterrupt as e:
    105         self.cancel()

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/s3transfer/futures.py:266, in TransferCoordinator.result(self)
    263 # Once done waiting, raise an exception if present or return the
    264 # final result.
    265 if self._exception:
--> 266     raise self._exception
    267 return self._result

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/s3transfer/tasks.py:269, in SubmissionTask._main(self, transfer_future, **kwargs)
    265     self._transfer_coordinator.set_status_to_running()
    267     # Call the submit method to start submitting tasks to execute the
    268     # transfer.
--> 269     self._submit(transfer_future=transfer_future, **kwargs)
    270 except BaseException as e:
    271     # If there was an exception raised during the submission of task
    272     # there is a chance that the final task that signals if a transfer
   (...)
    281 
    282     # Set the exception, that caused the process to fail.
    283     self._log_and_set_exception(e)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/s3transfer/download.py:354, in DownloadSubmissionTask._submit(self, client, config, osutil, request_executor, io_executor, transfer_future, bandwidth_limiter)
    325 """
    326 :param client: The client associated with the transfer manager
    327 
   (...)
    349     downloading streams
    350 """
    351 if transfer_future.meta.size is None:
    352     # If a size was not provided figure out the size for the
    353     # user.
--> 354     response = client.head_object(
    355         Bucket=transfer_future.meta.call_args.bucket,
    356         Key=transfer_future.meta.call_args.key,
    357         **transfer_future.meta.call_args.extra_args,
    358     )
    359     transfer_future.meta.provide_transfer_size(
    360         response['ContentLength']
    361     )
    363 download_output_manager = self._get_download_output_manager_cls(
    364     transfer_future, osutil
    365 )(osutil, self._transfer_coordinator, io_executor)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:535, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    531     raise TypeError(
    532         f"{py_operation_name}() only accepts keyword arguments."
    533     )
    534 # The "self" in this scope is referring to the BaseClient.
--> 535 return self._make_api_call(operation_name, kwargs)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:983, in BaseClient._make_api_call(self, operation_name, api_params)
    979     error_code = error_info.get("QueryErrorCode") or error_info.get(
    980         "Code"
    981     )
    982     error_class = self.exceptions.from_code(error_code)
--> 983     raise error_class(parsed_response, operation_name)
    984 else:
    985     return parsed_response

ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

[trungleduc]

Closing the issue, please feel free to re-open for more discussion.