How to debug deployed model endpoint?

[plynch-chwy]

Please fill out the form below.

System Information

• Framework: conda_mxnet_p36 kernel
• CPU or GPU: Deployed model, cpu
• Are you using a custom image: No

Describe the problem

I was able to successfully (no errors at least) deploy a model using a sagemaker notebook instance with the conda_mxnet_p36 kernel. I slightly modified this example and re-worked it using a MovieLens dataset. Here is my entry point code.

Trying to test a prediction with the SDK returns a ModelError/500 response and instructions to view the cloud watch logs. All I get from the cloud watch logs though is this: ERROR in serving: 'AssertionError' object has no attribute 'message'. My predict code is below

What is the best way to debug this error? I'd also like to know the best approach for updating the entry point. Do I need to re-deploy a new model with an updated script or can I edit the entry point without re-deploying?

• Exact command to reproduce:

predictor = sagemaker.mxnet.MXNetPredictor('sagemaker_name')
predictor.serializer = None
data = json.dumps({'userId':6, 'movieId':356})
predictor.predict(data)

[laurenyu]

hi @plynch-chewy, is there a stacktrace in the logs you could copy/paste here?

[plynch-chwy]

Here is the stack trace from my notebook instance, but it doesn't give details on the ModelError:

---------------------------------------------------------------------------
ModelError                                Traceback (most recent call last)
<ipython-input-22-af5deff2d3fe> in <module>()
----> 1 predictor.predict(data)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data)
     84             request_args['Accept'] = self.accept
     85 
---> 86         response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
     87 
     88         response_body = response['Body']

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    312                     "%s() only accepts keyword arguments." % py_operation_name)
    313             # The "self" in this scope is referring to the BaseClient.
--> 314             return self._make_api_call(operation_name, kwargs)
    315 
    316         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    610             error_code = parsed_response.get("Error", {}).get("Code")
    611             error_class = self.exceptions.from_code(error_code)
--> 612             raise error_class(parsed_response, operation_name)
    613         else:
    614             return parsed_response

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "". See *cloudwatch link* in *account number* for more information.

And these are the logs from cloud watch, which don't give the full stack trace:

[laurenyu]

I had a feeling the error message you were seeing meant there was an erroneous error.message call, but was searching through the wrong repositories. Anyway, I found it and submitted a PR - aws/sagemaker-containers#105

Once that is released, we'll hopefully be able to at least see a proper error message. If you want to apply the change sooner and not wait for us to release, you can apply the changes yourself locally, and rebuild the Docker image yourself.

[laurenyu]

@plynch-chewy sorry for the delay in getting back to you - the bug fix for our error handling has been released to the MXNet containers now. Can you try creating a new endpoint and running prediction again?

[plynch-chwy]

I modified original code I had bugs on but had different errors that I was able to view the trace in cloud watch logs. Seems to be fixed for me, thanks for the quick support!