SageMaker endpoint and Numpy Arrays

[NRauschmayr]

When hosting a Gluon model (MXNet 1.3.0, Python 2, SageMaker default container), the endpoint does not accept Numpy arrays, so the following code crashes:

predictor = mxnet_estimator.deploy(instance_type='local_gpu', initial_instance_count=1)
data = np.zeros((1,10,227,227))
print (predictor.predict(data))

In my entry point script I only defined model_fn for the inference:

def model_fn(model_dir):
    model = AutoEncoder()
    model.load_parameters("%s/model.params" %model_dir, ctx=ctx)
    return model

I get the following stacktrace:

algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in wsgi_app
algo-1-TZFX7_1_1307f2d8e109 |     response = self.full_dispatch_request()
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in full_dispatch_request
algo-1-TZFX7_1_1307f2d8e109 |     rv = self.handle_user_exception(e)
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1718, in handle_user_exception
algo-1-TZFX7_1_1307f2d8e109 |     reraise(exc_type, exc_value, tb)
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in full_dispatch_request
algo-1-TZFX7_1_1307f2d8e109 |     rv = self.dispatch_request()
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in dispatch_request
algo-1-TZFX7_1_1307f2d8e109 |     return self.view_functions[rule.endpoint](**req.view_args)
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/sagemaker_containers/_transformer.py", line 171, in transform
algo-1-TZFX7_1_1307f2d8e109 |     result = self._transform_fn(self._model, request.content, request.content_type, request.accept)
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/sagemaker_containers/_transformer.py", line 201, in _default_transform_fn
algo-1-TZFX7_1_1307f2d8e109 |     prediction = self._predict_fn(data, model)
algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
algo-1-TZFX7_1_1307f2d8e109 |     six.reraise(error_class, error_class(e), sys.exc_info()[2])
algo-1-TZFX7_1_1307f2d8e109 | TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
algo-1-TZFX7_1_1307f2d8e109 | 172.21.0.1 - - [13/Dec/2018:05:05:06 +0000] "POST /invocations HTTP/1.1" 500 291 "-" "-"
  File "run.py", line 34, in <module>
    print (predictor.predict(data))
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sagemaker/predictor.py", line 79, in predict
    return self._handle_response(response)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sagemaker/predictor.py", line 85, in _handle_response
    return self.deserializer(response_body, response['ContentType'])
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sagemaker/predictor.py", line 289, in __call__
    return json.load(codecs.getreader('utf-8')(stream))
  File "/home/ubuntu/anaconda3/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/home/ubuntu/anaconda3/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/ubuntu/anaconda3/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ubuntu/anaconda3/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I debugged the issue by running SageMaker locally and I got it working by setting the content_type:

data = np.zeros((1,10,227,227))
predictor.accept = 'application/x-npy'
predictor.content_type = 'application/x-npy'
predictor.deserializer =  numpy_deserializer
predictor.serializer =  npy_serializer

print (predictor.predict(data))

and in the entry point script I added transform_fn:

def transform_fn(model, data, content_type, accept):

    tmp = np.load(StringIO(data))
    mx_nd_array = mx.nd.array(tmp)
    mx_nd_array = mx_nd_array.as_in_context(ctx)
    output = model(mx_nd_array)
    np_array = output.asnumpy()
    np.save("output", np_array)
    f = open("output.npy")
    return f.read()

Is there an easier workaround to get this working? I was told SageMaker should accept Numpy arrays, and setting manually content types etc should actually not be done.

[ChoiByungWook]

Hello @NRauschmayr,

Our MXNet containers are public and can be found here:
https://github.com/aws/sagemaker-mxnet-container

As of now, by default our MXNet containers support only JSON for Gluon.
https://github.com/aws/sagemaker-mxnet-container/blob/master/src/sagemaker_mxnet_container/serving.py#L105

The reason you had to set the content_type is because by default we will default to JSON in our common container framework code: https://github.com/aws/sagemaker-containers/blob/master/src/sagemaker_containers/_worker.py#L143. Thus setting the content_type to numpy is needed.

Unfortunately, there isn't an easy way around this due to how our MXNet containers support the input and output format by default.

Modifying the transform function like you did above or both the input and output function would be the correct way around this ordeal, much like this example:
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/mxnet_gluon_mnist/mnist.py#L166

Our MXNet container is always improving and we are always open to contributions and suggestions! Providing a default input and output function for handling Numpy would be the best idea for future users.

Please let me know what you think and if there is anything I can clarify.

Thanks!

[NRauschmayr]

Hello @ChoiByungWook,

thanks a lot for detailed explanation. I checked the mxnet default container and i changed
VALID_CONTENT_TYPES = (content_types.JSON,) to VALID_CONTENT_TYPES = (content_types.JSON, content_types.NPY) in https://github.com/aws/sagemaker-mxnet-container/blob/master/src/sagemaker_mxnet_container/serving.py#L105
It seems to work fine (at least for Gluon models). I can send now Numpy arrays and receive the result as a list.

[nadiaya]

Thanks for confirming that suggested work around worked!
Resolving.

[andremoeller]

Reopening this with some follow-ups:

Shouldn't the MXNet container support NPY, especially if all that's needed is a one-line change in the container now that the MXNet container uses SageMaker Containers? I don't think a user should have to rebuild the MXNet container as a work-around for this.

And would it make sense to make NPY the default serializer for MXNet? The MXNet container itself deserializes JSON into NumPy arrays.

The stack trace also suggests that there's a bug in the error handling logic in MXNet containers or SageMaker Containers:

algo-1-TZFX7_1_1307f2d8e109 |   File "/usr/local/lib/python2.7/dist-packages/sagemaker_containers/_functions.py", line 86, in wrapper
algo-1-TZFX7_1_1307f2d8e109 |     six.reraise(error_class, error_class(e), sys.exc_info()[2])
algo-1-TZFX7_1_1307f2d8e109 | TypeError: 'NoneType' object is not callable

There's also the original issue, which is why the JSON deserialization failed in the MXNet container in the first place for what seems like a perfectly good NumPy array.

[nadiaya]

Great point, Andre! Thank you!

Adding bug and feature request labels and keeping it open.

[NRauschmayr]

Lauren added support for Numpy in MXNet container: aws/sagemaker-mxnet-training-toolkit#60 Closing the issue.