Serving a Tensorflow model fails with ConnectionClosedError #831
Comments
|
hi @tigerhawkvok, thanks for using SageMaker! Unfortunately, SageMaker's InvokeEndpoint API does have a 5MB limit on the size of incoming requests. For using numpy as the content type, you'll need to provide an inference script, or else the endpoint will reject any request that's neither JSON nor CSV, which is why you were getting a 415 back with "Unsupported Media Type." You can read more about how to write that script here: https://github.com/aws/sagemaker-tensorflow-serving-container/tree/6be54a389293340bde24a5c3c3a2ff6b16f7dca6#prepost-processing. |
|
@laurenyu FYI the all important bit:
In general the documentation is really scattershot, honestly ... the sagemaker predictor has documentation in
And not one place is complete! Also, the server doesn't have numpy? I added a very simple # To allow multiple people to work in one instance, we keep our work in a CONTAINER_DIR of our own
requirements = """
numpy ~= 1.15.1
"""
req = os.path.join(CONTAINER_DIR, "requirements.txt")
with open(req, "w") as fh:
fh.write(requirements)
model = Model(entry_point= entryPoint, dependencies= [req], model_data= "path/to/my/artifact.tar.gz", role= role)but still get the "No module named 'numpy'" error. with this import json
def input_handler(data, context):
""" Pre-process request input before it is sent to TensorFlow Serving REST API
Args:
data (obj): the request data, in format of dict or string
context (Context): an object containing request and configuration details
Returns:
(dict): a JSON-serializable dict that contains request body and headers
"""
if context.request_content_type == 'application/json':
# pass through json (assumes it's correctly formed)
d = data.read().decode('utf-8')
return d if len(d) else ''
if context.request_content_type == 'text/csv':
# very simple csv handler
return json.dumps({
'instances': [float(x) for x in data.read().decode('utf-8').split(',')]
})
if context.request_content_type in ('application/x-npy', "application/npy"):
import numpy as np
# If we're an array of numpy objects, handle that
if isinstance(data[0], np.ndarray):
data = [x.tolist() for x in data]
else:
data = data.tolist()
return json.dumps({
"instances": data
})
raise ValueError('{{"error": "unsupported content type {}"}}'.format(
context.request_content_type or "unknown"))
def output_handler(data, context):
"""Post-process TensorFlow Serving output before it is returned to the client.
Args:
data (obj): the TensorFlow serving response
context (Context): an object containing request and configuration details
Returns:
(bytes, string): data to return to client, response content type
"""
if data.status_code != 200:
raise ValueError(data.content.decode('utf-8'))
response_content_type = context.accept_header
prediction = data.content
return prediction, response_content_typeRemoving the explicit Numpy call and changing the def input_handler(data, context):
""" Pre-process request input before it is sent to TensorFlow Serving REST API
Args:
data (obj): the request data, in format of dict or string
context (Context): an object containing request and configuration details
Returns:
(dict): a JSON-serializable dict that contains request body and headers
"""
if context.request_content_type == 'application/json':
# pass through json (assumes it's correctly formed)
d = data.read().decode('utf-8')
return d if len(d) else ''
if context.request_content_type == 'text/csv':
# very simple csv handler
return json.dumps({
'instances': [float(x) for x in data.read().decode('utf-8').split(',')]
})
if context.request_content_type in ('application/x-npy', "application/npy"):
# If we're an array of numpy objects, handle that
if len(data.shape) is 5:
data = [x.tolist() for x in data]
elif len(data.shape) is 4:
data = data.tolist()
else:
raise ValueError("Invalid tensor shape "+str(data.shape))
return json.dumps({
"instances": data
})
raise ValueError('{{"error": "unsupported content type {}"}}'.format(
context.request_content_type or "unknown"))still has a Adding a
and changing the handler code to "ensure" that Numpy is installed if not absent: import sys
if sys.version_info < (3, 6):
class ModuleNotFoundError(Exception):
pass
import json
import io
try:
import numpy as np
except (ModuleNotFoundError, ImportError):
import os
os.system("pip install numpy")
import numpy as np
def input_handler(data, context):
""" Pre-process request input before it is sent to TensorFlow Serving REST API
Args:
data (obj): the request data, in format of dict or string
context (Context): an object containing request and configuration details
Returns:
(dict): a JSON-serializable dict that contains request body and headers
"""
if context.request_content_type == 'application/json':
# pass through json (assumes it's correctly formed)
d = data.read().decode('utf-8')
return d if len(d) else ''
if context.request_content_type == 'text/csv':
# very simple csv handler
return json.dumps({
'instances': [float(x) for x in data.read().decode('utf-8').split(',')]
})
if context.request_content_type == "application/x-npy":
# If we're an array of numpy objects, handle that
# See https://github.com/aws/sagemaker-python-sdk/issues/799#issuecomment-494564933
data = np.load(io.BytesIO(data), allow_pickle=True)
if len(data.shape) is 5:
data = [x.tolist() for x in data]
elif len(data.shape) is 4:
data = data.tolist()
else:
raise ValueError("Invalid tensor shape "+str(data.shape))
return json.dumps({
"instances": data
})
raise ValueError('{{"error": "unsupported content type {}"}}'.format(
context.request_content_type or "unknown"))
def output_handler(data, context):
"""Post-process TensorFlow Serving output before it is returned to the client.
Args:
data (obj): the TensorFlow serving response
context (Context): an object containing request and configuration details
Returns:
(bytes, string): data to return to client, response content type
"""
if data.status_code != 200:
raise ValueError(data.content.decode('utf-8'))
# May need to implement this
# https://github.com/aws/sagemaker-python-sdk/issues/799#issuecomment-494564933
# buffer = io.BytesIO()
# np.save(buffer, data.asnumpy())
# return buffer.getvalue()
response_content_type = context.accept_header
prediction = data.content
return prediction, response_content_typereturns |
|
Based on the numpy # The endpoint seems to run on py2 despite the traceback
# referencing py3.5 ... if we need it, explicitly declare
# ModuleNotFoundError so this works on both
import sys
if sys.version_info < (3, 6):
class ModuleNotFoundError(Exception):
pass
try:
# Numpy should be installed:
import numpy as np
except (ModuleNotFoundError, ImportError):
import os
try:
# Fine, it isn't. pip should be, though, to install the requirements file
# https://github.com/aws/sagemaker-tensorflow-serving-container/tree/6be54a389293340bde24a5c3c3a2ff6b16f7dca6#prepost-processing
os.system("pip install numpy")
import numpy as np
except (ModuleNotFoundError, ImportError):
# Double fine.
# Install pip first
# https://pip.pypa.io/en/stable/installing/
os.system("curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py")
os.system("python get-pip.py")
os.system("pip install numpy")
import numpy as npOnly to get the especially insane error:
@jesterhazy -- I based this somewhat on your example in #799 which is only two weeks old, any idea why this isn't working? |
|
hi @tigerhawkvok, thanks for your patience as we work through this. for the workaround - as for why the requirements.txt file isn't working, could you list the contents of the model tar that is being used for the endpoint? It's possible that the SDK is incorrectly packing the code with the model artifacts even though you have the directory structure correct locally. |
|
The model tar.gz looks like this:
So: |
|
can you try tar-ing it up as: |
|
Using This is a requests object, right? The requests doc implies I should access |
|
Seperately confirming that the Looks like the gateway timeout was from passing in a model where I saved it with the wrong input layer so it had a bad shape. Thanks! |
|
@tigerhawkvok were you able to fix the ConnectionClosedError error? |
|
@tigerhawkvok please show the inference.py working file if you fixed the error |
|
@akaraul Use the solution suggested by #831 (comment) |
|
Adding requirements.txt does not seem to work for me. Still see numpy package missing error. Interestingly Sagemaker appears to copy the *.tgz archive to different bucket without the requirements.txt file. Unfortunately it seems to only use that archive !! |
|
@whatdhack could you open a new issue (it'll help us track it) and include the code you're using to deploy your model? |
|
When I tried the solution @laurenyu suggested i got another error when specifying Model's entry point as inference.py as specified in https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst Error Message:
although the bucket where my artifacts resides is named test-sagemaker-bucket |
|
@JohnEmad the error message sounds like you're passing in |
|
@laurenyu Thank you for your quick response, here is the exact code I am running and I have tried tarring my model as and but I am still getting the same error |
|
@JohnEmad two questions:
|
|
@laurenyu I tried doing
and yes I am on windows |
|
@JohnEmad cool, that confirms my hypothesis. it's an issue in the SDK code - I've opened #1302 to fix it. Even after the fix is released, the As a workaround for now, can you try not specifying your entry point? (assuming your S3 model data is already packed as you described above) |
|
@laurenyu oh, it worked Thanks alot |
|
I still got the error Numpy module not found when deploying using By some reason, when I try to use the above lines to deploy in TF 2.10 with entry_point = 'inference.py', I raises Error. So, I opened a TF.1.15 kernel and to deploy the model. The inference.py file is below. Any specific things I need to notice? |
|
@quocdat32461997 numpy is not installed in the TFS images by default, so you'll need to provide a requirements.txt file (see an earlier comment for how to structure your |
|
`This is the input handler that worked for me: elif context.request_content_type == 'application/x-npy': |
How can we parse data from the input handler without using the json.dumps? If I don't use json.dumps I get a parsing error. My model expects a multiple NumPy array in a dictionary. If I use JSON dumps again in the input handler, predict function is not getting np array. Should I modify predict_fn for this purpose? |
@laurenyu my inference.py file is never called upon, even though I have packaged it correctly, and tried both with and without specifying the entry_point. Could using windows be causing this? |
|
@ana-pcosta it's hard to say without seeing your code/logs. can you open up a new issue and include as much info about your setup and the issue you're seeing? |
Please fill out the form below.
System Information
When I try to run a prediction / classification on an image, I get timeouts from Sagemaker. It seems like I'm not doing anything particularly complex
It seems I'm htting the 5 MB payload limit. This seems awfully small for image retraining, and I don't see an argument to adjust payload size (also here).
I tried changing the input to the raw Numpy array
but got
though #799 suggests that I I should be able to push Numpy directly, though I'd need to specify an entry point script to handle it on the endpoint's side (which isn't described in the documentation for
deploy, either).I get the same error when trying to directly create a
RealTimePredictor:The text was updated successfully, but these errors were encountered: