Deployment Prediction Error

[annatn998]

Hi, I've been following the tutorial but have been making minor adjustments to fit a GCN model rather than a text-classification one. I have gotten up to the section where we send a POST request for prediction.

When I follow the request sent on the tutorial:

curl -s -X POST \
  -H "Content-Type: application/json; charset=utf-8" \
  -d @./predictor/data_object.json \
  http://localhost:7080/predictions/test_model/

I get the error:

Error 1: 

{
  "code": 500,
  "type": "InternalServerException",
  "message": "Worker died."
}

I modified the curl command and sent:

curl -s -X POST \
  -d @./predictor/data_object.json \
  http://localhost:7080/predictions/test_model/

which got me this error:

Error 2: 
{
  "code": 503,
  "type": "InternalServerException",
  "message": "Prediction failed"
}

I think I've pinpointed the issues in the docker logs about what the issues might be, but I can't seem to find a solution.

Error 1 logs:

2022-01-26T17:29:44,750 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/model_service_worker.py", line 189, in <module>
2022-01-26T17:29:44,751 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     worker.run_server()
2022-01-26T17:29:44,752 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/model_service_worker.py", line 161, in run_server
2022-01-26T17:29:44,753 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     self.handle_connection(cl_socket)
2022-01-26T17:29:44,755 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/model_service_worker.py", line 116, in handle_connection
2022-01-26T17:29:44,756 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     cmd, msg = retrieve_msg(cl_socket)
2022-01-26T17:29:44,756 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/protocol/otf_message_handler.py", line 36, in retrieve_msg
2022-01-26T17:29:44,757 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     msg = _retrieve_inference_msg(conn)
2022-01-26T17:29:44,757 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/protocol/otf_message_handler.py", line 226, in _retrieve_inference_msg
2022-01-26T17:29:44,758 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     request = _retrieve_request(conn)
2022-01-26T17:29:44,759 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/protocol/otf_message_handler.py", line 261, in _retrieve_request
2022-01-26T17:29:44,759 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     input_data = _retrieve_input_data(conn)
2022-01-26T17:29:44,760 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/protocol/otf_message_handler.py", line 314, in _retrieve_input_data
2022-01-26T17:29:44,760 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     model_input["value"] = json.loads(value.decode("utf-8"))
2022-01-26T17:29:44,772 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
2022-01-26T17:29:44,772 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     return _default_decoder.decode(s)
2022-01-26T17:29:44,772 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
2022-01-26T17:29:44,773 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2022-01-26T17:29:44,773 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -   File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
2022-01-26T17:29:44,774 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG -     raise JSONDecodeError("Expecting value", s, err.value) from None
2022-01-26T17:29:44,775 [INFO ] W-9000-test_dict_1.0-stdout MODEL_LOG - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Error 2 Logs:

2022-01-26T17:33:27,447 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/service.py", line 102, in predict
2022-01-26T17:33:27,447 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -     ret = self._entry_point(input_batch, self.context)
2022-01-26T17:33:27,447 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/torch_handler/request_envelope/base.py", line 26, in handle
2022-01-26T17:33:27,447 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -     data = self.parse_input(data)
2022-01-26T17:33:27,447 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/torch_handler/request_envelope/json.py", line 19, in parse_input
2022-01-26T17:33:27,448 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -     lengths, batch = self._batch_from_json(data)
2022-01-26T17:33:27,448 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/torch_handler/request_envelope/json.py", line 30, in _batch_from_json
2022-01-26T17:33:27,448 [INFO ] W-9001-test_dict_1.0 ACCESS_LOG - /172.17.0.1:51646 "POST /predictions/test_dict/ HTTP/1.1" 503 6
2022-01-26T17:33:27,448 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -     mini_batches = [self._from_json(data_row) for data_row in data_rows]
2022-01-26T17:33:27,448 [INFO ] W-9001-test_dict_1.0 TS_METRICS - Requests5XX.Count:1|#Level:Host|#hostname:6c7c4274b7e0,timestamp:null
2022-01-26T17:33:27,449 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/torch_handler/request_envelope/json.py", line 30, in <listcomp>
2022-01-26T17:33:27,449 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -     mini_batches = [self._from_json(data_row) for data_row in data_rows]
2022-01-26T17:33:27,449 [DEBUG] W-9001-test_dict_1.0 org.pytorch.serve.job.Job - Waiting time ns: 162058, Inference time ns: 6406818
2022-01-26T17:33:27,451 [INFO ] W-9001-test_dict_1.0 TS_METRICS - WorkerThreadTime.ms:6|#Level:Host|#hostname:6c7c4274b7e0,timestamp:null
2022-01-26T17:33:27,452 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -   File "/home/venv/lib/python3.6/site-packages/ts/torch_handler/request_envelope/json.py", line 39, in _from_json
2022-01-26T17:33:27,452 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG -     rows = (data.get('data') or data.get('body') or data)['instances']
2022-01-26T17:33:27,452 [INFO ] W-9001-test_dict_1.0-stdout MODEL_LOG - KeyError: 'instances'

My json object:

{
       "instances": [
                     {
                         "data": {
                                 "x": [[-1.0], [0.0], [1.0]], 
                               "edge_index": [[0, 1, 1, 2], [1, 0, 2, 1]],
                                 "y": [1, 2, 3]
                    }
              }
       ]
  }

I've tested out my json object many times to make sure it loads properly, and I've followed the structure shown in the tutorial making sure that instances is a key in the json. So I'm really really stuck as to what's happening. Could someone please help me!

[kweinmeister]

Hi @annatn998, can you provide the link to the notebook that you're starting from?

[annatn998]

@kweinmeister I am using a workbench notebook from a private GCP project, but I'll copy and paste a link to the files that I'm using for deployment!

train.py - file used to train model
Dockerfile - torchserve dockerfile for prediction
custom_handler.py - custom handler for GCN model
models.py - because it's an eager model we include a model-file into the torch archive

pytorch_torchserve.zip

[kweinmeister]

Thanks! I am tagging @RajeshThallam, who wrote one of our PyTorch serving notebooks in this repo, in case he might be familiar with the errors you're seeing. Please keep in mind that we are not able to provide official support here, so I'll provide a couple more options to consider:

• Google Cloud Platform support options are at https://cloud.google.com/support/
• You can ask a question on Stack Overflow using the tag google-cloud-ml. This tag not only receives responses from the Stack Overflow community, but also from Google engineers, who monitor the tag and offer unofficial support.

[ivanmkc]

Closing as this is not the right forum for support requests.

[hanneshapke]

What was the conclusion? I am facing the same issue. Thank you for your hints.

[aronicq]

@annatn998 Hi!
Were you able to solve the issue?

[aronicq]

Nevermind, found an inconsistency between handler expectation and json envelope parser. Also was missing header in curl -H "Content-Type: application/json"...