tf-serving increase system memory usesge throwing an instance of 'std::bad_alloc'

[tiru1930]

tensorflow_model_server --version
TensorFlow ModelServer: 2.2.0-rc2+dev.sha.d22fc19
TensorFlow Library: 2.2.0

tensorflow_model_server --port=9000 --rest_api_port=8501 --model_name=eff_savd_model --model_base_path=s3://path

Error

2020-07-06 14:47:52.027866: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: eff_savd_model version: 1}
2020-07-06 14:47:52.030552: I tensorflow_serving/model_servers/server.cc:355] Running gRPC ModelServer at 0.0.0.0:9000 ...
2020-07-06 14:47:52.031977: I tensorflow_serving/model_servers/server.cc:375] Exporting HTTP/REST API at:localhost:8051 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2020-07-06 14:51:23.247636: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 4294967296 exceeds 10% of free system memory.
2020-07-06 14:51:24.768396: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 4294967296 exceeds 10% of free system memory.
2020-07-06 14:51:25.880967: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 4294967296 exceeds 10% of free system memory.
2020-07-06 14:51:27.950667: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 4294967296 exceeds 10% of free system memory.
2020-07-06 14:51:30.269902: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 12884901888 exceeds 10% of free system memory.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Request

def main():
  image_file = 'img.png'
  data = np.array(Image.open(image_file)).astype(np.float) #.reshape(-1, 512, 512, 3)

  print(data.shape)
  # np.set_printoptions(threshold=np.inf)       
  # json_request = '{{ "instances" : {} }}'.format(np.array2string(data, separator=',', formatter={'float':lambda x: "%.1f" % x}))
  image_file = 'img.png'
  img = Image.open(image_file)
  resize_image = np.array(img.resize((512,512)))
  print(resize_image.shape)
  image_inlist = resize_image.tolist()

  data = {"instances":image_inlist}

  # json_request = '{{ "instances" : {} }}'.format(resize_image)
  # json_request = '{{ "instances" : {} }}'.format(np.array2string(resize_image, separator=',', formatter={'float':lambda x: x}))
  with open("test.json","w") as f:
    json.dump(data,f)

  resp = requests.post(SERVER_URL, json=data)
  print('response.status_code: {}'.format(resp.status_code))     
  print('response.content: {}'.format(resp.content))

[tiru1930]

issues in inference request

 data = {"instances":image_inlist} -- >  data = {"instances":[image_inlist]}