Memory leak when reloading model config

[thomasdhc]

Bug Report

Memory leak when reloading model config

System information

OS Platform and Distribution: ubuntu:18.04
TensorFlow Serving installed from: binary
TensorFlow Serving version: 2.1.0
Bug produced using TFS docker image: tensorflow/serving:2.1.0-gpu

Describe the Problem

Using the grpc model management endpoints to load and unload models, specifically calling the function ReloadConfigRequest, we've loaded 22 copies of the same model each with size 208MiB and proceeded to unload them.

When all the models were loaded docker stats showed ~10GiB in memory usage. We expected it to return close to the base memory usage when we unloaded them all.

But after unloading them, we still saw a usage of 8.153GiB. No additional changes have been made to the TFS code.

Exact Steps to Reproduce

1. Pull Docker image sudo docker pull tensorflow/serving:2.1.0-gpu
2. Run Docker image sudo docker run -it --rm -v "/local/models:/models" -e MODEL_NAME=model_name tensorflow/serving:2.1.0-gpu
3. Have a separate window with tensorflow_serving_api==2.1.0 (binary)
4. Add python client side grpc code to tensorflow_serving (shown below)
5. Load model 22 (different copies of the same model) times using python client script
6. Record memory usage
7. Unload all models
8. Record memory usage

Source code / logs

Server side logs

Grpc Client Side Code

import grpc
from tensorflow_serving.apis import model_service_pb2_grpc
from tensorflow_serving.config import model_server_config_pb2
from tensorflow_serving.apis import model_management_pb2

server_address = "0.0.0.0:1234" # Replace with address of your server

def handle_reload_config_request(stub):
    model_server_config = model_server_config_pb2.ModelServerConfig()
    request = model_management_pb2.ReloadConfigRequest()
    config_list = model_server_config_pb2.ModelConfigList()

    model_server_config.model_config_list.CopyFrom(config_list)
    request.config.CopyFrom(model_server_config)

    response = stub.HandleReloadConfigRequest(request)

    print("Response: %s" % response)


def run():
    with grpc.insecure_channel(server_address) as channel:
	stub = model_service_pb2_grpc.ModelServiceStub(channel)
	print("-------------Handle Reload Config Request--------------")
	handle_reload_config_request(stub)


if __name__ == '__main__':
    run()

[shadowdragon89]

I cannot reproduce the issue following the steps. Attach the server side log I saw, which seems different comparing to the screen shot you provided.

Server side log:
2020-06-14 18:57:14.959220: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 18:57:15.040068: I tensorflow_serving/core/loader_harness.cc:138] Quiescing servable version {name: model version: 1538687196}
2020-06-14 18:57:15.040165: I tensorflow_serving/core/loader_harness.cc:145] Done quiescing servable version {name: model version: 1538687196}
2020-06-14 18:57:15.040186: I tensorflow_serving/core/loader_harness.cc:120] Unloading servable version {name: model version: 1538687196}
2020-06-14 18:57:15.058401: I ./tensorflow_serving/core/simple_loader.h:363] Calling MallocExtension_ReleaseToSystem() after servable unload with 123534814
2020-06-14 18:57:15.058440: I tensorflow_serving/core/loader_harness.cc:128] Done unloading servable version {name: model version: 1538687196}
2020-06-14 19:04:48.305008: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 19:04:57.627177: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 19:04:59.583243: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 19:05:01.173922: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 19:05:02.433362: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 19:05:03.683767: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-06-14 19:05:04.872638: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
...

Could you also confirm if you have seen the similar issue for CPU models?

[maxockner1]

We will take a look ASAP!

…

On Sun, Jun 14, 2020, 12:38 PM chaox ***@***.***> wrote: I cannot reproduce the issue following the steps. Attach the server side log I saw, which seems different comparing to the screen shot you provided. Server side log: 2020-06-14 18:57:14.959220: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 18:57:15.040068: I tensorflow_serving/core/loader_harness.cc:138] Quiescing servable version {name: model version: 1538687196} 2020-06-14 18:57:15.040165: I tensorflow_serving/core/loader_harness.cc:145] Done quiescing servable version {name: model version: 1538687196} 2020-06-14 18:57:15.040186: I tensorflow_serving/core/loader_harness.cc:120] Unloading servable version {name: model version: 1538687196} 2020-06-14 18:57:15.058401: I ./tensorflow_serving/core/simple_loader.h:363] Calling MallocExtension_ReleaseToSystem() after servable unload with 123534814 2020-06-14 18:57:15.058440: I tensorflow_serving/core/loader_harness.cc:128] Done unloading servable version {name: model version: 1538687196} 2020-06-14 19:04:48.305008: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 19:04:57.627177: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 19:04:59.583243: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 19:05:01.173922: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 19:05:02.433362: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 19:05:03.683767: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. 2020-06-14 19:05:04.872638: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models. ... Could you also confirm if you have seen the similar issue for CPU models? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1664 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGJLNXPZWB3P6P2RGVJXNGTRWURL5ANCNFSM4NYVOLMQ> .

[thomasdhc]

Hey @shadowdragon89 ,

We've reproduced the bug using a model provided by Tensorflow. Please find the reproduction steps below.

1. Download and copy models then start TFS
TFS starts with one model loaded:

export BASE_PATH=~/tensorflow-serving-issue-1664 ;
mkdir -p  $BASE_PATH ;
cd $BASE_PATH
curl -O https://s3-us-west-2.amazonaws.com/aws-tf-serving-ei-example/inception.zip
unzip inception.zip

cd SERVING_INCEPTION
mv SERVING_INCEPTION SERVING_INCEPTION_0
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_1
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_2
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_3
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_4
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_5
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_6
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_7
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_8
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_9
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_10
sudo docker run -it --rm -v "${BASE_PATH}/SERVING_INCEPTION:/models" -e MODEL_NAME="SERVING_INCEPTION_0" tensorflow/serving:2.1.0-gpu

2. Client side python script
Exec into the TFS docker to setup client side code:

sudo docker exec -it <TFS_DOCKER_ID> bash
# Install python and tensorflow_serving_api
>> apt update
>> apt upgrade
>> apt install python2.7 python-pip
>> pip install --upgrade pip
>> pip install tensorflow_serving_api==2.1.0

Add following code to tfs_grpc_client.py:

import argparse
import grpc
from tensorflow_serving.apis import model_service_pb2_grpc
from tensorflow_serving.config import model_server_config_pb2
from tensorflow_serving.apis import model_management_pb2

server_address = "0.0.0.0:8500"

def handle_reload_config_request(stub, load_model):
    model_server_config = model_server_config_pb2.ModelServerConfig()
    request = model_management_pb2.ReloadConfigRequest()
    config_list = model_server_config_pb2.ModelConfigList()

    model_name = "SERVING_INCEPTION_"
    base_path = "/models/SERVING_INCEPTION_"
    model_platform = "tensorflow"

    if (load_model=='True'):
        for x in xrange(1,11):
            new_config = config_list.config.add()
            new_config.name = model_name + str(x)
            new_config.base_path = base_path + str(x)
            new_config.model_platform = model_platform

    model_server_config.model_config_list.CopyFrom(config_list)
    request.config.CopyFrom(model_server_config)

    response = stub.HandleReloadConfigRequest(request)

    print("Response: %s" % response)


def run(args):
    with grpc.insecure_channel(server_address) as channel:
	stub = model_service_pb2_grpc.ModelServiceStub(channel)
	print("-------------Handle Reload Config Request--------------")
	handle_reload_config_request(stub, args.load_model)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("--load_model", type=str)
    args = parser.parse_args()
    run(args)

3. Load, unload and check memory usage

# Inside the TFS Docker
>> python tfs_grpc_client.py --load_model=True
# On a seperate terminal
sudo docker stats <TFS_DOCKER_ID> // Check memory usage when all models are loaded
# Inside the TFS Docker
>> python tfs_grpc_client.py --load_model=False
# On a seperate terminal
sudo docker stats <TFD_DOCKER_ID> // Check memory usage when all models are unloaded

4. Results we see
All models loaded:

All models unloaded:

[ratansingh98]

Hey @shadowdragon89 ,

We've reproduced the bug using a model provided by Tensorflow. Please find the reproduction steps below.

1. Download and copy models then start TFS
TFS starts with one model loaded:

export BASE_PATH=~/tensorflow-serving-issue-1664 ;
mkdir -p  $BASE_PATH ;
cd $BASE_PATH
curl -O https://s3-us-west-2.amazonaws.com/aws-tf-serving-ei-example/inception.zip
unzip inception.zip

cd SERVING_INCEPTION
mv SERVING_INCEPTION SERVING_INCEPTION_0
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_1
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_2
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_3
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_4
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_5
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_6
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_7
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_8
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_9
cp -R SERVING_INCEPTION_0 SERVING_INCEPTION_10
sudo docker run -it --rm -v "${BASE_PATH}/SERVING_INCEPTION:/models" -e MODEL_NAME="SERVING_INCEPTION_0" tensorflow/serving:2.1.0-gpu

2. Client side python script
Exec into the TFS docker to setup client side code:

sudo docker exec -it <TFS_DOCKER_ID> bash
# Install python and tensorflow_serving_api
>> apt update
>> apt upgrade
>> apt install python2.7 python-pip
>> pip install --upgrade pip
>> pip install tensorflow_serving_api==2.1.0

Add following code to tfs_grpc_client.py:

import argparse
import grpc
from tensorflow_serving.apis import model_service_pb2_grpc
from tensorflow_serving.config import model_server_config_pb2
from tensorflow_serving.apis import model_management_pb2

server_address = "0.0.0.0:8500"

def handle_reload_config_request(stub, load_model):
    model_server_config = model_server_config_pb2.ModelServerConfig()
    request = model_management_pb2.ReloadConfigRequest()
    config_list = model_server_config_pb2.ModelConfigList()

    model_name = "SERVING_INCEPTION_"
    base_path = "/models/SERVING_INCEPTION_"
    model_platform = "tensorflow"

    if (load_model=='True'):
        for x in xrange(1,11):
            new_config = config_list.config.add()
            new_config.name = model_name + str(x)
            new_config.base_path = base_path + str(x)
            new_config.model_platform = model_platform

    model_server_config.model_config_list.CopyFrom(config_list)
    request.config.CopyFrom(model_server_config)

    response = stub.HandleReloadConfigRequest(request)

    print("Response: %s" % response)


def run(args):
    with grpc.insecure_channel(server_address) as channel:
	stub = model_service_pb2_grpc.ModelServiceStub(channel)
	print("-------------Handle Reload Config Request--------------")
	handle_reload_config_request(stub, args.load_model)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("--load_model", type=str)
    args = parser.parse_args()
    run(args)

3. Load, unload and check memory usage

# Inside the TFS Docker
>> python tfs_grpc_client.py --load_model=True
# On a seperate terminal
sudo docker stats <TFS_DOCKER_ID> // Check memory usage when all models are loaded
# Inside the TFS Docker
>> python tfs_grpc_client.py --load_model=False
# On a seperate terminal
sudo docker stats <TFD_DOCKER_ID> // Check memory usage when all models are unloaded

4. Results we see
All models loaded:

All models unloaded:

I had reproduced the steps you provided. Yes, it has a memory leak if we load and unload multiple times. Might be TensorFlow-serving architecture was defined for creating and destroying instances. Considering the current build I would say it will be better if you create your own docker on top of TensorFlow-serving.

[thomasdhc]

Ok, what is your proposed solution? If the TensorFlow-serving architecture has a memory leak, how would building a docker on top of it find and deallocate the memory?

[ratansingh98]

Instead of unloading your model, you can destroy your docker container. And by creating your own docker on top reproduce step 2 for each run. And for step 3, you can call your container and even specify parameter through docker image CLI.

[mihaimaruseac]

Does this reproduce with any model?

[thomasdhc]

Hi @mihaimaruseac,

Yes, we were able to reproduce this with our own models as well as models provided by Tensorflow.

[shadowdragon89]

Thanks @thomasdhc! I can reproduce the problem with the updated steps. I am looking into the issue.

[shadowdragon89]

Hi @thomasdhc,
It looks to me the issue seems to be caused by some memory cache behavior from docker. When I tried to load and unload the model multiple times, the reported memory usage does not increase continuously.
More specifically, you could try to limit the docker memory by '-m 2GB' when starting the server, the models could be load and unload many times without problem.