Yet another problem about high latency when loading a new model

[weberxie]

I have noticed #385, and have read https://ai.googleblog.com/2017/11/latest-innovations-in-tensorflow-serving.html.

Now, even if loading new models in a isolated thread pool, but the main operation process is RestoreOpsV2, which is run in the thread pool of session run, This means that the load operation is not completely run in a separate thread pool. So, the serving query may be blocked by load operation.

In our environment, the model file are stored on HDFS, when loading new model, the latency from a few milliseconds to thousands of milliseconds, I confirm this does not include model warm up time.

[weberxie]

I have resolved this problem by modify some code in tensorflow:
Adding a new thread pool only for load new model when construct DirectSession,
before loading new model I choose the specific thread pool instead of the default thread pool by set run_options.

[weberxie]

I would like to ask whether this is a bug, whether the community welcomes the submission of a patch, or whether there is a better way to solve this problem.

[weberxie]

@chrisolston @nfiedel sorry to bother you, I have read your blog, can you give some advice?

[AlphaSue]

can you share how to do warm up? @weberxie

[weberxie]

@AlphaSue you can construct a fake input tensor from model signature, then call session run to warm up the whole model.

[AlphaSue]

thx

[tianyapiaozi]

@weberxie can you share your modification to tensorflow?

[ymodak]

@weberxie Just wanted to follow up. Is this still an issue?

[weberxie]

@ymodak I will see if I can reproduce it under the of version 1.10.

[ymodak]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!