TF serving using docker run well for first few days but error occurs after few days -No versions of servable half_plus_two found under base path

[janetkok]

Bug Report

System information

• Windows 10 Home : Single Language 2004
• TensorFlow Serving installed using docker toolbox
• TensorFlow Serving latest version

Describe the problem

I followed the official docs to run my model container. At the first few days, it works smoothly.
The command is as follows:
docker run -t --rm -p 8501:8501 -v "$(pwd)/ssdmod/ssd:/models/ssd" -e MODEL_NAME=ssd tensorflow/serving &
saved_model.pb and variables folder is placed in the following directory:
\ssdmod\ssd\00000123\

But after a few days of successful runs, the model couldn't run anymore and the following error appears.

Building single TensorFlow model file config:  model_name: ssd model_base_path: /models/ssd
2020-08-24 09:44:51.730529: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-08-24 09:44:51.730557: I tensorflow_serving/model_servers/server_core.cc:575]  (Re-)adding model: ssd
2020-08-24 09:44:51.731250: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable ssd found under base path /models/ssd
2020-08-24 09:44:52.732512: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable ssd found under base path /models/ssd
2020-08-24 09:44:53.734714: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable ssd found under base path /models/ssd
2020-08-24 09:44:54.734842: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable ssd found under base path /models/ssd

A lot of web sources are saying that its because a version number is not assigned to the model, but I have already assigned the version number by placing the .pb and variables folder under the folder '00000123' and it can run well and do predictions previously. I thought that it was my model problem, but when I tried to clone the demo models and run all command according to the official docs, the same problem occurs:

Building single TensorFlow model file config:  model_name: half_plus_two model_base_path: /models/half_plus_two
2020-08-24 09:37:52.148556: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-08-24 09:37:52.148747: I tensorflow_serving/model_servers/server_core.cc:575]  (Re-)adding model: half_plus_two
2020-08-24 09:37:52.149871: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable half_plus_two found under base path /models/half_plus_two
2020-08-24 09:37:53.151919: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable half_plus_two found under base path /models/half_plus_two
2020-08-24 09:37:54.152030: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable half_plus_two found under base path /models/half_plus_two

Steps to Reproduce

docker run a model several times for at least 3-4 days, check if it still run after a few days

Attempted solution

-reinstall docker toolbox and tensorflow/serving- error remains
-used older tf serving versions like 2.2.0, 2.1.0 and 1.15.0

[rmothukuru]

@janetkok,
This is interesting and a bit surprising. Can you please confirm

1. If you are performing inference in the same virtual environment as you did 3-4 days ago?
2. The path of the Model that you have passed is correct and that folder comprises the Model.

Thanks!

[janetkok]

@rmothukuru

1. yes, it was the same virtual environment. In fact, I have reproduced the same issue in another linux environment.
2. Yes, it is indeed correct and it does contain the model because I was using the same command to run the model previously and it was okay. I never modified the folder.

[janetkok]

I find out that using this command from the official docs

# Location of demo models
TESTDATA="$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata"

# Start TensorFlow Serving container and open the REST API port
docker run -t --rm -p 8501:8501 \
    -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" \
    -e MODEL_NAME=half_plus_two \
    tensorflow/serving &

and also my similar command docker run -t --rm -p 8501:8501 -v "$(pwd)/ssdmod/ssd:/models/ssd" -e MODEL_NAME=ssd tensorflow/serving &

did not mount my .pb and variables into the container. So I followed the instructions from the official docs to create my own serving image and copy my model into the container. Using this method, I was able to run my model. The fact that I can docker copy the model to the container in this way shows that this might not be a permission problem, maybe the following command just can't mount the .pb and variables folder after a few successful runs on the first few days.

docker run -t --rm -p 8501:8501 \
    -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" \
    -e MODEL_NAME=half_plus_two \
    tensorflow/serving &

[rmothukuru]

@janetkok,
Can you please confirm if we can close this issue as you can run the model now. Thanks!

[janetkok]

yes