Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Start serving failed with uninitialized value since path variable and lookup table can't initialize in order #1437
Describe the problem
When using lookup table(index_to_string_table_from_file) combined with assets(path assigned with a variable), since the serving initializer param
Log like this when start serving:
How to reproduce
Suppose we have a "test_index.assets" file with the following content:
In jupyter env, build graph:
Start session, run initilizers in order and test results:
This should print
Then in terminal, start serving the model with docker:
Serving would probably fail with log as follows:
I also tried using
Is there any workaround for this? Should the
Yes, I get the
Log for the same is shown below. Can you please wait for some time and confirm if Servable is loaded properly. Thanks!
I tried waiting retrying, though it may success，but that was just because of the luck (these initilizers incidentally run in an order we hoped). This log shows it served successfully on the 2nd retry:
It may success on the 2nd retry, or fail for all the 5 retries (max retry setting).
Starting multiple models at the same time could give a better understanding of the uncertainty of initialization.
Suppose we duplicate the SavedModel in
Some of the models may success, but some will fail, and the failed ones have to retry for the next time untill all the models loaded. This could take a long time.
The retry can't guarantee a successful serving. We need a way to initialize path assigner and lookup table in the given order.
Sorry for the late response.
The errors are the same when using
Since I'm not familiar with
These attempts failed. Actually, the error will just be thrown immediately after running
Could you try changing your code to the following
Then you can just use the following initializer:
The problem was that when we create a variable we create a default "read" op that has no control dependency. When you try to run the table's initializer TensorFlow's executor, which executes greedily, tries to run that "read" op first and fails because the
I tried the
Then I did some search, find out that the
So I replace the
Also with control dependency and
Hope things would be better in tf 2.0 since this post said that
In conclusion, we have to use
I think the problem is solved for now, and we're good to close this issue.