-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
USE latest version can't be used with MirroredStrategy ("Trying to access a placeholder that is not supposed to be executed.") #515
Comments
Any updates on this also facing this issue? |
Confirmed: there is an issue. Thank you, @eduardofv, for the very clear and reproducible report! @mercarikaicheung, there is no useful update yet, sorry. |
@arnoegw looking at the stacktrace, it seems that the |
From the documentation of USE large: https://tfhub.dev/google/universal-sentence-encoder-large/5 Changelog Version 1
Version 2
Version 3
Version 4
Version 5
|
Hi everyone, |
Not from the distribution strategy side since it appears that this model might have been saved in a non standard way. The TF Hub team is working on new version of the USE that will be built natively in TF2 and should work correctly. |
I am running into the same issue. Any timeline on the fixes, @guptapriya ? |
Hi all, the issue was because this model was converted from TF1 SavedModel to TF2 SavedModel. There's some tricky issues with loading such models under tf.distribute.Strategy. The team is working on a TF2 native version of the model. |
I am able to reproduce this same error using bert on 1 machine with 8 v100 gpus using mirrored strategy. It did not occur when running the same code (and data) on 4 v100 gpus
|
Changing from |
Great. Currently I'm not working on this but if I return to it I'll check this solution. Thanks |
Hi all, any news on this issue? |
@dkorkinof What issue are you facing exactly? Irrespective of the distribution strategy, defining ops within strategy.scope and accessing them via strategy.run resolved the issue for me. |
Basically, I am getting the exact same error with @eduardofv.
Throws this exception:
It seems what @crccw mentioned checks out, as the model's tensorflow version appears as |
Thank you, @jaxlaw! In light of @crccw's comment above, I think that's as much of a resolution we can hope for. |
hmm... after downloading the new model, I still face the same issue. |
Anyone has updates on this issue? |
It looks like this is working with the new model. Example: import tensorflow as tf
import tensorflow.keras as keras
import tensorflow_hub as hub
import tensorflow_text as text
def prepare_model():
inputs = keras.Input(shape=[], dtype=tf.string)
preprocessor = hub.KerasLayer("https://tfhub.dev/google/universal-sentence-encoder-cmlm/multilingual-preprocess/2")(inputs)
embedding = hub.KerasLayer("https://tfhub.dev/google/universal-sentence-encoder-cmlm/multilingual-base/1")(preprocessor)
return keras.Model(inputs=inputs, outputs=embedding["default"])
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
model = prepare_model()
model.compile(optimizer="adam", loss="binary_crossentropy")
# Make sure you're running the model outside of the strategy's scope!
model(tf.constant(["dog", "cat"])) If folks are still running into issues, could you please share some example code and specific error messages? |
Still having error on TF 2.3 with with https://tfhub.dev/google/universal-sentence-encoder/4 Function call stack: |
This is still an issue to me with TF2.3 and the latest version of USE: https://tfhub.dev/google/universal-sentence-encoder-large/5
|
Here is a minimum reproducible example for
Output after this :
@arnoegw , @MorganR : Any insights on how to resolve this error. |
The latest versions of USE throw this error when used in a MirroredStrategy. (tf: 2.1.0, keras: 2.2.4-tf, hub: 0.7.0)
Throws:
Interestingly, NNLM works fine and also USE-large v3 (albeit some warnings, not sure if they affect it's later performance). Also, if
OneDeviceStrategy
or no strategy is used everything works.Check this colab with code and test cases: https://colab.research.google.com/drive/1_YaGYje4tXPyQDx_hYaj9VNLAA5hi3Dg
The text was updated successfully, but these errors were encountered: