-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TensorFlow 2.0 with Colab TPU does not work. #1
Comments
Hi Huan, Could you try experimental_connect_to_cluster instead of experimental_connect_to_host? Basically resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR']) And what is os.environ['COLAB_TPU_ADDR'] value in your case? |
Hi @rxsang, Thank you for the suggestion! I had followed your suggestion and get another error result: resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
# tf.config.experimental_connect_to_host(resolver.master())
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.experimental.TPUStrategy(resolver) INFO:tensorflow:Initializing the TPU system: 10.88.139.146:8470
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
<ipython-input-10-343b36a70d10> in <module>()
10 tf.config.experimental_connect_to_cluster(resolver)
11 # tf.config.experimental_connect_to_host(resolver.master())
---> 12 tf.tpu.experimental.initialize_tpu_system(resolver)
13 strategy = tf.distribute.experimental.TPUStrategy(resolver)
14
8 frames
/tensorflow-2.0.0-rc0/python3.6/tensorflow_core/python/eager/context.py in add_function(self, fn)
987 """
988 self.ensure_initialized()
--> 989 pywrap_tensorflow.TFE_ContextAddFunction(self._handle, fn)
990
991 def add_function_def(self, fdef):
InvalidArgumentError: Unable to find a context_id matching the specified one (-5340281180391244817). Perhaps the worker was restarted, or the context was GC'd?
Additional GRPC error information:
{"created":"@1567762110.807927262","description":"Error received from peer","file":"external/grpc/src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Unable to find a context_id matching the specified one (-5340281180391244817). Perhaps the worker was restarted, or the context was GC'd?","grpc_status":3} You can reproduce it by open my Colab Notebook at here |
Hi guys, have the same issue. Tried experimental_connect_to_cluster suggestion and got the following error:
|
Hi, Does the issue still exist? If not, could you provide instructions how do you set up the environment and test? The thing I'm suspecting is the tpu_address may not point to the actual TPU worker correctly. |
Yuefeng pointed me the example https://colab.sandbox.google.com/github/huan/tensorflow-handbook-tpu/blob/master/tensorflow-handbook-tpu-example.ipynb#scrollTo=03EV61RS5jyR, I can reproduce this issue now. It seems a version mismatch between colab client and tpu worker. The client version is built with 2.0.0-rc2, but I'm not sure about the tpu worker version. I'll ask around about how does colab release their tpu worker. |
It's great to know that you had reproduced this issue, and I'm looking forward to using the TensorFlow 2.0 with colab, can not waiting for that! :) |
Update: Error message changed after the 2.0 released.
Related to: Reproducable Colab: https://colab.research.google.com/github/huan/tensorflow-handbook-tpu/blob/master/tensorflow-handbook-tpu-example.ipynb |
Hi Huan, Although TF2.0 has been released, but the colab version hasn't been updated. We are working on the colab update and will make sure this is ready with the 2.1 release (should be 2 weeks later). |
@rxsang It's great getting to know that the Colab with TPU will work with TF2.1 two weeks later! Thank you for the reply, and I'm looking forward to using them, Cheers! |
it seems like |
I have confirmed the @kpe solution to this issue, disable eager execution with tf.2.0 by adding |
Today I finally fixed this bug, with Colab & TensorFlow 2.2.0-rc1 The most important part is how to construct the tf.distribute strategy, there are lots of tricks, many developers struggling with that. At last, I learned from Martin Gorner, he has a great Colab notebook that uses TPU without any problem: 07_Keras_Flowers_TPU_xception_fine_tuned_best.ipynb So I copy/paste the tf.distribute strategy code to my notebook, run it, everything works like a charm! tpu = tf.distribute.cluster_resolver.TPUClusterResolver() # TPU detection
print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])
tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)
strategy = tf.distribute.experimental.TPUStrategy(tpu)
print("REPLICAS: ", strategy.num_replicas_in_sync) |
The issue persists with TF 2.2. (gcloud, v3-8 TPU, 8c, 30gbram) I was suspecting from resource limits, but it looks like the error is also related to tf.data. update-1: yes, it seems like the issue is tf.data.Dataset, yet I don't exactly know why. All works well if you feed your data without using tf.data class. |
Faced this issue on kaggle
|
@sourcecode369 i am also having same issue in kaggle today,1 day ago i trained a model using tpu,everything worked fine,,i changed nothing,,model is exactly same as it,everything is as before ,i once changed the dataset,previously the dataset i used for training was large but today the dataset i am trying for training is smaller than that,,, except changing dataset i changed nothing but 1 day ago it worked on large train set and today it is not working,please let me know if you find the solution for this issue |
|
@rxsang here is my code :
during executing code block above i get error InvalidArgumentError: Unable to find a context_id matching the specified one (1512067339570197782). Perhaps the worker was restarted, or the context was GC'd? |
@mobassir94 When you use a larger dataset, the |
@jianse thank you for your kind reply, i reduced the train set size,it's now (408520, 250)..in my code you can see i used .cache in validation set because the validation set is very small,,but ok after reading your comment i just deleted that line of code and ran the kernel and unfortunately i am getting this same error again : InvalidArgumentError: Unable to find a context_id matching the specified one (4466620674281380355). Perhaps the worker was restarted, or the context was GC'd? |
@mobassir94 did you manage to solve it later ? |
@zarif98sjs iirc i had problem in my dataset ( i had both int and float in label) i can't remember clearly this issue that i faced almost 2 year ago,but nowadays using latest tf version and tpu in kaggle i don't face issues like this anymore! |
@mobassir94 solved it later ! removing the cache worked for me . |
Tensorflow 2.0 RC on Colab can not work. @yuefengz
The text was updated successfully, but these errors were encountered: