-
Notifications
You must be signed in to change notification settings - Fork 45.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
metrics are not displayed during training #9136
Comments
Yup, same issue here. in TF 1 this is works properly, while in TF 2 it does not... |
I also have the same issue. I have to stop the training once in a while, run evaluation manually and resume the training process afterwards. |
Also hitting the same issue trying to port my code over from TF1 to TF2. I previously was using Weights & Biases to sync to Tensorboard so that I could monitor the progress of training... and now not sure what to do |
Just checked related issues, it seems that evaluation while training, as we did in TF 1 is not supported with TF2's model_main_tf2.py |
me too |
Hi, any update on this issue? |
I found this tutorial to be really useful to get evaluation on tensorboard while training the model with TF2. Check that for more details but basically you have to run your model_main_tf2.py script in parallel, one for training with the training dataset the other for evaluating with the validation dataset. You can either use 2 GPUs or if you have only one, use GPU for training, CPU for evaluating - it's explained how in the tutorial. |
I understand that we can run in parallel two scripts. But what should I do if I run everything in Colab? I thought most everyone uses Colab here... |
Hi
I install tensorflow 2.2, and use efficientdet_d2_coco17_tpu-32.
I managed to start training this model with this command:
But while training it outputs only loss without metrics evaluated on eval set as it was in Tensorflow Object Detection 1.
How can I see my metrics?
Also, in train folder there are events file. They are initialised in the beginning, and then nothing is happening with them. How can I update events to see model metrics and progress in Tensorboard?
The text was updated successfully, but these errors were encountered: