-
Notifications
You must be signed in to change notification settings - Fork 694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TFX 0.24 fails on/after trainer step in interactive run #2598
Comments
@lyschoening Please downgrade to tfx 0.22 and let me know if the issue still persists. Thanks! |
Any story moving forward? This is blocking upgrade. |
@gowthamkpr This issue does not occur in TFX 0.22, only in TFX 0.24. I ran the above Colab substituting |
@Iyschoening This is a know issue and will be fixed in the next release. Thanks! |
@gowthamkpr do we have any ETA for that release? |
CC: @hughmiao |
Hi @gowthamkpr @charlesccychen @hughmiao do we have any updates on the ETA? This blocks upgrade if we can't run training reliably. Thanks |
This fixes #2598. PiperOrigin-RevId: 336965037
This will be fixed in #2652. We will need a patch release to resolve this issue. CC: @zhitaoli @dhruvesh09 |
This fixes #2598. PiperOrigin-RevId: 336965037
1 similar comment
Reopening pending patch release. CC: @zhitaoli @dhruvesh09 |
TFX 0.24.1 is now available. Please feel free to test it and report any issues you find. |
It works indeed! Ty for the fast turnaround! |
The typical TFX pipeline fails with a kernel restart after the trainer is run for the first time.
For example, when running the following colab, the kernel restarts if the trainer step is re-run.
https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/components_keras.ipynb#scrollTo=429-vvCWibO0
The runtime logs list this error:
The text was updated successfully, but these errors were encountered: