Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC-FIX] TF autolog and non eager mode #11586

Open
konstantin-frolov opened this issue Apr 2, 2024 · 2 comments
Open

[DOC-FIX] TF autolog and non eager mode #11586

konstantin-frolov opened this issue Apr 2, 2024 · 2 comments
Labels
area/docs Documentation issues

Comments

@konstantin-frolov
Copy link

Willingness to contribute

No. I cannot contribute a documentation fix at this time.

URL(s) with the issue

https://mlflow.org/docs/latest/python_api/mlflow.tensorflow.html

Description of proposal (what needs changing)

Please add information about work tensorflow.autolog() and non eager mode for TF 2.
autolog() doesn't store any metrics if TF worked in non eager mode (compiled models).
I spend 2 hours for solve problem with not stored metrics. And I have no errors in any logs, just empty metrics in mlflow UI.
by default TF 2 eager mode enabled.

@konstantin-frolov konstantin-frolov added the area/docs Documentation issues label Apr 2, 2024
@chenmoneygithub
Copy link
Collaborator

@konstantin-frolov Thanks for raising the issue! It shouldn't be the case if you are using Keras model.fit(), for which graph mode is turned on by default, and we are explicitly supporting it. I suspect you are wrapping your whole training loop by tf.function, in which case you cannot get the numerics during training. Checking Keras code for how the compilation works as a reference: link

Could you share a reproducible github gist? We can take a closer look.

Copy link

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docs Documentation issues
Projects
None yet
Development

No branches or pull requests

2 participants