Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

**debug feature** for debugging numeric instability, like nan or infinity #37904

Open
bela127 opened this issue Mar 25, 2020 · 0 comments
Open
Assignees
Labels
comp:apis Highlevel API related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower type:feature Feature requests

Comments

@bela127
Copy link

bela127 commented Mar 25, 2020

System information

  • TensorFlow version (you are using): nightly
  • Are you willing to contribute it (Yes/No): cant do it alone, but willing to help

Describe the feature and the current behaviour/state.
I propose a debug feature for debugging numeric instability. The functionality should be similar to tf.debugging.enable_check_numerics() but more granular, but still more high level then tf.debugging.check_numerics.

Implementation Idea:
A tf.debugging.check_func_numerics annotation is added. It can be applied to ops warped in a python eager function and to tf.functons.
It will add numerics checks to all ops used in this function and, this is IMPORTANT, to all corresponding gradient ops.

The part of a model which courses numeric instability can be identified with this.

Will this change the current api? How?
it will add another debugging function/annotation, so the standard api is not changed.

Who will benefit with this feature?
Everyone with problems with nan and infinity in there models.
Especially for big models where tf.debugging.check_func_numerics can not be applied to the hole model.

@bela127 bela127 added the type:feature Feature requests label Mar 25, 2020
@ravikyram ravikyram added TF 2.2 Issues related to TF 2.2 comp:apis Highlevel API related issues labels Mar 26, 2020
@goldiegadde goldiegadde removed the TF 2.2 Issues related to TF 2.2 label Mar 26, 2020
@jvishnuvardhan jvishnuvardhan added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Apr 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:apis Highlevel API related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower type:feature Feature requests
Projects
None yet
Development

No branches or pull requests

5 participants