tf.gather Converting sparse IndexedSlices warning

[gasteigerjo]

System information

• Have I written custom code (as opposed to using a stock
  example script provided in TensorFlow): yes
• OS Platform and Distribution (e.g.,
  Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from: binary (Anaconda)
• TensorFlow version: 2.1
• Python version: 3.7.7
• CUDA/cuDNN version: 10.1/7.6.5
• GPU model and memory: GeForce 1080 Ti

Describe the current behavior
Whenever tf.gather (or tf.boolean_mask) is called with params (tensor) being the output of another function, this throws the warning UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. This behavior seems to be well known but largely ignored/accepted. Calling tf.gather on the output of another function is perfectly reasonable and should in my opinion not throw a warning.

Describe the expected behavior
The forward pass should infer the shape of the dense tensor required in the backward pass, if possible (issue with dynamic shapes?). No warning should be thrown or the user should be able to acknowledge/deactivate the warning (ideally a warning that is more specific than the current one).

Standalone code to reproduce the issue
See e.g. https://github.com/klicperajo/dimenet/blob/master/train.ipynb

Other info / logs
https://stackoverflow.com/questions/35892412/tensorflow-dense-gradient-explanation
Related: #23566

[amahendrakar]

@klicperajo,
I tried to reproduce the issue, but I'm facing an error stating KeyError: 'ceil'. Please find the gist of it here.

In order to expedite the trouble-shooting process, could you please provide a minimal code sample to reproduce the issue reported here. Thanks!

[gasteigerjo]

Thank you for looking into this!

That's because the runtime uses a completely outdated version of sympy. Just add a !pip install --upgrade sympy at the beginning and you're good to go.

The notebook also throws a deprecation warning from within tensorflow (calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated), so that should be worthwile to investigate as well.

[amahendrakar]

Was able to reproduce the issue with TF v2.1, however running the code with TF-nightly prints only a single warning. Please find the attached gist. Thanks!

[gasteigerjo]

Great! Throwing a single warning is certainly better than throwing 12. However, it does still throw a warning, so my criticism and thoughts from above still applies.