report_tensor_allocations_upon_oom and embedding_lookup together lead to memory leak

[bsnipers]

Describe the current behavior
The RunOptions report_tensor_allocations_upon_oom and embedding_lookup will lead to CPU memory leak.
This will only happen in tf 1.12, tf 1.10 didn't have this problem.

I thought tf.gather have almost same functionality, so I test it and it seems no problem at all.

I'm not sure if #21348 is related to this problem. My problem will only occur with runoptions.

Code to reproduce the issue

import tensorflow as tf

embeddings = tf.reshape(tf.range(1000), shape=(100,10))
ids = tf.reshape(tf.range(100), shape=(20,5))

embedding_lookup = tf.nn.embedding_lookup(embeddings, ids)
ga = tf.gather(embeddings, ids)

with tf.Session() as sess:
  sess.graph.finalize()
  run_opts = tf.RunOptions(report_tensor_allocations_upon_oom=True)

  for i in range(10000000):
    sess.run(embedding_lookup, options=run_opts)
    #sess.run(ga, options=run_opts)

The code above leaks cpu memory quite rapidly ~3 Mb/s.

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v1.12.0-0-ga6d8ffae09 1.12.0
• Python version: 3.6.7
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: 9.0/ 7.4.2
• GPU model and memory: GTX 1070 with 8GB memory

[tilakrayal]

@bsnipers,
Since the graph executor is multi-threaded, the order in which the various operations try to allocate memory will vary from run to run, which can create different amounts of fragmentation. And this can result in OOMs non-deterministically.

You can set the number of inter op threads to 1 to see if that helps. It will reduce performance but increase determinism.

GPU operations allocate memory as they execute (so tf.add(x, y) will allocate memory for its output every time it executes, and that memory will get free after the last use of the tensor it returned).

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[JyotiPDLr]

@tilakrayal , Here the issue is related to Memory leak not OOM or memory fragmentation.

As you said here in below.

GPU operations allocate memory as they execute (so tf.add(x, y) will allocate memory for its output every time it executes, and that memory will get free after the last use of the tensor it returned).

If memory getting freed then why Memory leak happening ? That's the question here.

As mentioned in #21348 it might have resoled in TF2. May be but not sure.

[tilakrayal]

Hi,

Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.

The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.

Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.