Memory leak when training simple LSTM Network #33139
Description
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No custom code written
== check python ===================================================
python version: 3.7.4
python build version: ('default', 'Aug 13 2019 20:35:49')
python compiler version: GCC 7.3.0
python implementation: CPython
== check os platform ===============================================
os: Linux
os kernel version: #31 18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019
os release version: 5.0.0-29-generic
os platform: Linux-5.0.0-29-generic-x86_64-with-debian-buster-sid
linux distribution: ('debian', 'buster/sid', '')
linux os distribution: ('debian', 'buster/sid', '')
architecture: ('64bit', '')
machine: x86_64
== are we in docker =============================================
No
== compiler =====================================================
c++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== check pips ===================================================
numpy 1.16.4
protobuf 3.8.0
tensorflow 1.14.0
tensorflow-estimator 1.14.0
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.version.VERSION = 1.14.0
tf.version.GIT_VERSION = unknown
tf.version.COMPILER_VERSION = 5.4.0
Describe the current behavior
When I run the code below the memory usage increases each epoch until my system is unresponsive.
Code to reproduce the issue
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, TimeDistributed, Dropout
import numpy as np
num_features = 205
time_lenth = 12
num_of_instances = 5000
trip_sets = np.random.rand(num_of_instances, time_lenth, num_features)
print('num features: ', num_features)
data_len = len(trip_sets)
test_split = np.arange(data_len)
np.random.shuffle(test_split)
new_dataset = np.array(trip_sets)
targets = np.random.rand(num_of_instances, time_lenth, 1)
test_data = new_dataset[test_split[:int(data_len*0.2)]]
y_test_data = targets[test_split[:int(data_len*0.2)]]
train_data = new_dataset[test_split[int(data_len*0.2):]]
y_train_data = targets[test_split[int(data_len*0.2):]]
model = Sequential()
model.add(LSTM(75, return_sequences=True, input_shape=(None, num_features)))
model.add(Dropout(0.3))
model.add(LSTM(75, return_sequences=True))
model.add(Dropout(0.3))
model.add(TimeDistributed(Dense(1)))
# Memory leak also occurs if i use a model.add(Dense(1)) below instead of Time Distributed
# model.add(TimeDistributed(Dense(1)))
adam = tf.keras.optimizers.Adam(lr=0.001)
model.compile(loss='mse', optimizer=adam)
history = model.fit(x=train_data, y=y_train_data, epochs=100, validation_data=(test_data,y_test_data))