K.ctc_decode() memory not released

[jkmiao]

memory not released bug with tf.python.ops.gen_ctc_ops (both with ctc_greedy_decoder and ctc_beam_search_decoder) ...

import time
from keras import backend as K

start_time = time.time()
while True:
     y_pred  = K.ctc_decode(args)
     cost_time = time.time()-start_time
     print 'cost time', cost_time   # time cost more and more with the same input args. memory cost bigger as well, why ??!! 
#  ...

when using K.ctc_decode() in a large loop, time cost longer loop become larger, why?
please help , thank you!!

[mtfcd]

I ran into the same problem. Have you found a solution?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[LiangHao92]

@jkmiao @cccong I ran into the same problem. Have you found a solution?

[mtfcd]

@LiangHao92 这个问题好久没人关注，我自己用itertools 的groupby写了一个简单的基本上接近ctc_decode的功能的函数，凑合着能用。
我在debug代码的时候发现tensorflow的每个tensor对象里都要引用其所在的graph对象，而每个graph对象都会引其包含的所有tensor对象，这样就出现一个循环引用，我怀疑就是这里出现的内存泄漏。
It seems no care about this issue in a very long time. I wirte a function that works almost the same with ctc_decode use groupby from itertools. It works for me.
When I debug the code, I found in tensorflow every tensor object references the graph object which contain the tensor object, and every graph object references all the tensor object in itself. I suspect this circular reference cause the momery leak.

[bzamecnik]

K.ctc_decode() add operation to the graph of tensors. It's mean to be called once (like defining layers) and then evaluated many times. Memory consumption is increasing due to this.

You can access it's results eg. by wrapping with K.function() or in a Model:

# ...
input_ima
input_length = Input(shape=[1], dtype='int64')
linear_output = Dense(...)
model = Model(...)
greedy_sparse_seqs, greedy_log_probs = ctc_decode(linear_output, input_length, greedy=True)
greedy_decoded_func = K.function([linear_output, input_length], [-greedy_log_probs] + greedy_sparse_seqs)

for i in range(big_number):
    ctc_outputs = ... # actual array with CTC probability map (batch_size, output width, char_count)
    greedy_log_probs, greedy_paths = greedy_decoded_func([ctc_outputs, np.array(batch_size * [output_width])])

[soulbyfeng]

@cccong 能分享一下你的实现吗？我最近也遇到了同样的问题。谢谢