Used RAM memory gradually increases killing the process

[ardianumam]

Hi,

I try to train, and the used RAM memory gradually increases until it kills the process due to out of RAM memory. I test it tf 1.12 and 1.10. Thanks.

[ardianumam]

Good news
I wanna share the cause of memory increase in train.py code (however, currently this code is already deleted). The root cause is in this code part:

_, _, _, summary = sess.run([tf.assign(rec_tensor, rec),
                            tf.assign(prec_tensor, prec),
                            tf.assign(mAP_tensor, mAP), write_op], feed_dict={is_training:True})

Putting tf.assign operation inside the training loop will create new additional graph repeatedly. So, I change those three tf.assign by using placeholder, and do feed_dict to them using rec, prec and mAP. The training time is also faster afterward.