Write in files during training

[martinjammes]

Hello,

I am using Tensorpack for FasterRCNN training and I want to know if it is possible to use the tf.io.write_file function (which returns a tf.Operation object) during training.

If I add these lines in generalized_rcnn.py, a file is written when building the graph but not during the training:

with tf.device('/cpu:0'), tf.Session(config=tf.ConfigProto(
        allow_soft_placement=True, log_device_placement=True), graph=tf.get_default_graph()).as_default():
        tf.io.write_file(file_path, text).run()

It seems that the tf.io.write_file function needs to be used with CPU and under a session, and removing allow_soft_placement=True, log_device_placement=True in the session config induce Colocation errors.

My question is: is it possible to make so that the tf.io.write_file function is run for each new image during training?

I would like to use it to store in a file the total cost associated to each image during training.

[ppwwyyxx]

is it possible to make so that the tf.io.write_file function is run for each new image during training?

Yes. You need to create the operation and make it a dependency of the training op. Session shall never be created when building the graph. You seem to have a misunderstanding of the concept of graph/session and how tensorflow runs things.

I would like to use it to store in a file the total cost associated to each image during training.

Using https://tensorpack.readthedocs.io/modules/callbacks.html#tensorpack.callbacks.DumpTensors or write a callback following DumpTensors is a much easier way to do that

[martinjammes]

Thanks a lot, DumpTensors callback was exactly what I need to do this.