Evaluation during training

[antoniosimunovic]

Hi,

is there a way to run model evaluation during training? It seems to me that I need to stop training process and run evaluation on checkpoints in train directory to get some metrics. It would be more elegant to run the evaluation process after X number of training steps.

[eshirima]

I just ran the eval on a separate terminal window without having to cancel the training

[antoniosimunovic]

Thanks for the tip, I am aware that it is possible to run evaluation in a different process if there is enough memory available. My point is that periodic model evaluation during training is common practice and it would be great to see how it can be implemented in this pipeline.

[cy89]

Is there a particular model you have in mind? And is there a particular set of TF libraries that you're using to help you with training?

In general, might this be a better discussion for StackOverflow? For example: https://stackoverflow.com/questions/41217953/tensorflow-evaluate-while-training-with-queues
https://stackoverflow.com/questions/42407483/tensorflow-supervisor-for-both-training-and-evaluating-operations

[antoniosimunovic]

For training I'm using standard mobilenet v1 ssd configuration for object detection on a custom training and validation dataset.

As for evaluation during training I was thinking of something along training procedure at https://github.com/davidsandberg/facenet/blob/master/src/train_softmax.py

Thanks for the links, it seems that preferred approach is to run evaluation in a separate process.

[gdelab]

@antoniosimunovic have you solved the memory problem then ? Or found another solution ?

[antoniosimunovic]

Currently I'm running evaluation on a seperate device in a different process as recommended. 2017-07-10 18:43 GMT+02:00 gdelab <notifications@github.com>:

…

@antoniosimunovic <https://github.com/antoniosimunovic> have you solved the memory problem then ? Or found another solution ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1885 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGIgjN0_STaWpNL7eqKkVZH7T3tiEyJ-ks5sMlS4gaJpZM4OQ1fh> .

[aselle]

You'd need to modify the code to do evaluation. Many people do evaluation as separate processes because it reduces the total training time by not putting an evaluation step in the middle of the training loop, but it requires another device or enough memory to run on the same device. You might try stackoverflow to see if anyone has a patch to do this, but if you extend the model to allow this option, we'd be happy to accept it.