Caffe not saving solverstate when using tee to redirect the output

[eamadord]

Hi,

I'm using Caffe over Ubuntu 16.04, and the command I am using to train my net is the following:
$CAFFE_ROOT/build/tools/caffe train -solver solver.prototxt 2>&1 | tee file.log

However, when I try to stop the training manually using Ctrl ^C, Caffe does not generate a solverstate file, neither a caffemodel. If I omit the redirection to a file and try to stop the training, that issue does not happen, and both my solverstate and caffemodel generate successfully.

Is there a way to fix this?

Thanks in advance.

[Noiredd]

When you redirect output to tee, you're actually interacting with tee itself in the console - so Ctrl+C actually sends an interrupt signal to tee, not Caffe. And it appears that tee just kills its input process when you do that.

You can go around it by sending a SIGINT signal directly to Caffe (from another console):
kill -2 <caffe_pid>

Protip: each line of the log
I0223 15:00:57.606747 12078 sgd_solver.cpp:105] Iteration 0, lr = 0.001
contains a PID on the third position (here: 12078).