Snapshot on signal #2253

jyegerlehner · Apr 3, 2015

This an implementation of the feature discussed in Issue 2012.

When you hit Ctrl-C to kill caffe (while training), it will now save a snapshot before exiting. Actually, the Solver just stops training, and a snapshot is only saved if snapshot_after_train is true.

This is the default behavior which is configurable via the sigint_effect and sighup_effect command line options. Also by default, SIGHUP signal causes caffe to save a snapshot and continue training. So you can make caffe save a snapshot by sending it SIGHUP signal, e.g.:

kill -SIGHUP PID

where PID is the process id of caffe, which you can find by doing ps -ef | grep caffe.

The design has two pieces to it. 1. Solver is modified slightly so that after each iteration it checks to see if its client wants it to either snapshot or exit. It does this via a callback function that a client can set on the Solver instance. If the callback hasn't been set, it just carries on as usual. So there is no breaking change to the Solver interface and the behavior of existing code shouldn't change. 2. The caffe executable provides the callback function to the Solver, and the callback is implemented on a SignalHandler that intercepts SIGINT and SIGHUP.

flx42 · Apr 18, 2015

Looks cool, but I don't think you can safely lock a mutex in a signal handler, nor try to perform I/O.

jyegerlehner · Apr 18, 2015

Thanks @flx42, I should have researched signals a bit more before I implemented this.

jyegerlehner · Apr 18, 2015

@flx42 do you see any problem with this new implementation?

flx42 · Apr 18, 2015

The type should be "sig_atomic_t volatile".
Also, since the variables are no longer of type "bool", it might be better to use "1" and "0" instead of "true" and "false".

After that, I think it will be fine :)
👍

jyegerlehner · Apr 18, 2015

Thanks for the reply.

The type should be "sig_atomic_t volatile".

Hrmm.. all the examples I see, the qualifier precedes the type name, same as with "const" or "mutable". Please let me know if I'm missing something.

since the variables are no longer of type "bool", it might be better to use "1" and "0" instead of "true" and "false".

true and false are promoted to 1 and 0 per the language spec I'm pretty sure. Since they better convey the meaning, I'm inclined to leave the true and false in there.

flx42 · Apr 18, 2015

On Sat, Apr 18, 2015 at 12:48 PM, jyegerlehner notifications@github.com
wrote:

Thanks for the reply.

The type should be "sig_atomic_t volatile".

Hrmm.. all the examples I see, the qualifier precedes the type name, same
as with "const" or "mutable". Please let me know if I'm missing something.

No, you're right. That's what I meant.

since the variables are no longer of type "bool", it might be better to
use "1" and "0" instead of "true" and "false".

true and false are promoted to 1 and 0 per the language spec I'm pretty
sure. Since they better convey the meaning, I'm inclined to leave the true
and false in there.

It was really a nitpicking comment about style. But both ways will be fine.

flx42 · Apr 20, 2015

Another nitpick: you should avoid unnecessary empty lines changes in your patch.
Like this: jyegerlehner/caffe@c0df909#diff-ed792b89816c46f109918495c88421bcL3

lukeyeager · May 13, 2015

Any update on this? I want it.

jyegerlehner · May 13, 2015

Hi Luke, I'm not aware of anything deficient about it.

jyegerlehner · May 14, 2015

Well there is one thing about this that is debatable. Solver doesn't check to see if it should exit when it is doing test, only when it is training. This means if you send it SIGINT and it happens to be in the middle of test (not train), caffe keeps going until the test is finished, and only then does it exit (or snapshot). So you might have to wait a bit if you try to request it to stop when it's testing. I thought that was a good thing since it allows the test to run to completion and produce a valid test result. However, one might prefer that it be more responsive and stop right away regardless of whether it is testing, and throw away the test that is in-progress. We could make it behave that way with a bit of extra complexity.

lukeyeager · May 14, 2015

one might prefer that it be more responsive and stop right away regardless of whether it is testing, and throw away the test that is in-progress

That sounds better to me. I would expect Ctrl+C to kill the process within a second or two. If you wait for testing to finish, you might have to wait several minutes.

jyegerlehner · May 17, 2015

I would expect Ctrl+C to kill the process within a second or two.

If anyone objects to Luke's preferred behaviour please speak up. Otherwise I'll plan to make that change.

shelhamer · May 18, 2015

I can see an argument for either finishing the testing or quitting
immediately, but to better fit expectations it should likely quit right
away no matter the action.

Thanks for the signal handling!
On Sun, May 17, 2015 at 08:38 jyegerlehner notifications@github.com wrote:

I would expect Ctrl+C to kill the process within a second or two.

If anyone objects to Luke's preferred behaviour please speak up. Otherwise
I'll plan to make that change.

—
Reply to this email directly or view it on GitHub
#2253 (comment).

flx42 · May 18, 2015

It would be nice to have both, if that's not too much complexity for you.
It's not uncommon to have different shutdown behaviors depending on which signal was received. For instance, nginx will do a fast shutdown on SIGTERM/SIGINT, and a graceful shutdown on SIGQUIT (wait for all current connections to finish). Apache has a similar behavior.

jyegerlehner · May 18, 2015

Behaviour modified to respond to signals during testing or training, whereas before it just responded during training. And rebased off master.

I tested these latest changes on a couple scenarios manually. But some of this code is so new if anyone else can test it that could give more confidence.

jyegerlehner · May 18, 2015

to better fit expectations it should likely quit right away

Makes sense to me.

Thanks for the signal handling!

Anything for the cause, comrade!

jyegerlehner · May 18, 2015

It would be nice to have both, if that's not too much complexity for you.

If we find we need or want that extra sophistication, I think my preference is to add it in a separate PR so that we may thereby proceed more incrementally. The changes to Solver started out very simple, and got more complex with the latest change and feels to me like pushing the limits of added risk in one PR.

That said, if everyone really really wants it right now, I can do it.

flx42 · May 18, 2015

No, I agree, this should be done in a separate patch.

jyegerlehner · Jun 11, 2015

I think this could use a "Ready for Review" label, if those make any difference.

Delete this comment.

Don't think we need boost/thread anymore. Delete if not needed.

Line break not needed for 80 cols? Remove if not needed.

ronghanghu · Aug 15, 2015

LGTM. I shall review this PR within next week.

jyegerlehner · Aug 16, 2015

OK thanks. I will squash the commits once the review is done and we know there aren't any more changes required.

ronghanghu · Aug 20, 2015

@jyegerlehner I just did a quick review today, and tested on my machine. Looks good to me! I'll finish reviewing this PR within tomorrow.

(As far as I know, this PR won't apply to windows. But since we are not officially supporting windows at the moment, I'm not too worried about that.)

One another potential enhancement related to this PR is that we may also support learning rate adjustment on signaling in the future (in a separate PR), so that one may adjust it during training based on e.g. learning curve from log, similar to some other deep learning tools. @jeffdonahue @longjon what do you think?

I'm sort of confused here... Why do we need a private constructor declaration in SignalHandler class?

If the default constructor is not declared private, the compiler will provide it implicitly. So by preventing default construction we force the use of the constructor taking explicitly-specified signal actions. We could provide a default constructor that, say, initializes the actions to sensible defaults. But it would be unused as it is and thus unnecessary.

If the default constructor is not declared private, the compiler will provide it implicitly.

C++ does generate a default constructor but only if you don't provide one of your own. Since you've already provided one: SignalHandler(SolverParameter_Action SIGINT_action, SolverParameter_Action SIGHUP_action);, a default constructor won't be generated by compiler implicitly. So a private constructor declaration here is unnecessary.

You may verify with this simple program:

class C {
 public:
  C(int) {}
};
int main() { C c; } // compilation fails because there's no default constructor in class C.

Anyway, this is only a small issue. The PR seems great to me :)

OK thanks for the correction and explanation. I will remove the private ctor dec.

Personally, I am not very in favor of putting this enum in caffe.proto, since it is never used in our net or solver prototxt but only in C++ code. @jeffdonahue what do you think?

I'm happy to move these out of proto into regular enum declaration. Wait for word from JD?

bhack · Aug 20, 2015

@willyd How this will impact with your windows plans?

willyd · Aug 20, 2015

@bhack On windows we would need to call SetConsoleCtrlHandler to handle SIGINT but don't think there is an equivalent to SIGHUP.

A cross-plaform implementation is available in boost.asio.

bhack · Aug 20, 2015

If there is still interest in #2537 I will avoid asio solution.

Maybe "unhook" these signals in SignalHandler::~SignalHandler()?

OK will do.

ronghanghu · Aug 20, 2015

Completed a thorough pass today. This PR seems in good shape to me, handles signals via POSIX sigaction and address actions from solver's train & test loop in a call-back fashion. A side effect: this PR is platform-specific and may impact community windows ports.

jyegerlehner · Aug 21, 2015

@ronghanghu The latest commits are intended to resolve the review issues you raised. Please let us know if they are not satisfactory.

As far as the Travis build failing: this looks like it happened due to an error installing a package. Does anyone know: should I make a dummy commit to provoke it to try building again? Or am I missing something that's an actual problem I need to fix?.

ronghanghu · Aug 21, 2015

I restarted Travis CI and now all tests pass. I'll try to take a look today.

ronghanghu · Aug 22, 2015

Seems ready to me :) Please squash into one commit.

ronghanghu · Aug 22, 2015

@jeffdonahue @longjon I would like to merge this PR if you don't have other concerns.

Community windows ports can perhaps simply strip this feature with #ifdef e.g. via removing the anonymous namespace section in caffe/util/signal_handler.cpp and always returning SolverAction::NONE in SignalHandler::CheckForSignals() (and Ctrl+C still terminates the program on windows).

jyegerlehner · Aug 22, 2015

OK, do you prefer commits to be squashed?

ronghanghu · Aug 22, 2015

Yes, please squash into one commit, so that I can merge in this weekend.

erogol · Aug 31, 2015

this is great. thanks for the effort @jyegerlehner

jyegerlehner · Aug 31, 2015

Sure thing @erogol. Glad to hear it's helpful.

Coderx7 · May 23, 2016

Can you add an option to the solver so that users can take snap-shots at any given time by pressing a key combination like Ctrl-S for example?

ajtulloch · May 23, 2016

@Coderx7 I think that'd add to much complexity to the feature. If you want to snapshot at an arbitrary time without stopping training, just send SIGHUP (kill -SIGHUP $(pidof caffe))

Coderx7 · May 24, 2016

@ajtulloch yes, but at the same time it provides a very convenient and good feature to have,
in linux thats a very good idea, but how does one do the exact same thing on windows, as far as I know, windows doesn't have signals in the linux sense

jyegerlehner changed the title from Add signal handler and early exit/snapshot to Solver. to Snapshot on signal Apr 4, 2015

jyegerlehner referenced this pull request Apr 6, 2015
Closed
Snapshot on signal #2012

shelhamer added the enhancement label Apr 9, 2015

lukeyeager referenced this pull request in NVIDIA/DIGITS May 13, 2015
Closed
Upgrades to scheduler module #104
4 of 9 tasks complete

shelhamer added the JL label May 13, 2015

shelhamer added focus RH labels Aug 5, 2015

ronghanghu added the ready for review label Aug 22, 2015

ronghanghu added a commit that referenced this pull request Aug 22, 2015

ronghanghu Merge pull request #2253 from jyegerlehner/snapshot_on_signal
Snapshot on signal
12e1432

ronghanghu merged commit 12e1432 into BVLC:master Aug 22, 2015
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

jyegerlehner deleted the jyegerlehner:snapshot_on_signal branch Aug 31, 2015

This was referenced Sep 16, 2015

Closed

Build the newest Caffe with Multi-GPU support willyd/caffe-builder#9

Open

Handle signals with boost Asio #3500

@@ -66,6 +77,8 @@ class Solver {
        string SnapshotToBinaryProto();
        string SnapshotToHDF5();
        // The test routine
+    +  // stop_was_requested will be set to true iff a request to stop training
+    +  // was received whilst testing.

@@ -0,0 +1,91 @@
+    +#include <boost/bind.hpp>
+    +#include <boost/thread/mutex.hpp>
+    +#include <boost/thread/thread.hpp>

+    +#ifndef INCLUDE_CAFFE_UTIL_SIGNAL_HANDLER_H_
+    +#define INCLUDE_CAFFE_UTIL_SIGNAL_HANDLER_H_
+    +
+    +#include "caffe/proto/caffe.pb.h"
+    +#include "caffe/solver.hpp"
+    +
+    +namespace caffe {
+    +
+    +class SignalHandler {
+    + public:
+    +  // Contructor. Specify what action to take when a signal is received.
+    +  SignalHandler(SolverParameter_Action SIGINT_action,
+    +                SolverParameter_Action SIGHUP_action);
+    +  ActionCallback GetActionFunction();
+    + private:
+    +  SignalHandler();  // Not implemented.

@@ -234,6 +234,19 @@ message SolverParameter {
        // If false, don't save a snapshot after training finishes.
        optional bool snapshot_after_train = 28 [default = true];
+    +
+    +  // Enumeration of actions that a client of the Solver may request by
+    +  // implementing the Solver's action request function, which a
+    +  // a client may optionally provide in order to request early termination
+    +  // or saving a snapshot without exiting. In the executable caffe, this
+    +  // mechanism is used to allow the snapshot to be saved when stopping
+    +  // execution with a SIGINT (Ctrl-C).
+    +  enum Action {
+    +    NONE = 0; // Take no special action.
+    +    STOP = 1; // Stop training. snapshot_after_train controls whether a snapshot
+    +              // is created.
+    +    SNAPSHOT = 2; // Take a snapshot, and keep training.
+    +  }

+    +
+    +    struct sigaction sa;
+    +    // Setup the sighub handler
+    +    sa.sa_handler = &handle_signal;
+    +    // Restart the system call, if at all possible
+    +    sa.sa_flags = SA_RESTART;
+    +    // Block every signal during the handler
+    +    sigfillset(&sa.sa_mask);
+    +    // Intercept SIGHUP and SIGINT
+    +    if (sigaction(SIGHUP, &sa, NULL) == -1) {
+    +      LOG(FATAL) << "Cannot install SIGHUP handler.";
+    +    }
+    +    if (sigaction(SIGINT, &sa, NULL) == -1) {
+    +      LOG(FATAL) << "Cannot install SIGINT handler.";
+    +    }
+    +  }

BVLC/caffe

Join GitHub today

Snapshot on signal #2253

Conversation

jyegerlehner commented Apr 3, 2015

jyegerlehner changed the title from Add signal handler and early exit/snapshot to Solver. to Snapshot on signal Apr 4, 2015

jyegerlehner referenced this pull request Apr 6, 2015

shelhamer added the enhancement label Apr 9, 2015

flx42 commented Apr 18, 2015

jyegerlehner commented Apr 18, 2015

jyegerlehner commented Apr 18, 2015

flx42 commented Apr 18, 2015

jyegerlehner commented Apr 18, 2015

flx42 commented Apr 18, 2015

flx42 commented Apr 20, 2015

lukeyeager referenced this pull request in NVIDIA/DIGITS May 13, 2015

lukeyeager commented May 13, 2015

jyegerlehner commented May 13, 2015

shelhamer added the JL label May 13, 2015

jyegerlehner commented May 14, 2015

lukeyeager commented May 14, 2015

jyegerlehner commented May 17, 2015

shelhamer commented May 18, 2015

flx42 commented May 18, 2015

jyegerlehner commented May 18, 2015

jyegerlehner commented May 18, 2015

jyegerlehner commented May 18, 2015

flx42 commented May 18, 2015

jyegerlehner commented Jun 11, 2015

shelhamer added focus RH labels Aug 5, 2015

Show 1 comment Hide comment jyegerlehner commented on an outdated diff Aug 15, 2015

Show 1 comment Hide comment jyegerlehner commented on an outdated diff Aug 15, 2015

Show 1 comment Hide comment jyegerlehner commented on an outdated diff Aug 15, 2015

ronghanghu commented Aug 15, 2015

jyegerlehner commented Aug 16, 2015

ronghanghu commented Aug 20, 2015

Show 4 comments Hide comments ronghanghu and 1 other commented on an outdated diff Aug 20, 2015

Show 2 comments Hide comments ronghanghu and 1 other commented on an outdated diff Aug 20, 2015

bhack commented Aug 20, 2015

willyd commented Aug 20, 2015

bhack commented Aug 20, 2015

ronghanghu commented on the diff Aug 20, 2015

ronghanghu commented Aug 20, 2015

jyegerlehner commented Aug 21, 2015

ronghanghu commented Aug 21, 2015

ronghanghu added the ready for review label Aug 22, 2015

ronghanghu commented Aug 22, 2015

ronghanghu commented Aug 22, 2015

jyegerlehner commented Aug 22, 2015

ronghanghu commented Aug 22, 2015

ronghanghu added a commit that referenced this pull request Aug 22, 2015

Hide details View details ronghanghu merged commit 12e1432 into BVLC:master Aug 22, 2015 1 check passed

1 check passed

erogol commented Aug 31, 2015

jyegerlehner commented Aug 31, 2015

jyegerlehner deleted the jyegerlehner:snapshot_on_signal branch Aug 31, 2015

This was referenced Sep 16, 2015

Coderx7 commented May 23, 2016

ajtulloch commented May 23, 2016

Coderx7 commented May 24, 2016 • edited

jyegerlehner commented on an outdated diff Aug 15, 2015

jyegerlehner commented on an outdated diff Aug 15, 2015

jyegerlehner commented on an outdated diff Aug 15, 2015

ronghanghu and 1 other commented on an outdated diff Aug 20, 2015

ronghanghu and 1 other commented on an outdated diff Aug 20, 2015

ronghanghu merged commit `12e1432` into BVLC:master Aug 22, 2015
1 check passed