Send raylet error logs through the log monitor

[ericl]

What do these changes do?

Now if your raylet crashes, you get the error sent via the log monitor process:

Related issue number

Closes #5322

Linter

• I've run scripts/format.sh to lint the changes in this PR.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15921/
Test PASSed.

[ericl]

(pid=raylet) WARNING: Logging before InitGoogleLogging() is written to STDERR
(pid=raylet) I0804 21:42:03.236798 32216 redis_gcs_client.cc:145] RedisGcsClient::Connect finished with status OK
(pid=raylet) WARNING: Logging before InitGoogleLogging() is written to STDERR
(pid=raylet) I0804 21:42:03.247087 32218 stats.h:48] Succeeded to initialize stats: exporter address is 127.0.0.1:8888
(pid=raylet) I0804 21:42:03.247763 32218 redis_gcs_client.cc:145] RedisGcsClient::Connect finished with status OK
(pid=raylet) I0804 21:42:03.248323 32218 grpc_server.cc:36] ObjectManager server started, listening on 0.0.0.0:44845
(pid=raylet) I0804 21:42:03.249888 32218 grpc_server.cc:36] NodeManager server started, listening on 0.0.0.0:37479
(pid=raylet) I0804 21:42:03.250210 32218 grpc_server.cc:36] Raylet server started, listening on unix:///tmp/ray/session_2019-08-04_21-42-03_009401_32194/sockets/raylet

cc @jovany-wang do you know why these log messages get written to STDERR? From what I can tell in the main, logging is initialized first thing. However, e.g., stats::Init() seems to print the above still to STDERR complaining that InitGoogleLogging hasn't been called yet.

[jovany-wang]

I looked into it and found the log_dir_ is empty here.
https://github.com/ray-project/ray/blob/master/src/ray/util/logging.cc#L126
And I can't see we pass the log dir anywhere.

I'll try to fix it later.

[jovany-wang]

btw, which file do you expect these logs to be written?

[ericl]

/tmp/ray/session_<id>/logs/raylet.out I believe (or whereever they end up after log init finishes). Eric

…

On Sun, Aug 4, 2019 at 10:49 PM Qing Wang ***@***.***> wrote: btw, which file do you expect these logs to be written? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5351?email_source=notifications&email_token=AAADUSWWTXRV5ILBDABP35LQC65OPA5CNFSM4IIZUVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QX7BY#issuecomment-518094727>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADUSVOOF7XSMV3XGZVSQTQC65OPANCNFSM4IIZUVCA> .

[jovany-wang]

@ericl I fixed it at #5374

[ericl]

@jovany-wang Thanks!

[robertnishihara]

Looks good to me (I tried it out) assuming that the messages below go away (these are currently printed at startup). I agree putting them in raylet.out makes sense. I don't think we should suppress them entirely.

(pid=raylet) I0805 13:56:15.351514 314066368 redis_gcs_client.cc:145] RedisGcsClient::Connect finished with status OK
(pid=raylet) I0805 13:56:15.362835 248083904 stats.h:48] Succeeded to initialize stats: exporter address is 127.0.0.1:8888
(pid=raylet) I0805 13:56:15.365021 248083904 redis_gcs_client.cc:145] RedisGcsClient::Connect finished with status OK
(pid=raylet) I0805 13:56:15.366199 248083904 grpc_server.cc:36] ObjectManager server started, listening on 0.0.0.0:50380
(pid=raylet) I0805 13:56:15.372203 248083904 grpc_server.cc:36] NodeManager server started, listening on 0.0.0.0:50381
(pid=raylet) I0805 13:56:15.372795 248083904 grpc_server.cc:36] Raylet server started, listening on unix:///tmp/ray/session_2019-08-05_13-56-15_102483_32599/sockets/raylet

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15985/
Test PASSed.

[ericl]

Added a StdoutLogger which is the new default logger for glog when no glog log dir is configured (as is the case by default). This achieves the desired behaviour of INFO in raylet.out.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15988/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15991/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16002/
Test PASSed.

[ericl]

Tests look unrelated.