cannot see log info by tf.logging to debug

[luchensk]

It looks that it failed to see the log info on STDOUT by tf.logging to debug when running by bazel run //tensorboard -- --logdir <log_path>.

Then, I tried to run by python main.py --logdir <log_path> and was able to see the log info by base_logging.getLogger as below, but still cannot see log info by tf.logging.info

TensorBoard 0.4.0a0 at http://xxx.local:6006 (Press CTRL+C to quit)

So, is there a way to enable tf.logging to see log info when tensorboard running by bazel run? Please correct me if something wrong with my operation, thanks.

BTW, it also failed to see log info by base_logging.getLogger on running by bazel run //tensorboard -- --logdir <log_path>.

[chihuahua]

Could you try starting with

bazel run //tensorboard -- \
  --logtostderr \
  --logdir <log_path>

?

[luchensk]

@chihuahua Thanks for the response. I tried to run with --logtostderr, and it was able to print some info by sys.stderr.write in main.py, but there was still nothing printed by tf.logging, such as tf.logging.info in debugger_plugin.py.

The output with --logtostderr was as follows:

$ bazel run //tensorboard -- --logdir <log_path> --debugger_data_server_grpc_port 8008 --logtostderr
INFO: Found 1 target...
Target //tensorboard:tensorboard up-to-date:
  bazel-bin/tensorboard/tensorboard
INFO: Elapsed time: 0.397s, Critical Path: 0.02s

INFO: Running command line: bazel-bin/tensorboard/tensorboard --logdir <log_path> --debugger_data_server_grpc_port 8008 --logtostderr
TensorBoard 0.4.0a0 at http://xxx.local:6006 (Press CTRL+C to quit)

[nfelt]

@luchensk I'm pretty sure that the lack of logging at INFO when using tf.logging is intentional, you can see here that we set the log level to WARN:
https://github.com/tensorflow/tensorboard/blob/master/tensorboard/util.py#L51

I don't have the full context on why that's set to WARN rather than INFO, but so it is. It sounds like we intentionally don't provide a flag for it either (see the comment inline), I'm also not sure why that is, but it's something we could possibly revisit.

What's your use case for seeing the INFO logs? If it's just for debugging purposes and you're building locally, I think you should be able to manually edit util.py to set it to INFO.

Also, I'm not sure I understand your reference to base_logging.getLogger. The "Tensorboard VVV at http://..." line appears to be written via sys.stdout.write() so it's bypassing the logging system completely. Perhaps that's what the debugger server startup line should be doing as well - if that's your specific concern, then we can probably make that change (or you're welcome to send a PR).

[luchensk]

@nfelt Sure, I will set it to INFO in util.py later, thanks.

BTW, for this issue, I was trying to use debugger server to see if tfdbg data would be shown in Graph tab from tensorboard, and found that there were just health pillls there.

[nfelt]

Sounds good, if that resolves your immediate issue I may go ahead and close this unless you'd specifically like a FR to make the minimum logging level adjustable. I will at least change the debugger plugin's use of tf.logging.info() so it just writes directly to stderr like the main startup message.

FYI for the debugger server, a big change went in just yesterday: #754 Not super familiar with it but you might want to give it another look.

[nfelt]

Closing as per the last comment, but for the record, it should suffice to just add --verbosity 0 for INFO logging to stderr or --verbosity 1 for DEBUG logging (by default, it only prints at WARNING and above).