kubectl: rationalize which messages to stderr/stdout

[alex-mohr]

A user reports to google-containers@googlegroups.com (https://groups.google.com/forum/#!topic/google-containers/qHDR-mvh7sM) the following. I verified similar behavior in kubectl v0.9.1.

I am using Kubernetes v0.7.0, when I run "kubectl" to create pod, I found:
$ kubectl create -s http://192.168.122.136:8080 -f ./mariadb-pod.yaml
I0128 22:09:44.258267 30016 restclient.go:133] Waiting for completion of operation 19 --> this is to stderr
mariadb --> this is to stdout

I do not know why the message "...Waiting for completion ..." is considered as an error, I think it should be to stdout too.

[j3ffml]

I've seen that log before, and actually, it should be log.V(4). Internal things like polling actions shouldn't appear in logs unless the command is run with --v=#

[brendandburns]

@smarterclayton @ghodss
I think another problem is actually that the kubectl outputs (at least the create.go one) is using fmt.Fprintf(...) which goes to stdout, while everything else is using glog... which goes to stderr.

What is the desired behavior? I thought that logging to stderr, and user facing output to stdout was sort of the expected behavior.

[smarterclayton]

Can sometimes be - in this case we're leveraging stdout for "things you might want to script" and stderr for "logs and pretty info". So create returns only the name of the thing created, so you can (in theory)

kubectl get pods $(kubectl create -f something.json)

Create/Delete/Update return the names of the things - get does stdout. I'm not sure these rules have to be followed (although get -q is something we need to add).

----- Original Message -----

@smarterclayton @ghodss
I think another problem is actually that the kubectl outputs (at least the
create.go one) is using fmt.Fprintf(...) which goes to stdout, while
everything else is using glog... which goes to stderr.

What is the desired behavior? I thought that logging to stderr, and user
facing output to stdout was sort of the expected behavior.

---

Reply to this email directly or view it on GitHub:
#3894 (comment)

[bgrant0607]

I do want to preserve and improve scriptability. It's a line item in cli-roadmap.md.

All user-friendly messages should go to stderr.

I'd like full objects and other machine-readable info to go to stdout.

We should provided an option to simply output names, partial URLs (for objects of multiple kinds, namespaces, etc.), whatever that can be provided back to kubectl as command-line arguments.

I could also imagine a completely silent mode that just produces a useful exit code.

[smarterclayton]

We also need -q or equiv for get so you can easily pipe to xargs. That's on my list, as well as delete all.

On Jan 28, 2015, at 8:36 PM, Brian Grant notifications@github.com wrote:

I do want to preserve and improve scriptability. It's a line item in cli-roadmap.md.

All user-friendly messages should go to stderr.

I'd like full objects and other machine-readable info to go to stdout.

We should provided an option to simply output names, partial URLs (for objects of multiple kinds, namespaces, etc.), whatever that can be provided back to kubectl as command-line arguments.

I could also imagine a completely silent mode that just produces a useful exit code.

—
Reply to this email directly or view it on GitHub.

[thockin]

My opinion: stderr is for errors, full stop.

kubectl help get -> stdout, user asked for it

kubectl gett -> stderr, user made a mistake
On Jan 28, 2015 5:45 PM, "Clayton Coleman" notifications@github.com wrote:

We also need -q or equiv for get so you can easily pipe to xargs. That's
on my list, as well as delete all.

On Jan 28, 2015, at 8:36 PM, Brian Grant notifications@github.com
wrote:

I do want to preserve and improve scriptability. It's a line item in
cli-roadmap.md.

All user-friendly messages should go to stderr.

I'd like full objects and other machine-readable info to go to stdout.

We should provided an option to simply output names, partial URLs (for
objects of multiple kinds, namespaces, etc.), whatever that can be provided
back to kubectl as command-line arguments.

I could also imagine a completely silent mode that just produces a
useful exit code.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#3894 (comment)
.

[smarterclayton]

Time to research!

http://www.jstorimer.com/blogs/workingwithcode/7766119-when-to-use-stderr-instead-of-stdout
http://en.wikipedia.org/wiki/Standard_streams

One thing that stands out - stdout is what you pipe to the next program, full stop. If you'd grep it, it would be on stdout. If you'd pipe it to xargs, it's stdout. If you need to see it, but don't care much past that... it's probably stderr.

The section on help docs in the first link matches my experience - usage "errors" are always stderr. Help output is probably more accurately stdout (I asked you for help).

[smarterclayton]

Also, if ordering matters, you probably want to keep it all on one stream. I.e. grep errors can be out of band (and potentially be flushed later than message), but trying to deliver things out of order.

[thockin]

Printing anything the user asked for is stdout. Printing the json of an
object can pipe to jq. Printing lines of a table can pipe to awk. Etc.
On Jan 28, 2015 7:13 PM, "Clayton Coleman" notifications@github.com wrote:

Also, if ordering matters, you probably want to keep it all on one stream.
I.e. grep errors can be out of band (and potentially be flushed later than
message), but trying to deliver things out of order.

Reply to this email directly or view it on GitHub
#3894 (comment)
.

[smarterclayton]

Yup, agree.

On Jan 28, 2015, at 10:35 PM, Tim Hockin notifications@github.com wrote:

Printing anything the user asked for is stdout. Printing the json of an
object can pipe to jq. Printing lines of a table can pipe to awk. Etc.
On Jan 28, 2015 7:13 PM, "Clayton Coleman" notifications@github.com wrote:

Also, if ordering matters, you probably want to keep it all on one stream.
I.e. grep errors can be out of band (and potentially be flushed later than
message), but trying to deliver things out of order.

Reply to this email directly or view it on GitHub
#3894 (comment)
.

—
Reply to this email directly or view it on GitHub.

[bgrant0607]

Help output to stdout -- sure, fine.

[smarterclayton]

Table headers should be to stderr.

----- Original Message -----

Help output to stdout -- sure, fine.

---

Reply to this email directly or view it on GitHub:
#3894 (comment)

[thockin]

no. They are not errors.

On Thu, Jan 29, 2015 at 7:37 AM, Clayton Coleman notifications@github.com
wrote:

Table headers should be to stderr.

----- Original Message -----

Help output to stdout -- sure, fine.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
#3894 (comment)
.

[smarterclayton]

I thought we just had the thing where the Unix way is to separate things that are output and part of the shell composition pattern from the things that are output and are not? How does osc get | grep Run -> returns header help me?

----- Original Message -----

no. They are not errors.

On Thu, Jan 29, 2015 at 7:37 AM, Clayton Coleman notifications@github.com
wrote:

Table headers should be to stderr.

----- Original Message -----

Help output to stdout -- sure, fine.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
#3894 (comment)
.

---

Reply to this email directly or view it on GitHub:
#3894 (comment)

[thockin]

Table headers are part of the output you asked for.

ps, df, netstate, etc all print table headers to stdout. I have never
heard of any program printing headers to stderr and rows to stdout. I
would find that INCREDIBLY annoying.

On Thu, Jan 29, 2015 at 9:45 AM, Clayton Coleman notifications@github.com
wrote:

I thought we just had the thing where the Unix way is to separate things
that are output and part of the shell composition pattern from the things
that are output and are not? How does osc get | grep Run -> returns header
help me?

----- Original Message -----

no. They are not errors.

On Thu, Jan 29, 2015 at 7:37 AM, Clayton Coleman <
notifications@github.com>
wrote:

Table headers should be to stderr.

----- Original Message -----

Help output to stdout -- sure, fine.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
<
#3894 (comment)

.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
#3894 (comment)
.

[smarterclayton]

Because you want to grep / xargs / table headers amrite? I'm not going to go to the wall for this one, but do you normally redirect things to files more often than you pipe them? Since we can turn off table headers via flag I don't really care that much.

On Jan 29, 2015, at 1:08 PM, Tim Hockin notifications@github.com wrote:

Table headers are part of the output you asked for.

ps, df, netstate, etc all print table headers to stdout. I have never
heard of any program printing headers to stderr and rows to stdout. I
would find that INCREDIBLY annoying.

On Thu, Jan 29, 2015 at 9:45 AM, Clayton Coleman notifications@github.com
wrote:

I thought we just had the thing where the Unix way is to separate things
that are output and part of the shell composition pattern from the things
that are output and are not? How does osc get | grep Run -> returns header
help me?

----- Original Message -----

no. They are not errors.

On Thu, Jan 29, 2015 at 7:37 AM, Clayton Coleman <
notifications@github.com>
wrote:

Table headers should be to stderr.

----- Original Message -----

Help output to stdout -- sure, fine.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
<
#3894 (comment)

.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
#3894 (comment)
.

—
Reply to this email directly or view it on GitHub.

[thockin]

Honestly, given a table with enough columns I actually DO want the header.
Likewise, if I am importing to CSV, etc.

I will go to the wall - dumping things that are not errors to stderr is
just wrong. :)

On Thu, Jan 29, 2015 at 10:41 AM, Clayton Coleman notifications@github.com
wrote:

Because you want to grep / xargs / table headers amrite? I'm not going to
go to the wall for this one, but do you normally redirect things to files
more often than you pipe them? Since we can turn off table headers via flag
I don't really care that much.

On Jan 29, 2015, at 1:08 PM, Tim Hockin notifications@github.com
wrote:

Table headers are part of the output you asked for.

ps, df, netstate, etc all print table headers to stdout. I have never
heard of any program printing headers to stderr and rows to stdout. I
would find that INCREDIBLY annoying.

On Thu, Jan 29, 2015 at 9:45 AM, Clayton Coleman <
notifications@github.com>
wrote:

I thought we just had the thing where the Unix way is to separate
things
that are output and part of the shell composition pattern from the
things
that are output and are not? How does osc get | grep Run -> returns
header
help me?

----- Original Message -----

no. They are not errors.

On Thu, Jan 29, 2015 at 7:37 AM, Clayton Coleman <
notifications@github.com>
wrote:

Table headers should be to stderr.

----- Original Message -----

Help output to stdout -- sure, fine.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
<

#3894 (comment)

.

---

Reply to this email directly or view it on GitHub:

#3894 (comment)

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3894#issuecomment-72070033>

.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#3894 (comment)
.

[bgrant0607]

Turning off headers is the solution for people who don't want them is fine. Or tail +2.

[zmerlynn]

Turning off headers via flag is the usual solution. I find the thought of
it going to stderr really odd, and agree with Tim's general principal of
"you asked for it" (this especially applies to --help ... I find it so
aggravating when --help gets dumped to stderr just because someone was lazy
and didn't change their usage() routine to do the right thing).

On Thu, Jan 29, 2015 at 10:55 PM, Brian Grant notifications@github.com
wrote:

Turning off headers is the solution for people who don't want them is
fine. Or tail +2.

—
Reply to this email directly or view it on GitHub
#3894 (comment)
.

[smarterclayton]

Yeah, since we have --no-headers we're good. Headers was maybe a bad example.

----- Original Message -----

Turning off headers via flag is the usual solution. I find the thought of
it going to stderr really odd, and agree with Tim's general principal of
"you asked for it" (this especially applies to --help ... I find it so
aggravating when --help gets dumped to stderr just because someone was lazy
and didn't change their usage() routine to do the right thing).

On Thu, Jan 29, 2015 at 10:55 PM, Brian Grant notifications@github.com
wrote:

Turning off headers is the solution for people who don't want them is
fine. Or tail +2.

—
Reply to this email directly or view it on GitHub
#3894 (comment)
.

---

Reply to this email directly or view it on GitHub:
#3894 (comment)

[ghodss]

So are we in fact saying that glog going to stderr is fine, and therefore this issue can be closed? Or should we parse out some glog messages (like info) to go to stdout? Just not clear on what actually needs to be changed here given the above thread.

[smarterclayton]

The glog prefixes on CLIs are terrible. Unless you're at --v=3 or higher writing the standard glog gorp to the screen is abusive to new users.

On Feb 11, 2015, at 7:22 PM, Sam Ghods notifications@github.com wrote:

So are we in fact saying that glog going to stderr is fine, and therefore this issue can be closed? Or should we parse out some glog messages (like info) to go to stdout? Just not clear on what actually needs to be changed here given the above thread.

—
Reply to this email directly or view it on GitHub.

[thockin]

This - all glog stuff < ERROR should be silent unless -v is set, though we
can debate the -v level impact

On Wed, Feb 11, 2015 at 6:02 PM, Clayton Coleman notifications@github.com
wrote:

The glog prefixes on CLIs are terrible. Unless you're at --v=3 or higher
writing the standard glog gorp to the screen is abusive to new users.

On Feb 11, 2015, at 7:22 PM, Sam Ghods notifications@github.com
wrote:

So are we in fact saying that glog going to stderr is fine, and
therefore this issue can be closed? Or should we parse out some glog
messages (like info) to go to stdout? Just not clear on what actually needs
to be changed here given the above thread.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#3894 (comment)
.

[smarterclayton]

Yeah, i just picked 3 randomly cause we said 2 is "normal" operation

----- Original Message -----

This - all glog stuff < ERROR should be silent unless -v is set, though we
can debate the -v level impact

On Wed, Feb 11, 2015 at 6:02 PM, Clayton Coleman notifications@github.com
wrote:

The glog prefixes on CLIs are terrible. Unless you're at --v=3 or higher
writing the standard glog gorp to the screen is abusive to new users.

On Feb 11, 2015, at 7:22 PM, Sam Ghods notifications@github.com
wrote:

So are we in fact saying that glog going to stderr is fine, and
therefore this issue can be closed? Or should we parse out some glog
messages (like info) to go to stdout? Just not clear on what actually needs
to be changed here given the above thread.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHub
#3894 (comment)
.

---

Reply to this email directly or view it on GitHub:
#3894 (comment)

[ghodss]

Seems like the specific info message mentioned at the top of this ticket was removed in #3921. Combined with the fact that we don't plan on changing the current stdout/stderr split seems means we can close this issue. If we want to change glog prefixes we can probably make a new issue for that.

[j3ffml]

Since the purpose of this issue was to decide what messages go to stdout/stderr, I think we can close this out and open separate bugs for specific output that isn't doing the right thing. Summary of decisions:

• Only errors, (log.fatal, log.error) go to stderr
• Anything the user requested goes to stdout. This includes table headers (they can be turned off via --no-headers), periodic updates ('waiting for blah...'), and crucially, --help output.
• All glog output < ERROR should be silent unless -v is set.
• glog prefixes should not be shown except in -v output.

[smarterclayton]

glog prefixing

----- Original Message -----

Since the purpose of this issue was to decide what messages go to
stdout/stderr, I think we can close this out and open separate bugs for
specific output that isn't doing the right thing. Summary of decisions:

• Only errors, (log.fatal, log.error) go to stderr
• Anything the user requested goes to stdout. This includes table headers
  (they can be turned off via --no-headers), periodic updates ('waiting for
  blah...'), and crucially, --help output.
• All glog output < ERROR should be silent unless -v is set.

---

Reply to this email directly or view it on GitHub:
#3894 (comment)