Consider using stdout in bin/build instead of launch.toml

[dzuelke]

As an alternative to writing a launch.toml, the bin/build script could be considered a proper program in the Unix sense: it writes status info (such as all build progress info) to stderr, and its program output, which more or less is the launch info, to stdout.

It's trivial to simply send everything that's written by programs or echo to stderr at the beginning of a script:

exec 1>&2

That way, buildpack authors must not pay special attention to e.g. programs they're calling misbehaving by printing informational output to stdout instead of stderr.

Same then for bin/develop, which would no longer have to write develop.toml.

/cc @jkutner

[jkutner]

@dzuelke exec 1>&2 works well for Bash, but we expect buildpacks to be written in other languages too. I'm so sure that redirection is as easy or reliable. Any solution would need to be 100% reliable because any cruft in stderr would result in invalid TOML, and fail a build.

[nebhale]

In addition, we've seen with the previous generation of buildpacks, that many authors and others who interacted with buildpacks considered the use of stdout for conveying information problematic in practice. We've seen requests to move further away from using stdout, rather than closer, so we now have input in both directions.

[dzuelke]

@jkutner but surely if you write a buildpack in another language, you wouldn't just puts or whatever text, but run it through some kind of minimalist I/O wrapper, in order for things to be properly testable etc, right?

@nebhale I'm not sure I follow. I'm saying that stdout shouldn't be used for informational output, but stderr, which is standard Unix practice, and which is what almost all programs do. It's the reason you can do curl foo | bar: data (in the case of bin/build that'd be the TOML output) goes to stdout, and status/info/error/debug messaging (such as curl's progress bar) to stderr.

[nebhale]

Sorry, I was sloppy with my terminology. What I mean to say is:

In addition, we've seen with the previous generation of buildpacks, that many authors and others who interacted with buildpacks considered the use of stdin/stdout for conveying programmatic input/output problematic in practice. We've seen requests to move further away from using stdin/stdout for programmatic input/output, rather than closer, so we now have design requests in both directions.

[sclevine]

Appreciate the feedback @dzuelke! Definitely want to make sure these interfaces are reasonable.

I'm saying that stdout shouldn't be used for informational output, but stderr, which is standard Unix practice, and which is what almost all programs do. It's the reason you can do curl foo | bar: data (in the case of bin/build that'd be the TOML output) goes to stdout, and status/info/error/debug messaging (such as curl's progress bar) to stderr.

I definitely agree with this practice for command line utilities on UNIX systems. That said, I'm not convinced that we should do this for /bin/build.

Here are some disadvantages I can think of:

• The primary responsibility of /bin/build is to execute other CLI utilities. Preserving the separation of stdout and stderr from these utilities seems desirable on cloud platforms, where logs sent to stderr are sometimes formatted differently (e.g., bold or red) compared to logs sent to stdout. In practice, many language tools (e.g., npm, gem) send logs to both stdout and stderr so they can split informational logs from diagnostic logs. Shouldn't we preserve that split?
• The entire output of /bin/build cannot be sent to stdout, just the buildpack metadata. Sending just the contents of launch.toml to stdout would split the output between stdout and disk (launch + cache layers). This seems less elegant to me than having just the launch and cache directories be the output.
• We would prevent buildpack authors from using libraries that are hardcoded to output to stdout or output to stdout by accident. Such libraries could even silently corrupt the metadata. (Note: This can be completely avoided for languages that let you make the dup2 syscall on Linux directly, like C or Go.)
• We would make writing a buildpack in a less common language difficult for users who are not familiar with output redirection.

Do those reasons make sense to you? What benefits do you see for sending the metadata to stdout?

[dzuelke]

I guess :) Thanks folks! It was just a minor nitpick really.

[sclevine]

@dzuelke #24 makes the build/detect interfaces match, so at least they are consistent 😄