Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: log: change Logger to be an interface #13182

Open
srinathh opened this issue Nov 7, 2015 · 15 comments

Comments

@srinathh
Copy link
Contributor

commented Nov 7, 2015

Motivation

Go stdlib provides a log.Logger that is widely used in both Go programs and Go libraries (eg. most notably in http.Server) as a standardized logging component.

However, the current implementation of the log.Logger enforces a specific logging format with limited flexibility which requires text parsing to allow it to be connected to any other logging system implementation through the swappable io.Writer interface. This adds complexity and overhead. The other alternative of directly using third party logging libraries is not very scalable both because existing libraries including components like http.Server in the stdlib expect log.Logger and because of network effects of stdlib.

Proposal

This proposes a minor change to the log package to make log.Logger more extensible in a fully backwards compatible by allowing Logger.Output() to write logs using custom logging implementations.

We do this by defining a function type called OutputFn with the same function signature as the current Logger.Output() and introducing an unexported field outputfn in Logger to hold the function that should be used for output much like we use the out field to hold the io.Writer currently. By default, log.New() will set the outputfn to the current implementation which is moved to Logger.DefOutputFn(). This can be swapped to any other OutputFn implementation by calling Logger.SetOutputFn() in the same way as we currently can swap io.Writer by calling Logger.SetOutput()

For the top level log package, we provide log.SetOutputFn() and log.SetDefOutputFn() to set the OutputFn of the log.std logger and reset it to default.

Proof of Concept

I have implemented a proof of concept for this proposed change and added an example in the test suite that demonstrates logging to a Tab Separated Variable format. While in the example i am merely writing to a Writer, the Output could be to anything including over the network etc.

If this approach looks ok, I can package this into a CL incorporating any feedback.

Rationale for the Approach

Most of the actual work of logging happens in Logger.Output() (and in the unexported Logger.formatHeader() called by it) which makes it a prime candidate for flexibility. We cannot change log.Logger into an interface due to Go1 compatibility promise and this approach therefore allows a lot of flexibility with a very minor change.

@srinathh

This comment has been minimized.

Copy link
Contributor Author

commented Nov 12, 2015

Could I request a review of the proposal? Should I submit a CL?

@adg

This comment has been minimized.

Copy link
Contributor

commented Nov 12, 2015

I don't see the advantage that this provides over the default logger. Instead of parsing colons you parse tabs?

@srinathh

This comment has been minimized.

Copy link
Contributor Author

commented Nov 12, 2015

No no.... the idea is to define a minimal function type mimicking the current Logger.Output() function that lets anyone use the standard library log package and log.Logger to do logging in any format and using any communication mechanism (eg. a logging to an API service exposed over the network) without any parsing.

This would allow for instance to pass a log.Logger to http.Server that can plug into any other logging service merely by implementing an OutputFn() and passing it to the Logger.

Purely as an example here, I have implemented a version of OutputFn() that logs into a Tab separated variable format file but the actual implementation can be anything at all.

@srinathh

This comment has been minimized.

Copy link
Contributor Author

commented Nov 23, 2015

Hi - could I pls. request another evaluation of the proposal as I believe it may have been mis-understood originally. The objective is NOT to write logs with tabs formatting or whatever. The objective is to introduce flexibility in the standard Logger to allow alternative logging implementations (which could include writing to files, over network, databases etc.) to plug into it with a minimal interface while retaining full backwards compatibility.

@rsc rsc added the Proposal label Dec 28, 2015

@rsc rsc added this to the Proposal milestone Dec 28, 2015

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 28, 2015

Like @adg, I don't believe this proposal has enough benefit to warrant the complexity suggested for the API. The log package already has a way to divert output, namely SetOutput. Instead of adding a second, why not use the first?

Note that the log package guarantees to deliver each message in a different Write call. So if you really need to tweak the format, it's possible to do today with no changes to the standard library: tweak the log message itself. Quoting the docs:

A Logger represents an active logging object that generates lines of output
to an io.Writer. Each logging operation makes a single call to the Writer's
Write method. A Logger can be used simultaneously from multiple goroutines;
it guarantees to serialize access to the Writer.

The only possible benefit I can see is that the current output writer doesn't know the call depth on each Write, so it can't reconstruct the file and line number for itself. But if that's really important, it's easy to pull them from the written data.

@rsc rsc changed the title Proposal: Make stdlib log.Logger more extensible in a backwards compatible way proposal: add SetOutputFn to log.Logger Dec 28, 2015

@srinathh

This comment has been minimized.

Copy link
Contributor Author

commented Dec 29, 2015

Thanks for reviewing Russ. Here is the benefit I see of enabling a more flexible logger output via an approach like OutputFn vs. using current SetOuptut

  • Currently, doing any non-standard logging (either in format or in action) with the log package is a four step process using the SetOutput approach.
    1. Setup the standard logger to generate the fields required for your custom logging through SetFlags() and/or SetPrefix(). The standard flags may be insufficient.
    2. Implement a custom logger that supports the writer interface & call SetOutput() with it
    3. In the custom logger, parse each line of output from the writer to extract the required fields from the string
    4. Perform whatever is the custom logging required.
  • Point 1 means the application code and the logger implementation get tightly coupled and you need to have knowledge of the input format that the custom logger implementation expects to use it. This is fragile and also depends on logging output format being guaranteed.
  • Point 3 is an unnecessary extra step and possibly a performance penalty. Why should a custom logger contain code to parse a []byte and extract timestamps and file-paths and so on vs. focusing on the mechanics of the custom logging?
  • The OutputFn type of approach lets logging implementations be swapped out cleanly without tying the implementation to the application code and also eliminates text parsing.
@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 29, 2015

With the exception of file name, you can shorten to:

  • SetFlags(0), SetPrefix("")
  • call SetOutput(w)
  • w.Write gets just the log message and can add whatever prefix or formatting it wants, without any parsing

If you want file names, then yes you have to use SetFlags(Llongfile) or Lshortfile and parse the file name. That's unfortunate, but it doesn't seem worth a whole new API. If we were starting over then yes I think we might reasonably make SetOutput take a func and have a separate SetWriter. But we're not starting over. We're working with what we have. If we created new API every time we realized we could have done something differently, we'd end up with a much larger (and arguably bloated) standard library. The bar is very high for additions.

@adg

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2016

Deferring to Go2.

@adg adg added the Go2 label Aug 15, 2016

@robpike

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2016

I think the right answer is to change Logger to be an interface, and to do that in Go 2.

@rsc rsc changed the title proposal: add SetOutputFn to log.Logger proposal: log: add SetOutputFn to Logger Jun 16, 2017

@ianlancetaylor ianlancetaylor changed the title proposal: log: add SetOutputFn to Logger proposal: log: change Logger to be an interface Jan 9, 2018

@ianlancetaylor ianlancetaylor changed the title proposal: log: change Logger to be an interface proposal: Go 2: log: change Logger to be an interface Jan 9, 2018

@universonic

This comment has been minimized.

Copy link

commented Mar 29, 2019

@robpike Totally agree. We usually use multiple third-party libraries in one project, and each of these third-party libraries usually has its own logger implementation. This is really a nightmare. A simple and generic logger interface will make this better.

@srinathh

This comment has been minimized.

Copy link
Contributor Author

commented Apr 3, 2019

I agree with the approach with Go 2. What would be the development process?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

Someone needs to develop a proposal. Past efforts have had trouble finding a coherent logging interface, as there are many different logging needs. So it's not a simple problem.

@montanaflynn

This comment has been minimized.

Copy link

commented Apr 5, 2019

There's a detailed proposal (#28412) which was well thought out in my opinion and has a decent amount of discussion in the comments that was closed as a duplicate of this issue.

Not sure where someone can develop a proposal if they get closed and referred back to here.

@gmosx

This comment has been minimized.

Copy link

commented Apr 11, 2019

Not sure where someone can develop a proposal if they get closed and referred back to here.

Maybe someone could sketch-out a preliminary proposal as a comment to this thread.

@MMulthaupt

This comment has been minimized.

Copy link

commented Apr 18, 2019

We're running into the need for a revision of the log package in a project rigtht now. Currently we have agreed on the convention that every non-main package, which is also not coupled tightly with a main-package, may only log if it wants to communicate a warning, by doing log.Printf(). Then we can set it up to be processed with log.SetOutput(pointerToMyGreatLogWriter) where we send it to logrus.Warn(). Somewhere in there, we scavenge the log.Llongfile-information from the log message string to bring it into our format. A fun little field trip through the logging pipeline, but not quite ideal: we don't know what to do with debug calls, and our log formatter has to make some guesses to figure out if the message came from a call to the standard log library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.