Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: log: change Logger to be an interface #13182

Open
srinathh opened this issue Nov 7, 2015 · 19 comments
Open

proposal: Go 2: log: change Logger to be an interface #13182

srinathh opened this issue Nov 7, 2015 · 19 comments

Comments

@srinathh
Copy link
Contributor

@srinathh srinathh commented Nov 7, 2015

Motivation

Go stdlib provides a log.Logger that is widely used in both Go programs and Go libraries (eg. most notably in http.Server) as a standardized logging component.

However, the current implementation of the log.Logger enforces a specific logging format with limited flexibility which requires text parsing to allow it to be connected to any other logging system implementation through the swappable io.Writer interface. This adds complexity and overhead. The other alternative of directly using third party logging libraries is not very scalable both because existing libraries including components like http.Server in the stdlib expect log.Logger and because of network effects of stdlib.

Proposal

This proposes a minor change to the log package to make log.Logger more extensible in a fully backwards compatible by allowing Logger.Output() to write logs using custom logging implementations.

We do this by defining a function type called OutputFn with the same function signature as the current Logger.Output() and introducing an unexported field outputfn in Logger to hold the function that should be used for output much like we use the out field to hold the io.Writer currently. By default, log.New() will set the outputfn to the current implementation which is moved to Logger.DefOutputFn(). This can be swapped to any other OutputFn implementation by calling Logger.SetOutputFn() in the same way as we currently can swap io.Writer by calling Logger.SetOutput()

For the top level log package, we provide log.SetOutputFn() and log.SetDefOutputFn() to set the OutputFn of the log.std logger and reset it to default.

Proof of Concept

I have implemented a proof of concept for this proposed change and added an example in the test suite that demonstrates logging to a Tab Separated Variable format. While in the example i am merely writing to a Writer, the Output could be to anything including over the network etc.

If this approach looks ok, I can package this into a CL incorporating any feedback.

Rationale for the Approach

Most of the actual work of logging happens in Logger.Output() (and in the unexported Logger.formatHeader() called by it) which makes it a prime candidate for flexibility. We cannot change log.Logger into an interface due to Go1 compatibility promise and this approach therefore allows a lot of flexibility with a very minor change.

@srinathh
Copy link
Contributor Author

@srinathh srinathh commented Nov 12, 2015

Could I request a review of the proposal? Should I submit a CL?

@adg
Copy link
Contributor

@adg adg commented Nov 12, 2015

I don't see the advantage that this provides over the default logger. Instead of parsing colons you parse tabs?

@srinathh
Copy link
Contributor Author

@srinathh srinathh commented Nov 12, 2015

No no.... the idea is to define a minimal function type mimicking the current Logger.Output() function that lets anyone use the standard library log package and log.Logger to do logging in any format and using any communication mechanism (eg. a logging to an API service exposed over the network) without any parsing.

This would allow for instance to pass a log.Logger to http.Server that can plug into any other logging service merely by implementing an OutputFn() and passing it to the Logger.

Purely as an example here, I have implemented a version of OutputFn() that logs into a Tab separated variable format file but the actual implementation can be anything at all.

@srinathh
Copy link
Contributor Author

@srinathh srinathh commented Nov 23, 2015

Hi - could I pls. request another evaluation of the proposal as I believe it may have been mis-understood originally. The objective is NOT to write logs with tabs formatting or whatever. The objective is to introduce flexibility in the standard Logger to allow alternative logging implementations (which could include writing to files, over network, databases etc.) to plug into it with a minimal interface while retaining full backwards compatibility.

@rsc rsc added the Proposal label Dec 28, 2015
@rsc rsc added this to the Proposal milestone Dec 28, 2015
@rsc
Copy link
Contributor

@rsc rsc commented Dec 28, 2015

Like @adg, I don't believe this proposal has enough benefit to warrant the complexity suggested for the API. The log package already has a way to divert output, namely SetOutput. Instead of adding a second, why not use the first?

Note that the log package guarantees to deliver each message in a different Write call. So if you really need to tweak the format, it's possible to do today with no changes to the standard library: tweak the log message itself. Quoting the docs:

A Logger represents an active logging object that generates lines of output
to an io.Writer. Each logging operation makes a single call to the Writer's
Write method. A Logger can be used simultaneously from multiple goroutines;
it guarantees to serialize access to the Writer.

The only possible benefit I can see is that the current output writer doesn't know the call depth on each Write, so it can't reconstruct the file and line number for itself. But if that's really important, it's easy to pull them from the written data.

@rsc rsc changed the title Proposal: Make stdlib log.Logger more extensible in a backwards compatible way proposal: add SetOutputFn to log.Logger Dec 28, 2015
@srinathh
Copy link
Contributor Author

@srinathh srinathh commented Dec 29, 2015

Thanks for reviewing Russ. Here is the benefit I see of enabling a more flexible logger output via an approach like OutputFn vs. using current SetOuptut

  • Currently, doing any non-standard logging (either in format or in action) with the log package is a four step process using the SetOutput approach.
    1. Setup the standard logger to generate the fields required for your custom logging through SetFlags() and/or SetPrefix(). The standard flags may be insufficient.
    2. Implement a custom logger that supports the writer interface & call SetOutput() with it
    3. In the custom logger, parse each line of output from the writer to extract the required fields from the string
    4. Perform whatever is the custom logging required.
  • Point 1 means the application code and the logger implementation get tightly coupled and you need to have knowledge of the input format that the custom logger implementation expects to use it. This is fragile and also depends on logging output format being guaranteed.
  • Point 3 is an unnecessary extra step and possibly a performance penalty. Why should a custom logger contain code to parse a []byte and extract timestamps and file-paths and so on vs. focusing on the mechanics of the custom logging?
  • The OutputFn type of approach lets logging implementations be swapped out cleanly without tying the implementation to the application code and also eliminates text parsing.
@rsc
Copy link
Contributor

@rsc rsc commented Dec 29, 2015

With the exception of file name, you can shorten to:

  • SetFlags(0), SetPrefix("")
  • call SetOutput(w)
  • w.Write gets just the log message and can add whatever prefix or formatting it wants, without any parsing

If you want file names, then yes you have to use SetFlags(Llongfile) or Lshortfile and parse the file name. That's unfortunate, but it doesn't seem worth a whole new API. If we were starting over then yes I think we might reasonably make SetOutput take a func and have a separate SetWriter. But we're not starting over. We're working with what we have. If we created new API every time we realized we could have done something differently, we'd end up with a much larger (and arguably bloated) standard library. The bar is very high for additions.

@adg
Copy link
Contributor

@adg adg commented Aug 15, 2016

Deferring to Go2.

@adg adg added the Go2 label Aug 15, 2016
@robpike
Copy link
Contributor

@robpike robpike commented Aug 15, 2016

I think the right answer is to change Logger to be an interface, and to do that in Go 2.

@rsc rsc changed the title proposal: add SetOutputFn to log.Logger proposal: log: add SetOutputFn to Logger Jun 16, 2017
@ianlancetaylor ianlancetaylor changed the title proposal: log: add SetOutputFn to Logger proposal: log: change Logger to be an interface Jan 9, 2018
@ianlancetaylor ianlancetaylor changed the title proposal: log: change Logger to be an interface proposal: Go 2: log: change Logger to be an interface Jan 9, 2018
@universonic
Copy link

@universonic universonic commented Mar 29, 2019

@robpike Totally agree. We usually use multiple third-party libraries in one project, and each of these third-party libraries usually has its own logger implementation. This is really a nightmare. A simple and generic logger interface will make this better.

@srinathh
Copy link
Contributor Author

@srinathh srinathh commented Apr 3, 2019

I agree with the approach with Go 2. What would be the development process?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 3, 2019

Someone needs to develop a proposal. Past efforts have had trouble finding a coherent logging interface, as there are many different logging needs. So it's not a simple problem.

@montanaflynn
Copy link

@montanaflynn montanaflynn commented Apr 5, 2019

There's a detailed proposal (#28412) which was well thought out in my opinion and has a decent amount of discussion in the comments that was closed as a duplicate of this issue.

Not sure where someone can develop a proposal if they get closed and referred back to here.

@gmosx
Copy link

@gmosx gmosx commented Apr 11, 2019

Not sure where someone can develop a proposal if they get closed and referred back to here.

Maybe someone could sketch-out a preliminary proposal as a comment to this thread.

@MMulthaupt
Copy link

@MMulthaupt MMulthaupt commented Apr 18, 2019

We're running into the need for a revision of the log package in a project rigtht now. Currently we have agreed on the convention that every non-main package, which is also not coupled tightly with a main-package, may only log if it wants to communicate a warning, by doing log.Printf(). Then we can set it up to be processed with log.SetOutput(pointerToMyGreatLogWriter) where we send it to logrus.Warn(). Somewhere in there, we scavenge the log.Llongfile-information from the log message string to bring it into our format. A fun little field trip through the logging pipeline, but not quite ideal: we don't know what to do with debug calls, and our log formatter has to make some guesses to figure out if the message came from a call to the standard log library.

@runner-mei
Copy link

@runner-mei runner-mei commented Nov 25, 2019

we need http://www.slf4j.org

@zimbatm
Copy link

@zimbatm zimbatm commented Apr 15, 2020

Logging is a bit of an overloaded term which creates quite a bit of confusion around what it really is. I don't think it's productive to try and define one "true" interface that would encompass all of its uses.

Instead, I propose the following changes to Go:

  1. Introduce a new fmt.Printer interface that would look like this:
    type Printer interface {
      Println(v ...interface{})
      Printf(format string, v ...interface{})
    }
  2. Add log.Default() that returns the default log.std logger.

This is the minimal subset that allows passing a log.Logger around to libraries. Since the consumers would import fmt, it means that in most cases functions can take log fmt.Printer as an argument without clashing with the log module. It also doesn't break backward-compatibility and can be added to Go 1.

Eg:

package xyz
import "fmt"

func MyLibFunction(log fmt.Printer, arg1 string) error {
  log.Println("something happened", arg1)
  return nil
}

This stays in the tradition of Go of defining very lean interfaces.

Functions such as log.Fatal and log.Panic are ignored as they shouldn't be used in library functions. Instead, the library should return an error.

Features such as log levels, tracing, and structured logging are also out of the scope of this proposal.

@k80w
Copy link

@k80w k80w commented Apr 16, 2020

   ```go
   type Printer interface {
    Println(v ...interface{})
    Printf(format string, v ...interface{})
   }
   ```

is a Printf method even necessary? I can't imagine a use case where it would make sense for a Printer to use an implementation of Printf that's different from fmt.Printf, so I'm assuming this is just for convenience. Would it be better for Printer to only implement Print and have a fmt.Pprintf(printer Printer, format string, v ...interface{}) method, similar to fmt.Fprintf? I feel it should be as difficult as possible to make a Printer that is non-compliant with fmt.Printf so that all packages can expect their output to be formatted properly, but in this case this would come at the cost of an admittedly less convenient syntax.

@zimbatm
Copy link

@zimbatm zimbatm commented Apr 19, 2020

Println and Printf have different data-structures that the logger might want to capture. A NullLogger would discard the arguments altogether and save the printf interpolation overhead. A JSONLogger might output different data structures. etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.