structured, leveled logging

[jba]

This discussion has led to a proposal and is now finished. Please comment on the proposal.

We would like to add structured logging with levels to the standard library. Structured logging is the ability to output logs with machine-readable structure, typically key-value pairs, in addition to a human-readable message. Structured logs can be parsed, filtered, searched and analyzed faster and more reliably than logs designed only for people to read. For many programs that aren't run directly by person, like servers, logging is the main way for developers to observe the detailed behavior of the system, and often the first place they go to debug it. Logs therefore tend to be voluminous, and the ability to search and filter them quickly is essential.

In theory, one can produce structured logs with any logging package:

log.Printf(`{"message": %q, "count": %d}`, msg, count)

In practice, this is too tedious and error-prone, so structured logging packages provide an API for expressing key-value pairs. This draft proposal contains such an API.

We also propose generalizing the logging "backend." The log package provides control only over the io.Writer that logs are written to. In the new package (tentative name: log/slog), every logger has a handler that can process a log event however it wishes. Although it is possible to have a structured logger with a fixed backend (for instance, zerolog outputs only JSON), having a flexible backend provides several benefits: programs can display the logs in a variety of formats, convert them to an RPC message for a network logging service, store them for later processing, and add to or modify the data.

Lastly, we include levels in our design, in a way that accommodates both traditional named levels and logr-style verbosities.

Our goals are:

• Ease of use. A survey of the existing logging packages shows that programmers want an API that is light on the page and easy to understand. This proposal adopts the most popular way to express key-value pairs, alternating keys and values.

• High performance. The API has been designed to minimize allocation and locking. It provides an alternative to alternating keys and values that is more cumbersome but faster (similar to Zap's Fields).

• Integration with runtime tracing. The Go team is developing an improved runtime tracing system. Logs from this package will be incorporated seamlessly into those traces, giving developers the ability to correlate their program's actions with the behavior of the runtime.

What Does Success Look Like?

Go has many popular structured logging packages, all good at what they do. We do not expect developers to rewrite their existing third-party structured logging code en masse to use this new package. We expect existing logging packages to coexist with this one for the foreseeable future.

We have tried to provide an API that is pleasant enough to prefer to existing packages in new code, if only to avoid a dependency. (Some developers may find the runtime tracing integration compelling.) We also expect newcomers to Go to become familiar with this package before learning third-party packages, so they will naturally prefer it.

But more important than any traction gained by the "frontend" is the promise of a common "backend." An application with many dependencies may find that it has linked in many logging packages. If all of the packages support the handler interface we propose, then the application can create a single handler and install it once for each logging library to get consistent logging across all its dependencies. Since this happens in the application's main function, the benefits of a unified backend can be obtained with minimal code churn. We hope that this proposal's handlers will be implemented for all popular logging formats and network protocols, and that every common logging framework will provide a shim from their own backend to a handler. Then the Go logging community can work together to build high-quality backends that all can share.

Prior Work

The existing log package has been in the standard library since the release of Go 1 in March 2012. It provides formatted logging, but not structured logging or levels.

Logrus, one of the first structured logging packages, showed how an API could add structure while preserving the formatted printing of the log package. It uses maps to hold key-value pairs, which is relatively inefficient.

Zap grew out of Uber's frustration with the slow log times of their high-performance servers. It showed how a logger that avoided allocations could be very fast.

zerolog reduced allocations even further, but at the cost of reducing the flexibility of the logging backend.

All the above loggers include named levels along with key-value pairs. Logr and Google's own glog use integer verbosities instead of named levels, providing a more fine-grained approach to filtering high-detail logs.

Other popular logging packages are Go-kit's log, HashiCorp's hclog, and klog.

Overview of the Design

Here is a short program that uses some of the new API:

import "log/slog"

func main() {
    slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr)))
    slog.Info("hello", "name", "Al")
    slog.Error("oops", net.ErrClosed, "status", 500)
    slog.LogAttrs(slog.ErrorLevel, "oops",
        slog.Int("status", 500), slog.Any("err", net.ErrClosed))
}

It begins by setting the default logger to one that writes log records in an easy-to-read format similar to logfmt . (There is also a built-in handler for JSON.)

The program then outputs three log messages augmented with key-value pairs. The first logs at the Info level, passing a single key-value pair along with the message. The second logs at the Error level, passing an error and a key-value pair.

The third produces the same output as the second, but more efficiently. Functions like Any and Int construct slog.Attr values, which are key-value pairs that avoid memory allocation for some values. slog.Attr is modeled on zap.Field.

The Design

Interaction Between Existing and New Behavior

The slog package works to ensure consistent output with the log package. Writing to slog's default logger without setting a handler will write structured text to log's default logger. Once a handler is set, as in the example above, the default log logger will send its text output to the structured handler.

Handlers

A slog.Handler describes the logging backend. It is defined as:

type Handler interface {
    // Enabled reports whether this handler is accepting records.
    Enabled(Level) bool

    // Handle processes the Record.
    Handle(Record) error

    // With returns a new Handler whose attributes consist of                                                                                                                        
    // the receiver's attributes concatenated with the arguments.                                                                                                                              
    With(attrs []Attr) Handler
}

The main method is Handle. It accepts a slog.Record with the timestamp, message, level, caller source position, and key-value pairs of the log event. Each call to a Logger output method, like Info, Error or LogAttrs, creates a Record and invokes the Handle method.

The Enabled method is an optimization that can save effort if the log event should be discarded. Enabled is called early, before any arguments are processed.

The With method is called by Logger.With, discussed below.

The slog package provides two handlers, one for simple textual output and one for JSON. They are described in more detail below.

The Record Type

The Record passed to a handler exports Time, Message and Level methods, as well as four methods for accessing the sequence of Attrs:

• Attrs() []Attr returns a copy of the Attrs as a slice.
• NumAttrs() int returns the number of Attrs.
• Attr(int) Attr returns the i'th Attr.
• SetAttrs([]Attr) replaces the sequence of Attrs with the given slice.

This API allows an efficient implementation of the Attr sequence that avoids copying and minimizes allocation. SetAttrs supports "middleware" handlers that want to alter the Attrs, say by removing those that contain sensitive data.

The Attr Type

The Attr type efficiently represents a key-value pair. The key is a string. The value can be any type, but Attr improves on any by storing common types without allocating memory. In particular, integer types and strings, which account for the vast majority of values in log messages, do not require allocation. The default version of Attr uses package unsafe to store any value in three machine words. The version without unsafe requires five.

There are convenience functions for constructing Attrs with various value types:

• Int(k string, v int) Attr
• Int64(k string, v int64) Attr
• Uint64(k string, v uint64) Attr
• Float64(k string, v float64) Attr
• String(k, v string) Attr
• Bool(k string, v bool) Attr
• Duration(k string, v time.Duration) Attr
• Time(k string, v time.Time) Attr
• Any(k string, v any) Attr

The last of these dispatches on the type of v, using a more efficient representation if Attr supports it and falling back to an any field in Attr if not.

The Attr.Key method returns the key. Extracting values from an Attr is reminiscent of reflect.Value: there is a Kind method that returns an enum, and a variety of methods like Int64() int64 and Bool() bool that return the value or panic if it is the wrong kind.

Attr also has an Equal method, and an AppendValue method that efficiently appends a string representation of the value to a []byte, in the manner of the strconv.AppendX functions.

Loggers

A Logger consists of a handler and a list of Attrs. There is a default logger with no attributes whose handler writes to the default log.Logger, as explained above. Create a Logger with New:

func New(h Handler) *Logger

To add attributes to a Logger, use With:

l2 := l1.With("url", "http://example.com/")

The arguments are interpreted as alternating string keys and and arbitrary values, which are converted to Attrs. Attrs can also be passed directly. Loggers are immutable, so this actually creates a new Logger with the additional attributes. To allow handlers to preprocess attributes, the new Logger’s handler is obtained by calling Handler.With on the old one.

You can obtain a logger's handler with Logger.Handler.

The basic logging methods are

func (*Logger) Log(level Level, message string, kvs ...any)

which logs a message at the given level with a list of attributes that are interpreted just as in Logger.With, and the more efficient

func (Logger) LogAttrs(level Level, message string, attrs ...Attr)

These functions first call Handler.Enabled(level) to see if they should proceed. If so, they create a Record with the current time, the given level and message, and a list of attributes that consists of the receiver's attributes followed by the argument attributes. They then pass the Record to Handler.Handle.

Each of these methods has an alternative form that takes a call depth, so other functions can wrap them and adjust the source line information.

There are four convenience methods for common levels:

func (*Logger) Info(message string, kvs ...any)
func (*Logger) Warn(message string, kvs ...any)
func (*Logger) Debug(message string, kvs ...any)
func (*Logger) Error(message string, err error, kvs ...any)

They all call Log with the appropriate level. Error first appends Any("err", err) to the attributes.

There are no convenience methods for LogAttrs. We expect that most programmers will use the more convenient API; those few who need the extra speed will have to type more, or provide wrapper functions.

All the methods described in this section are also names of top-level functions that call the corresponding method on the default logger.

Context Support

Passing a logger in a context.Context is a common practice and a good way to include dynamically scoped information in log messages. For instance, you could construct a Logger with information from an http.Request and pass it through the code that handles the request by adding it to r.Context().

The slog package has two functions to support this pattern. One adds a Logger to a context:

func NewContext(ctx context.Context, l *Logger) context.Context

As an example, an HTTP server might want to create a new Logger for each request. The logger would contain request-wide attributes and be stored in the context for the request:

func handle(w http.ResponseWriter, r *http.Request) {
    rlogger := slog.With(
        "method", r.Method,
        "url", r.URL,
        "traceID", getTraceID(r))
    ctx := slog.NewContext(r.Context(), rlogger)
    // ... use ctx ...
}

To retrieve a Logger from a context, call FromContext:

slog.FromContext(ctx).Info(...)

FromContext returns the default logger if it can't find one in the context.

Levels

A level is a positive integer, where lower numbers designate more severe or important log events. The slog package provides names for common levels, with gaps between the assigned numbers to accommodate other level schemes. (For example, Google Cloud Platform supports a Notice level between Info and Warn.)

Some logging packages like glog and Logr use verbosities instead, where a verbosity of 0 corresponds to the Info level and higher values represent less important messages. To use a verbosity of v with this design, pass slog.InfoLevel + v to Log or LogAttrs.

Provided Handlers

The slog package includes two handlers, which behave similarly except for their output format. TextHandler emits attributes as KEY=VALUE, and JSONHandler writes line-delimited JSON objects. Both can be configured with the same options:

• The boolean AddSource option controls whether the file and line of the log call. It is false by default, because there is a small cost to extracting this information.

• The LevelRef option, of type LevelRef, provides control over the maximum level that the handler will output. For example, setting a handler's LevelRef to Info will suppress output at Debug and higher levels. A LevelRef is a safely mutable pointer to a level, which makes it easy to dynamically and atomically change the logging level for an entire program.

• To provide fine control over output, the ReplaceAttr option is a function that both accepts and returns an Attr. If present, it is called for every attribute in the log record, including the four built-in ones for time, message, level and (if AddSource is true) the source position. ReplaceAttr can be used to change the default keys of the built-in attributes, convert types (for example, to replace a time.Time with the integer seconds since the Unix epoch), sanitize personal information, or remove attributes from the output.

Interoperating with Other Log Packages

As stated earlier, we expect that this package will interoperate with other log packages.

One way that could happen is for another package's frontend to send slog.Records to a slog.Handler. For instance, a logr.LogSink implementation could construct a Record from a message and list of keys and values, and pass it to a Handler. To facilitate that, slog provides a way to construct Records directly and add attributes to it:

func NewRecord(t time.Time, level Level, msg string, calldepth int) Record

func (*Record) AddAttr(Attr)

Another way for two log packages to work together is for the other package to wrap its backend as a slog.Handler, so users could write code with the slog package's API but connect the results to an existing logr.LogSink, for example. This involves writing a slog.Handler that wraps the other logger's backend. Doing so doesn't seem to require any additional support from this package.

Acknowledgements

Ian Cottrell's ideas about high-performance observability, captured in the golang.org/x/exp/event package, informed a great deal of the design and implementation of this proposal.

Seth Vargo’s ideas on logging were a source of motivation and inspiration. His comments on an earlier draft helped improve the proposal.

Michael Knyszek explained how logging could work with runtime tracing.

Tim Hockin helped us understand logr's design choices, which led to significant improvements.

Abhinav Gupta helped me understand Zap in depth, which informed the design.

Russ Cox provided valuable feedback and helped shape the final design.

Appendix: API

package slog // import "golang.org/x/exp/slog"


FUNCTIONS

func Debug(msg string, args ...any)
    Debug calls Logger.Debug on the default logger.

func Error(msg string, err error, args ...any)
    Error calls Logger.Error on the default logger.

func Info(msg string, args ...any)
    Info calls Logger.Info on the default logger.

func Log(level Level, msg string, args ...any)
    Log calls Logger.Log on the default logger.

func LogAttrs(level Level, msg string, attrs ...Attr)
    LogAttrs calls Logger.LogAttrs on the default logger.

func NewContext(ctx context.Context, l *Logger) context.Context
    NewContext returns a context that contains the given Logger. Use FromContext
    to retrieve the Logger.

func SetDefault(l *Logger)
    SetDefault makes l the default Logger. After this call, output from the
    log package's default Logger (as with log.Print, etc.) will be logged at
    InfoLevel using l's Handler.

func Warn(msg string, args ...any)
    Warn calls Logger.Warn on the default logger.


TYPES

type Attr struct {
	// Has unexported fields.
}
    An Attr is a key-value pair. It can represent some small values without an
    allocation. The zero Attr has a key of "" and a value of nil.

func Any(key string, value any) Attr
    Any returns an Attr for the supplied value.

    Any does not preserve the exact type of integral values. All signed integers
    are converted to int64 and all unsigned integers to uint64. Similarly,
    float32s are converted to float64.

    However, named types are preserved. So given

        type Int int

    the expression

        log.Any("k", Int(1)).Value()

    will return Int(1).

func Bool(key string, value bool) Attr
    Bool returns an Attr for a bool.

func Duration(key string, value time.Duration) Attr
    Duration returns an Attr for a time.Duration.

func Float64(key string, value float64) Attr
    Float64 returns an Attr for a floating-point number.

func Int(key string, value int) Attr
    Int converts an int to an int64 and returns an Attr with that value.

func Int64(key string, value int64) Attr
    Int64 returns an Attr for an int64.

func String(key, value string) Attr
    String returns a new Attr for a string.

func Time(key string, value time.Time) Attr
    Time returns an Attr for a time.Time.

func Uint64(key string, value uint64) Attr
    Uint64 returns an Attr for a uint64.

func (a Attr) AppendValue(dst []byte) []byte
    AppendValue appends a text representation of the Attr's value to dst.
    The value is formatted as with fmt.Sprint.

func (a Attr) Bool() bool
    Bool returns the Attr's value as a bool. It panics if the value is not a
    bool.

func (a Attr) Duration() time.Duration
    Duration returns the Attr's value as a time.Duration. It panics if the value
    is not a time.Duration.

func (a1 Attr) Equal(a2 Attr) bool
    Equal reports whether two Attrs have equal keys and values.

func (a Attr) Float64() float64
    Float64 returns the Attr's value as a float64. It panics if the value is not
    a float64.

func (a Attr) Format(s fmt.State, verb rune)
    Format implements fmt.Formatter. It formats a Attr as "KEY=VALUE".

func (a Attr) HasValue() bool
    HasValue returns true if the Attr has a value.

func (a Attr) Int64() int64
    Int64 returns the Attr's value as an int64. It panics if the value is not a
    signed integer.

func (a Attr) Key() string
    Key returns the Attr's key.

func (a Attr) Kind() Kind
    Kind returns the Attr's Kind.

func (a Attr) String() string
    String returns Attr's value as a string, formatted like fmt.Sprint.
    Unlike the methods Int64, Float64, and so on, which panic if the Attr is of
    the wrong kind, String never panics.

func (a Attr) Time() time.Time
    Time returns the Attr's value as a time.Time. It panics if the value is not
    a time.Duration.

func (a Attr) Uint64() uint64
    Uint64 returns the Attr's value as a uint64. It panics if the value is not
    an unsigned integer.

func (a Attr) Value() any
    Value returns the Attr's value as an any. If the Attr does not have a value,
    it returns nil.

func (a Attr) WithKey(key string) Attr
    WithKey returns an attr with the given key and the receiver's value.

type Handler interface {
	// Enabled reports whether this handler is accepting records
	// at the given level.
	Enabled(Level) bool

	// Handle processes the Record.
	// Handle methods that produce output should observe the following rules:
	//   - If r.Time() is the zero time, do not output it.
	//   - If r.Level() is Level(0), do not output it.
	Handle(Record) error

	// With returns a new Handler whose attributes consist of
	// the receiver's attributes concatenated with the arguments.
	With(attrs []Attr) Handler
}
    A Handler processes log records produced by Logger output. Any of the
    Handler's methods may be called concurrently with itself or with other
    methods. It is the responsibility of the Handler to manage this concurrency.

type HandlerOptions struct {
	// Add a "source" attributes to the output whose value is of the form
	// "file:line".
	AddSource bool

	// Ignore records with levels above LevelRef.Level.
	// If nil, accept all levels.
	LevelRef *LevelRef

	// If set, ReplaceAttr is called on each attribute of the message,
	// and the returned value is used instead of the original. If the returned
	// key is empty, the attribute is omitted from the output.
	//
	// The built-in attributes with keys "time", "level", "source", and "msg"
	// are passed to this function first, except that time and level are omitted
	// if zero, and source is omitted if AddSource is false.
	ReplaceAttr func(a Attr) Attr
}
    HandlerOptions are options for a TextHandler or JSONHandler. A zero
    HandlerOptions consists entirely of default values.

func (opts HandlerOptions) NewJSONHandler(w io.Writer) *JSONHandler
    NewJSONHandler creates a JSONHandler with the given options that writes to
    w.

func (opts HandlerOptions) NewTextHandler(w io.Writer) *TextHandler
    NewTextHandler creates a TextHandler with the given options that writes to
    w.

type JSONHandler struct {
	// Has unexported fields.
}
    JSONHandler is a Handler that writes Records to an io.Writer as
    line-delimited JSON objects.

func NewJSONHandler(w io.Writer) *JSONHandler
    NewJSONHandler creates a JSONHandler that writes to w, using the default
    options.

func (h JSONHandler) Enabled(l Level) bool
    Enabled reports whether l is less than or equal to the maximum level.

func (h *JSONHandler) Handle(r Record) error
    Handle formats its argument Record as a JSON object on a single line.

    If the Record's time is zero, it is omitted. Otherwise, the key is "time"
    and the value is output in RFC3339 format with millisecond precision.

    If the Record's level is zero, it is omitted. Otherwise, the key is "level"
    and the value of Level.String is output.

    If the AddSource option is set and source information is available,
    the key is "source" and the value is output as "FILE:LINE".

    The message's key is "msg".

    To modify these or other attributes, or remove them from the output,
    use [HandlerOptions.ReplaceAttr].

    Values are formatted as with encoding/json.Marshal.

    Each call to Handle results in a single, mutex-protected call to
    io.Writer.Write.

func (h *JSONHandler) With(attrs []Attr) Handler
    With returns a new JSONHandler whose attributes consists of h's attributes
    followed by attrs.

type Kind int
    Kind is the kind of an Attr's value.

const (
	AnyKind Kind = iota
	BoolKind
	DurationKind
	Float64Kind
	Int64Kind
	StringKind
	TimeKind
	Uint64Kind
)
func (k Kind) String() string

type Level int
    A Level is the importance or severity of a log event. The higher the level,
    the less important or severe the event.

const (
	ErrorLevel Level = 10
	WarnLevel  Level = 20
	InfoLevel  Level = 30
	DebugLevel Level = 31
)
    Names for common levels.

func (l Level) String() string
    String returns a name for the level. If the level has a name, then that
    name in uppercase is returned. If the level is between named values, then an
    integer is appended to the uppercased name. Examples:

        WarnLevel.String() => "WARN"
        (WarnLevel-2).String() => "WARN-2"

type LevelRef struct {
	// Has unexported fields.
}
    A LevelRef is a reference to a level. LevelRefs are safe for use by multiple
    goroutines. Use NewLevelRef to create a LevelRef.

    If all the Handlers of a program use the same LevelRef, then a single Set on
    that LevelRef will change the level for all of them.

func NewLevelRef(l Level) *LevelRef
    NewLevelRef creates a LevelRef initialized to the given Level.

func (r *LevelRef) Level() Level
    Level returns the LevelRef's level. If LevelRef is nil, it returns the
    maximum level.

func (r *LevelRef) Set(l Level)
    Set sets the LevelRef's level to l.

type Logger struct {
	// Has unexported fields.
}
    A Logger generates Records and passes them to a Handler.

    Loggers are immutable; to create a new one, call New or Logger.With.

func Default() *Logger
    Default returns the default Logger.

func FromContext(ctx context.Context) *Logger
    FromContext returns the Logger stored in ctx by NewContext, or the default
    Logger if there is none.

func New(h Handler) *Logger
    New creates a new Logger with the given Handler.

func With(attrs ...any) *Logger
    With calls Logger.With on the default logger.

func (l *Logger) Debug(msg string, args ...any)
    Debug logs at DebugLevel.

func (l *Logger) Enabled(level Level) bool
    Enabled reports whether l emits log records at level.

func (l *Logger) Error(msg string, err error, args ...any)
    Error logs at ErrorLevel. If err is non-nil, Error appends Any("err",
    err) to the list of attributes.

func (l *Logger) Handler() Handler
    Handler returns l's Handler.

func (l *Logger) Info(msg string, args ...any)
    Info logs at InfoLevel.

func (l *Logger) Log(level Level, msg string, args ...any)
    Log emits a log record with the current time and the given level and
    message. The Record's Attrs consist of the Logger's attributes followed by
    the Attrs specified by args.

    The attribute arguments are processed as follows:
      - If an argument is an Attr, it is used as is.
      - If an argument is a string and this is not the last argument, the
        following argument is treated as the value and the two are combined into
        an Attr.
      - Otherwise, the argument is treated as a value with key "!BADKEY".

func (l *Logger) LogAttrs(level Level, msg string, attrs ...Attr)
    LogAttrs is a more efficient version of Logger.Log that accepts only Attrs.

func (l *Logger) LogAttrsDepth(calldepth int, level Level, msg string, attrs ...Attr)
    LogAttrsDepth is like Logger.LogAttrs, but accepts a call depth argument
    which it interprets like Logger.LogDepth.

func (l *Logger) LogDepth(calldepth int, level Level, msg string, args ...any)
    LogDepth is like Logger.Log, but accepts a call depth to adjust the file
    and line number in the log record. 0 refers to the caller of LogDepth;
    1 refers to the caller's caller; and so on.

func (l *Logger) Warn(msg string, args ...any)
    Warn logs at WarnLevel.

func (l *Logger) With(attrs ...any) *Logger
    With returns a new Logger whose handler's attributes are a concatenation of
    l's attributes and the given arguments, converted to Attrs as in Logger.Log.

type Record struct {
	// Has unexported fields.
}
    A Record holds information about a log event.

func NewRecord(t time.Time, level Level, msg string, calldepth int) Record
    NewRecord creates a new Record from the given arguments. Use Record.AddAttr
    to add attributes to the Record. If calldepth is greater than zero,
    Record.SourceLine will return the file and line number at that depth.

    NewRecord is intended for logging APIs that want to support a Handler as a
    backend. Most users won't need it.

func (r *Record) AddAttr(a Attr)
    AddAttr appends a to the list of r's attributes. It does not check for
    duplicate keys.

func (r *Record) Attr(i int) Attr
    Attr returns the i'th Attr in r.

func (r *Record) Attrs() []Attr
    Attrs returns a copy of the sequence of Attrs in r.

func (r *Record) Level() Level
    Level returns the level of the log event.

func (r *Record) Message() string
    Message returns the log message.

func (r *Record) NumAttrs() int
    NumAttrs returns the number of Attrs in r.

func (r *Record) SourceLine() (file string, line int)
    SourceLine returns the file and line of the log event. If the Record
    was created without the necessary information, or if the location is
    unavailable, it returns ("", 0).

func (r *Record) Time() time.Time
    Time returns the time of the log event.

type TextHandler struct {
	// Has unexported fields.
}
    TextHandler is a Handler that writes Records to an io.Writer as a sequence
    of key=value pairs separated by spaces and followed by a newline.

func NewTextHandler(w io.Writer) *TextHandler
    NewTextHandler creates a TextHandler that writes to w, using the default
    options.

func (h TextHandler) Enabled(l Level) bool
    Enabled reports whether l is less than or equal to the maximum level.

func (h *TextHandler) Handle(r Record) error
    Handle formats its argument Record as a single line of space-separated
    key=value items.

    If the Record's time is zero, it is omitted. Otherwise, the key is "time"
    and the value is output in RFC3339 format with millisecond precision.

    If the Record's level is zero, it is omitted. Otherwise, the key is "level"
    and the value of Level.String is output.

    If the AddSource option is set and source information is available,
    the key is "source" and the value is output as FILE:LINE.

    The message's key "msg".

    To modify these or other attributes, or remove them from the output,
    use [HandlerOptions.ReplaceAttr].

    Keys are written as unquoted strings. Values are written according to their
    type:
      - Strings are quoted if they contain Unicode space characters or are over
        80 bytes long.
      - If a value implements [encoding.TextMarshaler], the result of
        MarshalText is used.
      - Otherwise, the result of fmt.Sprint is used.

    Each call to Handle results in a single, mutex-protected call to
    io.Writer.Write.

func (h *TextHandler) With(attrs []Attr) Handler
    With returns a new TextHandler whose attributes consists of h's attributes
    followed by attrs.

[evanphx]

👋🏻 hclog developer here! Curious about seeing if the stdlib could mostly contain the interface surface area for different implementations to step into. slog.Logger could be something an implementation of that interface.

Were that route taken, I'd propose the surface area that looks something like this:

type Logger interface {
   Log(level int, msg string, args ...any)
   Debug(msg string, args ...any)
   Info(msg string, args ...any)
   Warn(msg string, args ...any)
   Error(msg string, args ...any)
   With(args ...any) Logger
}

var DefaultLogger Logger

func Debug(msg string, args ...any) {
   DefaultLogger.Debug(msg, args...)
}
// Toplevel for Info, Warn, Error, and With

func InContext(ctx context.Context, log Logger) context.Context { ... }
func FromContext(ctx context.Context) Logger { ... } // returns DefaultLogger if none available

[akshayjshah]

Curious about seeing if the stdlib could mostly contain the interface surface area for different implementations to step into.

Is this suggesting that the slog package export the Logger interface you outline, which third-party packages may then implement ("step into")?

If so, the interface you propose ends up being fairly slow because passing ints, strings, and other values as any often allocates. These allocations are problematic because they're nearly impossible to avoid. Even if debug-level logging is disabled,

logger.Debug("some high-volume thing happened", "key1", someString, "key2", someOtherString)

will make multiple allocations just to call the logger implementation, which then realizes that debug-level logs are disabled and does nothing. These unavoidable allocations make logging difficult on code paths that need to be fast.

This is often just fine - most code doesn't need to be particularly high-performance. If we're going to put structured logging into the stdlib, though, the logger interface should probably expose enough internals to be useful to performance-sensitive applications.

[evanphx]

That's a fair concern! Perhaps adding In(level int) bool to allow a user to guard the statements in high throughput areas is prudent here. It's been a little while since I benchmarked the allocations of string->any and int->any conversions, which are the 2 vast majority of argument cases.

With the more complex APIs, the trade is carefully crafting them to not allocate and having them sometimes suffer ergonomically and decrease the developer usage.

[evanphx]

Looking at the benchmarking, the only allocation today is is creating the slice it appears, no individual allocs for the arguments.

[vearutop]

I have a use case where I would like to control logging level based on context (e.g. enabling Debug for requests with magical header). Seems I can do this by creating Logger instance using debug-enabled Handler and then making NewContext with it.

However, this context instrumentation needs to happen before any other (for example context may already have a Logger.With(...)) to avoid loss of state.
Maybe it would make sense to allow replacing Handler using WithHandler(h Handler) *Logger to make such manipulations more flexible.

[jba]

The question is, is this common enough to warrant built-in support?

[jochumdev]

@jba that would mean the "root" handler (in this case "logger1") needs to have the lowest supported Level in subloggers, right?

[jba]

First of all, the comparison was wrong in this code because we changed the direction of levels. I made an edit to fix that.

Now to answer your question, I don't think it matters what logger1's level is. The "level" of a logger is just a way of talking about how its handler's Enabled method is written, and in this code the outermost handler has complete control—it does not delegate to the handler it wraps.

[jochumdev]

@jba ohh I see, so Enabled will not be called on the wrapped Handler, that's nice.

[Davincible]

The question is, is this common enough to warrant built-in support?

@jba if possible it would be very nice to have built-in support for this. Having to create a wrapper like that for a sub-logger with a different level feels a bit clunky. I feel like enabling different log levels for different components of your code could generally be quite useful.

[vearutop]

What would happen if kvs ...any arguments are not well-formed (odd number or key is not a string)?

[jba]

Is the intention of this library to be abstracted by other libraries

Intention is a strong word, but as I've written elsewhere in this discussion, I think it's inevitable.

alternatives such as zerolog exist, which "box" you into not making mistakes

Zerolog has a lovely API. I experimented with something like .String("k1", "s").Int("k2", 1)... at one point. There are a few downsides, though.

First, you have to remember to write Msg at the end of the line.

Second, it's a bit redundant to have to write Warn().....Msg(...). But you need the level at the beginning to filter, and something at the end to say you're done, so there's no alternative.

Third, it's hard to get good performance from the chaining API if you want a general backend that remembers the attributes. Either you allocate for each attribute, or you copy something that can hold several attributes, which gets expensive. zerolog doesn't have that problem, because it converts each attribute to bytes immediately and appends to a []byte. If that buffer is pool-allocated with a large enough size, you won't see any allocations (hence the name).

[meling]

It has probably been proposed by others in another context and rejected, but it would seem like a nice addition to the language to be able to specify variadic pairs. Something like ...{string, any}.

[edgmnt]

@meling, a struct containing a string and an any should be enough to prevent unbalanced key-value pairs (although other improvements may be considered):

type KV struct {
        Key   string
        Value any
}

func (log *Logger) Error(err error, msg string, kvs ...KV)

// Usage:
log.Error(err, "message",
        slog.KV{"handle", handle},
        slog.KV{"user", user})

[jba]

specify variadic pairs

I think this would fall under the category of eliding type names in literals. For example, #21496.

[jba]

type KV struct {
        Key   string
        Value any
}

Our Attr type is essentially this, but will not allocate memory for many values that this would.

[willfaught]

Looks promising. Some thoughts:

But more important than any traction gained by the "frontend" is the promise of a common "backend." An application with many dependencies may find that it has linked in many logging packages. If all of the packages support the handler interface we propose, then the application can create a single handler and install it once for each logging library to get consistent logging across all its dependencies. Since this happens in the application's main function, the benefits of a unified backend can be obtained with minimal code churn. We hope that this proposal's handlers will be implemented for all popular logging formats and network protocols, and that every common logging framework will provide a shim from their own backend to a handler. Then the Go logging community can work together to build high-quality backends that all can share.

Why is this being proposed now? Why not 5–8 years ago?

This seems to tout handlers as an innovation over the state of the art. Do none of the existing solutions have a comparable handler design?

Will this effort to establish a common interface for the community extend to other problem domains, like audio?

The program then outputs three log messages augmented with key-value pairs.

In my opinion, the loose ...any key/value pairs parameter design, like in go-kit, feels a little too "loose". I understand going with it, because I can't think of a better way to do it either, but it always felt like there was a better way out there somewhere. It's not worth getting hung up on, but I'd be very interested in any viable alternatives.

What happens if a value is missing?

The second logs at the Error level, passing an error and a key-value pair.

What if there isn't a Go error value when logging an error condition? What value should be used?

FromContext returns the default logger if it can't find one in the context.

Could this hide configuration or setup mistakes, where there should have been a logger, but there wasn't?

InfoLevel Level = 30
DebugLevel Level = 31

Why Is DebugLevel 31, when the other levels increment by 10?

TextHandler emits attributes as KEY=VALUE

What is the exact intended initial format, whether or not you want to document it?

What is the order of the fields? Are the "built-in" fields first?

Are the pairs delimited by a space?

Are strings containing whitespace quoted?

JSONHandler writes line-delimited JSON objects

Is the JSON minimized?

What is the order of the fields? Are the "built-in" fields first?

ReplaceAttr can be used to change the default keys of the built-in attributes

What are the default keys?

Functions like Any and Int construct slog.Attr values
slog.LogAttrs

Nit: Consider using the names Field/Fields, like zap. "Attr" is a little more abstract than "field," and "WithFields" is a little clearer and reads a little better than "WithAttrs," in my humble opinion.

Any(k string, v any) Attr

Why not include variants for all built-in types, like Int8, Rune or Complex64? Would generics help here?

The boolean AddSource option controls whether the file and line of the log call.

Have you considered the potential demand for a stack trace option (as opposed to just the current file name and line)?

func (l *Logger) With(attrs ...any) *Logger

I don't see a Logger.WithAttrs variant. Is that because it wouldn't help avoid allocations? If so, why is that?

[willfaught]

A common, popular logger design that has an error parameter

That would be logr

I don't think that design counts as common, as I showed with the design survey; only logr has this design.

I don't think that design counts as popular, either, but I acknowledge that logr is one of the more notable log packages, albeit in the lowest "tier" of notable (i.e. less than logrus and zap; at the same level as go-kit).

No, the main point is that it's more convenient. That is just a side benefit.

No, I assumed that none of them were. That is what I meant when I said the percentages alone would be evidence for an error argument: because even if none were mistakes, programmers would have an easier time overall.

OK, so this isn't about foot-guns, or about user-perceived convenience that we've measured, it's about observing that supplying error values correlates quite a bit with a particular log level, specializing part of the design around that particular use case, and assuming that users will find that more preferable. The correlation threshold for doing this is ~2/3 of the time; ~4/10 of the time is too low.

Let's consider your zap numbers again. Your analysis takes into account the convenience that the error parameter would have had ~2/3 of the time when there was an error value, but it doesn't take into account the inconvenience it would have had ~1/3 of the time when there wasn't an error value. It also doesn't take into account the design downsides of having a different method signature from the other levels. While weighing those trade-offs does involve some subjectivity, I think the consensus of the community is against the design of logr, and against this design, in this particular respect.

[smlx]

It also doesn't take into account the design downsides of having a different method signature from the other levels.

I also find this choice of inconsistency confusing. In my experience you often want to log errors at any log level, not just Error. This design also implies that if you have an error then you should be logging at Error, which is not correct.

[jba]

It's new and unfamiliar.
There's evidence that it is a net benefit. (It makes some things worse, but more things better.)
Let's try it. There will be plenty of time to kick the tires.

[hherman1]

Putting a vote in for the convenience of the error argument. Not clear to me why the slight inconsistency with other log levels matters.

[thockin]

As one of the logr authors, I can see both sides of this. My own argument AGAINST the logr style error arg is the need to give it a hard-coded "magic" key ("err").

I can buy the argument that fewer magic keys would be better. The counter argument is that there is value in a consistent name across all call-sites.

logr was informed by kubernetes, which really wanted the consistency and really does have an error in almost all Error() calls

[akshayjshah]

👋🏽 I'm glad to see this discussion reignited! I'm one of the original zap authors, and may have a little additional context to share. Points below notwithstanding, this proposal seems carefully thought-through and researched. Many thanks to the authors for all the work they've put in 😍

Previous art

Peter Bourgon and Chris Hines made a proposal with similar goals in 2017. The overall situation hasn't changed much since then - if anything, the proliferation of log-like APIs in distributed tracing packages has made it even worse. If the authors of this proposal haven't seen the previous doc, it's worth reviewing. I'm not sure if Chris and Peter are still interested in logging, but their perspectives would be valuable here.

I particularly liked the very small interface they proposed and the Valuer interface, even if both come at some performance cost.

Namespacing

I notice that this proposal doesn't include support for namespacing key-value pairs. For example, we might want a gRPC interceptor to produce JSON records like this:

{ ... standard fields ..., "grpc": {"method": "some.package/Method", ... other gRPC stuff...}}

Zap exposes this via Namespace. At least at Uber, this was critical for practical use: like many storage backends, ElasticSearch stops indexing if a field changes type. Implicitly, this forces each binary to have a schema for its log output. It's very hard to do this if dozens of loosely-coordinated packages are writing to a shared, flat keyspace. Namespacing cuts the problem into manageable pieces: each package uses its own name as a namespace, within which it can attempt to adhere to some implicit schema. This problem is significant enough that people have attempted to write static analysis to mitigate it.

Of course, not all storage engines have this limitation - but many do. Food for thought.

First-class error support

The error interface is infamously common in Go code, and logging errors is also common. It seems worth building in special support, rather than forcing users to call err.Error() eagerly. This also opens the door to a richer structured representation of errors. For example, the proposed multi-error support in the stdlib will be nearly unreadable if users shove err.Error() into JSON. If the old x/errors proposal for automatically capturing stacktraces re-emerges, we'd also want a sane way to include them in structured log output.

Zap spends a fair bit of effort to special-case go.uber.org/multierr and github.com/pkg/errors to paper over some of these complexities.

Edit: on a second reading, I see the error argument to slog.Error. I still think that it may be valuable to have a func Error(err error) Attr to use with other log levels (particularly debug).

Delegating serialization & complex objects

It's convenient for end users to let types implement their own serialization logic (a la json.Marshaler or zap.ObjectMarshaler). I don't see an equivalent here. How do you imagine types customizing their representation in logs? Implementing encoding-specific interfaces like json.Marshaler is difficult, since there's no telling what encoding the owner of main prefers. Should packages expose a function that allocates a []log.Attr for each loggable type? Is this wholly out of scope? Do users log each field of interest by hand?

Generics

My experience with zap is that the mere existence of a faster API, even it's painfully verbose, puts a lot of pressure on developers to use it. Making the lower-allocation API as ergonomic as possible has a disproportionate effect on developer experience.

One of my enduring regrets about zap is its fussy API for producing fields. Nowadays, this seems like the sort of problem we ought to be able to solve nicely with a

type Attr[T any] struct {
  key string
  value T
}

Minimizing allocations would require a fast path through marshaling that doesn't go through any, which I think would require support for type switches on parametric types. Personally, I'd love to see a sketch of what might be possible with #45380.

vet support

It'd be nice to also have go vet check for mistakes in the friendlier, alternating key-value API. People don't make many mistakes in practice, but it's nice to have a little extra assurance that you haven't left off an argument or mixed up types. Perhaps there's space for function names as unique as printf and friends, so that the same vet rule could apply to any similarly-named functions?

It's great though!

Even with the concerns above, I want to reiterate how encouraging this is. Large sections of the Go community seem to really want structured logs, and the current fragmentation is painful. Working around the limitations of a single stdlib implementation will be much better than working around the quirks of the many structured loggers today.

Edit: I realize that this proposal isn't a referendum on structured logging in general. Nevertheless, as one of the perpetrators of zap, I do feel compelled to say that I'm personally still not convinced that the fussiness of structured logging is worth it. On balance, I prefer leaving this degree of structure to distributed tracing and Prometheus-style tagged metrics and keeping logs as simple printfs. 🤷🏽‍♂️ I seem to be out of step with the rest of the community on this, though, and I do see this proposal as a step forward for structured logging.

[rabbbit]

I'm a biased zap user but would like to +1 @akshayjshah ideas on:

• namespacing: it is quite frustrating when you import a new library, it starts logging, and now either all my dashboards are garbage, or just disappear when ELK invisibly stops ingesting. This is painful when you don't control either of the libraries.
• specialized func Error(err error) Attr. It's surprisingly pleasant not to even have to deal with naming the error (slog.Info("e", err.Error()) || slog.Info("err", err.Error()) || slog.Info("error", err.Error()) vs just slog.Info(slog.Err, err)

[AndrewHarrisSPU]

Nevertheless, as one of the perpetrators of zap, I do feel compelled to say that I'm personally still not convinced that the fussiness of structured logging is worth it. On balance, I prefer leaving this degree of structure to distributed tracing and Prometheus-style tagged metrics and keeping logs as simple printfs. 🤷🏽‍♂️

This made me grin. FWIW, my own experience with zap was that I did end up finding structured logging useful; I enjoyed it a lot more when I switched from zap's sugared logger to DIY, local sugar.

[thockin]

Regarding a specific method for types to marshal log-data -- logr defines such a method, too. It seemed "obviously useful", so +1 on that

[riking]

Specific method for types to marshal log data - isn't this just MarshalJSON, MarshalText, and friends in other encoding schemes? Or are we talking about something producing a bunch of Attr?

[thockin]

In logr (which does not use an Attr equivalent), this method returns an any, which is then rendered in the normal way (whatever the given backend would do).

This means you can return a string, but you can also return an int, a map, a slice, a struct, etc. The original key for this value still holds.

So, for example, if I am logging a struct, I can use this method to wipe sensitive fields or convert to a simpler structure entirely.

[seankhliao]

ReplaceAttr func(a Attr) Attr

This allows changing or omitting an Attr, but not splitting it out into multiple.
At our company, logging calls take a context.Context,
and TraceID/SpanID (and other metadata) is extracted from there and injected into log records.
We do it this was as it's easier to remember, and span ids aren't yet visible at higher levels (eg wrapping handlers).

type Level int

I'd like to see it implement encoding.TextMarshaler / encoding.TextUnmarshaler for use in flags

type Handler interface

I'd like to see a standard handler that forwards to testing.TB.Logf

func SetDefault

zerolog provides a global logger as a separate package,
I've found it quite useful in catching if you use the wrong logger (global vs context-derived).

---

The other recent (big?) project that I'm aware of in logging standardization is OpenTelemetry's Log Data Model, which has recently been stabilized.

There they chose a different mapping of Levels / Severity to numbers,
I wonder if it would be worth using the same numbering as theirs?
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-severitynumber

---

I'll echo the desire for namespaced key-values,
but I think having it on the handler level isn't quite right,
as there are cases when you want to write into different namespaces partitioned by business logic rather than hard package boundaries.
For JSON it looks halfway possible with Any,
maybe there could be another one for Attrs(key string, attrs...Attr) Attr?

[willfaught]

Perhaps there should be (*Logger).WithNamespace(string) *Logger and NewLoggerNamespace(string) *Logger that prepend/wrap namespaces around keys.

[jba]

I'll echo the desire for namespaced key-values,
but I think having it on the handler level isn't quite right,
as there are cases when you want to write into different namespaces partitioned by business logic rather than hard package boundaries.

Can you give an example of how this would look?

[seankhliao]

I've seen 2 patterns used:

• log with dotted key strings and rely on the backend to merge them:
  in middleware log = log.With("workflow.request_tyoe","...", "request.source", "...")
  and later log.Info("done", "workflow.result")
  but this can be somewhat difficult to process properly
• update a context object that's passed along
  and leave it to the final caller or logger processing hooks to expand it into the appropriate fields,
  the above log would be {"msg":"done", "request":{"source":"..."},"workflow":{"request_type":"...","result":"..."}}

would be really nice if somehow interoperated with opentelemetry's attribute package, our developers are already exposed to that for annotating traces/spans.

[hherman1]

What if there was just a new Attr type that was a namespace attr, and supported passing in a collection of sub-attrs? Seems performant + tiny extension to the existing api

[hherman1]

E.g ‘sllog.Namespace(ns string, attrs Attr…)

[sagikazarmark]

Thanks for starting this discussion. The sheer amount of logging libraries out there seems to be an indicator that there should be a solution for this in the standard library.

A couple thoughts about the proposal:

OpenTelemetry

I think it would make sense to look at the work the OpenTelemetry working group does...at least to make sure this proposal is not incompatible with what they are working on.

I know they don't focus on libraries itself at the moment, but a wire format for logs.

Sugared logger

Personally, I'm quite fond of Zap's default logger enforcing attributes to be Field instances. Strong type safety sounds like a good default. For people who are willing to sacrifice that can always use the Sugared logger.

Maybe reflecting that in this proposal would also make sense by creating separate implementations.

Attr

I was wondering if it would make sense to make Attr an interface:

type Attr interface {
    Key() string
    Value() string
}

That way converting a value to string could be handed off to a custom Attr implementation. I don't know if that would affect allocations though. Maybe they would, but it's good enough for Zap. This would also allow to create separate types instead of using a single Attr with an any value.

I was wondering what the purpose of Kind is. Where would it be useful?

Context

Personally, I prefer extracting information from the context in the logger, not the other way around.

Let's say a context travels through a set of layers, each of them adding information to the context that you would like to log.
The best place to extract all those information is at the place you actually want to log something.

Therefore I'd want to pass the context to the logger somehow, not the other way around. For example, I could implement a handler that extracts all the relevant information from the context. The only question is: how do we pass a context to the logger?

[pohly]

How does that work if two different, unrelated packages both want to get additional values logged? If two functions add such an interface to the context, the handler will only retrieve the more recent one. Each function which adds that interface would need to check whether there is already one and support chaining in its interface implementation. This all feels rather clunky and would still depend on adding multiple things to a context (values plus interface for extracting them).

Or do you suggest that the main function adds the interface only once? That only works if it knows about all additional values that packages may want to log. If that's a long list, then the handler will have to check every single one on each log call, including the ones that were not added in the current call chain.

It also only solves "what to log" problem. It doesn't address "how to log". A handler could be attached to a context instead of a logger, but IMHO attaching a logger and encapsulating additional values in that logger is simpler overall.

[edgmnt]

A helper or set of helpers is all you need to log using a context, while the core API just takes a logger. E.g.

func A(log slog.Logger) {
        log.Info(...)
}

func B(ctx context.Context) {
        slog.FromContext(ctx).Info(...)
        // ... or ...
        slog.Info(ctx, ...)
}

func C(ctx context.Context) {
        A(slog.FromContext(ctx).WithValues(...))
        B(slog.WithValues(ctx, ...))
}

You could find a way to track attributes separately from the logger, perhaps lumping them up in a single context-like object, but I don't see the point.

I also think some comments may be missing the subtle fact that this doesn't have to be a complete, heavy logger that's expensive to create, pass around or swap out, it's simply a harness that does something with the logged parameters to allow decoupling. It's a sort of callback/closure with a bit of extra structure.

[edgmnt]

I think it's more worthwhile exploring whether you can collect and stuff attributes into normal error values. Because then you'll also be able to return errors as usual and log them higher in the call chain, where they may ve recoverable. All while avoiding logging the same error multiple times, which is fairly problematic when packages decide to both log and return the same base error.

[mgood]

Why couldn't you do exactly that (and pass the context down as a Logger attribute)?

I'll try to provide a more concrete example of a similar concern I have about the context and tracing. Here's a simple adaptation of one of the examples from https://pkg.go.dev/go.opentelemetry.io/otel/trace to add a logging line. It uses FromContext to look up a logger, but that logger won't have info about the current span:

var tracer trace.Tracer

func init() {
	tracer = otel.Tracer("instrumentation/package/name")
}

func operation(ctx context.Context) {
	var span trace.Span
	ctx, span = tracer.Start(ctx, "operation")
	defer span.End()
	
	slog.FromContext(ctx).Info("msg")
}

So, IIUC the way you would have to do that is wrap each tracer.Start call to create a new logger with the span info, and return slog.NewContext to include the attributes.

That would work, however there may be cases where the connection between tracing and logging is more decoupled. E.g. let's assume these functions are split across packages:

func TracedOp(ctx context.Context) {
	var span trace.Span
	ctx, span = tracer.Start(ctx, "operation")
	defer span.End()
	foo.LoggedOp(ctx)
}

package foo
func LoggedOp(ctx context.Context) {
	slog.FromContext(ctx).Info("msg")
}

Here, we've lost the context info about the span again, but now we have to make one of these places aware of either logging or tracing. If these are 3rd party packages that's even more difficult.

So, we have the necessary info available on the context at the point of the logging call, but to actually get to that information we would need to add additional coupling between logging and tracing throughout the application.

Providing a way to pass the context through would enable you to configure a Context-aware logger at some top-level entry point instead. Then each logging call would extract the info needed from the Context provided at that point.

[svzhl]

I agree with the original comment about working with context and believe that extracting logger from a context is a design smell.
Lets look at the following code:

func (f *Foo) Bar(ctx context.Contex) {
	...
	f.logger.Info(ctx, "Bar processed")
	...
}

If you read this line you know that:

1. A customized logger configured for Foo struct is used. You can easily find out of logger configuration if required.
2. All logging attributes added to context will be logged.

The semantics is pretty strait forward and clear.

Compare to the proposal:

func (f *Foo) Bar(ctx context.Contex) {
	logger := slog.FromContext(ctx)
	...
	logger.Info("Bar processed")
	...
}

Looking at Info method call it is unclear which logger is used. Is it taken from context? Is it a brand new instance of logging? Does the logger contain all necessary baggage attributes? Things are getting worse if a function works with 2 instances of context. Every log method call requires careful investigation.

If we use another way:

func (f *Foo) Bar(ctx context.Contex) {
	...
	slog.FromContext(ctx).Info(ctx, "Bar processed")
	...
}

It is clear that a logger from context is used. But it would be difficult to find all the places where the instance of logger is modified and added to context for a particular call stack. So we really don't know here how logger is configured. A third party library could override the logger in a wrong way and add it to context which is not safe. Fixing a bug would be not an easy journey.

Although most of popular existing log libraries don't accept context as a parameter it is possible to wrap the logger. This is what we do usually. The proposed slog package also has such opportunity. But I believe a modern log package should enforce accepting context.

[rogpeppe]

I think this is promising. A bunch of thoughts:

• a few times in the docs, it talks about an Attr "having" a value. I'm not sure that's very helpful (and it wasn't very obvious what it meant to me). How about just calling it "nil" as that's presumably what it is under the hood. Then HasValue becomes redundant - it's just a.Kind() == NilValue
• why not make the zero value of LevelRef usable?
• I'd like to see more justification for the arbitrary-seeming level constants. Why 10 apart and not 1, 100 or 1000? Why no gap between Info and Debug? It seems to me like the arbitrary numbers make it easier to get things wrong and are less efficient (no trivial mapping to an array)
• some mention could be made of how NaN and Infinity are handled when marshaling to JSON.
• "Strings are quoted if they contain Unicode space characters" - they should surely be quoted if they contain a quote character too? The quote format should also be defined (presumably Go string syntax, but this could be made clear)
• It seems a bit odd to me that the handler constructors are methods on the options, rather than just regular functions that take the options as arguments. If there are any other handlers defined externally, that's the pattern they'll need to use, and as methods, there's a distinct possibility that they might be pulled into a binary even when a json dependency isn't desired.

[jba]

a few times in the docs, it talks about an Attr "having" a value

Agreed. We can remove Attr.HasValue and any talk of an Attr "having" a value.

It seems a bit odd to me that the handler constructors are methods on the options, rather than just regular functions that take the options as arguments.

This pattern avoids a nil argument to the constructor. It has some precedent in the standard library:

net: Listen, ListenConfig.Listen
go/printer: Fprint, Config.Fprint
go/types: Config.Check (although there is no top-level Check)

If there are any other handlers defined externally, that's the pattern they'll need to use

Why?

[pohly]

they should surely be quoted if they contain a quote character too

We got complaints from Kubernetes developers after switching to structured logging and doing exactly that in the klog text output for structured log calls: the problem was that in a few places, the value was a long, multi-line string with plenty of quotation marks. The output then became completely unreadable.

The solution was a different quoting scheme for strings with line breaks - see kubernetes/klog#273

[firelizzard18]

I'm working with a framework that catches and logs panics (not a behavior I like but I don't really have a choice). It quotes the stack trace and replaces newlines and tabs with \n and \t. This makes the long output entirely unusable until I copy it into an editor and unescape those characters. I'd love something like what kubernetes/klog#273 describes.

[rogpeppe]

Using generics

In one of the comments above, there was mention of the possibility of using generics to mitigate some of the API blowup from having many explicitly different Attr constructors and methods on Attr to retrieve values.

I'll expand a bit on my reply above. I think we can use generics to mitigate allocations and reduce the API, even in the absence of #45380.

Here's how the API might look. We'd remove the Duration, Time, etc constructors and methods and replace them with this:

// Mk returns an Attr for the supplied value.
// Mk does not preserve the exact type of integral values. All signed integers
// are converted to int64 and all unsigned integers to uint64. Similarly,
// float32s are converted to float64.
//
// It special-cases the following types to avoid allocations:
//
//	bool
//	int, int8, int16, etc
//	uint, uintptr, uint8, uint16, etc
//	time.Duration
//	string
// 	time.Time
//	float32, float64
//
//   However, named types are preserved. So given
// 
//       type Int int
// 
//   the expression
// 
//       log.Any("k", Int(1)).Value()
// 
//   will return Int(1).
func Mk[T any](key string, value T) Attr

// Value returns the value part of a. It panics if T is not [any]
// and doesn't match the kind reported by a.Kind().
//
// The above condition implies that T must be one of the
// following types:
//	any, int, uint, time.Duration, time.Time, string, bool, float64
func Value[T any](a Attr) T

Another possibility to make the performance implications of creating an Attr a bit clearer might be to define a constraint that allows all the known concrete types, and then keep Any but define the above Mk in terms of that:

type Types interface {
    time.Duration | time.Time | int | uint | ...
}

func Mk[T Types](key string, value T) Attr

Another possible spelling for Mk might be A (or if Attr was renamed to Field, F).

As I mentioned in the original comment, it's possible to implement this API without incurring allocations, although there is some performance overhead.

[jba]

Thanks for finding this trick. (I had gotten as far as switching on *T, but never considered that any(val) wouldn't allocate.)

One possibility is to replace Any with Any[T]. Any[any] would behave like the current Any. Most or all of the constructors could go away.

I'm less convinced of Value[T](Attr). It's much more clumsy to write slog.Value[string](a) than a.String(), and the constraints on T make it no more powerful than the handful of methods we have now. It might be worth it if

slog.Value[MyInt](slog.Any("k", 1))

would work, where MyInt's underlying type is int. But that would require switching on ~int.

[AndrewHarrisSPU]

But that would require switching on ~int.

FWIW, there's an unsafe trick that bypasses Kind() or switching, and could be used on ~int:

func value[T any](rhs any) T {
	var t T
	lhs := any(t)
	lhsPtr := (*uint64)(unsafe.Pointer(&lhs))
	rhsPtr := (*uint64)(unsafe.Pointer(&rhs))
	if *lhsPtr != *rhsPtr {
		panic("unsafe assertion")
	}

	return rhs.(T)
}

This must take an any because it relies on the first word of an interface being associated with dynamic type/index into runtime itab table. Any constraint on what an Attr can hold is exogenous to this call. Works on godbolt.org for a lot of things. I don't know if it's a great idea.

edit: this is a dumb example, just a type assertion in practice - the trick is really that the unsafe pointer value for rhs can be grabbed and stored ahead of time

[jba]

You might see if reflect.TypeOf(lhs) == reflect.TypeOf(rhs) works for you. The implementation of reflect.TypeOf is not much more than what you have.

Anyway, this is an exact check, and to make slog.Value[MyInt](slog.Any("k", 1)) work we need to ask if MyInt's underlying type is int.

[AndrewHarrisSPU]

You might see if reflect.TypeOf(lhs) == reflect.TypeOf(rhs) works for you

Pragmatically it's fine. Mostly I've been interested/distracted in exploring what's possible where the FGG/FG paper suggests covariant method receiver types. I feel like the typing and type checking of Attrs is a case where exhaustive pattern matching on a type switch would be reasonable way to structure code, it's just not quite in the language.

Another fuzzy idea, in slog this would be a bit nasty:

type TypicalInt[T constraints.Signed ] int64

func (TypicalInt[T]) Typical() {}

Attrs:

type Attr struct {
	k string
	n int64
	some interface{ Typical() }
}

func Any[T interface{ Typical() }]( key string, value T ) Attr {
	if v, ok := any(value).( interface{ UnwrapInt() int64 }); ok {
		return Attr{key, v.UnwrapInt(), value } // or nil for the last field is possible
	}

	return Attr{key, 0, value}
}

func (attr Attr) Int() int64 {
	if _, ok := attr.some.(interface{ UnwrapInt() int64 }); ok {
		return attr.n
	}
	panic("oops")
}

But, downstream, in myPkg importing slog, this is possible, and any TypicalInt is known to be convertible to int64

// package myPkg
type myInt slog.TypicalInt[myInt]

func(Int) Typical() {}
func (n Int) UnwrapInt() int64 {
	return int64(n)
}

Dunno if this would actually help to coordinate backends / loggers or just pushes the problem elsewhere. It's messy, and it's type erasure and coupling.

edit: fixed exporting

[rogpeppe]

The docs should document what should happen when With is used to attach the same attribute key twice. Does it override the original attribute, get ignored, or is the behaviour undefined?

[jba]

No duplicate checking is done anywhere. Too expensive. The Record we deliver to your Handler will have a raw sequence of Attrs, exactly as you added them. A Handler can of course dedup if it wishes, but the ones we provide do not.

I will make this clear in the next version of the doc.

[willfaught]

"attrs" in

func (l *Logger) With(attrs ...any) *Logger

should be "args" like in

func (l *Logger) Warn(msg string, args ...any)

to not suggest that attrs must be Attrs.