Multiple writers, different levels on each?

[bfreis]

Hi,

I'm trying to configure zerolog to output logs both to the console and a file, with different formats. I can easily do that with ConsoleWriter and io.MultiWriter.

But I'm also trying to have the different log destinations log different levels.

I couldn't find a way do it. I tried to see if a Hook would solve it, but apparently the hooks seem to apply before the logs are sent to the Writers.

Is there a way to do it?

Thanks

[R0bson]

Huh I'm working on that also now. So far I managed to do that by implementing custom writer:

package main
import (
	"os"
	"github.com/rs/zerolog"
)

type FilteredWriter struct {
	w zerolog.LevelWriter
	level zerolog.Level
}
func (w *FilteredWriter) Write(p []byte) (n int, err error) {
	return w.w.Write(p)
}
func (w *FilteredWriter) WriteLevel(level zerolog.Level, p []byte) (n int, err error) {
	if level >= w.level {
		return w.w.WriteLevel(level, p)
	}
	return len(p), nil
}

func main() {
	fAll, _ := os.OpenFile("./test-all.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fWarn, _ := os.OpenFile("./test-warn.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)

	errWriter := zerolog.MultiLevelWriter(fWarn)
	filteredWriter := &FilteredWriter{errWriter, zerolog.WarnLevel}
	w := zerolog.MultiLevelWriter(fAll, zerolog.ConsoleWriter{Out: os.Stdout}, filteredWriter)
	mainLog := zerolog.New(w).With().Str("some_key", "some_val").Timestamp().Logger()
	mainLog.Debug().Msg("🛠️ Some trace")
	mainLog.Info().Msg("👌 Should work")
	mainLog.Warn().Msg("⚠️ My first warning")
	mainLog.Error().Msg("🛑 My last warning")
	mainLog.Fatal().Msg("💣 Kaboom!")
}

It works although solution is far from being perfect because there is no way to attach different fields to different outputs. I would expect something like that to work but unfortunately it doesn't and I have no idea how to make it to work.

Expected but doesn't work:

package main

import (
	"os"
	"github.com/rs/zerolog"
)

func main() {
	fAll, _ := os.OpenFile("./test-all.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fWarn, _ := os.OpenFile("./test-warn.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)

	rootLogger := zerolog.New(fAll).With().Str("some_key", "some_val").Timestamp().Logger()
	warnLogger := zerolog.New(fWarn).With().Str("additional", "context").Logger().Level(zerolog.WarnLevel)
	consoleLogger := zerolog.New(zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: "15:04:05"})

	w := zerolog.MultiLevelWriter(rootLogger, warnLogger, consoleLogger)
	mainLog := zerolog.New(w)
	mainLog.Debug().Msg("🛠️ Some trace")
	mainLog.Info().Msg("👌 Should work")
	mainLog.Warn().Msg("⚠️ My first warning")
	mainLog.Error().Msg("🛑 My last warning")
	mainLog.Fatal().Msg("💣 Kaboom!")
}

[rs]

I guess chaining loggers this way would double encode your logs, you can't chain zerologgers this way. This type of complexe logging routing with custom per output fields is not supported by zerolog.

[uniqss]

this is so simple.
./log_wrapper/log_wrapper.go:

package log_wrapper

import (
	"github.com/rs/zerolog"
	"os"
)

var MainLog zerolog.Logger

type FilteredWriter struct {
	w     zerolog.LevelWriter
	level zerolog.Level
}

func (w *FilteredWriter) Write(p []byte) (n int, err error) {
	return w.w.Write(p)
}
func (w *FilteredWriter) WriteLevel(level zerolog.Level, p []byte) (n int, err error) {
	if level == w.level {
		return w.w.WriteLevel(level, p)
	}
	return len(p), nil
}

func InitLog(serverName string, serverNameMini string) bool {
	fAll, _ := os.OpenFile("./" + serverName + "-all.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fDebug, _ := os.OpenFile("./" + serverName + "-debug.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fInfo, _ := os.OpenFile("./" + serverName + "-info.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fWarn, _ := os.OpenFile("./" + serverName + "-warn.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fError, _ := os.OpenFile("./" + serverName + "-error.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	fFatal, _ := os.OpenFile("./" + serverName + "-fatal.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)

	writerDebug := zerolog.MultiLevelWriter(fDebug)
	writerInfo := zerolog.MultiLevelWriter(fInfo)
	writerWarn := zerolog.MultiLevelWriter(fWarn)
	writerError := zerolog.MultiLevelWriter(fError)
	writerFatal := zerolog.MultiLevelWriter(fFatal)
	filteredWriterDebug := &FilteredWriter{writerDebug, zerolog.DebugLevel}
	filteredWriteInfo := &FilteredWriter{writerInfo, zerolog.InfoLevel}
	filteredWriterWarn := &FilteredWriter{writerWarn, zerolog.WarnLevel}
	filteredWriterError := &FilteredWriter{writerError, zerolog.ErrorLevel}
	filteredWriterFatal := &FilteredWriter{writerFatal, zerolog.FatalLevel}
	w := zerolog.MultiLevelWriter(fAll, zerolog.ConsoleWriter{Out: os.Stdout},
		filteredWriterDebug, filteredWriteInfo, filteredWriterWarn, filteredWriterError, filteredWriterFatal)

	MainLog = zerolog.New(w).With().Str("stype", serverNameMini).Timestamp().Logger()

	return true
}

main.go:

package main

import (
	"log_test_zerolog/log_wrapper"
)

func main() {
	log_wrapper.InitLog("game_server", "gs")

	log_wrapper.MainLog.Debug().Str("hello", "world").Msg(" Some Debug log 我是中文哈哈")
	log_wrapper.MainLog.Info().Strs("xixi", []string{"haha", "hehe"}).Msg(" Some  Info log 呵呵吼吼")
	log_wrapper.MainLog.Warn().Msg(" Some warning 哦吼！！着火了")
	log_wrapper.MainLog.Error().Msg(" Some Error log 卧槽，，服务器挂了")
	log_wrapper.MainLog.Fatal().Msg(" Some Fatal log! 不得了了。。。机房冒烟了")
}

check https://github.com/uniqss/gostudy/tree/master/log_test/log_test_zerolog for detail.

[meshenka]

in k8s we try to do this:
every logs debug info warn should go to stdout
every logs error, critical, etc should go to stderr

i did not found elegant way to implement that with zerolog

[rs]

Implement a writer like this:

type LevelWriter struct {
	io.Writer
	ErrorWriter io.Writer
}

func (lw *LevelWriter) WriteLevel(l zerolog.Level, p []byte) (n int, err error) {
        w := lw
	if l > zerolog.InfoLevel {
		w = lw.ErrorWriter
	}
        return w.Write(p)
}

[meshenka]

Implement a writer like this:

type LevelWriter struct {
	io.Writer
	ErrorWriter io.Writer
}

func (lw *LevelWriter) WriteLevel(l zerolog.Level, p []byte) (n int, err error) {
        w := lw
	if l > zerolog.InfoLevel {
		w = lw.ErrorWriter
	}
        return w.Write(p)
}

thanks!! work like a charm
only change i made
w := lw.Writer

[Decmoe47]

I changed a bit more:

type LevelWriter struct {  
   io.Writer   
   Level       zerolog.Level  
}  
  
func (lw *LevelWriter) WriteLevel(l zerolog.Level, p []byte) (n int, err error) {  
   if l >= lw.Level {  // Notice that it's ">=", not ">"
	   return lw.Writer.Write(p)  
	}
   return len(p), nil
}

Since I just want to skip logging when the log level is lower than the set level, so I return len(p), nil after reading the source code.

zerolog/writer.go

Lines 76 to 88 in d894f12

	func (t multiLevelWriter) WriteLevel(l Level, p []byte) (n int, err error) {
	for _, w := range t.writers {
	if _n, _err := w.WriteLevel(l, p); err == nil {
	n = _n
	if _err != nil {
	err = _err
	} else if _n != len(p) {
	err = io.ErrShortWrite
	}
	}
	}
	return n, err
	}

Use case:

consoleWriter := zerolog.NewConsoleWriter(  
   func(w *zerolog.ConsoleWriter) {
        w.Out = os.Stderr
    },  
)  
// "Writer" is just the the anonymous field "io.Writer" in the  struct
consoleWriterLeveled := &LevelWriter{Writer: consoleWriter, Level: zerolog.DebugLevel}  
  
fileWriter := &lumberjack.Logger{  
   Filename:   "log/server.log",  
   MaxSize:    1,  
   MaxAge:     30,  
   MaxBackups: 5,  
   LocalTime:  false,  
   Compress:   false,  
}  
fileWriterLeveled := &LevelWriter{Writer: fileWriter, Level: zerolog.WarnLevel}

log.Logger = log.Output(zerolog.MultiLevelWriter(consoleWriterLeveled, fileWriterLeveled))

[parthAjagiya]

I am having similar use case.

I have multiple log categories ie access logs, application logs, transactional logs. I want them to store in separate files. I tried multiple way but no luck.

Is there anything Zerolog provides?

[mitar]

What about you tried and what exactly didn't work. This works well for me.

[parthAjagiya]

@mitar its for LevelWriter. ZeroLog provides method WriteLevel to implement to replace it with Write. But I want to store in multiple files based on field specified.

[mitar]

I think you should open another issue about your use case. This issue is about multiple writers with different levels.

[mitar]

#573 added FilteredLevelWriter which allows you to combine it with MultiLevelWriter to write to multiple destinations at different levels. Use it only when you use MultiLevelWriter because otherwise it is simpler to filter at the logger level to filter out events early.

[baruchiro]

Hi @mitar, can you please give me an example of how to print different levels to different outputs (the common usage- print errors to stderr and info to stdout)?

I'm trying to do that but I even can't find the FilteredLevelWriter (I'm on 1.29.0)

[mitar]

I do not think FilteredLevelWriter has been released yet. You have to update to the latest master in meantime (go get github.com/rs/zerolog@master).

With that, you can do something like (untested code):

writers := []io.Writer{
        &zerolog.FilteredLevelWriter{
	        Writer: zerolog.LevelWriterAdapter{os.Stdout},
	        Level:  zerolog.InfoLevel,
        },
        &zerolog.FilteredLevelWriter{
	        Writer: zerolog.LevelWriterAdapter{os.Stderr},
	        Level:  zerolog.ErrorLevel,
        },
}
writer := zerolog.MultiLevelWriter(writers...)
logger := zerolog.New(writer).Level(zerolog.InfoLevel).With().Timestamp().Logger()

Do notice that this will write both info and error levels to stdout, and error levels to stderr.

[baruchiro]

I don't like it prints errors to both stdout and stderr.

I think I will use a hook to print errors with a second logger.

[mitar]

You can check the code of FilteredLevelWriter and you will see it is very simple. You can easily make something similar where you have one writer which filters everything above a level out, and another which filters everything under a level out.

[baruchiro]

Thanks, I did it!

I moved the discussion to StackOverflow 🙂
https://stackoverflow.com/questions/76858037/how-to-use-zerolog-to-filter-info-logs-to-stdout-and-error-logs-to-stderr