Rework logging throughout the code.

[beorn7]

There are two aspects of this:

• Use structured logging consistently.
• Re-assess the different logging philosophies applied, pick one, and apply consistently. (Inspiration: http://peter.bourgon.org/blog/2016/02/07/logging-v-instrumentation.html )

[beorn7]

Most (if not all) warnings and errors should probably have counters attached (as it is the case for some already, cf. prometheus_local_storage_out_of_order_samples_total, prometheus_local_storage_inconsistencies_total, ...

[beorn7]

Also see #1315 , in particular consider query logging (separated or at least separatable) and separate out the crash recovery logs.

[brian-brazil]

I think #1315 covers the main aspects of the 2nd point, and for the 1st point more focused bugs would be useful.

[beorn7]

Can we close this once the more focused bugs are filed and not the other way round?
I think this is important.

[gouthamve]

Hi @beorn7 Could you elaborate on what exactly is missing right now? I think most of structured logging is fixed, is it missing counters for errors/warnings?

[beorn7]

It might be all fixed in 2.0. I haven't assessed the state. Happy to close this if that's what you think. I was just objecting the notion of "let's close this and file more focused bugs later". (It should be the other way round. ;)

[gouthamve]

Yep, absolutely.

If its just counters for errors, I will open a new issue for that labelled low-hanging-fruit but if you think logging is still remaining, I think this should stay open but with a little more description on what is missing.

Also, logging has not changed at all in 2.0. So if you could comment on the state of 1.x, I think it would cover the issue.

[beorn7]

I think we currently have no clarity what our logging philosophy is or should be. We follow different strategies in different parts of the code base. Regularly people complain about doing logging the "wrong" way. Sometimes, somebody angrily rips out logging code that others are then missing dearly (not necessarily in the way that they think it was perfect before but more in the way that the previous state was better than nothing). I think the first step in tackling this would be to write an RFC/design-doc where we come to terms with ourselves what logging should look like. After that, we can adjust reality to our grand plan. In general, I think that many individual aspects will be controversial, and some discussion will be needed to get consensus or at least buy-in.

#1315 covers query logging, which is a big chunk of the work once we get it done. But there is much more.

The notion of having a counter metric for each type of error logged – and vice versa a line of error log for each error metric incremented, might be something to add in there, but even there I'm not sure if it is uncontroversial.

Bottom-line: Writing that RFC and getting clarity about the course of action is the actual work for this issue. Once that's done, filing more concrete bugs is easy (and arguably implementing better logging will be not much more than chore work).