Log smarter, not harder

[geoffjentry]

This is more of an epic and not a right now thing, but I wanted to capture the thought for future consideration.

In the author's not-so-humble opinion we log way too much stuff, to the point that logs aren't particularly useful unless you know exactly what you're looking for.

We also know that Cromwell's logs are blowing apart loggly and stuff like that.

We also manage to log some stuff (e.g. stacktraces) multiple times for one incident.

We also have many issues where logs would be super helpful yet all we see is something like "an error occurred" without any context.

We should sit down as a group and with focus groups of downstream clients (e.g. firecloud, gotc) and go through what we're logging and look for ways to both massively debulk our logs as well as making sure that we're logging the most useful stuff.

[geoffjentry]

Another thing I've noticed is that a ton of our log messages are at debug level but most people aren't using debug level logging. Oopsie Daisy

[kcibul]

from Ruchi

Currently we have a lot of information that gets logged by Cromwell and some of it seems like it may not be useful to our customers. For example, every actor state transition gets logged, but it's unclear who is using that information.

We tech talked today an the suggestion was to meet with some key customer's, figure out if there are aspects of our logging they don't need to help reduce, and instead just redirect the unwanted logging to debug level, so that it can still be used for debugging purposes.

[geoffjentry]

That's exactly what the initial ticket writeup suggested. Whomever wrote the ticket in the first place is a genius ;)

[geoffjentry]

@kcibul @ruchim Can we decide what exactly this ticket entails in terms of 0.21? I don't think what I originally meant it to be is really appropriate for feature party, but we've decided that this ticket is a placeholder for something which is currently ill defined

[cjllanwarne]

I believe we're much smarter now that we no longer rely on detailed logs for test cases to pass. Closing for now