[FR] Support log level configuration + document this feature

[pierluigilenoci]

Trying to change the verbosity of the software to produce a lower number of logs (only warnings and errors, the info level logs do not interest us) I searched for the documentation that specified all the args of the software but I could not find it.
Exists?

If it doesn't exist in what part of the software are they handled?
Maybe I can only extricate myself by reading the source code.

I tried searching both the documentation and the repos but couldn't find anything.

Thanks in advance.

gz#1529

[ivankube]

Thanks for submitting this feature request! Although it's not possible to configure log verbosity now, It’s one we’ve heard before, so we’ll make sure it’s on our list of features to consider when we’re ready to make some changes.

[pierluigilenoci]

@ivankube thanks. Is there any documentation of the possible arguments that exist to date somewhere?

[mbolt35]

@pierluigilenoci You can pass -v which gets passed through to the klog library for logging. The only other arguments accepted are cost-model (which is automatically applied) and agent which is undocumented.

[pierluigilenoci]

@mbolt35 thanks!

Is only the -v option passed? --logtostderr or --stderrthreshold?

Unfortunately klog is affected by a fairly serious bug. 😢
kubernetes/klog#212

[mbolt35]

@pierluigilenoci I have not tried to use the flags mentioned; however, it looks like we are piping logtostderr through as well. We're using the cobra library to process command line args, and it captures flags differently than the built-in go flags, which unfortunately means that we have to pipe flags through if they're used by other libraries. For klog, this is done here:

https://github.com/kubecost/cost-model/blob/5eb640394da9815976f561f32b07e652883a22f3/pkg/cmd/commands.go#L46-L47

I'll be honest, klogs decision to hijack command line arguments as a library seems odd, as I wish they would've just provided an programmable initialization which accepted arguments directly. If there are any specific klog options you'd like passed through, they can be added in the code I referenced above. Glad to take a look at any PRs submitted there.

[pierluigilenoci]

@mbolt35 thank you very much for the explanation and for your availability.
To complete the picture it might be interesting to have stderrthreshold and alsologtostderr passed to Klog.
In this way, it is possible to direct the logs according to personal preferences.

[pzimno]

+1 on this feature request. We've had kubecost logs spam our logging infrastructure with millions of records for an ETL misconfig. Currently our only fix is to filter it out from log collection.

[dwbrown2]

@pzimno, is there a specific log that you can share that's been spamming you? My understanding is that this request would provide general log level configuration but I'd want to do our best to address your very specific case.

[pzimno]

@dwbrown2, cost-model had some sort of etl failure so we would get a couple million logs per minute with messages like

E0330 16:41:39.748529       1 log.go:17] [Error] ETL: Allocation: automatic reconciliation: error reconciling range for [2022-03-30T15:00:00+0000, 2022-03-30T16:00:00+0000): error querying for AllocationSetRange to reconcile: boundary error: requested [2022-03-30T15:00:00+0000, 2022-03-30T16:00:00+0000); supported [2022-03-28T15:00:00+0000, 2022-03-30T14:25:43+0000): ETL: Allocation[1h] is 100.0% complete
I0330 16:41:39.748535       1 log.go:47] [Info] ETL: Allocation: automatically reconciling [2022-03-30T15:00:00+0000, 2022-03-30T16:00:00+0000) (0.00% complete)
I0330 16:41:39.748545       1 log.go:47] [Info] ETL: Asset[1h]: Query [2022-03-30T15:00:00Z, 2022-03-30T16:00:00Z)

[dwbrown2]

@AjayTripathy is that particular log worth de-duping or handling differently?

[dramich]

@pzimno I opened a new issue to keep track of this as I don't think it falls under the original issue or the fix added for said issue. If you have more details, like what was misconfigured, please add them to the new issue.

@dwbrown2 This sounds like we something we should at least look into if it caused millions of logs