Consider switching to structured logs (slog)

[manugarg]

Go comes with its own structured logger now:
https://pkg.go.dev/log/slog.

We should consider switching to it for generic logging:

cloudprober/logger/logger.go

Line 355 in 2332d30

func genericLog(severity logging.Severity, name string, s string) {

[manugarg]

This will also take care of #406.

[manugarg]

Note: slog is available only with Go 1.21, so we'll have to first upgrade to Go 1.21. Alternatively we can use the experimental version for now:
https://pkg.go.dev/golang.org/x/exp/slog

[manugarg]

The way I envision this:

• We'll make cloudprober logger use slog.Logger.
• When we create new Logger for each module, we'll add some attributes to the logger.. something like:
  logger.New().With("probe", "classic_http_probe")
• We'll also allow adding attributes to individual log lines. For this we'll implement new set of functions: InfoAttrs, WarnAttrs, etc, similar to slog.Logger's functions.
• We'll keep the existing implementation of Info, Infof, Warning, Warningf, etc.

[manugarg]

#462 with the possible implementation.

[manugarg]

And, it's done. New logs look like this:

time=2023-08-17T14:06:35.809-07:00 level=INFO source=cloudprober/probes/http/http.go:533 msg="Probe(staging_companies) targets updated: map[api-gateway_0:ADD]" system=cloudprober probe=staging_companies
time=2023-08-17T14:06:37.095-07:00 level=INFO source=cloudprober/common/oauth/http_token.go:62 msg="oauth2: response from token URL: {\"access_token\":\"eyJ ........ z6i7SxS35qzREAcjJQ\"}" system=cloudprober probe=staging_companies

We can now export logs in JSON format as well, using the flag -logfmt=json

{"time":"2023-08-17T14:08:57.574018-07:00","level":"INFO","source":{"function":"github.com/cloudprober/cloudprober/sysvars.Start","file":"/Users/mgarg/code/cloudprober/sysvars/sysvars.go","line":185},"msg":"1692306537 labels=ptype=sysvars,probe=sysvars hostname=\"aaa.local\" start_timestamp=\"1692306537\" version=\"\"","system":"cloudprober","component":"sysvars"}
{"time":"2023-08-17T14:08:58.007281-07:00","level":"INFO","source":{"function":"github.com/cloudprober/cloudprober/common/oauth.(*httpTokenSource).tokenFromHTTP","file":"cloudprober/common/oauth/http_token.go","line":62},"msg":"oauth2: response from token URL: {\"access_token\":\"eyJ ........ z6i7SxS35qzREAcjJQ\"}","system":"cloudprober","probe":"staging_companies"}