CLOUDP-253916 - readinessProbe: send to stdout as well, give option for compression and reduce default size

[nammn]

Summary:

example output with the console/stdout logger

  Warning  Unhealthy  2m28s  kubelet            Readiness probe failed: {"level":"error","ts":1718279186.4724512,"msg":"health status file not avaible yet: open /var/log/mongodb-mms-automation/agent-health-status.json: no such file or directory "}
  Warning  Unhealthy  83s    kubelet            Readiness probe failed: {"level":"debug","ts":1718279251.0533059,"msg":"ExpectedToBeUp: true, IsInGoalState: false, LastMongoUpTime: 1970-01-01 00:00:00 +0000 UTC"}
{"level":"info","ts":1718279251.053355,"msg":"Mongod is not ready"}
{"level":"info","ts":1718279251.0533628,"msg":"Reached the end of the check. Returning not ready."}
  Warning  Unhealthy  78s  kubelet  Readiness probe failed: {"level":"debug","ts":1718279256.0862045,"msg":"ExpectedToBeUp: true, IsInGoalState: false, LastMongoUpTime: 1970-01-01 00:00:00 +0000 UTC"}
{"level":"info","ts":1718279256.0862503,"msg":"Mongod is not ready"}
{"level":"info","ts":1718279256.086258,"msg":"Reached the end of the check. Returning not ready."}
  Warning  Unhealthy  33s  kubelet  Readiness probe failed: {"level":"debug","ts":1718279301.0727608,"msg":"ExpectedToBeUp: true, IsInGoalState: false, LastMongoUpTime: 2024-06-13 11:48:16 +0000 UTC"}
{"level":"debug","ts":1718279301.0728135,"msg":"The Agent hasn't reported working on the new config yet, the last plan finished at 2024-06-13T11:48:15Z"}
{"level":"info","ts":1718279301.0728211,"msg":"Reached the end of the check. Returning not ready."}
  Warning  Unhealthy  28s  kubelet  Readiness probe failed: {"level":"debug","ts":1718279306.0484068,"msg":"ExpectedToBeUp: true, IsInGoalState: false, LastMongoUpTime: 2024-06-13 11:48:16 +0000 UTC"}
{"level":"debug","ts":1718279306.0484557,"msg":"The Agent hasn't reported working on the new config yet, the last plan finished at 2024-06-13T11:48:15Z"}
{"level":"info","ts":1718279306.0484633,"msg":"Reached the end of the check. Returning not ready."}

All Submissions:

• Have you opened an Issue before filing this PR?
• Have you signed our CLA?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• Put closes #XXXX in your comment to auto-close the issue that your PR fixes (if such).

[nammn]

sending the same information to the console/stdout means that kubectl describe will show if the readinessProbe has failed. This was the prior reason we used panics. Now - no panics are needed for the same level of information.

[nammn]

We can now also just deactivate the logging to the file

[nammn]

prior to that we set 0, which will default to 100mb. That means the readinessProbe might will up to 5x100mb=500mb of logs. Note - those logs are not that useful, while still filling up the disk

[mircea-cosbuc]

LGTM!