Added logging decorator for CuratorFramework to Audit ZK Endpoints

[rosalind210]

Created a decorator for the CuratorFramework to handle logging calls to ZooKeeper. It holds a maps that connects the thread ID, classname, and method nameto the number of zookeeper calls it has made over certain interval of time.

The most important updates:

• LoggingCuratorFramework created that keeps track of call counts and logs when that amount
• New framework is set as CuratorFramework in the SingularityCuratorProvider, if a boolean useLoggingCuratorFramework is true in the SingularityConfiguration
• logging curator's clear() performed by a timer, clears every 30 milliseconds.

[1/28 Update] Added timer to LoggingCuratorFramework that performs a clear (logs and empties everything in the map) every 3 minutes.

[1/29] Logs tested in iad03-test, output:

INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 49: TaskManager's getTasks called ZK 1 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 45: TaskManager's getTaskIdsForRequest called ZK 6 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 49: TaskManager's filterInactiveTaskIds called ZK 12 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 47: TaskManager's getTaskHistoryUpdates called ZK 12 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 43: StateManager's getState called ZK 4 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 49: TaskManager's getTaskMetadata called ZK 2 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 49: TaskManager's getHealthcheckResults called ZK 2 times in 180000 milliseconds
INFO  [2021-01-29 21:22:58,168] com.hubspot.singularity.data.LoggingCuratorFramework: 49: DeployManager's getRequestDeployState called ZK 2 times in 180000 milliseconds

[pschoenfelder]

Looks solid overall 👍 Couple minor things but the core is all there

I'm curious — will these logs mostly be read by humans or machines (presumably aggregating for further analysis)? If the latter, may want to make the output format more easily parsable, but otherwise looks good.

[pschoenfelder]

🚢

[ssalinas]

🚢