[strimzi-canary] Excessive logging by zookeeper

[hhahn-tw]

I deployed strimzi-canary about a month ago and just recently got alerted to problems with zookeeper instances in associated clusters filling up their log volumes. Looking at disk usage historically, this volume was previously static in usage, < 5%, but since deploying strimzi-canary, every single zookeeper instance has shown a steady linear increase in usage.

Inspecting the log files shows records such as the following accounting for 99% of the usage (these are apparently binary files, excuse the formatting):

�9/kafka/brokers/topics/__strimzi_canary/partitions/2/stateQ{"controller_epoch":9,"leader":2,"version":1,"leader_epoch":183200,"isr":[/kafka/controller_epoch
2&/kafka/brokers/topics/__strimzi_canary{"version":2,"partitions":{"2":[2,3,4],"5":[5,8,7],"27":[28,29,27],"12":[13,14,12],"15":[15,16,17],"8":[6,10,9],"21":[23,22,21],"18":[18,20,19],"24":[26,25,24],"7":[7,6,10],"1":[1,2,3],"17":[17,18,20],"23":[21,26,25],"4":[4,5,8],"26":[24,28,29],"11":[11,13,14],"14":[12,15,16],"29":[27,0,1],"20":[19,23,22],"28":[29,27,0],"6":[8,7,6],"9":[10,9,11],"0":[0,1,2],"22":[22,21,26],"16":[16,17,18],"19":[20,19,23],"10":[9,11,13],"3":[3,4,5],"25":[25,24,28],"13":[14,12,15]},"adding_replicas":{},"removi/kafka/controller_epoch�Uj������h

It varies from cluster to cluster in ways I can't figure out. One reached 100% but other, even larger, clusters with the same sized log volume are still under 20% (though climbing). I'm working on configuring zookeeper to automatically purge these regularly but I was wondering if this was an understood issue, if there's any way to reduce the logging volume in general, and why one cluster blew up so much more than the others.