Log takes all space and mongodb is down

[ossinkine]

What did you do to encounter the bug?
We are running a cluster with 3 nodes, it works about 2 months.
Once one replica gone down.
After some investigation we found that the file /var/log/mongodb-mms-automation/mongodb.log takes all spaces in /var/log/mongodb-mms-automation mount.

What did you expect?
I expect /var/log/mongodb-mms-automation stores something for agent or similar but not mongod logs.
I expect mongod writes logs into stdout as containerized applications typically behave.

What happened instead?
No space left in /var/log/mongodb-mms-automation, mongod and agent works improperly.

Operator Information

• Operator Version: 0.6.0
• MongoDB Image used: registry.hub.docker.com/library/mongo:4.4.4

Kubernetes Cluster Information

• Distribution: GKE
• Version: 1.18
• Image Registry location (quay, or an internal registry): default operator configuration

[ossinkine]

It looks like the operator configures mongod to write logs to a local folder. This is rather strange, but even if it is, who is in charge of rotating and deleting old logs? If you do not do this, the disk will inevitably fill up, which happened to me.

[priyolahiri]

Thank you for reporting this. We are looking into this.