Rocketchat unstable after mongoDB restart (docker-composer v3 depends_on issue?)

[jalbstmeijer]

Description:

After a mongoDB restart (docker recreate), rocketchat becomes unstable.

Not sure if depends_on in docker-composer v2 forced a rocketchat restart.
In that case, how to handle that in newer docker-composer syntax versions?

Steps to reproduce:

1. recreate the mongoDB container
2. see some clients not being able to send all messages (feels random)
3. restart rocketchat container

Expected behavior:

Rocketchat fully recovers after a mongoDB restart/recreate or docker forces a Rocketchat restart in such case.

Actual behavior:

Rocketchat becomes unstable, and stays that way till it's restarted (left it hours)

Server Setup Information:

• Version of Rocket.Chat Server: 4.6.3
• Operating System: official docker image
• Deployment Method: docker
• Number of Running Instances: 1
• DB Replicaset Oplog: 1
• NodeJS Version: 14.18.3
• MongoDB Version: 5.0.8

Additional context

Reading online this issue might not be there with the older docker-composer v2 "depends_on" syntax behavior.
I don't specify any version on top of my docker-composer file now.

Relevant logs:

Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================
Exception in setInterval callback: MongoServerSelectionError: getaddrinfo ENOTFOUND mongo
     at Timeout._onTimeout (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/sdam/topology.js:437:30)
     at listOnTimeout (internal/timers.js:557:17)
     at processTimers (internal/timers.js:500:7) {
   reason: TopologyDescription {
     type: 'Single',
     setName: null,
     maxSetVersion: null,
     maxElectionId: null,
     servers: Map(1) { 'mongo:27017' => [ServerDescription] },
     stale: false,
     compatible: true,
     compatibilityError: null,
     logicalSessionTimeoutMinutes: null,
     heartbeatFrequencyMS: 10000,
     localThresholdMS: 15,
     commonWireVersion: 13
   }
 }
 
 {"level":50,"time":"2022-05-26T14:46:12.417Z","pid":10,"hostname":"3d65f933ba3f","name":"LivechatEnterprise","section":"Queue","msg":"Error processing queue public","err":{"type":"MongoServerSelectionError","message":"getaddrinfo ENOTFOUND mongo","stack":"MongoServerSelectionError: getaddrinfo ENOTFOUND mongo\n    at Timeout._onTimeout (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/sdam/topology.js:437:30)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at app/models/server/raw/OmnichannelQueue.ts:43:15\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at ee/app/livechat-enterprise/server/lib/LivechatEnterprise.js:269:4\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","name":"MongoServerSelectionError","reason":{"type":"Single","setName":null,"maxSetVersion":null,"maxElectionId":null,"servers":{},"stale":false,"compatible":true,"compatibilityError":null,"logicalSessionTimeoutMinutes":null,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"commonWireVersion":13}},"msg":"getaddrinfo ENOTFOUND mongo"}

[skreddy6673]

@jalbstmeijer Did you find any solution for this? we are in Rocketchat version 4.4.2 and mongo version 4.4.15 and since then mongo pods are restarting.

[WebsliceNL]

No, I think for now I went back to docker-composer file version 2, so RocketChat gets a reboot too if mongo does.

So not sure what you mean with "since then mongo pods are restarting"

[skreddy6673]

@WebsliceNL Sorry i mean since when we upgraded to those versions we are seeing the mongo pod restarts.

[WebsliceNL]

In that case it's not the same issue. If you really think this is RocketChat related, open an new issue.

[Sirttas]

I encountered the same issue tried restarting RC/mongo even host. It would be great this team start fixing things instead of removing free features.