Uncaught exception MongoServerSelectionError during primary node election

[francescov1]

Do you want to request a feature or report a bug?

Report a bug

What is the current behavior?

We are currently running a MongoDB Atlas replica set. We are finding that when our cluster needs to re-elect a new primary, our application loses connection to MongoDB, regains connection, then throws an uncaught exception MongoServerSelectionError: The server is in quiesce mode and will shut down.

If the current behavior is a bug, please provide the steps to reproduce.

Here is the code we use for connecting:

  mongoose
    .connect(config.database, {
      maxPoolSize: 100,
      autoIndex: config.mongoose.autoCreateIndicesAndCollections,
      autoCreate: config.mongoose.autoCreateIndicesAndCollections,
      promiseLibrary: Bluebird
    })
    .catch(handleError);

  mongoose.connection.on('connected', () => log.info('Mongodb connected'));
  mongoose.connection.on('open', () => log.info('Mongodb connection opened'));
  mongoose.connection.on('error', handleError);

  mongoose.connection.on('disconnected', () =>
    log.warn('Mongodb disconnected')
  );

The error can be reproduced using the Test Failover option in Atlas or by scaling the cluster up or down. When doing this, here are the logs we see:

# initial connection
info: Mongodb connected {"timestamp":"2022-04-13T17:06:57.590Z"}
info: Mongodb connection opened {"timestamp":"2022-04-13T17:06:57.595Z"}

# disconnect event from primary re-election
warn: Mongodb disconnected {"timestamp":"2022-04-13T17:14:48.557Z"}

# reconnect events immediately after connection loss
info: Mongodb connected {"timestamp":"2022-04-13T17:14:48.738Z"}
info: Mongodb connection opened {"timestamp":"2022-04-13T17:14:48.739Z"}

# uncaught exception immediately after reconnect
error: MongoServerSelectionError: The server is in quiesce mode and will shut down {"type":"uncaughtException","timestamp":"2022-04-13T17:15:19.418Z"}

What is the expected behavior?

• Mongoose does not throw an uncaught exception, the error is reported through the mongoose.connection.on('error', ...) listener.
• Mongoose does not lose connection during a re-election (this could be an underlying Mongo issue so might not be relevant here).

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.

Node.js: 16.8
Mongoose: 6.1.6
MongoDB: 5.0

[francescov1]

Not sure how to create a repro script since my only way to repro is to perform a Test Failover in Mongo Atlas. All you need is a local Node.js process running with an opened connection to the database to see the issue.

[Uzlopak]

Die you try Mongoose 6.3? We updated the Mongodb native Driver. Maybe they fixed the issue?

[francescov1]

I'll try that out!

[francescov1]

After updating to mongoose 6.3.0 the uncaught exception is not thrown anymore but I am still seeing the disconnect event occur during a re-election (logs look the exact same as above, other than the error log at the end).

[vkarpov15]

@francescov1 seeing the 'disconnected' event during re-election is expected. The unhandled 'error' is not expected, but that looks fixed.

The 'disconnected' event is purely for debugging and monitoring. It doesn't necessarily mean that Mongoose can't talk to any members in the replica set or execute any operations. It just means that Mongoose lost connectivity to the replica set primary, which is expected during replica set failover. As long as your replica set failover happens quickly (faster than serverSelectionTimeoutMS) none of your operations should fail, you can still save(), find(), etc. But those operations won't succeed until after the replica set failover completes.

[francescov1]

@vkarpov15 Thanks for the thorough explanation, this is great to hear 🙌