topology was destroyed #4890

Open
TrejGun opened this Issue Jan 11, 2017 · 7 comments

Projects

None yet

3 participants

@TrejGun
Contributor
TrejGun commented Jan 11, 2017

Do you want to request a feature or report a bug?
bug

What is the current behavior?

MongoError: topology was destroyed
at executeCommand (~/node_modules/mongoose/node_modules/mongodb/lib/db.js:289:76)
at Db.command (~/node_modules/mongoose/node_modules/mongodb/lib/db.js:334:44)
at count (~/node_modules/mongoose/node_modules/mongodb/lib/collection.js:2049:13)
at Collection.count (~/node_modules/mongoose/node_modules/mongodb/lib/collection.js:2004:44)
at NativeCollection.(anonymous function) [as count] (~/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:126:28)
at NodeCollection.count (~/node_modules/mquery/lib/collection/node.js:46:19)
at model.Query.Query._count (~/node_modules/mongoose/lib/query.js:1391:20)
at ~/node_modules/kareem/index.js:239:8
at ~/node_modules/kareem/index.js:18:7
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
From previous event:
	at model.Query.exec (~/node_modules/mongoose/lib/query.js:2294:17)
at model.Query.Query.then (~/node_modules/mongoose/lib/query.js:2341:15)
at AbstractModel.abstractModelWrapper [as count] (~/server/shared/models/abstract.js:138:5)
at VehicleController.wrapper [as count] (~/server/shared/controllers/abstract.js:107:10)
at paginate (~/server/shared/utils/response.js:5:20)
at VehicleController.list (~/server/office/controllers/vehicle.js:75:10)
at ~/server/shared/routes/wrapper.js:40:3
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at next (~/node_modules/express/lib/router/route.js:131:13)
at ~/server/shared/routes/utils.js:65:11
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at next (~/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (~/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at ~/node_modules/express/lib/router/index.js:277:22
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at Function.handle (~/node_modules/express/lib/router/index.js:176:3)
at router (~/node_modules/express/lib/router/index.js:46:12)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at ~/node_modules/express/lib/router/index.js:618:15
From previous event:
	at _passport2.default.deserializeUser (~/server/shared/auth/pathport.js:14:5)
at pass (~/node_modules/passport/lib/authenticator.js:347:9)
at Authenticator.deserializeUser (~/node_modules/passport/lib/authenticator.js:352:5)
at SessionStrategy.authenticate (~/node_modules/passport/lib/strategies/session.js:53:28)
at attempt (~/node_modules/passport/lib/middleware/authenticate.js:348:16)
at authenticate (~/node_modules/passport/lib/middleware/authenticate.js:349:7)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at initialize (~/node_modules/passport/lib/middleware/initialize.js:53:5)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at ~/node_modules/express-session/index.js:473:7
at Command.callback (~/node_modules/connect-redis/lib/connect-redis.js:160:14)
at normal_reply (~/node_modules/redis/index.js:714:21)
at RedisClient.return_reply (~/node_modules/redis/index.js:816:9)
at JavascriptRedisParser.returnReply (~/node_modules/redis/index.js:188:18)
at JavascriptRedisParser.execute (~/node_modules/redis-parser/lib/parser.js:491:12)
at Socket.<anonymous> (~/node_modules/redis/index.js:267:27)
at emitOne (events.js:96:13)

I'm doing nothing special but having this error at list once a day

Please mention your node.js, mongoose and MongoDB version.
mongoose 4.7.6
➜ node -v
v7.0.0
➜ npm -v
4.0.2
➜ mongo --version
MongoDB shell version v3.4.0

@varunjayaraman
Collaborator
varunjayaraman commented Jan 12, 2017 edited

hey @TrejGun I get this sometimes too, usually during my tests because sometimes i forget to close my db connections in my tests. Couple of questions:

Can you give us more info on what environment this is happening in? Can you paste your mongoose configuration and how the connection is established, any options you pass?

@TrejGun
Contributor
TrejGun commented Jan 16, 2017 edited

on my localhost
I'm not using shards or replication

and using pm2, webpack, babel, react with hot reloading
but none of them can't cause destroyed topology

mongoose.Promise = bluebird.Promise;

const url = "mongodb://localhost/dbName";
const options = {
	server: {
		socketOptions: {
			keepalive: 1
		}
	}
}

const db = mongoose.createConnection();

db.on("disconnected", () => {
	db.open(url, Object.assign({}, options, {promiseLibrary: bluebird}));
});

db.on("error", error => {
	db.close();
});

db.open(url, Object.assign({}, options, {promiseLibrary: bluebird}));
@varunjayaraman
Collaborator

Ok thanks. I was gonna ask if you were running this behind a load balancer but it looks like your db lives on the same instance as your app.

Do you have error logs for your application and can you cross reference them for when your db errors and your app issues db.close().

I wonder if your db is closing while a db call is ongoing or about to happen in your app. For instance, are there any db operations that happen on application startup?

Another theory is that your db error event listener is never triggered and the db maintains an open connection if an error crashes your server. I generally like to handle db connections in a SIGINT or SIGKILL or SIGTERM listener as well.

Either way, this might be difficult to debug unless you have some logs in place.

@TrejGun
Contributor
TrejGun commented Jan 17, 2017

probably you are right this can be caused by webpack's --watch option which restarts dev server at any time, ie while db call is ongoing

not sure how to handle this gracefully, may be it make sense to add connectTimeoutMS: 30000 to connection options?

here are some more errors

MongoError: topology was destroyed
at executeCommand (~/node_modules/mongoose/node_modules/mongodb/lib/db.js:289:76)
at Db.command (~/node_modules/mongoose/node_modules/mongodb/lib/db.js:334:44)
at Collection.aggregate (~/node_modules/mongoose/node_modules/mongodb/lib/collection.js:2686:13)
at NativeCollection.(anonymous function) [as aggregate] (~/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:126:28)
at ~/node_modules/mongoose/lib/aggregate.js:576:10
at Aggregate.exec (~/node_modules/mongoose/lib/aggregate.js:562:10)
at Function.aggregate (~/node_modules/mongoose/lib/model.js:2405:13)
at AbstractModel.aggregate (~/server/shared/models/abstract.js:111:10)
at VehicleController.wrapper [as aggregate] (~/server/shared/controllers/abstract.js:107:10)
at VehicleController.typeahead (~/server/office/controllers/vehicle.js:28:15)
at ~/server/shared/routes/wrapper.js:40:3
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at next (~/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (~/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at ~/node_modules/express/lib/router/index.js:277:22
From previous event:
	at AbstractModel.aggregate (~/server/shared/models/abstract.js:111:10)
at VehicleController.wrapper [as aggregate] (~/server/shared/controllers/abstract.js:107:10)
at VehicleController.typeahead (~/server/office/controllers/vehicle.js:28:15)
at ~/server/shared/routes/wrapper.js:40:3
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at next (~/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (~/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at ~/node_modules/express/lib/router/index.js:277:22
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at Function.handle (~/node_modules/express/lib/router/index.js:176:3)
at router (~/node_modules/express/lib/router/index.js:46:12)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at ~/node_modules/express/lib/router/index.js:618:15
at next (~/node_modules/express/lib/router/index.js:256:14)
at Function.handle (~/node_modules/express/lib/router/index.js:176:3)
at router (~/node_modules/express/lib/router/index.js:46:12)
From previous event:
	at _passport2.default.deserializeUser (~/server/shared/auth/pathport.js:14:5)
at pass (~/node_modules/passport/lib/authenticator.js:347:9)
at Authenticator.deserializeUser (~/node_modules/passport/lib/authenticator.js:352:5)
at SessionStrategy.authenticate (~/node_modules/passport/lib/strategies/session.js:53:28)
at attempt (~/node_modules/passport/lib/middleware/authenticate.js:348:16)
at authenticate (~/node_modules/passport/lib/middleware/authenticate.js:349:7)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at initialize (~/node_modules/passport/lib/middleware/initialize.js:53:5)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at ~/node_modules/express-session/index.js:473:7
at Command.callback (~/node_modules/connect-redis/lib/connect-redis.js:160:14)
at normal_reply (~/node_modules/redis/index.js:714:21)
at RedisClient.return_reply (~/node_modules/redis/index.js:816:9)
at JavascriptRedisParser.returnReply (~/node_modules/redis/index.js:188:18)
at JavascriptRedisParser.execute (~/node_modules/redis-parser/lib/parser.js:491:12)
at Socket.<anonymous> (~/node_modules/redis/index.js:267:27)
at emitOne (events.js:96:13)
MongoError: Topology was destroyed
at nextFunction (~/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:541:25)
at Cursor.next [as _next] (~/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:693:3)
at nextObject (~/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:671:8)
at Cursor.next (~/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:261:12)
at findOne (~/node_modules/mongoose/node_modules/mongodb/lib/collection.js:1403:10)
at Collection.findOne (~/node_modules/mongoose/node_modules/mongodb/lib/collection.js:1389:44)
at NativeCollection.(anonymous function) [as findOne] (~/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:126:28)
at NodeCollection.findOne (~/node_modules/mquery/lib/collection/node.js:38:19)
at model.Query.Query.findOne (~/node_modules/mquery/lib/mquery.js:1787:20)
at model.Query.Query._findOne (~/node_modules/mongoose/lib/query.js:1260:22)
at ~/node_modules/kareem/index.js:239:8
at ~/node_modules/kareem/index.js:18:7
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
From previous event:
	at model.Query.exec (~/node_modules/mongoose/lib/query.js:2294:17)
at AbstractModel.enchant (~/server/shared/models/abstract.js:30:16)
at AbstractModel.findOne (~/server/shared/models/abstract.js:62:15)
at VehicleController.wrapper (~/server/shared/controllers/abstract.js:107:10)
at VehicleController.getByUId (~/server/shared/controllers/stateful.js:20:15)
at VehicleController.getById (~/server/office/controllers/vehicle.js:92:16)
at ~/server/shared/routes/wrapper.js:40:3
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at next (~/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (~/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at ~/node_modules/express/lib/router/index.js:277:22
at param (~/node_modules/express/lib/router/index.js:349:14)
at paramCallback (~/node_modules/express/lib/router/index.js:401:21)
at ~/server/shared/routes/private.js:15:5
at paramCallback (~/node_modules/express/lib/router/index.js:404:7)
at param (~/node_modules/express/lib/router/index.js:384:5)
at Function.process_params (~/node_modules/express/lib/router/index.js:410:3)
at next (~/node_modules/express/lib/router/index.js:271:10)
at Function.handle (~/node_modules/express/lib/router/index.js:176:3)
at router (~/node_modules/express/lib/router/index.js:46:12)
From previous event:
	at _passport2.default.deserializeUser (~/server/shared/auth/pathport.js:14:5)
at pass (~/node_modules/passport/lib/authenticator.js:347:9)
at Authenticator.deserializeUser (~/node_modules/passport/lib/authenticator.js:352:5)
at SessionStrategy.authenticate (~/node_modules/passport/lib/strategies/session.js:53:28)
at attempt (~/node_modules/passport/lib/middleware/authenticate.js:348:16)
at authenticate (~/node_modules/passport/lib/middleware/authenticate.js:349:7)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at initialize (~/node_modules/passport/lib/middleware/initialize.js:53:5)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at ~/node_modules/express-session/index.js:473:7
at Command.callback (~/node_modules/connect-redis/lib/connect-redis.js:160:14)
at normal_reply (~/node_modules/redis/index.js:714:21)
at RedisClient.return_reply (~/node_modules/redis/index.js:816:9)
at JavascriptRedisParser.returnReply (~/node_modules/redis/index.js:188:18)
at JavascriptRedisParser.execute (~/node_modules/redis-parser/lib/parser.js:491:12)
at Socket.<anonymous> (~/node_modules/redis/index.js:267:27)
at emitOne (events.js:96:13)
WriteError({"code":11000,"index":0,"errmsg":"E11000 duplicate key error index: DB.users.$email-duplicate dup key: { : \"trejgun@gmail.com\" }","op":{"email":"trejgun@gmail.com"}})
at Function.MongoError.create (~/node_modules/mongoose/node_modules/mongodb-core/lib/error.js:31:11)
at toError (~/node_modules/mongoose/node_modules/mongodb/lib/utils.js:114:22)
at ~/node_modules/mongoose/node_modules/mongodb/lib/collection.js:658:23
at handleCallback (~/node_modules/mongoose/node_modules/mongodb/lib/utils.js:95:56)
at ~/node_modules/mongoose/node_modules/mongodb/lib/bulk/unordered.js:465:9
at handleCallback (~/node_modules/mongoose/node_modules/mongodb/lib/utils.js:95:56)
at resultHandler (~/node_modules/mongoose/node_modules/mongodb/lib/bulk/unordered.js:413:5)
at ~/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:457:18
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
From previous event:
	at Function.create (~/node_modules/mongoose/lib/model.js:1852:17)
at AbstractModel.abstractModelWrapper [as create] (~/server/shared/models/abstract.js:137:10)
at UserController.wrapper [as create] (~/server/shared/controllers/abstract.js:107:10)
at UserController.insert (~/server/oauth2/controllers/user.js:80:15)
at ~/server/shared/routes/wrapper.js:40:3
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at next (~/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (~/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at ~/node_modules/express/lib/router/index.js:277:22
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at Function.handle (~/node_modules/express/lib/router/index.js:176:3)
at router (~/node_modules/express/lib/router/index.js:46:12)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
at ~/node_modules/express/lib/router/index.js:280:7
at Function.process_params (~/node_modules/express/lib/router/index.js:330:12)
at next (~/node_modules/express/lib/router/index.js:271:10)
at ~/node_modules/express/lib/router/index.js:618:15
at next (~/node_modules/express/lib/router/index.js:256:14)
at Function.handle (~/node_modules/express/lib/router/index.js:176:3)
at router (~/node_modules/express/lib/router/index.js:46:12)
at Layer.handle [as handle_request] (~/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (~/node_modules/express/lib/router/index.js:312:13)
@varunjayaraman
Collaborator

Ah I was under the impression this was happening in your production env too. Is this only in dev mode when you run webpack dev server?

Anyway, you can try increasing the connection timeout, that might work. But this is kind of outside mongoose's scope to handle since throwing an error when you try to make a db call when the db isn't accessible is probably a good thing and it is actually a mongo node driver issue moreso than it is a mongoose issue.

That being said, you could probably easily replicate the bug to confirm by making a query that is guaranteed to take a few seconds and then change some server side code that would force a server restart. If you can make it reproducible, I'd recommend you open an issue on https://github.com/mongodb/node-mongodb-native. Sorry if this seems like a cop out but mongoose doesn't do much in the way of db connection besides simply exposing the underlying driver connection to the db.

@TrejGun
Contributor
TrejGun commented Jan 19, 2017 edited

@varunjayaraman yes sure, just tell me please who is responsible for reconnection? because this is the main issue, after that Topology was destroyed mongoose can't reconnect and I should manually restart server

@vkarpov15
Collaborator

So if mongoose reports a connection error then it's your job to try to reconnect. Mongoose will try to reconnect on its own but if it gets Topology was destroyed then that's up to you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment