New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoError: no mongos proxy available, after upgrading from 4.1.2 to 4.4.6 #3945

Closed
aartiles opened this Issue Mar 4, 2016 · 15 comments

Comments

Projects
None yet
10 participants
@aartiles

aartiles commented Mar 4, 2016

After upgrading from 4.1.2 to 4.4.6 we are randomly getting MongoError: no mongos proxy available.
Downgrading to 4.1.2 makes it work stable.

MongoDB cluster with 6 shards (Primary/Secondary/Arbiter each), 2 mongos and 3 config servers MongoDB v3.0.9
Node.js v0.10.24

Options

    options.server = {
      auto_reconnect: true,
      poolSize: 5,
      socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 },
      reconnectTries: 3000
    };
    options.replset = {
      auto_reconnect: true,
      poolSize: 5,
      socketOptions: { keepAlive: 1, connectTimeoutMS : 30000 },
      reconnectTries: 3000
    };
    options.mongos = true;

The connection string is something like this:

var uri = "mongodb://mongodb-streaming-mongos01.mymongo.com:15000,mongodb-streaming-mongos02.mymongo.com:15000/dbname";
var db = mongoose.createConnection(uri, options);

Stack trace

MongoError: no mongos proxy available
MongoError: no mongos proxy available
at [object Object].State.pickServer (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/mongos.js:192:33)
at [object Object].Mongos.command (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/mongos.js:595:33)
at [object Object].Mongos.command (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/lib/mongos.js:342:17)
at executeCommand (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/lib/db.js:298:19)
at [object Object].Db.command (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/lib/db.js:322:44)
at [object Object].wrapped (/data/home/socialb/node_modules/newrelic/lib/instrumentation/mongodb.js:185:25)
at count (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/lib/collection.js:1963:13)
at [object Object].Collection.count (/data/home/socialb/node_modules/mongoose/node_modules/mongodb/lib/collection.js:1923:44)
at [object Object].wrapped (/data/home/socialb/node_modules/newrelic/lib/transaction/tracer/index.js:155:28)
at [object Object].wrappedQuery (/data/home/socialb/node_modules/newrelic/lib/instrumentation/mongodb.js:218:53)
at [object Object].wrapped [as count] (/data/home/socialb/node_modules/newrelic/lib/instrumentation/mongodb.js:188:22)
at NativeCollection.(anonymous function) [as count] (/data/home/socialb/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:124:26)
at NodeCollection.count (/data/home/socialb/node_modules/mongoose/node_modules/mquery/lib/collection/node.js:46:19)
at Query._count (/data/home/socialb/node_modules/mongoose/lib/query.js:1276:20)
at /data/home/socialb/node_modules/mongoose/node_modules/kareem/index.js:156:8
at /data/home/socialb/node_modules/mongoose/node_modules/kareem/index.js:18:7
at wrapped (/data/home/socialb/node_modules/newrelic/lib/transaction/tracer/index.js:155:28)
at process._tickDomainCallback (node.js:459:13)
at process.wrappedFunction (/data/home/socialb/node_modules/newrelic/lib/transaction/tracer/index.js:245:51)

Given the complexity of the topology and the random nature of the error we can't provide a test case.

@vkarpov15 vkarpov15 modified the milestones: 4.4.7, 4.4.8 Mar 6, 2016

@xavimb

This comment has been minimized.

xavimb commented Mar 12, 2016

I'm having the same issue in 4.4.4

@bendytree

This comment has been minimized.

bendytree commented Mar 16, 2016

Same issue on mongoose 4.4.7: MongoError: no mongos proxy available.
MongoDB v3.2.1
Node.js v0.10.24

Restarting usually fixes it, but not always.

Stack trace

at State.pickServer (/mnt/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/mongos.js:192:33)
at Mongos.getServer (/mnt/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/mongos.js:740:29)
at nextFunction (/mnt/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:505:35)
at Cursor.next [as _next] (/mnt/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:685:3)
at fetchDocs (/mnt/app/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:844:10)
at toArray (/mnt/app/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:871:3)
at Cursor.toArray (/mnt/app/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:824:44)
at /mnt/app/node_modules/mongoose/node_modules/mquery/lib/collection/node.js:29:12
at handleCallback (/mnt/app/node_modules/mongoose/node_modules/mongodb/lib/utils.js:96:12)
at Collection.find (/mnt/app/node_modules/mongoose/node_modules/mongodb/lib/collection.js:354:44)
at NativeCollection.(anonymous function) [as find] (/mnt/app/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:125:28)
at NodeCollection.find (/mnt/app/node_modules/mongoose/node_modules/mquery/lib/collection/node.js:26:19)
at Query.find (/mnt/app/node_modules/mongoose/node_modules/mquery/lib/mquery.js:1630:20)
at Query._find (/mnt/app/node_modules/mongoose/lib/query.js:1054:26)
at /mnt/app/node_modules/mongoose/node_modules/kareem/index.js:156:8
at /mnt/app/node_modules/mongoose/node_modules/kareem/index.js:18:7

Here's how I connect:

var connectionString = "mongodb://username:password@aws-us-east-1-portal.4.dblayer.com:10437,aws-us-east-1-portal.3.dblayer.com:10200/dbname?ssl=true";
var options = {
    "mongos": {
      "ssl": true,
      "sslValidate": false
    }
};
var mongoose = require('mongoose');
mongoose.connect(connectionString, options, function(err, db) { ... });
@vkarpov15

This comment has been minimized.

Collaborator

vkarpov15 commented Mar 17, 2016

Some repro instructions would be helpful - @xavimb @bendytree can you show connection strings and configuration options?

@aartiles

This comment has been minimized.

aartiles commented Mar 17, 2016

I've update the description with a connection string example.

@vkarpov15 vkarpov15 modified the milestones: 4.4.9, 4.4.8 Mar 18, 2016

@bendytree

This comment has been minimized.

bendytree commented Mar 21, 2016

I've added connection information as well.

@crossman

This comment has been minimized.

crossman commented Mar 21, 2016

We're seeing the same issue too. It's impossible to reproduce predictably.

MongoDB v3.0.4
Node v5.7
mongoose v4.4.7

Like @bendytree, restarting usually fixes it.

Stack Trace

Unhandled rejection MongoError: no mongos proxy available
     at [object Object].State.pickServer (/app/node_modules/mongoose/node_modules/mongodb-core/lib/topologies/mongos.js:192:33)
     at [object Object].Mongos.getServer (/app/node_modules/mongoose/node_modules/mongodb-core/lib/topologies/mongos.js:740:29)
     at nextFunction (/app/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:505:35)
     at [object Object].Cursor.next [as _next] (/app/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:685:3)
     at fetchDocs (/app/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:844:10)
     at toArray (/app/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:871:3)
     at [object Object].Cursor.toArray (/app/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:824:44)
     at indexInformation (/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1607:39)
     at [object Object].Db.indexInformation (/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1571:44)
     at ensureIndex (/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1066:8)
     at [object Object].Db.ensureIndex (/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1042:44)
     at ensureIndex (/app/node_modules/mongoose/node_modules/mongodb/lib/collection.js:1814:13)
     at [object Object].Collection.ensureIndex (/app/node_modules/mongoose/node_modules/mongodb/lib/collection.js:1802:44)
     at [object Object].tryCatcher (/app/node_modules/bluebird/js/main/util.js:26:23)
     at [object Object].ret [as ensureIndexAsync] (eval at <anonymous> (/app/node_modules/bluebird/js/main/promisify.js:163:12), <anonymous>:13:39)
     at MongoStore.setAutoRemoveAsync (/app/node_modules/connect-mongo/src/index.js:123:40)
     at MongoStore.handleNewConnectionAsync (/app/node_modules/connect-mongo/src/index.js:116:18)
     at NativeConnection.<anonymous> (/app/node_modules/connect-mongo/src/index.js:90:72)
     at NativeConnection.g (events.js:261:16)
     at emitNone (events.js:73:20)
     at NativeConnection.emit (events.js:167:7)
     at open (/app/node_modules/mongoose/lib/connection.js:511:11)

Connection settings

var mongo_connection = "mongodb://<username>:<password>@aws-us-east-1-portal.3.dblayer.com:10660,aws-us-east-1-portal.0.dblayer.com:10844/admin?ssl=true"

var mongo_options = {
  ssl: true,
  sslValidate: true,
  sslCA: [process.env.COMPOSE_MONGO_CA],
  ca: [process.env.COMPOSE_MONGO_CA],
  poolSize: 5,
  reconnectTries: 1,
  socketOptions: {keepAlive: 1, connectTimeoutMS: 30000},
  auto_reconnect: true,
}

mongoose.connect(mongo_connection, {mongos: mongo_options, db: mongo_options})

@vkarpov15 vkarpov15 modified the milestones: 4.5, 4.4.9, 4.4.10 Mar 23, 2016

@enicholson

This comment has been minimized.

enicholson commented Mar 23, 2016

I'm encountering the same error using the mongodb driver directly after upgrading from 2.1.7 to 2.1.10. It's intermittent/spontaneous and I haven't seen it on my dev system, so I can't give any repro advice.

@vkarpov15 vkarpov15 modified the milestones: 4.4.11, 4.4.10 Mar 24, 2016

@crispy1989

This comment has been minimized.

crispy1989 commented Mar 25, 2016

I can confirm this using the mongodb driver directly. Downgrading fixes the issue.

@crossman

This comment has been minimized.

crossman commented Mar 25, 2016

Hmm, I just checked our versions and we're using 2.1.7
@crispy1989 what version did you downgrade to?

@bendytree

This comment has been minimized.

bendytree commented Mar 28, 2016

Should we file a bug with the driver (or has someone already done that)?
What's the latest stable version of mongoose? 4.1.2?

@crossman

This comment has been minimized.

crossman commented Mar 29, 2016

mongodb-core issue: mongodb-js/mongodb-core#88
A new version was just released which should in theory solve this mongodb-js/mongodb-core@2b4ebbb

@vkarpov15 would you like a pull request to update the mongodb-core version that mongoose depends on?

@letsgolesco

This comment has been minimized.

letsgolesco commented Mar 29, 2016

@vkarpov15 the (potential) fix for this is in mongodb-core v1.3.12, which is being used by node-mongodb-native v2.1.13 (mongoose is on v2.1.10 right now).

@james075

This comment has been minimized.

james075 commented Mar 31, 2016

Is there an official fix yet ?

@vitali-ausianik

This comment has been minimized.

vitali-ausianik commented Mar 31, 2016

unfortunately not yet

@vkarpov15 vkarpov15 modified the milestones: 4.4.12, 4.4.11 Apr 2, 2016

@vkarpov15

This comment has been minimized.

Collaborator

vkarpov15 commented Apr 3, 2016

There is, just upgraded to mongodb driver 2.1.14 yesterday

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment