Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to create a new connection for the connection pool. #337

Open
Panoplos opened this issue May 31, 2017 · 25 comments
Open

Fail to create a new connection for the connection pool. #337

Panoplos opened this issue May 31, 2017 · 25 comments
Labels
question

Comments

@Panoplos
Copy link

Just trying out the library for the first time, with a simple

> r = require('rethinkdbdash')({db: 'test'})

I am getting the following:

Creating a pool connected to localhost:28015
{ [Function: _r]
  row: { [Function: term] _query: [ 13 ], _r: [Circular] },
  monday: { [Function: term] _query: [ 107 ], _r: [Circular] },
  tuesday: { [Function: term] _query: [ 108 ], _r: [Circular] },
  wednesday: { [Function: term] _query: [ 109 ], _r: [Circular] },
  thursday: { [Function: term] _query: [ 110 ], _r: [Circular] },
  friday: { [Function: term] _query: [ 111 ], _r: [Circular] },
  saturday: { [Function: term] _query: [ 112 ], _r: [Circular] },
  sunday: { [Function: term] _query: [ 113 ], _r: [Circular] },
  january: { [Function: term] _query: [ 114 ], _r: [Circular] },
  february: { [Function: term] _query: [ 115 ], _r: [Circular] },
  march: { [Function: term] _query: [ 116 ], _r: [Circular] },
  april: { [Function: term] _query: [ 117 ], _r: [Circular] },
  may: { [Function: term] _query: [ 118 ], _r: [Circular] },
  june: { [Function: term] _query: [ 119 ], _r: [Circular] },
  july: { [Function: term] _query: [ 120 ], _r: [Circular] },
  august: { [Function: term] _query: [ 121 ], _r: [Circular] },
  september: { [Function: term] _query: [ 122 ], _r: [Circular] },
  october: { [Function: term] _query: [ 123 ], _r: [Circular] },
  november: { [Function: term] _query: [ 124 ], _r: [Circular] },
  december: { [Function: term] _query: [ 125 ], _r: [Circular] },
  minval: { [Function: term] _query: [ 180 ], _r: [Circular] },
  maxval: { [Function: term] _query: [ 181 ], _r: [Circular] },
  nextVarId: 1,
  _Term: [Function: Term],
  _poolMaster: 
   PoolMaster {
     _r: [Circular],
     _line: Dequeue { start: 0, end: 0, buffer: [Object] },
     _pools: { unknownPools: [Object] },
     _healthyPools: [ [Object] ],
     _healthy: true,
     _init: false,
     _index: 0,
     _indexUnknown: 0,
     _discovery: false,
     _options: { db: 'test', buffer: 50, max: 1000 },
     _log: [Function],
     _draining: false,
     _numConnections: 0,
     _numAvailableConnections: 0,
     _hasPrintWarningLocalhost: false,
     _feed: null,
     _consecutiveFails: -1,
     _timeoutError: 1000,
     _maxExponent: 6,
     _seed: 0,
     _servers: [ [Object] ] },
  _options: {} }
> Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Exiting slow growth mode

RethinkDB is running, as I am able to access it via both the dashboard and the rethinkdb package, and the port is default.

I am able to use the resultant r object to query the DB, too.

> r.tableList().run().then( console.dir )
Promise {
  _bitField: 0,
  _fulfillmentHandler0: undefined,
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
> [ 'consumers' ]

Environment Info

Machine: Intel 64bit
OS: Ubuntu 17.04 (Zesty)
Node: v7.10.0
RethinkDB: built from git (next branch)
rethinkdbdash: 2.3.29
@neumino
Copy link
Owner

neumino commented Jun 1, 2017

Did you start rethinkdb after starting your nodejs script?

@Panoplos
Copy link
Author

Panoplos commented Jun 1, 2017

No, it is running as a daemon on the system.

@Panoplos
Copy link
Author

Panoplos commented Jun 2, 2017

Any insight into this? This is obviously a show-stopper for us.

@Panoplos
Copy link
Author

Panoplos commented Jun 2, 2017 •
edited

I tested a slightly different configuration, where I set pool: false, then attempted the following:

import Rethinkdbdash from 'rethinkdbdash'
const r = Rethinkdbdash({pool: false})
r.connect()
  .then(debug.log)
  .error(debug.error)

Running as follows:

⟩ env DEBUG=test node --require 'babel-register' test.js

Results in:

  test
---------------------------------------
ReqlDriverError
Failed to connect to localhost:28015 in less than 20s.
ReqlDriverError
    at Timeout._onTimeout (/home/panoplos/test/node_modules/rethinkdbdash/lib/connection.js:103:12)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)
---------------------------------------

Note that I have also tested the following:

import r from 'rethinkdb'
r.connect( {host: 'localhost', port: 28015}, (err, conn) => {
  if (err) {
    debug.error(err)
  } else {
    debug.log("Connected successfully!")
  }
})

With the following result:

⟩ env DEBUG=test node --require 'babel-register' test.js 
  test   Connected successfully!

@Panoplos
Copy link
Author

Panoplos commented Jun 2, 2017 •
edited

More testing:

I have tested this against next, v2.4.x and v2.3.x, and it works only on v2.3.x of rethinkdb, so it seems the driver is not working with v2.4+ versions.

@neumino
Copy link
Owner

neumino commented Jun 2, 2017

I don't see a 2.4.x release for RethinkDB - Does it work with the official driver?

@Panoplos
Copy link
Author

Panoplos commented Jun 2, 2017 •
edited

v2.4.x is a branch of the git repo. It is not an official release, but it is stable, as is next.
And, yes, the official driver works against these.

@neumino
Copy link
Owner

neumino commented Jun 5, 2017

Hum, rethinkdb doesn't build on my workstation (next or v2.4.x). That being said, it's probably an issue with rethinkdb though.

I don't see any change in the official driver for 2.4, so it's supposed to use the same protocol (and I don't see any mention of a different API in the changelog)

@Panoplos
Copy link
Author

Panoplos commented Jun 6, 2017

What system are you attempting to build on? I had a small issue with building it on Ubuntu, but it was just a matter of compiling with the correct version of gcc.

@BhaskaranR
Copy link

I get same error with rethinkdb docker image package verion - 2.3.5, Initially it connects and then throws the error

{
    servers: [
        { host: '192.168.99.100', port: 32780 },
        { host: '192.168.99.100', port: 32779 }
    ],
    pool: true,
    buffer: 300,
    max: 3000
}


Creating a pool connected to 192.168.99.100:32780
Creating a pool connected to 192.168.99.100:32780
(node:24627) [DEP0016] DeprecationWarning: 'root' is deprecated, use 'global'
{"kind":"notice","notice":"hello seneca cmj0k3p0xwii/1502666893090/24627/3.3.0/-","level":"info","when":1502666895646}
Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to 192.168.99.100:32780 in less than 20s","message":"Failed to connect to 192.168.99.100:32780 in less than 20s.","isOperational":true}
Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to 192.168.99.100:32780 in less than 20s","message":"Failed to connect to 192.168.99.100:32780 in less than 20s.","isOperational":true}

@BhaskaranR
Copy link

@Panoplos were you able to fix this?

@Panoplos
Copy link
Author

No. I had to downgrade my RethinkDB installation.

@neumino
Copy link
Owner

neumino commented Aug 14, 2017

@BhaskaranR - you are running the stable version, you probably have a different problem. Please open a new bug instead

@dustinrouillard
Copy link

This really needs to be fixed.. I hate the default rdb module.. I love this for the ease of use and would love to continue to use it. But this is preventing me from doing so... This doesn't seem to be a rethinkdb server issue.. This is only happening since updating rdb.

@adamierymenko
Copy link

Yes this an issue here as well.

@adamierymenko
Copy link

With next.

@adamierymenko
Copy link

It looks as if _compareDigest is not being called with newer versions which is never calling clearTimeout to clear the connection timeout timer.

@Zalasanjay
Copy link

Zalasanjay commented Dec 13, 2017 •
edited

i'm also getting this same issue i have installed
rethinkdb 2.3.5 (GCC 5.4.0) rethinkdbdash 2.3.31,

@heathweaver
Copy link

heathweaver commented Feb 14, 2018 •
edited

Everything was running well and suddenly this started happening for me as well.
rethinkdb 2.3.6~0xenial (GCC 5.3.1)

(restarted my docker container, not rethinkdb, but the application and it was okay).

@SatyaVaraprasad1978
Copy link

I am also facing the same problem, Is it resolved after restarting the docker container?

@gamebak
Copy link

gamebak commented Aug 6, 2018 •
edited

Same problem over here. I have 2 containers, one works while the other gets connection error mentioned above and it gets stuck...

I even attempted to try healthy checks as mentioned in the documentation, but those aren't working either.
Example of attempts:

r.getPoolMaster().on('healthy', function(healthy) {
  if (healthy === true) {
    console.log('We can run queries.');
  }
  else {
    console.log('No queries can be run.');
  }
});

Another attempt:

//attempt to get connections length and available connections, always seems to be 0
r.getPoolMaster().getLength()

What works is to restart container with a timeout... but that's a bad practice on a big project in production...
Any ideas on this? (I downgraded from the recent August update, but it seems I' still having the same issue)

@sudomoose
Copy link

Still having this issue too, it makes rethink nearly unusable. I still haven't been able to find a solution. This issue really needs more attention.

@gamebak
Copy link

gamebak commented Aug 8, 2018

My current solution is to have my own rate limit, because my current architecture allows it.
I am reading from the message queue at an interval of time and I kept a close eye when I had timeouts, when that happen I reduced the number of messages read.

I hope this gives someone at least a solution on how to deal with this sort of thing. Another solution that I had in mind was to drain the pool periodically, but you will need to add extra checks in your code, in case if a pool is being drained you will need to not do any query and to wait for that to finish, by default you'll get an error if it's not treated.

@ssribeiro
Copy link

My current solution is to have my own rate limit, because my current architecture allows it.
I am reading from the message queue at an interval of time and I kept a close eye when I had timeouts, when that happen I reduced the number of messages read.

I hope this gives someone at least a solution on how to deal with this sort of thing. Another solution that I had in mind was to drain the pool periodically, but you will need to add extra checks in your code, in case if a pool is being drained you will need to not do any query and to wait for that to finish, by default you'll get an error if it's not treated.

good solution, but a hell code situation. better dive into the origin of the bug. lots of headaches here

@MattMacGregor
Copy link

I was wondering if anyone has figured out what is going on that is causing this bug. I dug around and it does seem like _compareDigest isn't being called. After looking even more only one data event is being sent per a connection, which seems to be causing the time out, as the self.state in connection.js is only reaches 1 then stops, where to execute _compareDigest it needs to be equal to 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests