fix(transaction): fixed unhandled rejection when connection acquiring times out (#9218) #9879

AlexKorn · 2018-09-05T13:44:44Z

Pull Request check-list

Please make sure to review and check all of these items:

Does npm run test or npm run test-DIALECT pass with this change (including linting)?
Does the description below contain a link to an existing issue (Closes #[issue]) or a description of the issue you are solving?
Have you added new tests to prevent regressions?
Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
Did you follow the commit message conventions explained in CONTRIBUTING.md?

Description of change

Error described in issue #9218 happens when connection acquiring for auto-callback transaction times out: sequelize tries to automatically rollback transaction, but there is no connection to send rollback command.
Added check for inexistent connection, test and special erorr type.

… times out (sequelize#9218)

sushantdhiman · 2018-09-06T06:06:42Z

lib/dialects/abstract/connection-manager.js

@@ -273,6 +274,7 @@ class ConnectionManager {
    return promise.then(() => {
      return this.pool.acquire(options.priority, options.type, options.useMaster)
        .then(mayBeConnection => this._determineConnection(mayBeConnection))
+        .catch(/TimeoutError/, err => { throw new errors.ConnectionAcquireTimeoutError(err); })


Throw this error in _determineConnection, we also we want to cover cases of replication

Maybe I miss your point, but when TimeoutError happens, process doesn't fall in _determineConnection? Or the point is that check should be copied to _determineConnection?

In case of timeout error mayBeConnection should be TimeError instance, which can be used to throw ConnectionAcquireTimeoutError inside _determineConnection

This is important because _determineConnection is used by both common pool and replication pool. Otherwise you will need to apply this same check to three different places (where pool.acquire is used)

Timeout error is unfortunately exception from case described in #8330 - it is really thrown by pool, so _determineConnection is not called. By the way, is replication pool used directly, bypassing getConnection() method of connection-manager? Search didn't show me such cases, isn't call to _determineConnection redundant in replication pool acquire() method?

sushantdhiman · 2018-09-06T06:10:35Z

Dedicated error for acquire timeout has been talked about before and I think your approach is good.

Could you also work toward an integration test which demonstrate issue with transaction. Limit pool to 1 connection and try to run two transactions. That should be sufficient as a test case. Please also test if similar error which we are trying to resolve in this PR appears by that test without your fix.

codecov · 2018-09-06T09:57:17Z

Codecov Report

Merging #9879 into master will increase coverage by 5.03%.
The diff coverage is 100%.

AlexKorn · 2018-09-06T14:12:05Z

fixed pool integration test concerning this issue: not in pretty way but I'm stuck trying to imagine another. The problem is that this unhandled error is so tricky that it bypasses promise catches, so .eventually.rejectedWith doesn't catch it either. Now if test doesn't pass, it just crashes node process

AlexKorn · 2018-09-06T15:08:12Z

The worst about this error that error message is misleading: first line of call stack says that it happens in promise at this line (for postgres):

new Promise((resolve, reject) => this.client.query(this.sql, (error, result) => error ? reject(error) : resolve(result)));

but in fact it's not the main culprit. It's the line

this.sequelize.log('Executing (' + (this.client.uuid || 'default') + '): ' + this.sql, this.options);

where uuid param of inexistent this.client is referenced, causing process to fail with synchronous error before promise is returned, that's why all .catch()-es doesn't work

sushantdhiman · 2018-09-07T04:55:18Z

lib/dialects/abstract/connection-manager.js

@@ -273,6 +274,7 @@ class ConnectionManager {
    return promise.then(() => {
      return this.pool.acquire(options.priority, options.type, options.useMaster)
        .then(mayBeConnection => this._determineConnection(mayBeConnection))
+        .catch(err => err.name === 'TimeoutError', err => { throw new errors.ConnectionAcquireTimeoutError(err); })


Short version catch('TimeoutError', err => { ... })

Add this check to

sequelize/lib/dialects/abstract/connection-manager.js

Line 154 in 342e18f

return this.pool.read.acquire(priority)

and

sequelize/lib/dialects/abstract/connection-manager.js

Line 157 in 342e18f

return this.pool.write.acquire(priority)

Don't know whether you've seen my question: it seems that acquire() method of replication pool proxy object is never called directly except from getConnection() method, where all necessary checks are made, so wouldn't be catch() here and moreover check for mayBeConnection redundant?

You are right, for both replication and general case, this.pool.acquire in getConnection represent all calls. Please go ahead and remove mayBeConnection check from replication pool proxy's acquire case.

Apart from that just simplify catch('TimeoutError', err => { ... }) and this should be good to go

Removed checks.
Considering catch() simplification: string didn't work. It could be a regex, but then it tries to match with whole stack, therefore fails test for errors in mssql/connection-manager. Found in bluebird docs the way to make it a bit shorter with object predicate.

sushantdhiman · 2018-09-07T04:57:43Z

test/integration/pool.test.js

-      .to.eventually.be.rejectedWith('ResourceRequest timed out');
+    return expect(this.testInstance.transaction(() => {
+      return this.testInstance.transaction(() => {});
+    })).to.eventually.be.rejectedWith(Sequelize.ConnectionAcquireTimeoutError);


sushantdhiman · 2018-09-07T08:38:39Z

lib/dialects/abstract/connection-manager.js

@@ -273,6 +272,7 @@ class ConnectionManager {
    return promise.then(() => {
      return this.pool.acquire(options.priority, options.type, options.useMaster)
        .then(mayBeConnection => this._determineConnection(mayBeConnection))
+        .catch({name: 'TimeoutError'}, err => { throw new errors.ConnectionAcquireTimeoutError(err); })


We could have used https://github.com/coopernurse/node-pool/blob/master/lib/errors.js#L18 but looks like it is not exported. Not super critical either

Yeah, I also wondered whether it was exposed, but not

sushantdhiman · 2018-09-07T08:40:59Z

Thanks @AlexKorn

papandreou · 2018-09-07T08:57:15Z

Do you think this would be backportable to v4?

Cherry-pick of 27e4494

papandreou · 2018-09-07T09:29:44Z

Do you think this would be backportable to v4?

Seems to apply cleanly. Here goes nothing: #9889

fix(transaction): fixed unhandled rejection when connection acquiring…

8719b5a

… times out (sequelize#9218)

sushantdhiman reviewed Sep 6, 2018

View reviewed changes

AlexKorn force-pushed the bugfix/9218 branch from 7468e32 to 8719b5a Compare September 6, 2018 09:57

AlexKorn added 2 commits September 6, 2018 12:59

fix(connection-manager): correct TimeoutError handling

79a4dfa

fix(tests): test on unhandled rejection when transaction rolled back

0773be7

sushantdhiman reviewed Sep 7, 2018

View reviewed changes

fix(connection-manager): removed redundant connection checks

dc4faf3

sushantdhiman approved these changes Sep 7, 2018

View reviewed changes

sushantdhiman merged commit 27e4494 into sequelize:master Sep 7, 2018

sushantdhiman mentioned this pull request Sep 7, 2018

Unhandled rejection TypeError: Cannot read property 'query' of undefined #9218

Closed

papandreou added a commit to peakon/sequelize that referenced this pull request Sep 7, 2018

fix: Backport sequelize#9879 to v4

43a7a56

Cherry-pick of 27e4494

papandreou mentioned this pull request Sep 7, 2018

fix: Backport #9879 to v4 #9889

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(transaction): fixed unhandled rejection when connection acquiring times out (#9218) #9879

fix(transaction): fixed unhandled rejection when connection acquiring times out (#9218) #9879

AlexKorn commented Sep 5, 2018 •

edited

Loading

sushantdhiman Sep 6, 2018

AlexKorn Sep 6, 2018 •

edited

Loading

sushantdhiman Sep 6, 2018

AlexKorn Sep 6, 2018 •

edited

Loading

sushantdhiman commented Sep 6, 2018

codecov bot commented Sep 6, 2018 •

edited

Loading

AlexKorn commented Sep 6, 2018

AlexKorn commented Sep 6, 2018 •

edited

Loading

sushantdhiman Sep 7, 2018

AlexKorn Sep 7, 2018

sushantdhiman Sep 7, 2018

AlexKorn Sep 7, 2018 •

edited

Loading

sushantdhiman Sep 7, 2018

sushantdhiman Sep 7, 2018

AlexKorn Sep 7, 2018

sushantdhiman commented Sep 7, 2018

papandreou commented Sep 7, 2018

papandreou commented Sep 7, 2018

fix(transaction): fixed unhandled rejection when connection acquiring times out (#9218) #9879

fix(transaction): fixed unhandled rejection when connection acquiring times out (#9218) #9879

Conversation

AlexKorn commented Sep 5, 2018 • edited Loading

Pull Request check-list

Description of change

Choose a reason for hiding this comment

AlexKorn Sep 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexKorn Sep 6, 2018 • edited Loading

Choose a reason for hiding this comment

sushantdhiman commented Sep 6, 2018

codecov bot commented Sep 6, 2018 • edited Loading

Codecov Report

AlexKorn commented Sep 6, 2018

AlexKorn commented Sep 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexKorn Sep 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sushantdhiman commented Sep 7, 2018

papandreou commented Sep 7, 2018

papandreou commented Sep 7, 2018

AlexKorn commented Sep 5, 2018 •

edited

Loading

AlexKorn Sep 6, 2018 •

edited

Loading

AlexKorn Sep 6, 2018 •

edited

Loading

codecov bot commented Sep 6, 2018 •

edited

Loading

AlexKorn commented Sep 6, 2018 •

edited

Loading

AlexKorn Sep 7, 2018 •

edited

Loading