New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix corrupt transaction state caused by `before_commit` exceptions #23407

Merged
merged 1 commit into from Feb 1, 2016

Conversation

Projects
None yet
4 participants
@jeremy
Member

jeremy commented Feb 1, 2016

When a before_commit callback raises, the database is rolled back but
AR's record of the current transaction is not, leaving the connection in
a perpetually broken state that affects all future users of the
connection: subsequent requests, jobs, etc. They'll think a transaction
is active when none is, so they won't BEGIN on their own. This manifests
as missing after_commit callbacks and broken ROLLBACKs.

This happens because before_commit callbacks fire before the current
transaction is popped from the stack, but the exception-handling path
they hit assumes that the current transaction was already popped. So the
database ROLLBACK is issued, but the transaction stack is left intact.

Common cause: deadlocked #touch, which is now implemented with
before_commit callbacks.

What's next:

  • We shouldn't allow active transaction state when checking in or out
    from the connection pool. Verify that conns are clean.
  • Closer review of txn manager sad paths. Are we missing other spots
    where we'd end up with incorrect txn state? What's the worst that can
    happen if txn state drifts? How can we guarantee it doesn't and
    contain the fallout if it does?

Thanks for @tomafro for expert diagnosis!

@tomafro

This comment has been minimized.

Contributor

tomafro commented Feb 1, 2016

Just double-double-checked this fixes my reproducible case and it does. Thanks @jeremy.

@rafaelfranca

This comment has been minimized.

Member

rafaelfranca commented Feb 1, 2016

LGTM

@vipulnsward vipulnsward added this to the 5.0.0 milestone Feb 1, 2016

Fix corrupt transaction state caused by `before_commit` exceptions
When a `before_commit` callback raises, the database is rolled back but
AR's record of the current transaction is not, leaving the connection in
a perpetually broken state that affects all future users of the
connection: subsequent requests, jobs, etc. They'll think a transaction
is active when none is, so they won't BEGIN on their own. This manifests
as missing `after_commit` callbacks and broken ROLLBACKs.

This happens because `before_commit` callbacks fire before the current
transaction is popped from the stack, but the exception-handling path
they hit assumes that the current transaction was already popped. So the
database ROLLBACK is issued, but the transaction stack is left intact.

Common cause: deadlocked `#touch`, which is now implemented with
`before_commit` callbacks.

What's next:
* We shouldn't allow active transaction state when checking in or out
  from the connection pool. Verify that conns are clean.
* Closer review of txn manager sad paths. Are we missing other spots
  where we'd end up with incorrect txn state? What's the worst that can
  happen if txn state drifts? How can we guarantee it doesn't and
  contain the fallout if it does?

Thanks for @tomafro for expert diagnosis!

@jeremy jeremy force-pushed the jeremy:corrupt-before-commit branch to 8fd123f Feb 1, 2016

jeremy added a commit that referenced this pull request Feb 1, 2016

Merge pull request #23407 from jeremy/corrupt-before-commit
Fix corrupt transaction state caused by `before_commit` exceptions

@jeremy jeremy merged commit 633969f into rails:master Feb 1, 2016

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@jeremy jeremy deleted the jeremy:corrupt-before-commit branch Feb 1, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment