Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Omit BEGIN/COMMIT statements for empty transactions #32647

Merged
merged 1 commit into from Aug 23, 2018

Conversation

@eugeneius
Copy link
Member

@eugeneius eugeneius commented Apr 19, 2018

Fixes #17937.

If a transaction is opened and closed without any queries being run, we can safely omit the BEGIN and COMMIT statements, as they only exist to modify the connection's behaviour inside the transaction. This removes the overhead of those statements when saving a record with no changes, which makes workarounds like save if changed? unnecessary.

This implementation wraps the raw database connection in a proxy which buffers begin_transaction calls, and materializes them the next time the connection is used, or discards them if end_transaction is called first. This approach makes new connection usage inside the adapters behave correctly by default. Third party adapters can use the begin_transaction/end_transaction APIs to opt in to the new behaviour, but will still work even if they don't.

This implementation buffers transactions inside the transaction manager and materializes them the next time the connection is used. For this to work, the adapter needs to guard all connection use with a call to materialize_transactions. Because of this, adapters must opt in to get this new behaviour by implementing supports_lazy_transactions?.

If raw_connection is used to get a reference to the underlying database connection, the behaviour is disabled and transactions are opened eagerly, as we can't know how the connection will be used. However when the connection is checked back into the pool, we can assume that the application won't use the reference again and reenable lazy transactions. This prevents a single raw_connection call from disabling lazy transactions for the lifetime of the connection.

@rails-bot
Copy link

@rails-bot rails-bot commented Apr 19, 2018

r? @pixeltrix

(@rails-bot has picked a reviewer for you, use r? to override)

@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch Apr 19, 2018
activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb Outdated
@@ -112,7 +112,7 @@ def discard! # :nodoc:
private

def connect
@connection = Mysql2::Client.new(@config)
@connection = LazyTransactionProxy.new(Mysql2::Client.new(@config))

This comment has been minimized.

@rafaelfranca

rafaelfranca Apr 19, 2018
Member

I think we can push this lazy proxy creation up to the abstract class and call that method to avoiding having to remember to wrap it every time a driver connection is create.

This comment has been minimized.

@eugeneius

eugeneius Apr 19, 2018
Author Member

👍 I've added a private connection= method to the abstract adapter that wraps the connection in a proxy, and changed the concrete adapters to use it when they create a connection.

@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch 3 times, most recently Apr 19, 2018
@yahonda
Copy link
Contributor

@yahonda yahonda commented Apr 20, 2018

@eugeneius Let me take a look at if it "breaks" Oracle enhanced adapter. If it does I am happy to update Oracle enhanced adapter.

@eugeneius
Copy link
Member Author

@eugeneius eugeneius commented Apr 20, 2018

Thanks @yahonda! 🙌 I've tried to make the implementation backwards compatible, so please let me know if it does break - it might mean that I made an incorrect assumption about the adapter interface.

@eugeneius
Copy link
Member Author

@eugeneius eugeneius commented Apr 20, 2018

This test failure was legitimate: https://travis-ci.org/rails/rails/jobs/368924286#L1171-L1175

Executing a prepared statement doesn't touch the connection object (at least on the SQLite adapter), but does require the connection's transaction state to be up to date.

I've fixed the problem by materializing transactions in exec_query before prepared statements are executed, but that weakens the "it just works" property that I hoped to get from the proxy approach.

😕

@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch 3 times, most recently Apr 20, 2018
@kbrock
Copy link
Contributor

@kbrock kbrock commented Apr 22, 2018

Thanks @eugeneius This turned out very nicely

/cc @Fryguy

@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch Apr 22, 2018
@eugeneius
Copy link
Member Author

@eugeneius eugeneius commented Apr 22, 2018

I've moved the materialize_transactions call from exec_query into a proxy wrapping the statement object (more proxies!), which feels a bit more robust.

@jeremy jeremy requested review from matthewd, kamipo and sgrif Apr 22, 2018
@sgrif
sgrif approved these changes Apr 22, 2018
Copy link
Contributor

@sgrif sgrif left a comment

This feels like a lot of complexity for very little gain, but the implementation seems fine.

activerecord/lib/active_record/connection_adapters/abstract/lazy_transaction_proxy.rb Outdated
module ActiveRecord
module ConnectionAdapters
class LazyTransactionProxy < SimpleDelegator # :nodoc:
class StatementProxy < SimpleDelegator

This comment has been minimized.

@sgrif

sgrif Apr 22, 2018
Contributor

Does it make sense to use DelegateClass instead for both of these?

This comment has been minimized.

@eugeneius

eugeneius Apr 23, 2018
Author Member

I don't think so, because the type of object they wrap depends on the adapter in use; there's no single class we can provide with the full set of methods they need to support.

This comment has been minimized.

@eugeneius

eugeneius Apr 23, 2018
Author Member

This comment did make me realise that I forgot to :nodoc: the inner class, though.

@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch Apr 23, 2018
yahonda added a commit to yahonda/oracle-enhanced that referenced this pull request Apr 23, 2018
@yahonda
Copy link
Contributor

@yahonda yahonda commented Apr 23, 2018

Oracle enhanced adapter needs some update to support this pull request. Here are CI results for this pull request https://travis-ci.org/yahonda/oracle-enhanced/builds/370051142 .

I am still having some jet lags due after a long flight, I'll take a look at this result in detail tomorrow.

activerecord/lib/active_record/connection_adapters/abstract/lazy_transaction_proxy.rb Outdated
@connection = connection
end

def method_missing(*)

This comment has been minimized.

@sgrif

sgrif Apr 23, 2018
Contributor

This needs to implement respond_to_missing? as well

This comment has been minimized.

@eugeneius

eugeneius Apr 23, 2018
Author Member

Delegator already provides an implementation of respond_to_missing?:
https://github.com/ruby/ruby/blob/v2_4_4/lib/delegate.rb#L95-L104

Since we call super in method_missing, we should do the same for respond_to_missing?:

def respond_to_missing?(*)
  super
end

I've added this to both classes, if only to document that using Delegator's behaviour unmodified is intentional.

This comment has been minimized.

@sgrif

sgrif Apr 24, 2018
Contributor

I'm a bit confused as to why we need to call materialize_transactions in method_missing, but not respond_to_missing?. These two methods should always be in sync.

This comment has been minimized.

@eugeneius

eugeneius Apr 24, 2018
Author Member

The contract between the two methods is that respond_to_missing? will return true iff called with a method that method_missing would dispatch. Usually this means making equivalent changes to both methods to keep them in sync, as you've noted. In this case however, calling materialize_transactions in method_missing doesn't affect which methods are dispatched; it's purely a side effect, and so doesn't require an equivalent change in respond_to_missing?.

Another way of looking at it: we need to call materialize_transactions in method_missing because we don't know whether the call to super will touch the socket. We don't need to call it in respond_to_missing? because we know that calling super won't touch the socket.

activerecord/lib/active_record/connection_adapters/abstract/lazy_transaction_proxy.rb Outdated
__getobj__
end

def method_missing(*)

This comment has been minimized.

@sgrif

sgrif Apr 23, 2018
Contributor

This needs to implement respond_to_missing? as well

@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch 2 times, most recently Apr 23, 2018
@eugeneius
Copy link
Member Author

@eugeneius eugeneius commented Apr 23, 2018

@yahonda can you try triggering another build? I renamed LazyTransactionProxy#raw_connection to LazyTransactionProxy#proxied_connection so it wouldn't clash with this method.

@matthewd
Copy link
Member

@matthewd matthewd commented Apr 23, 2018

A few thoughts from an initial read-through:

Yay! ❤️ As I mentioned in Pittsburgh, it's awesome to have an implementation to look at -- this one's been sitting on the "nice to have" list for a while.

My gut feeling is that this "should" be closer to the transaction manager rather than the connection -- particularly the deferral stack, though the connection of course needs to be responsible for triggering the materialize.

Failing that, I also wonder about putting the functionality on the abstract connection itself rather than a proxy. I don't mind the code complexity needed for more thorough bookkeeping, but the proxy feels architecturally heavyweight.

Side-thought that would probably just distract us to pursue right now, but I'll mention anyway: this has some interesting parallels (deferring work on the connection until we need to run a real statement, and then potentially giving that some special handling) with #28200 and friends.

Finally, the big one, because it's about behaviour rather than implementation: I believe it's important that savepoints don't materialize the transaction -- that they too get deferred. To me, the ideal version of this change allows us to create transactions more freely, which means we can default requires_new to true, eliminating some very surprising behaviours where changes escape a transaction block. Reducing a no-op save from two queries to zero is great, of course.. I just see some other shinies in the distance.

@yahonda
Copy link
Contributor

@yahonda yahonda commented Apr 24, 2018

CI is green now with Oracle enhanced adapter. As a nature of Oracle database transaction management begin and end statements are not executed to start and finish transaction. Then some of newly added tests get "failed" but it is okay for me.

If a transaction is opened and closed without any queries being run, we
can safely omit the `BEGIN` and `COMMIT` statements, as they only exist
to modify the connection's behaviour inside the transaction. This
removes the overhead of those statements when saving a record with no
changes, which makes workarounds like `save if changed?` unnecessary.

This implementation buffers transactions inside the transaction manager
and materializes them the next time the connection is used. For this to
work, the adapter needs to guard all connection use with a call to
`materialize_transactions`. Because of this, adapters must opt in to get
this new behaviour by implementing `supports_lazy_transactions?`.

If `raw_connection` is used to get a reference to the underlying
database connection, the behaviour is disabled and transactions are
opened eagerly, as we can't know how the connection will be used.
However when the connection is checked back into the pool, we can assume
that the application won't use the reference again and reenable lazy
transactions. This prevents a single `raw_connection` call from
disabling lazy transactions for the lifetime of the connection.
@eugeneius eugeneius force-pushed the eugeneius:lazy_transactions branch 3 times, most recently to 0ac81ee Aug 13, 2018
@matthewd matthewd merged commit 123fe0c into rails:master Aug 23, 2018
2 checks passed
2 checks passed
codeclimate All good!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@matthewd
Copy link
Member

@matthewd matthewd commented Aug 23, 2018

Sorry I left this sitting for ages, and thanks again!

❤️ 💚 💙 💛 💜

@jeremy
Copy link
Member

@jeremy jeremy commented Aug 23, 2018

@kbrock
Copy link
Contributor

@kbrock kbrock commented Aug 23, 2018

Thanks so much Matthew and Eugene and all others who took a stab at this and helped it go forward

@Nondv
Copy link

@Nondv Nondv commented Aug 23, 2018

Wow. I was just wondering about omiting empty transactions and found out that it was merged several hours ago.

Thank you, rails community. You are awesome <3

kamipo added a commit that referenced this pull request Oct 4, 2018
`test_update_does_not_run_sql_if_record_has_not_changed` would pass
without #18501 since `assert_queries` ignores BEGIN/COMMIT unless
`ignore_none: true` is given.

Since #32647, empty BEGIN/COMMIT is ommited. So we no longer need to use
`assert_queries(0)` to ignore BEGIN/COMMIT in the queries.
Envek added a commit to Envek/after_commit_everywhere that referenced this pull request Sep 10, 2019
It broke due new to [Lazy transactions](rails/rails#32647) feature.

Workaround is taken from the Isolator gem: palkan/isolator#20
@f3ndot
Copy link

@f3ndot f3ndot commented Sep 11, 2019

I would love if this would be backported to 5.2. Is that something not out of the realm of possibility?

@eugeneius
Copy link
Member Author

@eugeneius eugeneius commented Sep 11, 2019

I don't think so; 5.2 is only supported for security fixes at this point.

@pixeltrix
Copy link
Member

@pixeltrix pixeltrix commented Sep 15, 2019

@eugeneius I'm pretty sure that 5.2 is still receiving some bug fixes but they'll be limited in scope unless warranted by a new Ruby version or some other similar event (e.g. we backported a whole bunch of stuff to Rails 4.2 to make it work with Ruby 2.4's changes to to_time).

However, even given the above I agree that it's unlikely to be backported unless something else makes it necessary.

JesseChavez added a commit to JesseChavez/activerecord-jdbc-adapter that referenced this pull request Dec 19, 2019
YehudaGold added a commit to YehudaGold/activerecord-sqlserver-adapter that referenced this pull request Apr 24, 2020
aidanharan added a commit to aidanharan/activerecord-sqlserver-adapter that referenced this pull request Apr 27, 2020
for implementing supports for lazy transactions rails/rails#32647 (comment)
wpolicarpo pushed a commit to rails-sqlserver/activerecord-sqlserver-adapter that referenced this pull request Apr 29, 2020
* Support lazy transactions

* aaded materialize_transactions guard to connection uses.
for implementing supports for lazy transactions rails/rails#32647 (comment)

* Moved materialize_transactions calls

* Fix typo

* Materialize transaction in do_execute

Co-authored-by: Yehuda Goldberg <hvusvd@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

You can’t perform that action at this time.