Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Current transaction records retained by transactional fixtures (master) #3300

Closed
wants to merge 2 commits into from

7 participants

Will Bryant Sean Walbran Isaac Sanders Steve Klabnik Jon Leighton Łukasz Strzałkowski Carlos Antonio da Silva
Will Bryant

As discussed on the mailing list, this fixes the memory bloat when using transactional fixtures to run large test suites, which is basically a regression since 2.3 due to the side effects of the better record rollback code. This substantially improves test runtime too if you save a lot of objects in tests.

This branch also exposes the option to not retain references to the records saved in transactions, which is useful for applications that have large transactions that save a lot of objects but don't need their state in memory to be rolled back if the transaction fails nor need the transaction callbacks.

willbryant added some commits
Will Bryant willbryant when the transactional fixtures keeps a database transaction alive wi…
…thout using the transaction method, don't retain references to all the records saved in the fixtures transaction once their subtransaction completes
d731f5c
Will Bryant willbryant expose :remember_record_state => false option on transaction(), so ap…
…plications that don't need state rollback (or after_commit/rollback hooks) can garbage collect the records during long transactions
837b369
Sean Walbran

Just fought with this leak myself, happy to see this in play. The approach I took was to add the below to the teardown_fixtures method in fixtures.rb, which seems consistent with the original callbacks commit ( b070739 ) and should allow for the actual rollback callbacks to take effect in tests (for, e.g., cleanup of external references if any, etc.). Any thoughts on this spin instead/as well?

def teardown_fixtures
...
# Rollback changes if a transaction is active.

if run_in_transaction? && ActiveRecord::Base.connection.open_transactions != 0
ActiveRecord::Base.connection.rollback_db_transaction

  • ActiveRecord::Base.connection.send(:rollback_transaction_records,true) if ActiveRecord::Base.connection.instance_variable_get('@_current_transaction_records') ActiveRecord::Base.connection.decrement_open_transactions end ActiveRecord::Base.clear_active_connections! end
Will Bryant

Sean, firing rollback event handlers seems reasonable to me as an additional thing. Haven't been able to produce much interest in that when I asked about it on the rails-core mailing list (see posts around 12 October) though.

Isaac Sanders

Is this still an issue?

Will Bryant

Yes.

Steve Klabnik
Collaborator

This is gonna need a rebase if it's ever gonna get included.

Steve Klabnik
Collaborator

@willbryant ping! are you still interested in keeping up with this pull request?

Carlos Antonio da Silva carlosantoniodasilva commented on the diff
...d/connection_adapters/abstract/database_statements.rb
@@ -170,6 +170,11 @@ def transaction(options = {})
else
@transaction_joinable = true
end
+ if options.has_key?(:remember_record_state)
+ remember_record_state = options[:remember_record_state]
+ else
+ remember_record_state = true
+ end

This can be simplified with options.fetch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Will Bryant

@steveklabnik: Yes I am, we use it in production and it's pretty critical for us.

Both commits apply cleanly to 3-0-stable, 3-1-stable, 3-2-stable; I've updated current_transaction_records_3-2-stable in my fork.

For master, first see @jonleighton's comments on b89ffe7 - note that weakrefs are completely broken on 1.9.

@jonleighton, could you please give me some feedback about what your intention is on master? There's no issue tracker numbers on the conflicting commit that have refactored the transaction internals, so I'm just not sure if you plan to achieve this by other means, or if not, whether you'd consider merging this feature if I update it to match the refactoring master.

Jon Leighton
Collaborator

@willbryant:

I discussed more with @tenderlove after that commit. He intends that each record will hold its transaction in an ivar, rather than the other way round. Then, calling e.g. record.new_record? will query the transaction state. It will still be necessary to hold a list of records in the transaction for records with commit/rollback hooks, that's unavoidable. But for the usual case it should work.

Are you up for implementing that?

Will Bryant

@jonleighton: that's an interesting approach. That would mean also that #id, for example, and any other things currently reset by the transaction rollback code would need to query the transaction state?

Jon Leighton
Collaborator

Unsure, I haven't really looked into it..

Will Bryant

Looking at the remember_transaction_record_state code, here's the list of things that we'd need to give the special behavior to:

  • id
  • new_record
  • destroyed
  • frozen? Not sure about level (transaction nesting level), that would depend on how the reference to the transaction is maintained.

id is the only one that's an attribute and that could make it a little messy to get the special semantics. The others wouldn't be too hard.

Łukasz Strzałkowski
Collaborator

Guys, what's the status?

/cc @jonleighton @willbryant

Will Bryant

I think the above alternate solution got committed to master for 4.0. I haven't tested it but I'll close this in the meantime.

Will Bryant willbryant closed this
Łukasz Strzałkowski
Collaborator

Thanks @willbryant for an update and taking action :+1:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 11, 2011
  1. Will Bryant

    when the transactional fixtures keeps a database transaction alive wi…

    willbryant authored
    …thout using the transaction method, don't retain references to all the records saved in the fixtures transaction once their subtransaction completes
  2. Will Bryant

    expose :remember_record_state => false option on transaction(), so ap…

    willbryant authored
    …plications that don't need state rollback (or after_commit/rollback hooks) can garbage collect the records during long transactions
This page is out of date. Refresh to see the latest.
12 activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb
View
@@ -162,7 +162,7 @@ def supports_statement_cache?
# end # RELEASE SAVEPOINT active_record_1 <--- BOOM! database error!
# end
def transaction(options = {})
- options.assert_valid_keys :requires_new, :joinable
+ options.assert_valid_keys :requires_new, :joinable, :remember_record_state
last_transaction_joinable = defined?(@transaction_joinable) ? @transaction_joinable : nil
if options.has_key?(:joinable)
@@ -170,6 +170,11 @@ def transaction(options = {})
else
@transaction_joinable = true
end
+ if options.has_key?(:remember_record_state)
+ remember_record_state = options[:remember_record_state]
+ else
+ remember_record_state = true
+ end

This can be simplified with options.fetch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
requires_new = options[:requires_new] || !last_transaction_joinable
transaction_open = false
@@ -185,7 +190,7 @@ def transaction(options = {})
end
increment_open_transactions
transaction_open = true
- @_current_transaction_records.push([])
+ @_current_transaction_records.push(remember_record_state ? [] : nil)
end
yield
end
@@ -217,8 +222,7 @@ def transaction(options = {})
else
release_savepoint
save_point_records = @_current_transaction_records.pop
- unless save_point_records.blank?
- @_current_transaction_records.push([]) if @_current_transaction_records.empty?
+ unless save_point_records.blank? || @_current_transaction_records.empty?
@_current_transaction_records.last.concat(save_point_records)
end
end
25 activerecord/test/cases/fixtures_test.rb
View
@@ -335,6 +335,31 @@ def test_destroy_just_kidding
end
end
+class TransactionalFixturesTest < ActiveRecord::TestCase
+ self.use_instantiated_fixtures = true
+ self.use_transactional_fixtures = true
+
+ fixtures :topics
+
+ def test_records_saved_in_fixture_transaction_not_recorded
+ @first.update_attributes!(:title => "New title")
+ assert !Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@first)
+ end
+
+ def test_records_saved_in_explicit_transaction_recorded
+ Topic.transaction(:requires_new => true) do
+ @first.update_attributes!(:title => "New title")
+ assert Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@first)
+ end
+ Topic.transaction do
+ @second.update_attributes!(:title => "New title")
+ assert Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@second)
+ end
+ assert !Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@first)
+ assert !Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@second)
+ end
+end
+
class MultipleFixturesTest < ActiveRecord::TestCase
fixtures :topics
fixtures :developers, :accounts
23 activerecord/test/cases/transactions_test.rb
View
@@ -389,6 +389,29 @@ def test_restore_active_record_state_for_all_records_in_a_transaction
assert !@second.destroyed?, 'not destroyed'
end
+ def test_optional_no_restore_active_record_state_so_records_not_retained_in_memory
+ topic_1 = Topic.new(:title => 'test_1')
+ Topic.transaction(:remember_record_state => false) do
+ assert topic_1.save
+ @first.destroy
+ assert topic_1.persisted?, 'persisted'
+ assert_not_nil topic_1.id
+ assert @first.destroyed?, 'destroyed'
+ raise ActiveRecord::Rollback
+ end
+
+ assert topic_1.persisted?, 'state was rolled back, so record must have been retained'
+ assert_not_nil topic_1.id
+ assert_raises(ActiveRecord::RecordNotFound) do
+ topic_1.reload
+ flunk 'database was not rolled back'
+ end
+ assert @first.destroyed?, 'state was rolled back, so record must have been retained'
+ assert_nothing_raised do
+ Topic.find(@first.id)
+ end
+ end
+
if current_adapter?(:PostgreSQLAdapter) && defined?(PGconn::PQTRANS_IDLE)
def test_outside_transaction_works
assert Topic.connection.outside_transaction?
Something went wrong with that request. Please try again.