Skip to content
This repository

Current transaction records retained by transactional fixtures (master) #3300

Closed
wants to merge 2 commits into from

7 participants

Will Bryant Sean Walbran Isaac Sanders Steve Klabnik Jon Leighton Łukasz Strzałkowski Carlos Antonio da Silva
Will Bryant

As discussed on the mailing list, this fixes the memory bloat when using transactional fixtures to run large test suites, which is basically a regression since 2.3 due to the side effects of the better record rollback code. This substantially improves test runtime too if you save a lot of objects in tests.

This branch also exposes the option to not retain references to the records saved in transactions, which is useful for applications that have large transactions that save a lot of objects but don't need their state in memory to be rolled back if the transaction fails nor need the transaction callbacks.

added some commits October 11, 2011
Will Bryant when the transactional fixtures keeps a database transaction alive wi…
…thout using the transaction method, don't retain references to all the records saved in the fixtures transaction once their subtransaction completes
d731f5c
Will Bryant expose :remember_record_state => false option on transaction(), so ap…
…plications that don't need state rollback (or after_commit/rollback hooks) can garbage collect the records during long transactions
837b369
Sean Walbran

Just fought with this leak myself, happy to see this in play. The approach I took was to add the below to the teardown_fixtures method in fixtures.rb, which seems consistent with the original callbacks commit ( b070739 ) and should allow for the actual rollback callbacks to take effect in tests (for, e.g., cleanup of external references if any, etc.). Any thoughts on this spin instead/as well?

def teardown_fixtures
...
# Rollback changes if a transaction is active.

if run_in_transaction? && ActiveRecord::Base.connection.open_transactions != 0
ActiveRecord::Base.connection.rollback_db_transaction

  • ActiveRecord::Base.connection.send(:rollback_transaction_records,true) if ActiveRecord::Base.connection.instance_variable_get('@_current_transaction_records') ActiveRecord::Base.connection.decrement_open_transactions end ActiveRecord::Base.clear_active_connections! end
Will Bryant

Sean, firing rollback event handlers seems reasonable to me as an additional thing. Haven't been able to produce much interest in that when I asked about it on the rails-core mailing list (see posts around 12 October) though.

Isaac Sanders

Is this still an issue?

Will Bryant

Yes.

Steve Klabnik
Collaborator

This is gonna need a rebase if it's ever gonna get included.

Steve Klabnik
Collaborator

@willbryant ping! are you still interested in keeping up with this pull request?

Carlos Antonio da Silva carlosantoniodasilva commented on the diff November 17, 2012
activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb
@@ -170,6 +170,11 @@ def transaction(options = {})
170 170
         else
171 171
           @transaction_joinable = true
172 172
         end
  173
+        if options.has_key?(:remember_record_state)
  174
+          remember_record_state = options[:remember_record_state]
  175
+        else
  176
+          remember_record_state = true
  177
+        end
1

This can be simplified with options.fetch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Will Bryant

@steveklabnik: Yes I am, we use it in production and it's pretty critical for us.

Both commits apply cleanly to 3-0-stable, 3-1-stable, 3-2-stable; I've updated current_transaction_records_3-2-stable in my fork.

For master, first see @jonleighton's comments on b89ffe7 - note that weakrefs are completely broken on 1.9.

@jonleighton, could you please give me some feedback about what your intention is on master? There's no issue tracker numbers on the conflicting commit that have refactored the transaction internals, so I'm just not sure if you plan to achieve this by other means, or if not, whether you'd consider merging this feature if I update it to match the refactoring master.

Jon Leighton
Owner

@willbryant:

I discussed more with @tenderlove after that commit. He intends that each record will hold its transaction in an ivar, rather than the other way round. Then, calling e.g. record.new_record? will query the transaction state. It will still be necessary to hold a list of records in the transaction for records with commit/rollback hooks, that's unavoidable. But for the usual case it should work.

Are you up for implementing that?

Will Bryant

@jonleighton: that's an interesting approach. That would mean also that #id, for example, and any other things currently reset by the transaction rollback code would need to query the transaction state?

Jon Leighton
Owner

Unsure, I haven't really looked into it..

Will Bryant

Looking at the remember_transaction_record_state code, here's the list of things that we'd need to give the special behavior to:

  • id
  • new_record
  • destroyed
  • frozen? Not sure about level (transaction nesting level), that would depend on how the reference to the transaction is maintained.

id is the only one that's an attribute and that could make it a little messy to get the special semantics. The others wouldn't be too hard.

Łukasz Strzałkowski
Collaborator

Guys, what's the status?

/cc @jonleighton @willbryant

Will Bryant willbryant closed this March 15, 2013
Will Bryant

I think the above alternate solution got committed to master for 4.0. I haven't tested it but I'll close this in the meantime.

Łukasz Strzałkowski
Collaborator

Thanks @willbryant for an update and taking action :+1:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Oct 11, 2011
Will Bryant when the transactional fixtures keeps a database transaction alive wi…
…thout using the transaction method, don't retain references to all the records saved in the fixtures transaction once their subtransaction completes
d731f5c
Will Bryant expose :remember_record_state => false option on transaction(), so ap…
…plications that don't need state rollback (or after_commit/rollback hooks) can garbage collect the records during long transactions
837b369
This page is out of date. Refresh to see the latest.
12  activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb
@@ -162,7 +162,7 @@ def supports_statement_cache?
162 162
       #     end  # RELEASE SAVEPOINT active_record_1  <--- BOOM! database error!
163 163
       #   end
164 164
       def transaction(options = {})
165  
-        options.assert_valid_keys :requires_new, :joinable
  165
+        options.assert_valid_keys :requires_new, :joinable, :remember_record_state
166 166
 
167 167
         last_transaction_joinable = defined?(@transaction_joinable) ? @transaction_joinable : nil
168 168
         if options.has_key?(:joinable)
@@ -170,6 +170,11 @@ def transaction(options = {})
170 170
         else
171 171
           @transaction_joinable = true
172 172
         end
  173
+        if options.has_key?(:remember_record_state)
  174
+          remember_record_state = options[:remember_record_state]
  175
+        else
  176
+          remember_record_state = true
  177
+        end
173 178
         requires_new = options[:requires_new] || !last_transaction_joinable
174 179
 
175 180
         transaction_open = false
@@ -185,7 +190,7 @@ def transaction(options = {})
185 190
               end
186 191
               increment_open_transactions
187 192
               transaction_open = true
188  
-              @_current_transaction_records.push([])
  193
+              @_current_transaction_records.push(remember_record_state ? [] : nil)
189 194
             end
190 195
             yield
191 196
           end
@@ -217,8 +222,7 @@ def transaction(options = {})
217 222
             else
218 223
               release_savepoint
219 224
               save_point_records = @_current_transaction_records.pop
220  
-              unless save_point_records.blank?
221  
-                @_current_transaction_records.push([]) if @_current_transaction_records.empty?
  225
+              unless save_point_records.blank? || @_current_transaction_records.empty?
222 226
                 @_current_transaction_records.last.concat(save_point_records)
223 227
               end
224 228
             end
25  activerecord/test/cases/fixtures_test.rb
@@ -335,6 +335,31 @@ def test_destroy_just_kidding
335 335
   end
336 336
 end
337 337
 
  338
+class TransactionalFixturesTest < ActiveRecord::TestCase
  339
+  self.use_instantiated_fixtures = true
  340
+  self.use_transactional_fixtures = true
  341
+
  342
+  fixtures :topics
  343
+
  344
+  def test_records_saved_in_fixture_transaction_not_recorded
  345
+    @first.update_attributes!(:title => "New title")
  346
+    assert !Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@first)
  347
+  end
  348
+
  349
+  def test_records_saved_in_explicit_transaction_recorded
  350
+    Topic.transaction(:requires_new => true) do
  351
+      @first.update_attributes!(:title => "New title")
  352
+      assert Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@first)
  353
+    end
  354
+    Topic.transaction do
  355
+      @second.update_attributes!(:title => "New title")
  356
+      assert Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@second)
  357
+    end
  358
+    assert !Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@first)
  359
+    assert !Topic.connection.instance_variable_get("@_current_transaction_records").flatten.include?(@second)
  360
+  end
  361
+end
  362
+
338 363
 class MultipleFixturesTest < ActiveRecord::TestCase
339 364
   fixtures :topics
340 365
   fixtures :developers, :accounts
23  activerecord/test/cases/transactions_test.rb
@@ -389,6 +389,29 @@ def test_restore_active_record_state_for_all_records_in_a_transaction
389 389
     assert !@second.destroyed?, 'not destroyed'
390 390
   end
391 391
 
  392
+  def test_optional_no_restore_active_record_state_so_records_not_retained_in_memory
  393
+    topic_1 = Topic.new(:title => 'test_1')
  394
+    Topic.transaction(:remember_record_state => false) do
  395
+      assert topic_1.save
  396
+      @first.destroy
  397
+      assert topic_1.persisted?, 'persisted'
  398
+      assert_not_nil topic_1.id
  399
+      assert @first.destroyed?, 'destroyed'
  400
+      raise ActiveRecord::Rollback
  401
+    end
  402
+
  403
+    assert topic_1.persisted?, 'state was rolled back, so record must have been retained'
  404
+    assert_not_nil topic_1.id
  405
+    assert_raises(ActiveRecord::RecordNotFound) do
  406
+      topic_1.reload
  407
+      flunk 'database was not rolled back'
  408
+    end
  409
+    assert @first.destroyed?, 'state was rolled back, so record must have been retained'
  410
+    assert_nothing_raised do
  411
+      Topic.find(@first.id)
  412
+    end
  413
+  end
  414
+
392 415
   if current_adapter?(:PostgreSQLAdapter) && defined?(PGconn::PQTRANS_IDLE)
393 416
     def test_outside_transaction_works
394 417
       assert Topic.connection.outside_transaction?
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.