Permalink
Browse files

Add connection_validator extension for automatically checking connect…

…ions and transparently handling disconnects

This extension is primarily designed for users whose idle
database connections are automatically disconnected from the
database after a given period of time.  However, it can also
be used to handle all disconnects by validating connections
at every checkout, though that may hurt performance significantly.

This shouldn't cause any problems with transactions or special
connection states, since code that relies on either isn't even
yielded a connection until after the validation check has been
run.
  • Loading branch information...
1 parent 029254e commit eee00604f1dae74c35ff00be6934cf30790c792d @jeremyevans committed Oct 24, 2012
View
@@ -1,5 +1,7 @@
=== HEAD
+* Add connection_validator extension for automatically checking connections and transparently handling disconnects (jeremyevans)
+
* Add Database#valid_connection? for checking whether a given connection is valid (jeremyevans)
* Make dataset.limit(nil, nil) reset offset as well as limit (jeremyevans) (#571)
@@ -0,0 +1,98 @@
+# The connection_validator extension modifies a database's
+# connection pool to validate that connections checked out
+# from the pool are still valid, before yielding them for
+# use. If it detects an invalid connection, it removes it
+# from the pool and tries the next available connection,
+# creating a new connection if no available connection is
+# valid. Example of use:
+#
+# DB.extension(:connection_validator)
+#
+# As checking connections for validity involves issuing a
+# query, which is potentially an expensive operation,
+# the validation checks are only run if the connection has
+# been idle for longer than a certain threshold. By default,
+# that threshold is 3600 seconds (1 hour), but it can be
+# modified by the user, set to -1 to always validate
+# connections on checkout:
+#
+# DB.pool.connection_validation_timeout = -1
+#
+# Note that if you set the timeout to validate connections
+# on every checkout, you should probably manually control
+# connection checkouts on a coarse basis, using
+# Database#synchonrize. In a web application, the optimal
+# place for that would be a rack middleware. Validating
+# connections on every checkout without setting up coarse
+# connection checkouts will hurt performance, in some cases
+# significantly. Note that setting up coarse connection
+# checkouts reduces the concurrency level acheivable. For
+# example, in a web application, using Database#synchronize
+# in a rack middleware will limit the number of concurrent
+# web requests to the number to connections in the database
+# connection pool.
+#
+# Note that this extension only affects the default threaded
+# and the sharded threaded connection pool. The single
+# threaded and sharded single threaded connection pools are
+# not affected. As the only reason to use the single threaded
+# pools is for speed, and this extension makes the connection
+# pool slower, there's not much point in modifying this
+# extension to work with the single threaded pools. The
+# threaded pools work fine even in single threaded code, so if
+# you are currently using a single threaded pool and want to
+# use this extension, switch to using a threaded pool.
+
+module Sequel
+ module ConnectionValidator
+ class Retry < Error; end
+
+ # The number of seconds that need to pass since
+ # connection checkin before attempting to validate
+ # the connection when checking it out from the pool.
+ # Defaults to 3600 seconds (1 hour).
+ attr_accessor :connection_validation_timeout
+
+ # Initialize the data structures used by this extension.
+ def self.extended(pool)
+ pool.instance_eval do
+ @connection_timestamps ||= {}
+ @connection_validation_timeout = 3600
+ end
+ end
+
+ private
+
+ # Record the time the connection was checked back into the pool.
+ def checkin_connection(*)
+ conn = super
+ @connection_timestamps[conn] = Time.now
+ conn
+ end
+
+ # If an available connection is being checked out, and it has
+ # has been idle for longer than the connection validation timeout,
+ # test the connection for validity. If it is not valid,
+ # disconnect the connection, and retry with the next available
+ # connection. If no avialable connections are valid, this will
+ # return nil.
+ def next_available(*)
+ begin
+ if (conn = super) &&
+ (t = @connection_timestamps.delete(conn)) &&
+ Time.now - t > @connection_validation_timeout &&
+ !db.valid_connection?(conn)
+ db.disconnect_connection(conn)
+ raise Retry
+ end
+ rescue Retry
+ retry
+ end
+
+ conn
+ end
+ end
+
+ Database.register_extension(:connection_validator){|db| db.pool.extend(ConnectionValidator)}
+end
+
@@ -0,0 +1,111 @@
+require File.join(File.dirname(File.expand_path(__FILE__)), "spec_helper")
+
+shared_examples_for "Sequel::ConnectionValidator" do
+ before do
+ @db.extend(Module.new do
+ def disconnect_connection(conn)
+ @sqls << 'disconnect'
+ end
+ def valid_connection?(conn)
+ super
+ conn.valid
+ end
+ def connect(server)
+ conn = super
+ conn.extend(Module.new do
+ attr_accessor :valid
+ end)
+ conn.valid = true
+ conn
+ end
+ end)
+ @db.extension(:connection_validator)
+ end
+
+ it "should still allow new connections" do
+ @db.synchronize{|c| c}.should be_a_kind_of(Sequel::Mock::Connection)
+ end
+
+ it "should only validate if connection idle longer than timeout" do
+ c1 = @db.synchronize{|c| c}
+ @db.sqls.should == []
+ @db.synchronize{|c| c}.should equal(c1)
+ @db.sqls.should == []
+ @db.pool.connection_validation_timeout = -1
+ @db.synchronize{|c| c}.should equal(c1)
+ @db.sqls.should == ['SELECT NULL']
+ @db.pool.connection_validation_timeout = 1
+ @db.synchronize{|c| c}.should equal(c1)
+ @db.sqls.should == []
+ @db.synchronize{|c| c}.should equal(c1)
+ @db.sqls.should == []
+ end
+
+ it "should disconnect connection if not valid" do
+ c1 = @db.synchronize{|c| c}
+ @db.sqls.should == []
+ c1.valid = false
+ @db.pool.connection_validation_timeout = -1
+ c2 = @db.synchronize{|c| c}
+ @db.sqls.should == ['SELECT NULL', 'disconnect']
+ c2.should_not equal(c1)
+ end
+
+ it "should disconnect multiple connections repeatedly if they are not valid" do
+ q, q1 = Queue.new, Queue.new
+ c1 = nil
+ c2 = nil
+ @db.pool.connection_validation_timeout = -1
+ @db.synchronize do |c|
+ Thread.new do
+ @db.synchronize do |cc|
+ c2 = cc
+ end
+ q1.pop
+ q.push nil
+ end
+ q1.push nil
+ q.pop
+ c1 = c
+ end
+ c1.valid = false
+ c2.valid = false
+
+ c3 = @db.synchronize{|c| c}
+ @db.sqls.should == ['SELECT NULL', 'disconnect', 'SELECT NULL', 'disconnect']
+ c3.should_not equal(c1)
+ c3.should_not equal(c2)
+ end
+
+ it "should not leak connection references" do
+ c1 = @db.synchronize do |c|
+ @db.pool.instance_variable_get(:@connection_timestamps).should == {}
+ c
+ end
+ @db.pool.instance_variable_get(:@connection_timestamps).should have_key(c1)
+
+ c1.valid = false
+ @db.pool.connection_validation_timeout = -1
+ c2 = @db.synchronize do |c|
+ @db.pool.instance_variable_get(:@connection_timestamps).should == {}
+ c
+ end
+ c2.should_not equal(c1)
+ @db.pool.instance_variable_get(:@connection_timestamps).should_not have_key(c1)
+ @db.pool.instance_variable_get(:@connection_timestamps).should have_key(c2)
+ end
+end
+
+describe "Sequel::ConnectionValidator with threaded pool" do
+ before do
+ @db = Sequel.mock
+ end
+ it_should_behave_like "Sequel::ConnectionValidator"
+end
+describe "Sequel::ConnectionValidator with sharded threaded pool" do
+ before do
+ @db = Sequel.mock(:servers=>{})
+ end
+ it_should_behave_like "Sequel::ConnectionValidator"
+end
+
@@ -94,3 +94,10 @@ def INTEGRATION_DB.drop_table(*tables)
super
end
end
+
+if ENV['SEQUEL_CONNECTION_VALIDATOR']
+ ENV['SEQUEL_NO_CHECK_SQLS'] = '1'
+ INTEGRATION_DB.extension(:connection_validator)
+ INTEGRATION_DB.pool.connection_validation_timeout = -1
+end
+
View
@@ -90,6 +90,7 @@
<li><a href="rdoc-plugins/files/lib/sequel/extensions/blank_rb.html">blank</a>: Adds blank? instance methods to all objects.</li>
<li><a href="rdoc-plugins/files/lib/sequel/extensions/core_extensions_rb.html">core_extensions</a>: Extends the Array, Hash, String, and Symbol classes with methods that return Sequel expression objects (loaded by default).</li>
<li><a href="rdoc-plugins/files/lib/sequel/extensions/columns_introspection_rb.html">columns_introspection</a>: Attemps to skip database queries by introspecting the selected columns if possible.</li>
+<li><a href="rdoc-plugins/files/lib/sequel/extensions/connection_validator_rb.html">connection_validator</a>: Automatically validates connections on pool checkout and handles disconnections transparently.</li>
<li><a href="rdoc-plugins/files/lib/sequel/extensions/constraint_validations_rb.html">constraint_validations</a>: Creates database constraints when creating/altering tables, with metadata for <a href="rdoc-plugins/classes/Sequel/Plugins/ConstraintValidations.html">automatic model validations via the constraint_validations plugin</a>.</li>
<li><a href="rdoc-plugins/files/lib/sequel/extensions/eval_inspect_rb.html">eval_inspect</a>: Makes inspect on Sequel's expression objects attempt to return a string suitable for eval.</li>
<li><a href="rdoc-plugins/files/lib/sequel/extensions/inflector_rb.html">inflector</a>: Adds instance-level inflection methods to String.</li>

0 comments on commit eee0060

Please sign in to comment.