New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make RoundhousE fault tolerant to transient exceptions that often occur for Azure Database #234

Closed
vansha opened this Issue Nov 17, 2015 · 3 comments

Comments

Projects
None yet
2 participants
@vansha
Contributor

vansha commented Nov 17, 2015

We're using RoundhousE to manage migrations on Azure Database. As it occurred, transient faults (exceptions that could be easily handled by retrying the operation) are often happen in cloud environment.

Here the typical places where exceptions occur:

  • Exception when opening a sql connection
  • Exception when trying to execute update script
  • Exception when dealing with RoundhousE internal tables (like querying from or inserting to ScriptsRun table).

It would be great if RoundhousE had a built-in ability to handle such exceptions via some retry policy.

Below are some of the exceptions we've seen so far:

Timeout on opening connection:

Creating SomeDb database on [azureserver].database.windows.net server if it doesn't exist.
roundhouse.databases.sqlserver.SqlServerDatabase with provider System.Data.SqlClient does not provide a facility for .
Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
RoundhousE encountered an error.
System.Data.SqlClient.SqlException (0x80131904): Database 'SomeDb' on server 'azureserver' is not currently available.ter.  If the problem persists, contact customer support, and provide them the session tracing ID of 'guid'
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean  asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader Set bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, Handler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
   at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
   at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
   at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString  credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString  credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, onOptions, SessionData reconnectSessionData)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey rInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection s options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions l oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions l oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 oolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 ptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, ConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, ctory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory nSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at roundhouse.connections.AdoNetConnection.open()
   at roundhouse.databases.AdoNetDatabase.open_connection(Boolean with_transaction)
   at roundhouse.migrators.DefaultDatabaseMigrator.open_connection(Boolean with_transaction)
   at roundhouse.runners.RoundhouseMigrationRunner.run()

Network issue when executing an update script:

 Running 100.SomeScript.sql on server.database.windows.net - SomeDb.
 Running 101.SomeOtherScript.sql on server.database.windows.net - SomeDb.
RoundhousE encountered an error.
System.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) ---> System.ComponentModel.Win32Exception (0x80004005): The specified network name is no longer available
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at roundhouse.databases.AdoNetDatabase.run_command_with(String sql_to_run, ConnectionType connection_type, IList`1 parameters)
   at roundhouse.databases.AdoNetDatabase.run_sql(String sql_to_run, ConnectionType connection_type, IList`1 parameters)
   at roundhouse.databases.DefaultDatabase`1.run_sql(String sql_to_run, ConnectionType connection_type)
   at roundhouse.migrators.DefaultDatabaseMigrator.run_sql(String sql_to_run, String script_name, Boolean run_this_script_once, Boolean run_this_script_every_time, Int64 version_id, Environment environment, String repository_version, String repository_path, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.traverse_files_and_run_sql(String directory, Int64 version_id, MigrationsFolder migration_folder, Environment migrating_environment, String repository_version, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.traverse_files_and_run_sql(String directory, Int64 version_id, MigrationsFolder migration_folder, Environment migrating_environment, String repository_version, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.log_and_traverse(MigrationsFolder folder, Int64 version_id, String new_version, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.run()

Internal server error when creating a database:

Creating SomeDb database on server.database.windows.net server if it doesn't exist.
roundhouse.databases.sqlserver.SqlServerDatabase with provider System.Data.SqlClient does not provide a facility for creating a database at this time.
CREATE DATABASE operation failed. Internal service error.
RoundhousE encountered an error.
System.Data.SqlClient.SqlException (0x80131904): Cannot open database "SomeDb" requested by the login. The login failed.
Login failed for user 'someuser'.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
   at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
   at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
   at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at roundhouse.connections.AdoNetConnection.open()
   at roundhouse.databases.AdoNetDatabase.open_connection(Boolean with_transaction)
   at roundhouse.migrators.DefaultDatabaseMigrator.open_connection(Boolean with_transaction)
   at roundhouse.runners.RoundhouseMigrationRunner.run()

Semaphore timeout when executing an update script:

 Running 0001.SomeScript.sql on server.database.windows.net - SomeDb.
RoundhousE encountered an error.
System.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.) ---> System.ComponentModel.Win32Exception (0x80004005): The semaphore timeout period has expired
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at roundhouse.databases.AdoNetDatabase.run_command_with(String sql_to_run, ConnectionType connection_type, IList`1 parameters)
   at roundhouse.databases.AdoNetDatabase.run_sql(String sql_to_run, ConnectionType connection_type, IList`1 parameters)
   at roundhouse.databases.DefaultDatabase`1.run_sql(String sql_to_run, ConnectionType connection_type)
   at roundhouse.migrators.DefaultDatabaseMigrator.run_sql(String sql_to_run, String script_name, Boolean run_this_script_once, Boolean run_this_script_every_time, Int64 version_id, Environment environment, String repository_version, String repository_path, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.traverse_files_and_run_sql(String directory, Int64 version_id, MigrationsFolder migration_folder, Environment migrating_environment, String repository_version, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.log_and_traverse(MigrationsFolder folder, Int64 version_id, String new_version, ConnectionType connection_type)
   at roundhouse.runners.RoundhouseMigrationRunner.run()

Exception when retrieving list of already executed scripts:

0002.SomeScript.sql - I didn't find this script executed before.
NHibernate.Exceptions.GenericADOException: could not execute query
[ SELECT this_.id as id1_0_, this_.version_id as version2_1_0_, this_.script_name as script3_1_0_, this_.text_of_script as text4_1_0_, this_.text_hash as text5_
1_0_, this_.one_time_script as one6_1_0_, this_.entry_date as entry7_1_0_, this_.modified_date as modified8_1_0_, this_.entered_by as entered9_1_0_ FROM Roundho
usE.ScriptsRun this_ ]
 ---> System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception: An existing connection was forcibly closed by the remote host
   --- End of inner exception stack trace ---
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[] buff, offset, Int32 len, ref Int32 totalRead)
   at System.Data.SqlClient.TdsParserStateObject.TryReadStringWithEncoding(Int32 length, Encoding encoding, Boolean isPlp, ref String value)
   at System.Data.SqlClient.TdsParser.TryReadSqlStringValue(SqlBuffer value, type, Int32 length, Encoding encoding, Boolean isPlp, teObj)
   at System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, md, Int32 length, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.TryReadColumnData()
   at System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly)
   at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
   at System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32 i)
   at System.Data.SqlClient.SqlDataReader.GetValue(Int32 i)
   at System.Data.SqlClient.SqlDataReader.get_Item(Int32 i)
   at NHibernate.Driver.NHybridDataReader.System.Data.IDataRecord.get_Item(Int32 i)
   at NHibernate.Type.AbstractStringType.Get(IDataReader rs, Int32 index)
   at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String name)
   at NHibernate.Type.NullableType.NullSafeGet(IDataReader rs, String[] names, session, Object owner)
   at NHibernate.Type.AbstractType.Hydrate(IDataReader rs, String[] names, session, Object owner)
   at NHibernate.Persister.Entity.AbstractEntityPersister.Hydrate(IDataReader rs, Object id, Object obj, ILoadable rootLoadable, String[][] suffixedPropertyColumns, Boolean allProperties, ISessionImplementor session)
   at NHibernate.Loader.Loader.LoadFromResultSet(IDataReader rs, Int32 i, Object obj, String instanceClass, EntityKey key, String rowIdAlias, LockMode lockMode, ILoadable rootPersister, ISessionImplementor session)
   at NHibernate.Loader.Loader.InstanceNotYetLoaded(IDataReader dr, Int32 i, persister, EntityKey key, LockMode lockMode, String rowIdAlias, EntityKey optionalObjectKey, Object optionalObject, IList hydratedObjects, session)
   at NHibernate.Loader.Loader.GetRow(IDataReader rs, ILoadable[] persisters, EntityKey[] keys, Object optionalObject, EntityKey optionalObjectKey, LockMode[] lockModes, IList hydratedObjects, ISessionImplementor session)
   at NHibernate.Loader.Loader.GetRowFromResultSet(IDataReader resultSet, session, QueryParameters queryParameters, LockMode[] lockModeArray, EntityKey optionalObjectKey, IList hydratedObjects, EntityKey[] keys, Boolean returnProxies)
   at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, queryParameters, Boolean returnProxies)
   at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(session, QueryParameters queryParameters, Boolean returnProxies)
   at NHibernate.Loader.Loader.DoList(ISessionImplementor session, queryParameters)
   --- End of inner exception stack trace ---
   at NHibernate.Loader.Loader.DoList(ISessionImplementor session, queryParameters)
   at NHibernate.Loader.Loader.ListIgnoreQueryCache(ISessionImplementor session, QueryParameters queryParameters)
   at NHibernate.Loader.Loader.List(ISessionImplementor session, QueryParameters queryParameters, ISet`1 querySpaces, IType[] resultTypes)
   at NHibernate.Impl.SessionImpl.List(CriteriaImpl criteria, IList results)
   at NHibernate.Impl.CriteriaImpl.List(IList results)
   at NHibernate.Impl.CriteriaImpl.List<T>()
   at NHibernate.Criterion.QueryOver`1.List<U>()
   at NHibernate.Criterion.QueryOver`1.NHibernate.IQueryOver<TRoot>.List<U>()
   at roundhouse.infrastructure.persistence.Repository.get_all<T>()
   at roundhouse.databases.DefaultDatabase`1.ensure_script_cache()
   at roundhouse.databases.DefaultDatabase`1.get_from_script_cache(_name)
   at roundhouse.databases.DefaultDatabase`1.get_current_script_hash(pt_name)
   at roundhouse.migrators.DefaultDatabaseMigrator.this_script_has_changed_since_last_run(String script_name, String sql_to_run)
@vansha

This comment has been minimized.

Show comment
Hide comment
@vansha

vansha Nov 17, 2015

Contributor

I'm currently integrating Transient Fault Handling library from Microsoft to RoundhosuE to handle our issues. Going to test the result against our deployments for a week or so and if everything goes smoothly will make a PR.

Contributor

vansha commented Nov 17, 2015

I'm currently integrating Transient Fault Handling library from Microsoft to RoundhosuE to handle our issues. Going to test the result against our deployments for a week or so and if everything goes smoothly will make a PR.

@ferventcoder

This comment has been minimized.

Show comment
Hide comment
@ferventcoder

ferventcoder Nov 17, 2015

Member

We may already have Lokad Retry Policy here.

Member

ferventcoder commented Nov 17, 2015

We may already have Lokad Retry Policy here.

@vansha

This comment has been minimized.

Show comment
Hide comment
@vansha

vansha Nov 17, 2015

Contributor

As far as I understand there is only simple retry for just a single case - https://github.com/chucknorris/roundhouse/blob/master/product/roundhouse/databases/AdoNetDatabase.cs#L139

Contributor

vansha commented Nov 17, 2015

As far as I understand there is only simple retry for just a single case - https://github.com/chucknorris/roundhouse/blob/master/product/roundhouse/databases/AdoNetDatabase.cs#L139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment