Add retry for BeginTransaction #183

paulegradie · 2022-02-17T05:17:55Z

Octopus Deploy dev teams have noticed transient connectivity issues when calling BeginTransaction in ReacTransaction.cs.
We are also behind 2 versions of SqlClient that may have beneficial bug fixes or performance improvements..

The objective of this PR is to:

Bump the version of SqlClient
Wrap these calls with retry logic provided by the RetryManager

Slack Thread

Benchmark Results

Before

TODO: get these running (currently hitting errors)

After

TODO: Same

paulegradie · 2022-02-17T05:20:58Z

source/Nevermore/Advanced/ReadTransaction.cs

@@ -89,7 +89,7 @@ public async Task OpenAsync(IsolationLevel isolationLevel)

            // We use the synchronous overload here even though there is an async one, because the BeginTransactionAsync calls


Doesn't seem like a async overload will be very helpful.

paulegradie · 2022-02-17T05:23:22Z

source/Nevermore/Nevermore.csproj

@@ -45,7 +45,7 @@
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
    <PackageReference Include="System.Diagnostics.Contracts" Version="4.3.0" />
    <PackageReference Include="System.ValueTuple" Version="4.4.0" />
-    <PackageReference Include="Microsoft.Data.SqlClient" Version="2.1.4" />
+    <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />


I'd recommend against using the sqlClient retry provider for now until we have a much more stable build environment. Currently, I'm only aware of this option for opening connections (not beginning transactions, but this advice applies to both in case that comes to light). Perhaps we don't want to make that change at all regardless.

paulegradie · 2022-02-17T05:28:29Z

source/Nevermore.IntegrationTests/SetUp/IntegrationTestDatabase.cs

@@ -14,7 +14,7 @@ public IntegrationTestDatabase()
            var username = Environment.GetEnvironmentVariable("NevermoreTestUsername");
            var password = Environment.GetEnvironmentVariable("NevermoreTestPassword");
            testDatabaseName = Environment.GetEnvironmentVariable("NevermoreTestDatabase") ?? "Nevermore-IntegrationTests";
-            var builder = new SqlConnectionStringBuilder($"Server={sqlInstance};Database={testDatabaseName};{(username == null ? "Trusted_connection=true;" : string.Empty)}")
+            var builder = new SqlConnectionStringBuilder($"Server={sqlInstance};Database={testDatabaseName};{(username == null ? "Trusted_connection=true;" : string.Empty)}; Encrypt=False;")


Bumping the SqlClient altered the default for encryption. Most devs probably don't have this enabled on their dev SQL Server instance, so we can set this to avoid forcing folks to have to set another env var, or change their server settings.

Will we need to add this on all connection strings that we have on Octopus?

Yeah possibly / probably 👍 - good call

paulegradie · 2022-02-17T06:16:00Z

source/Nevermore.Benchmarks/SetUp/IntegrationTestDatabase.cs

@@ -13,7 +13,7 @@ public IntegrationTestDatabase()
            var username = Environment.GetEnvironmentVariable("NevermoreTestUsername");
            var password = Environment.GetEnvironmentVariable("NevermoreTestPassword");
            testDatabaseName = Environment.GetEnvironmentVariable("NevermoreBenchmarkDatabase") ?? "Nevermore-Benchmarks";
-            var builder = new SqlConnectionStringBuilder($"Server={sqlInstance};Database={testDatabaseName};{(username == null ? "Trusted_connection=true;" : string.Empty)}")
+            var builder = new SqlConnectionStringBuilder($"Server={sqlInstance};Database={testDatabaseName};{(username == null ? "Trusted_connection=true;" : string.Empty)}; Encrypt=False;")


akirayamamoto

I left some comments in there. Thanks Paul!

akirayamamoto · 2022-02-21T06:38:13Z

source/Nevermore.IntegrationTests/SetUp/IntegrationTestDatabase.cs

@@ -14,7 +14,7 @@ public IntegrationTestDatabase()
            var username = Environment.GetEnvironmentVariable("NevermoreTestUsername");
            var password = Environment.GetEnvironmentVariable("NevermoreTestPassword");
            testDatabaseName = Environment.GetEnvironmentVariable("NevermoreTestDatabase") ?? "Nevermore-IntegrationTests";
-            var builder = new SqlConnectionStringBuilder($"Server={sqlInstance};Database={testDatabaseName};{(username == null ? "Trusted_connection=true;" : string.Empty)}")
+            var builder = new SqlConnectionStringBuilder($"Server={sqlInstance};Database={testDatabaseName};{(username == null ? "Trusted_connection=true;" : string.Empty)}; Encrypt=False;")


Will we need to add this on all connection strings that we have on Octopus?

akirayamamoto · 2022-02-21T06:44:18Z

source/Nevermore/Transient/TransactionExtensions.cs

+
+        public static DbTransaction BeginTransactionWithRetry(this SqlConnection connection, IsolationLevel isolationLevel, string sqlServerTransactionName, RetryPolicy retryPolicy)
+        {
+            return (retryPolicy ?? RetryPolicy.NoRetry).LoggingRetries("Beginning Database Transaction").ExecuteAction(() => connection.BeginTransaction());


Suggested change

return (retryPolicy ?? RetryPolicy.NoRetry).LoggingRetries("Beginning Database Transaction").ExecuteAction(() => connection.BeginTransaction());

return (retryPolicy ?? RetryPolicy.NoRetry).LoggingRetries("Beginning Database Transaction").ExecuteAction(() => connection.BeginTransaction(isolationLevel, sqlServerTransactionName));

Don't we need to continue passing those arguments?

🤦 Yes, haha of course. Thanks for catching this :D

droyad · 2022-02-21T09:46:10Z

Will we need to add this on all connection strings that we have on Octopus?

@akirayamamoto We might have to set it to False just like with this setting

paulegradie commented Feb 17, 2022

View reviewed changes

paulegradie force-pushed the core/pg/add-retry-begin-transaction branch from 6cc0ac8 to 61aa573 Compare February 17, 2022 05:27

paulegradie commented Feb 17, 2022

View reviewed changes

paulegradie force-pushed the core/pg/add-retry-begin-transaction branch from 61aa573 to 0c6b31e Compare February 17, 2022 05:41

paulegradie commented Feb 17, 2022

View reviewed changes

paulegradie requested review from a team and droyad February 17, 2022 22:38

paulegradie marked this pull request as ready for review February 20, 2022 23:18

add retry for BeginTransaction

14bb02d

paulegradie force-pushed the core/pg/add-retry-begin-transaction branch from 0c6b31e to 14bb02d Compare February 21, 2022 03:13

akirayamamoto approved these changes Feb 21, 2022

View reviewed changes

pass arguments correctly

7971f63

paulegradie merged commit de7c60e into master Mar 1, 2022

paulegradie deleted the core/pg/add-retry-begin-transaction branch March 1, 2022 02:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add retry for BeginTransaction #183

Add retry for BeginTransaction #183

paulegradie commented Feb 17, 2022 •

edited

Loading

paulegradie Feb 17, 2022

paulegradie Feb 17, 2022

paulegradie Feb 17, 2022

akirayamamoto Feb 21, 2022

paulegradie Feb 28, 2022

paulegradie Feb 17, 2022

akirayamamoto left a comment

akirayamamoto Feb 21, 2022

akirayamamoto Feb 21, 2022

paulegradie Feb 28, 2022

droyad commented Feb 21, 2022 •

edited

Loading

		@@ -89,7 +89,7 @@ public async Task OpenAsync(IsolationLevel isolationLevel)

		// We use the synchronous overload here even though there is an async one, because the BeginTransactionAsync calls

	return (retryPolicy ?? RetryPolicy.NoRetry).LoggingRetries("Beginning Database Transaction").ExecuteAction(() => connection.BeginTransaction());
	return (retryPolicy ?? RetryPolicy.NoRetry).LoggingRetries("Beginning Database Transaction").ExecuteAction(() => connection.BeginTransaction(isolationLevel, sqlServerTransactionName));

Add retry for BeginTransaction #183

Add retry for BeginTransaction #183

Conversation

paulegradie commented Feb 17, 2022 • edited Loading

Benchmark Results

Before

After

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akirayamamoto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

droyad commented Feb 21, 2022 • edited Loading

paulegradie commented Feb 17, 2022 •

edited

Loading

droyad commented Feb 21, 2022 •

edited

Loading