Fixed racing conditions on sql-based snapshot stores

[Horusiath]

This is another approach of fixing problem solved by #1471 . Instead of working on making collection of cancellation tokens thread safe, we can remove it completely and use single shared cancellation token source between all async database calls.

[Aaronontheweb]

This PR has a compilation failure - why was it merged without checking the build log?

[rogeralsing]

??
========== Rebuild All: 85 succeeded, 0 failed, 0 skipped ==========

[Aaronontheweb]

[09:48:54]Checking for changes
[09:48:54]Collecting changes in 1 VCS root
[09:48:55]Clearing temporary directory: D:\BuildAgent\temp\buildTmp
[09:48:55]Publishing internal artifacts
[09:48:55]Using vcs information from agent file: 49b164d63843fb4.xml
[09:48:55]Clean build enabled: removing old files from D:\BuildAgent\work\49b164d63843fb4
[09:48:55]Checkout directory: D:\BuildAgent\work\49b164d63843fb4
[09:48:55]Updating sources: server side checkout (4s)
[09:48:59]Step 1/1: FAKE Build (Command Line) (19s)
[09:48:59][Step 1/1] Starting: D:\BuildAgent\temp\agentTmp\custom_script8667200924586386103.cmd
[09:48:59][Step 1/1] in directory: D:\BuildAgent\work\49b164d63843fb4
[09:49:00][Step 1/1] Checking for updates from https://www.nuget.org/api/v2/.
[09:49:01][Step 1/1] Currently running NuGet.exe 3.3.0.
[09:49:01][Step 1/1] NuGet.exe is up to date.
[09:49:01][Step 1/1] Checking for updates from https://www.nuget.org/api/v2/.
[09:49:02][Step 1/1] Currently running NuGet.exe 3.3.0.
[09:49:02][Step 1/1] NuGet.exe is up to date.
[09:49:03][Step 1/1] Feeds used:
[09:49:03][Step 1/1]   C:\Windows\system32\config\systemprofile\AppData\Local\NuGet\Cache
[09:49:03][Step 1/1]   C:\Windows\system32\config\systemprofile\.nuget\packages\
[09:49:03][Step 1/1]   https://api.nuget.org/v3/index.json
[09:49:03][Step 1/1] 
[09:49:03][Step 1/1] Attempting to gather dependencies information for package 'FAKE.4.1.0' with respect to project 'src\packages', targeting 'Any,Version=v0.0'
[09:49:04][Step 1/1] Attempting to resolve dependencies for package 'FAKE.4.1.0' with DependencyBehavior 'Lowest'
[09:49:04][Step 1/1] Resolving actions to install package 'FAKE.4.1.0'
[09:49:04][Step 1/1] Resolved actions to install package 'FAKE.4.1.0'
[09:49:04][Step 1/1] Adding package 'FAKE.4.1.0' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages'
[09:49:04][Step 1/1] Added package 'FAKE.4.1.0' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages'
[09:49:04][Step 1/1] Successfully installed 'FAKE 4.1.0' to src\packages
[09:49:05][Step 1/1] Feeds used:
[09:49:05][Step 1/1]   C:\Windows\system32\config\systemprofile\AppData\Local\NuGet\Cache
[09:49:05][Step 1/1]   C:\Windows\system32\config\systemprofile\.nuget\packages\
[09:49:05][Step 1/1]   https://api.nuget.org/v3/index.json
[09:49:05][Step 1/1] 
[09:49:05][Step 1/1] Attempting to gather dependencies information for package 'xunit.runner.console.2.0.0' with respect to project 'src\packages\FAKE', targeting 'Any,Version=v0.0'
[09:49:05][Step 1/1] Attempting to resolve dependencies for package 'xunit.runner.console.2.0.0' with DependencyBehavior 'Lowest'
[09:49:05][Step 1/1] Resolving actions to install package 'xunit.runner.console.2.0.0'
[09:49:05][Step 1/1] Resolved actions to install package 'xunit.runner.console.2.0.0'
[09:49:05][Step 1/1] Adding package 'xunit.runner.console.2.0.0' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages\FAKE'
[09:49:05][Step 1/1] Added package 'xunit.runner.console.2.0.0' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages\FAKE'
[09:49:05][Step 1/1] Successfully installed 'xunit.runner.console 2.0.0' to src\packages\FAKE
[09:49:06][Step 1/1] Feeds used:
[09:49:06][Step 1/1]   C:\Windows\system32\config\systemprofile\AppData\Local\NuGet\Cache
[09:49:06][Step 1/1]   C:\Windows\system32\config\systemprofile\.nuget\packages\
[09:49:06][Step 1/1]   https://api.nuget.org/v3/index.json
[09:49:06][Step 1/1] 
[09:49:06][Step 1/1] Attempting to gather dependencies information for package 'nunit.runners.2.6.4' with respect to project 'src\packages\FAKE', targeting 'Any,Version=v0.0'
[09:49:07][Step 1/1] Attempting to resolve dependencies for package 'nunit.runners.2.6.4' with DependencyBehavior 'Lowest'
[09:49:07][Step 1/1] Resolving actions to install package 'nunit.runners.2.6.4'
[09:49:07][Step 1/1] Resolved actions to install package 'nunit.runners.2.6.4'
[09:49:07][Step 1/1] Adding package 'NUnit.Runners.2.6.4' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages\FAKE'
[09:49:07][Step 1/1] Added package 'NUnit.Runners.2.6.4' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages\FAKE'
[09:49:07][Step 1/1] Successfully installed 'NUnit.Runners 2.6.4' to src\packages\FAKE
[09:49:07][Step 1/1] Feeds used:
[09:49:07][Step 1/1]   https://api.nuget.org/v3/index.json
[09:49:07][Step 1/1] 
[09:49:08][Step 1/1] Attempting to gather dependencies information for package 'SourceLink.Fake.1.1.0' with respect to project 'src\packages', targeting 'Any,Version=v0.0'
[09:49:08][Step 1/1] Attempting to resolve dependencies for package 'SourceLink.Fake.1.1.0' with DependencyBehavior 'Lowest'
[09:49:08][Step 1/1] Resolving actions to install package 'SourceLink.Fake.1.1.0'
[09:49:08][Step 1/1] Resolved actions to install package 'SourceLink.Fake.1.1.0'
[09:49:08][Step 1/1] Adding package 'SourceLink.Fake.1.1.0' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages'
[09:49:08][Step 1/1] Added package 'SourceLink.Fake.1.1.0' to folder 'D:\BuildAgent\work\49b164d63843fb4\src\packages'
[09:49:08][Step 1/1] Successfully installed 'SourceLink.Fake 1.1.0' to src\packages
[09:49:15][Step 1/1] Assembly version: 1.0.6
[09:49:15][Step 1/1] Nuget version; 1.0.6
[09:49:15][Step 1/1] 
[09:49:15][Step 1/1] Building project with version: 1699
[09:49:15][Step 1/1] Shortened DependencyGraph for Target All:
[09:49:15][Step 1/1] <== All
[09:49:15][Step 1/1]    <== BuildRelease
[09:49:15][Step 1/1]       <== CopyOutput
[09:49:15][Step 1/1]          <== Build
[09:49:15][Step 1/1]             <== RestorePackages
[09:49:15][Step 1/1]                <== AssemblyInfo
[09:49:15][Step 1/1]                   <== Clean
[09:49:15][Step 1/1]       <== CleanNuget
[09:49:15][Step 1/1]    <== RunTests
[09:49:15][Step 1/1]       <== CleanTests
[09:49:15][Step 1/1]    <== MultiNodeTests
[09:49:15][Step 1/1]    <== Nuget
[09:49:15][Step 1/1] 
[09:49:15][Step 1/1] The resulting target order is:
[09:49:15][Step 1/1]  - Clean
[09:49:15][Step 1/1]  - AssemblyInfo
[09:49:15][Step 1/1]  - RestorePackages
[09:49:15][Step 1/1]  - Build
[09:49:15][Step 1/1]  - CopyOutput
[09:49:15][Step 1/1]  - CleanNuget
[09:49:15][Step 1/1]  - BuildRelease
[09:49:15][Step 1/1]  - CleanTests
[09:49:15][Step 1/1]  - RunTests
[09:49:15][Step 1/1]  - MultiNodeTests
[09:49:15][Step 1/1]  - Nuget
[09:49:15][Step 1/1]  - All
[09:49:15][Step 1/1] Starting Target: Clean 
[09:49:15][Step 1/1] Target: Clean
[09:49:15][Step 1/1] D:\BuildAgent\work\49b164d63843fb4\bin does not exist.
[09:49:15][Step 1/1] Finished Target: Clean
[09:49:15][Step 1/1] Starting Target: AssemblyInfo (==> Clean)
[09:49:15][Step 1/1] Target: AssemblyInfo
[09:49:15][Step 1/1] Task: AssemblyInfo src/SharedAssemblyInfo.cs
[09:49:15][Step 1/1] Created AssemblyInfo file "src/SharedAssemblyInfo.cs".
[09:49:15][Step 1/1] Task: AssemblyInfo D:\BuildAgent\work\49b164d63843fb4\src\core\Akka.FSharp\Properties\AssemblyInfo.fs
[09:49:15][Step 1/1] Created AssemblyInfo file "D:\BuildAgent\work\49b164d63843fb4\src\core\Akka.FSharp\Properties\AssemblyInfo.fs".
[09:49:15][Step 1/1] Finished Target: AssemblyInfo
[09:49:15][Step 1/1] Starting Target: RestorePackages (==> AssemblyInfo)
[09:49:15][Step 1/1] Target: RestorePackages
[09:49:15][Step 1/1] Task: RestoreSolutionPackages ./src/Akka.sln
[09:49:15][Step 1/1] D:\BuildAgent\work\49b164d63843fb4\src\.nuget\NuGet.exe "restore" "D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln" "-OutputDirectory" "D:\BuildAgent\work\49b164d63843fb4\src\packages"
[09:49:16][Step 1/1] MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
[09:49:16][Step 1/1] Error parsing solution file at D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln: Exception has been thrown by the target of an invocation.
[09:49:16][Step 1/1] Task failed with Package restore of ./src/Akka.sln failed
[09:49:16][Step 1/1] Retry.
[09:49:16][Step 1/1] D:\BuildAgent\work\49b164d63843fb4\src\.nuget\NuGet.exe "restore" "D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln" "-OutputDirectory" "D:\BuildAgent\work\49b164d63843fb4\src\packages"
[09:49:16][Step 1/1] MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
[09:49:16][Step 1/1] Error parsing solution file at D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln: Exception has been thrown by the target of an invocation.
[09:49:16][Step 1/1] Task failed with Package restore of ./src/Akka.sln failed
[09:49:16][Step 1/1] Retry.
[09:49:16][Step 1/1] D:\BuildAgent\work\49b164d63843fb4\src\.nuget\NuGet.exe "restore" "D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln" "-OutputDirectory" "D:\BuildAgent\work\49b164d63843fb4\src\packages"
[09:49:17][Step 1/1] MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
[09:49:17][Step 1/1] Error parsing solution file at D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln: Exception has been thrown by the target of an invocation.
[09:49:17][Step 1/1] Task failed with Package restore of ./src/Akka.sln failed
[09:49:17][Step 1/1] Retry.
[09:49:17][Step 1/1] D:\BuildAgent\work\49b164d63843fb4\src\.nuget\NuGet.exe "restore" "D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln" "-OutputDirectory" "D:\BuildAgent\work\49b164d63843fb4\src\packages"
[09:49:17][Step 1/1] MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
[09:49:17][Step 1/1] Error parsing solution file at D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln: Exception has been thrown by the target of an invocation.
[09:49:17][Step 1/1] Task failed with Package restore of ./src/Akka.sln failed
[09:49:17][Step 1/1] Retry.
[09:49:17][Step 1/1] D:\BuildAgent\work\49b164d63843fb4\src\.nuget\NuGet.exe "restore" "D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln" "-OutputDirectory" "D:\BuildAgent\work\49b164d63843fb4\src\packages"
[09:49:18][Step 1/1] MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'.
[09:49:18][Step 1/1] Error parsing solution file at D:\BuildAgent\work\49b164d63843fb4\src\Akka.sln: Exception has been thrown by the target of an invocation.
[09:49:18][Step 1/1] ##teamcity[buildStatus status='FAILURE' text='System.Exception: Package restore of ./src/Akka.sln failed   at Fake.RestorePackageHelper.RestoreMSSolutionPackages@189-1.Invoke(String message) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 189   at Fake.RestorePackageHelper.runNuGet(String toolPath, TimeSpan timeOut, String args, FSharpFunc`2 failWith) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 90   at Fake.RestorePackageHelper.runNuGetTrial@94.Invoke(Unit unitVar0) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 94   at Fake.TaskRunnerHelper.runWithRetries|[a|](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32   at Fake.TaskRunnerHelper.runWithRetries|[a|](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32   at Fake.TaskRunnerHelper.runWithRetries|[a|](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32   at Fake.TaskRunnerHelper.runWithRetries|[a|](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32   at Fake.RestorePackageHelper.runNuGetTrial(Int32 retries, String toolPath, TimeSpan timeOut, String args, FSharpFunc`2 failWith) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 95   at Fake.RestorePackageHelper.RestoreMSSolutionPackages(FSharpFunc`2 setParams, String solutionFile) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 189   at FSI_0001.Build.clo@63-2.Invoke(Unit _arg1) in D:\BuildAgent\work\49b164d63843fb4\build.fsx:line 65   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 411']
[09:49:18][Step 1/1] Running build failed.
[09:49:18][Step 1/1] Error:
[09:49:18][Step 1/1] System.Exception: Package restore of ./src/Akka.sln failed
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.RestoreMSSolutionPackages@189-1.Invoke(String message) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 189
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.runNuGet(String toolPath, TimeSpan timeOut, String args, FSharpFunc`2 failWith) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 90
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.runNuGetTrial@94.Invoke(Unit unitVar0) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 94
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.runNuGetTrial(Int32 retries, String toolPath, TimeSpan timeOut, String args, FSharpFunc`2 failWith) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 95
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.RestoreMSSolutionPackages(FSharpFunc`2 setParams, String solutionFile) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 189
[09:49:18][Step 1/1]    at FSI_0001.Build.clo@63-2.Invoke(Unit _arg1) in D:\BuildAgent\work\49b164d63843fb4\build.fsx:line 65
[09:49:18][Step 1/1]    at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 411
[09:49:18][Step 1/1] 
[09:49:18][Step 1/1] ---------------------------------------------------------------------
[09:49:18][Step 1/1] Build Time Report
[09:49:18][Step 1/1] ---------------------------------------------------------------------
[09:49:18][Step 1/1] Target         Duration
[09:49:18][Step 1/1] ------         --------
[09:49:18][Step 1/1] Clean          00:00:00.0013530
[09:49:18][Step 1/1] AssemblyInfo   00:00:00.1094329
[09:49:18][Step 1/1] Total:         00:00:03.0194226
[09:49:18][Step 1/1] ---------------------------------------------------------------------
[09:49:18][Step 1/1] Status:        Failure
[09:49:18][Step 1/1] ---------------------------------------------------------------------
[09:49:18][Step 1/1]   1) System.Exception: Package restore of ./src/Akka.sln failed
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.RestoreMSSolutionPackages@189-1.Invoke(String message) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 189
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.runNuGet(String toolPath, TimeSpan timeOut, String args, FSharpFunc`2 failWith) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 90
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.runNuGetTrial@94.Invoke(Unit unitVar0) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 94
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.TaskRunnerHelper.runWithRetries[a](FSharpFunc`2 f, Int32 retries) in C:\code\fake\src\app\FakeLib\TaskRunnerHelper.fs:line 32
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.runNuGetTrial(Int32 retries, String toolPath, TimeSpan timeOut, String args, FSharpFunc`2 failWith) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 95
[09:49:18][Step 1/1]    at Fake.RestorePackageHelper.RestoreMSSolutionPackages(FSharpFunc`2 setParams, String solutionFile) in C:\code\fake\src\app\FakeLib\RestorePackageHelper.fs:line 189
[09:49:18][Step 1/1]    at FSI_0001.Build.clo@63-2.Invoke(Unit _arg1) in D:\BuildAgent\work\49b164d63843fb4\build.fsx:line 65
[09:49:18][Step 1/1]    at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 411
[09:49:18][Step 1/1] Process exited with code 42
[09:49:18][Step 1/1] Step FAKE Build (Command Line) failed
[09:49:19]Publishing internal artifacts
[09:49:19]Publishing artifacts

[Aaronontheweb]

Malformed Akka.SLN file that can't be parsed for NuGet package restore

[Aaronontheweb]

Build server has to be the single source of truth for this stuff.

[Aaronontheweb]

What's weird about this though, is Akka.SLN wasn't updated at all... Dafuq

[Aaronontheweb]

I think this is the culprit: NuGet/Home#1631

[Aaronontheweb]

Nevermind, culprit was the big Akka.Cluster.Tools merge earlier