[perf] Performance runs report the database is not correctly configured

[kotlarmilos]

All perf runs fail to send payloads to the Helix due to a database misconfiguration.

  Starting Azure Pipelines Test Run Ubuntu.1804.Amd64.Tiger.Perf
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : System.Net.Http.HttpRequestException: Request to https://dev.azure.com/dnceng/internal/_apis/test/runs?api-version=5.0 returned failed status 500 Internal Server Error [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :  [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : {"$id":"1","innerException":null,"message":"TF30040: The database is not correctly configured. Contact your Azure DevOps Server administrator.","typeName":"Microsoft.VisualStudio.Services.Common.VssServiceException, Microsoft.VisualStudio.Services.Common","typeKey":"VssServiceException","errorCode":0,"eventId":3000} [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :    at Microsoft.DotNet.Helix.AzureDevOps.AzureDevOpsTask.HandleFailedRequest(HttpRequestMessage req, HttpResponseMessage res) in /_/src/Microsoft.DotNet.Helix/Sdk/AzureDevOpsTask.cs:line 139 [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :    at Microsoft.DotNet.Helix.AzureDevOps.AzureDevOpsTask.ParseResponseAsync(HttpRequestMessage req, HttpResponseMessage res) in /_/src/Microsoft.DotNet.Helix/Sdk/AzureDevOpsTask.cs:line 147 [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :    at Microsoft.DotNet.Helix.AzureDevOps.StartAzurePipelinesTestRun.<>c__DisplayClass32_0.<<ExecuteCoreAsync>b__0>d.MoveNext() in /_/src/Microsoft.DotNet.Helix/Sdk/StartAzurePipelinesTestRun.cs:line 52 [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : --- End of stack trace from previous location --- [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :    at Microsoft.DotNet.Helix.AzureDevOps.AzureDevOpsTask.<>c__DisplayClass13_0.<<RetryAsync>b__0>d.MoveNext() in /_/src/Microsoft.DotNet.Helix/Sdk/AzureDevOpsTask.cs:line 89 [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : --- End of stack trace from previous location --- [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :    at Microsoft.DotNet.Helix.AzureDevOps.AzureDevOpsTask.RetryAsync[T](Func`1 function, Action`1 logRetry, Func`2 isRetryable, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/AzureDevOpsTask.cs:line 206 [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error :    at Microsoft.DotNet.Helix.AzureDevOps.AzureDevOpsTask.RetryAsync(Func`1 function) in /_/src/Microsoft.DotNet.Helix/Sdk/AzureDevOpsTask.cs:line 86 [/__w/1/s/eng/testing/performance/microbenchmarks.proj]
##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.23364.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : (NETCORE_ENGINEERING_TELEMETRY=Helix) System.Net.Http.HttpRequestException: Request to https://dev.azure.com/dnceng/internal/_apis/test/runs?api-version=5.0 returned failed status 500 Internal Server Error

https://dev.azure.com/dnceng/internal/_build/results?buildId=2225018&view=results

[caaavik-msft]

This issue seems to be caused by an issue/outage with Azure DevOps which should hopefully be resolved soon. It is affecting a few other .NET pipelines too. there is a thread in First Responders with an ICM linked to track this. I'll close this here since it's not caused by anything we can control.

[dkurepa]

We've seen this issue in other repos and opened an ICM with AzDo to resolve it https://portal.microsofticm.com/imp/v3/incidents/details/407504779/home