[BUG]: Self-hosted agents stuck/stop/freeze/time out at step "Try to append 1 batches web console lines for record..."

[LeaCCC]

What happened?

We have 7 self-hosted agents running on the server for ReadyAPI tests.
When there are 2 to 7 tests running in parallel, one of the parallel tests (sometimes 2) can get stuck at the ReadyAPI task, and eventually time out. This happens 1 out of 3 trial.

Versions

Azure DevOps Self Hosted Agent version: 3.225.0, 2.239.1

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2019 Standard

Version controll system

Azure DevOps Repo (Git)

Relevant log output

[2024-05-23 03:55:42Z INFO StepsRunner] Processing step: DisplayName='parallel readyAPI - Rating Unit Financial Business Service - DEV - CCOTFSDPLYUI01', ContinueOnError=False, Enabled=True
[2024-05-23 03:55:42Z INFO ExpressionManager] Evaluating: succeeded()
[2024-05-23 03:55:42Z INFO ExpressionManager] Result: True
[2024-05-23 03:55:42Z INFO StepsRunner] Starting the step.
[2024-05-23 03:55:42Z INFO TaskRunner] Allow publishing telemetry for SoapUIProForAzureDevOpsTask@2.1.7 task: False
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Bin': 'C:\agentDevOps4\bin'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Root': 'C:\agentDevOps4'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Work': 'C:\agentDevOps4\_work'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Tasks': 'C:\agentDevOps4\_work\_tasks'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Bin': 'C:\agentDevOps4\bin'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Root': 'C:\agentDevOps4'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'Work': 'C:\agentDevOps4\_work'
[2024-05-23 03:55:42Z INFO HostContext] Well known directory 'TaskZips': 'C:\agentDevOps4\_work\_taskzips'
[2024-05-23 03:55:42Z INFO TaskManager] Loading task definition 'C:\agentDevOps4\_work\_tasks\SoapUIProForAzureDevOpsTask_712dda37-2b62-4e68-950d-e85a85d51fc4\2.1.7\task.json'.
[2024-05-23 03:55:42Z INFO TaskRunner] Get handler data for target platform Windows
[2024-05-23 03:55:42Z INFO TaskRunner] Handler data is of type Microsoft.VisualStudio.Services.Agent.Worker.PowerShell3HandlerData
[2024-05-23 03:55:42Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Worker.IJobExtension'
[2024-05-23 03:55:42Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Worker.Build.ISourceProvider'
[2024-05-23 03:55:42Z INFO BuildJobExtension] Build JobExtension resolving path use source provide: CCC-BusinessService-RatingUnitFinancial-3-0-readyapi-project.xml
[2024-05-23 03:55:42Z INFO BuildJobExtension] The Default Path Root of Build JobExtension is repository.path: C:\agentDevOps4\_work\2\s
[2024-05-23 03:55:42Z INFO BuildJobExtension] After prefix Default Path Root provide by JobExtension: C:\agentDevOps4\_work\2\s\CCC-BusinessService-RatingUnitFinancial-3-0-readyapi-project.xml
[2024-05-23 03:55:42Z INFO BuildJobExtension] Return absolute path after prefix DefaultPathRoot: C:\agentDevOps4\_work\2\s\CCC-BusinessService-RatingUnitFinancial-3-0-readyapi-project.xml
[2024-05-23 03:55:42Z INFO TaskRunner] Build JobExtension resolved a rooted path:: C:\agentDevOps4\_work\2\s\CCC-BusinessService-RatingUnitFinancial-3-0-readyapi-project.xml
[2024-05-23 03:55:42Z INFO PowerShell3Handler] Handler.AfterExecutionContextInitialized _continueAfterCancelProcessTreeKillAttempt = True
[2024-05-23 03:55:42Z INFO PowerShellExeUtil] Generation: '1'
[2024-05-23 03:55:42Z INFO PowerShellExeUtil] Key name 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellEngine', value name 'PowerShellVersion': '2.0'
[2024-05-23 03:55:42Z INFO PowerShellExeUtil] Unsupported version. Skipping.
[2024-05-23 03:55:42Z INFO PowerShellExeUtil] Generation: '3'
[2024-05-23 03:55:42Z INFO PowerShellExeUtil] Key name 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\3\PowerShellEngine', value name 'PowerShellVersion': '5.1.17763.1'
[2024-05-23 03:55:42Z INFO PowerShellExeUtil] Key name 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\3\PowerShellEngine', value name 'ApplicationBase': 'C:\Windows\System32\WindowsPowerShell\v1.0'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper] Starting process:
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   File name: 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Arguments: '-NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". ([scriptblock]::Create('if ([Console]::InputEncoding -is [Text.UTF8Encoding] -and [Console]::InputEncoding.GetPreamble().Length -ne 0) { [Console]::InputEncoding = New-Object Text.UTF8Encoding $false } if (!$PSHOME) { $null = Get-Item -LiteralPath ''variable:PSHOME'' } else { Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Management\Microsoft.PowerShell.Management.psd1'')) ; Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Utility\Microsoft.PowerShell.Utility.psd1'')) }')) 2>&1 | ForEach-Object { Write-Verbose $_.Exception.Message -Verbose } ; Import-Module -Name 'C:\agentDevOps4\_work\_tasks\SoapUIProForAzureDevOpsTask_712dda37-2b62-4e68-950d-e85a85d51fc4\2.1.7\ps_modules\VstsTaskSdk\VstsTaskSdk.psd1' -ArgumentList @{ NonInteractive = $true } -ErrorAction Stop ; $VerbosePreference = 'Continue' ; $DebugPreference = 'Continue' ; Invoke-VstsTaskScript -ScriptBlock ([scriptblock]::Create('. ''C:\agentDevOps4\_work\_tasks\SoapUIProForAzureDevOpsTask_712dda37-2b62-4e68-950d-e85a85d51fc4\2.1.7\SoapUIProFunctionalTestingTask.ps1'''))"'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Working directory: 'C:\agentDevOps4\_work\_tasks\SoapUIProForAzureDevOpsTask_712dda37-2b62-4e68-950d-e85a85d51fc4\2.1.7'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Require exit code zero: 'True'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'False'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Persist current code page: 'True'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   ContinueAfterCancelProcessTreeKillAttempt: 'True'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Sigint timeout: '00:00:07.5000000'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Sigterm timeout: '00:00:02.5000000'
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper]   Try to use graceful shutdown: False
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper] OOM score adjustment is Linux-only.
[2024-05-23 03:55:42Z INFO ProcessInvokerWrapper] Process started with process id 9652, waiting for process exit.
[2024-05-23 03:55:43Z INFO JobServerQueue] Try to upload 1 log files or attachments, success rate: 1/1.
[2024-05-23 03:55:44Z INFO Worker] Metadata update message received.
[2024-05-23 03:55:44Z INFO Worker] Listening for cancel message from the channel.
[2024-05-23 03:55:44Z INFO Worker] Waiting for the job to complete or for a cancel message from the channel.
[2024-05-23 03:55:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '771d55ec-34a7-5ddd-398e-3e8e57b4f3f0', success rate: 1/1.
[2024-05-23 03:55:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:47Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:49Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:50Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:52Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:53Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:55:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:56:01Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:56:09Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:56:15Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:56:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:00Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:05Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:11Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:17Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:23Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:29Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:34Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:39Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:49Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:57:55Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:00Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:05Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:10Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:15Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:20Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:26Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:31Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:36Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:52Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:58:57Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:02Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:07Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:13Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:18Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:23Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:28Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:33Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:49Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 03:59:59Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:04Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:10Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:15Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:21Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:26Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:31Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:36Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:00:57Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:02Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:08Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:12Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:17Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:23Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:28Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:33Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:43Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:48Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:01:59Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:04Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:10Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:14Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:20Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:25Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:30Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:35Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:40Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:45Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:50Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:02:56Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:01Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:06Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:11Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:16Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:21Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:27Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:32Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:37Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:47Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:53Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:03:58Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:03Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:08Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:13Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:18Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:24Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:29Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:34Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:39Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:49Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:04:55Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:00Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:05Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:11Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:16Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:21Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:26Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:31Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:36Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:52Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:05:57Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:02Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:08Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:13Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:18Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:23Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:28Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:34Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:39Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:49Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:06:55Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:00Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:05Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:10Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:16Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:21Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:25Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:31Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:36Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:07:57Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:02Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:07Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:12Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:18Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:23Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:28Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:33Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:48Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:08:59Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:05Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:10Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:15Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:20Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:25Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:30Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:36Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:09:56Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:01Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:07Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:12Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:17Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:22Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:27Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:32Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:43Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:48Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:10:59Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:03Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:09Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:14Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:20Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:25Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:30Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:35Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:40Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:45Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:11:56Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:01Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:06Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:11Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:16Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:21Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:27Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:32Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:37Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:42Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:47Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:53Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:12:57Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:03Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:08Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:13Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:18Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:23Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:28Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:33Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:44Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:49Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:54Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:13:59Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:04Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:09Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:15Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:20Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:25Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:30Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:35Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:40Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:51Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:14:56Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:15:02Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:15:02Z INFO Worker] Cancellation/Shutdown message received.
[2024-05-23 04:15:02Z INFO ExpressionManager] Evaluating: succeeded()
[2024-05-23 04:15:02Z INFO ExpressionManager] Result: False
[2024-05-23 04:15:02Z INFO StepsRunner] Cancel current running step.
[2024-05-23 04:15:02Z INFO ProcessInvokerWrapper] Sending CTRL_C to process 9652.
[2024-05-23 04:15:02Z INFO ProcessInvokerWrapper] Successfully sent CTRL_C to process 9652.
[2024-05-23 04:15:02Z INFO ProcessInvokerWrapper] Waiting for process exit or 7.5 seconds after CTRL_C signal fired.
[2024-05-23 04:15:02Z INFO ProcessInvokerWrapper] Ignore Ctrl+C to current process.
[2024-05-23 04:15:03Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6bc11752-9934-5965-4fc2-cd3b5ece4252', success rate: 1/1.
[2024-05-23 04:15:03Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2024-05-23 04:15:03Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2024-05-23 04:15:03Z INFO ProcessInvokerWrapper] Exited process 9652 with exit code 1
[2024-05-23 04:15:03Z INFO ProcessInvokerWrapper] Process exited successfully.
[2024-05-23 04:15:03Z INFO ProcessInvokerWrapper] Process cancelled successfully through Ctrl+C/SIGINT.
[2024-05-23 04:15:03Z INFO ProcessInvokerWrapper] Finished process 9652 with exit code 1, and elapsed time 00:19:20.8288261.
[2024-05-23 04:15:03Z ERR  StepsRunner] Caught cancellation exception from step: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.CancellationToken.ThrowIfCancellationRequested()
   at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Agent.Sdk\ProcessInvoker.cs:line 384
   at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.VisualStudio.Services.Agent\ProcessInvoker.cs:line 329
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, Boolean continueAfterCancelProcessTreeKillAttempt, TimeSpan sigintTimeout, TimeSpan sigtermTimeout, Boolean useGracefulShutdown, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Agent.Worker\Handlers\StepHost.cs:line 85
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.PowerShell3Handler.RunAsync() in D:\a\_work\1\s\src\Agent.Worker\Handlers\PowerShell3Handler.cs:line 82
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsyncInternal() in D:\a\_work\1\s\src\Agent.Worker\TaskRunner.cs:line 454
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync() in D:\a\_work\1\s\src\Agent.Worker\TaskRunner.cs:line 76
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken) in D:\a\_work\1\s\src\Agent.Worker\StepsRunner.cs:line 264
[2024-05-23 04:15:03Z INFO StepsRunner] Step result: Canceled
[2024-05-23 04:15:03Z INFO StepsRunner] No need for updating job result with current step result 'Canceled'.
[2024-05-23 04:15:03Z INFO StepsRunner] Current state: job state = 'Canceled'

[DmitriiBobreshev]

Hi @LeaCCC, Thanks for the feedback! Could you set env. variables VSTSAGENT_TRACE and VSTS_AGENT_HTTPTRACE for detailed logs?

[LeaCCC]

Hi Dmitrii, Thank you for looking at this issue. Do I add these two variables to the DevOps pipeline and set them to 'true'? This is where I set system.debug = true? Thanks, lea From: Dmitrii Bobreshev ***@***.***> Sent: Friday, May 24, 2024 10:37 PM To: microsoft/azure-pipelines-agent ***@***.***> Cc: Deng, Lea ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/azure-pipelines-agent] [BUG]: Self-hosted agents stuck/stop/freeze/time out at step "Try to append 1 batches web console lines for record..." (Issue #4812) You don't often get email from ***@***.******@***.***>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi @LeaCCC<https://github.com/LeaCCC>, Thanks for the feedback! Could you set env. variables VSTSAGENT_TRACE<https://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Sdk/Knob/AgentKnobs.cs#L206> and VSTS_AGENT_HTTPTRACE<https://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Sdk/Knob/AgentKnobs.cs#L306> for detailed logs? - Reply to this email directly, view it on GitHub<#4812 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEF6JH5TF6QH77JKVN2VLLTZD4J3VAVCNFSM6AAAAABIGQNMQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRZGIYDQOJWGI>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> This electronic email and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed. The views expressed in this message are those of the individual sender and may not necessarily reflect the views of the Christchurch City Council. If you are not the correct recipient of this email please advise the sender and delete the email. [Christchurch City Council logo]<https://ccc.govt.nz/>

[DmitriiBobreshev]

Hi @LeaCCC, You need to set them as the system environment variables , please make sure that you re-run agent after you set them.

[LeaCCC]

Hi Dmitrii, thank you for your reply. I've added VSTSAGENT_TRACE and VSTS_AGENT_HTTPTRACE as environment variables on the window server where the self-hosted agents are running.
Please find agent log and worker log attached.
Pipeline started around 2024-05-27T22:21:48, then froze, and time out after about 20 minutes (20 minutes is the time out defined for the pipeline)

Agent_20240527-221243-utc_froze.log
Worker_20240527-222136-utc_froze.log

[ptemmer]

Very much interested in the outcome of this investigation as we're experiencing the exact same problem.

[ptemmer]

These are the latest log lines of our agent job execution, where you can observe there is almost a 1h gap

[2024-05-28 07:46:07Z INFO ProcessInvokerWrapper] Finished process 21272 with exit code 0, and elapsed time 00:08:01.6348290.
[2024-05-28 07:46:07Z INFO AsyncCommandContext] Start flush buffered output.
[2024-05-28 07:46:07Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1f06056c-1ae7-54e3-0278-7970ab3d0383', success rate: 1/1.
[2024-05-28 08:37:33Z INFO Worker] Cancellation/Shutdown message received.
[2024-05-28 08:37:33Z INFO ExpressionManager] Evaluating: always()
[2024-05-28 08:37:33Z INFO ExpressionManager] Result: True`

[ptemmer]

running v3.236.1 btw