Agents timing out inside private vnet... looking to increase timeout threshold on Agent.Listener

[es-cs1]

Hello,

we are experimenting putting Docker Agents in a private Vnet, everything works fine about 90% of the time.... however there are cases when our agents take too long to send network traffic through the Vnet to the azure devops server (Dev.azure.com).

we get the classic

##[error]We stopped hearing from agent <agent-name>. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see:
https://go.microsoft.com/fwlink/?linkid=846610

In our logs we get messages like this:

   at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.GetNextMessageAsync(CancellationToken token) in /mnt/vss/_work/1/s/src/Agent.Listener/MessageListener.cs:line 178
[2024-03-11 14:04:10Z INFO MessageListener] Retriable exception: The HTTP request timed out after 00:01:00.
[2024-03-11 14:04:10Z INFO AgentServer] Refresh MessageQueue VssConnection to get on a different AFD node.

This is an Azure Agent v 3.236.1 running inside an Ubuntu Docker Container (using the reference Dockerfile on learn.microsoft.com)

How can we increase that Agent.Listener threshold so the HTTP Requests dont time out after only 1 minute?

This issue only appeared when the agents are put behind the Vnet.

[es-cs1]

We tried piping the logs of the container group into a log analytics workspace, but it ended up giving the message

2024-03-11, 5:52:32.745 p.m.
2024-03-11 17:52:32Z: Agent reconnected.
2024-03-11, 5:51:14.059 p.m.
2024-03-11 17:51:14Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
2024-03-11, 5:45:09.128 p.m.
2024-03-11 17:45:09Z: Agent reconnected.
2024-03-11, 5:43:43.729 p.m.
2024-03-11 17:43:43Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
2024-03-11, 5:43:43.724 p.m.
- /azp/_diag/Agent_20240311-173244-utc.log
2024-03-11, 5:43:43.723 p.m.
Error reported in diagnostic logs. Please examine the log for more details.

so the contents of /azp/_diag/Agent_20240311-173244-utc.log is not actually passed over to the log analytics workspace, and in the terraform definition, there doesnt seem to be any mention of diagnostic settings other than the log_analytics block https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/container_group

# terraform code:

  dynamic "diagnostics" {
    for_each = toset(range(var.log_analytics == null ? 0: 1))
    content {
      log_analytics {
        workspace_id  = data.azurerm_log_analytics_workspace.my_law[0].workspace_id
        workspace_key = data.azurerm_log_analytics_workspace.my_law[0].primary_shared_key
      }
    }
  }