Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Location.GetConnectionData : The HTTP request timed out (With agent 2.177.0) #3176

Closed
ghost opened this issue Nov 27, 2020 · 11 comments
Closed

Comments

@ghost
Copy link

ghost commented Nov 27, 2020

Agent Version and Platform

Version of your agent? 2.177.1

OS of the machine running the agent? Ubuntu 18.04, CentOS8

Azure DevOps Type and Version

dev.azure.com

If dev.azure.com, what is your organization name? Accelize

What's not working?

The command :

./config.sh --unattended --acceptteeeula --pool Default --agent "125061 Acid AWS EC2 OSname" --replace --url https://dev.azure.com/Accelize/ --work /dev/shm/_work --auth PAT --token ***

Returns the error:

The HTTP request timed out after 00:01:40.

We use an Ansible role to provision the agent on differents OS and cloud (Azure or AWS). This works good since almost one year now. But recently, we starded get this issues on some case like (100% reproductible):

  • AWS + Ubuntu 18.04
  • AWS + CentOS 8

Strangely, this works on:

  • AWS + CentOS 7
  • Azure + Ubuntu 18.04
  • Azure + CentOS 7
  • Azure + CentOS 8

There is no differences between theses cases (Except base image and cloud provider), so I do not see from where come the issue. Especially since this worked perfectly before.

Agent and Worker's Diagnostic Logs

[2020-11-27 11:22:12Z INFO AgentProcess] Agent package linux-x64
[2020-11-27 11:22:12Z INFO AgentProcess] Running on Linux (X64)
[2020-11-27 11:22:12Z INFO AgentProcess] RuntimeInformation: Linux 4.18.0-193.6.3.el8_2.x86_64 #1 SMP Wed Jun 10 11:09:32 UTC 2020
[2020-11-27 11:22:12Z INFO AgentProcess] Version: 2.177.1
[2020-11-27 11:22:12Z INFO AgentProcess] Commit: 772540a8a7439472b715cd4a8a8c126db751d077
[2020-11-27 11:22:12Z INFO AgentProcess] Culture: 
[2020-11-27 11:22:12Z INFO AgentProcess] UI Culture: 
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO AgentProcess] Validating directory permissions for: '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO CommandSettings] Configure {
  "AcceptTeeEula": true,
  "AddDeploymentGroupTags": false,
  "AddEnvironmentVirtualMachineResourceTags": false,
  "AddMachineGroupTags": false,
  "AlwaysExtractTask": false,
  "Agent": "125061 Acid AWS EC2 CentOS 8",
  "CollectionName": null,
  "DeploymentGroup": false,
  "DeploymentGroupName": null,
  "DeploymentGroupTags": null,
  "DeploymentPool": false,
  "DeploymentPoolName": null,
  "EnvironmentVMResource": false,
  "EnvironmentName": null,
  "EnvironmentVMResourceTags": null,
  "GitUseSChannel": false,
  "DisableLogUploads": false,
  "MachineGroup": false,
  "MachineGroupName": null,
  "MachineGroupTags": null,
  "MonitorSocketAddress": null,
  "NotificationPipeName": null,
  "NotificationSocketAddress": null,
  "NoRestart": false,
  "OverwriteAutoLogon": false,
  "Pool": "Default",
  "ProjectName": null,
  "ProxyPassword": null,
  "ProxyUserName": null,
  "ProxyUrl": null,
  "Replace": true,
  "RunAsAutoLogon": false,
  "RunAsService": false,
  "RunOnce": false,
  "SslCACert": null,
  "SslClientCert": null,
  "SslClientCertArchive": null,
  "SslClientCertKey": null,
  "SslClientCertPassword": null,
  "SslSkipCertValidation": false,
  "Url": "https://dev.azure.com/Accelize/",
  "WindowsLogonAccount": null,
  "WindowsLogonPassword": null,
  "Work": "/dev/shm/_work",
  "Auth": "PAT",
  "LaunchBrowser": false,
  "Password": null,
  "Token": "***",
  "Unattended": true,
  "UserName": null,
  "Help": false,
  "Version": false
}
[2020-11-27 11:22:12Z INFO AgentProcess] Arguments parsed
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Proxy': '/opt/azure_pipeline_agent/.proxy'
[2020-11-27 11:22:12Z INFO VstsAgentWebProxy] No proxy setting found
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Certificates': '/opt/azure_pipeline_agent/.certificates'
[2020-11-27 11:22:12Z INFO AgentCertificateManager] No certificate setting found
[2020-11-27 11:22:12Z INFO Agent] ExecuteCommand
[2020-11-27 11:22:12Z INFO ConfigurationStore] currentAssemblyLocation: /opt/azure_pipeline_agent/bin/Agent.Listener.dll
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO ConfigurationStore] binPath: /opt/azure_pipeline_agent/bin
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO ConfigurationStore] RootFolder: /opt/azure_pipeline_agent
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Agent': '/opt/azure_pipeline_agent/.agent'
[2020-11-27 11:22:12Z INFO ConfigurationStore] ConfigFilePath: /opt/azure_pipeline_agent/.agent
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Credentials': '/opt/azure_pipeline_agent/.credentials'
[2020-11-27 11:22:12Z INFO ConfigurationStore] CredFilePath: /opt/azure_pipeline_agent/.credentials
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Service': '/opt/azure_pipeline_agent/.service'
[2020-11-27 11:22:12Z INFO ConfigurationStore] ServiceConfigFilePath: /opt/azure_pipeline_agent/.service
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Autologon': '/opt/azure_pipeline_agent/.autologon'
[2020-11-27 11:22:12Z INFO ConfigurationStore] AutoLogonSettingsFilePath: /opt/azure_pipeline_agent/.autologon
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'Options': '/opt/azure_pipeline_agent/.options'
[2020-11-27 11:22:12Z INFO ConfigurationStore] RuntimeOptionsFilePath: /opt/azure_pipeline_agent/.options
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known config file 'SetupInfo': '/opt/azure_pipeline_agent/.setup_info'
[2020-11-27 11:22:12Z INFO ConfigurationStore] SetupInfoFilePath: /opt/azure_pipeline_agent/.setup_info
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: 
  ___                      ______ _            _ _
 / _ \                     | ___ (_)          | (_)
/ /_\ \_____   _ _ __ ___  | |_/ /_ _ __   ___| |_ _ __   ___  ___
|  _  |_  / | | | '__/ _ \ |  __/| | '_ \ / _ \ | | '_ \ / _ \/ __|
| | | |/ /| |_| | | |  __/ | |   | | |_) |  __/ | | | | |  __/\__ \
\_| |_/___|\__,_|_|  \___| \_|   |_| .__/ \___|_|_|_| |_|\___||___/
                                   | |
        agent v2.177.1             |_|          (commit 772540a)

[2020-11-27 11:22:12Z INFO ConfigurationManager] ConfigureAsync
[2020-11-27 11:22:12Z INFO ConfigurationStore] IsConfigured()
[2020-11-27 11:22:12Z INFO ConfigurationStore] IsConfigured: False
[2020-11-27 11:22:12Z INFO ConfigurationManager] Is configured: False
[2020-11-27 11:22:12Z INFO CommandSettings] Flag 'sslskipcertvalidation': 'False'
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: 
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: >> End User License Agreements:
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: 
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Bin': '/opt/azure_pipeline_agent/bin'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Root': '/opt/azure_pipeline_agent'
[2020-11-27 11:22:12Z INFO HostContext] Well known directory 'Externals': '/opt/azure_pipeline_agent/externals'
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: Building sources from a TFVC repository requires accepting the Team Explorer Everywhere End User License Agreement. This step is not required for building sources from Git repositories

A copy of the Team Explorer Everywhere license agreement can be found at:
  /opt/azure_pipeline_agent/externals/tee/license.html
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: 
[2020-11-27 11:22:12Z INFO CommandSettings] Flag 'acceptteeeula': 'True'
[2020-11-27 11:22:12Z INFO CommandSettings] Flag 'deploymentgroup': 'False'
[2020-11-27 11:22:12Z INFO CommandSettings] Flag 'deploymentpool': 'False'
[2020-11-27 11:22:12Z INFO CommandSettings] Flag 'environment': 'False'
[2020-11-27 11:22:12Z INFO ExtensionManager] Getting extensions for interface: 'Microsoft.VisualStudio.Services.Agent.Listener.Configuration.IConfigurationProvider'
[2020-11-27 11:22:12Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.BuildReleasesAgentConfigProvider, Agent.Listener
[2020-11-27 11:22:12Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.DeploymentGroupAgentConfigProvider, Agent.Listener
[2020-11-27 11:22:12Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.SharedDeploymentAgentConfigProvider, Agent.Listener
[2020-11-27 11:22:12Z INFO ExtensionManager] Creating instance: Microsoft.VisualStudio.Services.Agent.Listener.Configuration.EnvironmentVMResourceConfigProvider, Agent.Listener
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: 
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: >> Connect:
[2020-11-27 11:22:12Z INFO Terminal] WRITE LINE: 
[2020-11-27 11:22:12Z INFO CommandSettings] Arg 'url': 'https://dev.azure.com/Accelize/'
[2020-11-27 11:22:12Z INFO ConfigurationManager] GetCredentialProvider
[2020-11-27 11:22:12Z INFO CommandSettings] Arg 'auth': 'PAT'
[2020-11-27 11:22:12Z INFO ConfigurationManager] Creating credential for auth: PAT
[2020-11-27 11:22:12Z INFO CredentialManager] GetCredentialProvider
[2020-11-27 11:22:12Z INFO CredentialManager] Creating type PAT
[2020-11-27 11:22:12Z INFO CredentialManager] Creating credential type: PAT
[2020-11-27 11:22:12Z INFO PersonalAccessToken] EnsureCredential
[2020-11-27 11:22:12Z INFO CommandSettings] Arg 'token': '***'
[2020-11-27 11:22:12Z INFO PersonalAccessToken] GetVssCredentials
[2020-11-27 11:22:12Z INFO PersonalAccessToken] token retrieved: 52 chars
[2020-11-27 11:22:12Z INFO PersonalAccessToken] cred created
[2020-11-27 11:22:12Z INFO ConfigurationManager] cred retrieved
[2020-11-27 11:22:12Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-11-27 11:23:52Z WARN VisualStudioServices] GET request to https://dev.azure.com/Accelize/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 timed out after 100 seconds
[2020-11-27 11:23:52Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-11-27 11:23:52Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2020-11-27 11:25:32Z WARN VisualStudioServices] GET request to https://dev.azure.com/Accelize/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 timed out after 100 seconds
[2020-11-27 11:25:32Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2020-11-27 11:25:32Z INFO LocationServer] Unable to connect to https://dev.azure.com/Accelize/
[2020-11-27 11:25:32Z ERR  LocationServer] System.TimeoutException: The HTTP request timed out after 00:01:40
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
[2020-11-27 11:25:32Z INFO CommandSettings] Flag 'unattended': 'True'
[2020-11-27 11:25:32Z ERR  Agent] System.TimeoutException: The HTTP request timed out after 00:01:40
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection)
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.IsHostedServer(String serverUrl, VssCredentials credentials)
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.ConfigureAsync(CommandSettings command)
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.ExecuteCommand(CommandSettings command)
[2020-11-27 11:25:32Z ERR  Terminal] WRITE ERROR: The HTTP request timed out after 00:01:40
@sdobrodeev sdobrodeev added the bug label Dec 4, 2020
@ghost
Copy link
Author

ghost commented Dec 8, 2020

Hello, is there some extra information I can give or debug option I can use to help in this issue resolution?

@anatolybolshakov
Copy link
Contributor

Hi @xlz-jgoutin, is 'https://dev.azure.com/Accelize/' reachable from agent machine? If you experience some network issues - you can also use agent knobs - just by increasing value for them by setting VSTS_HTTP_RETRY and VSTS_HTTP_TIMEOUT environment variables - to increase HTTP timeout and retry count for agent

@anatolybolshakov anatolybolshakov self-assigned this Dec 8, 2020
@ghost
Copy link
Author

ghost commented Dec 8, 2020

Hello,

I did the following test on a fresh Ubuntu 18.04 instance:

curl https://dev.azure.com/Accelize/ -L

=> Success (with a very short response delay)

Trying to configure the agent

./config.sh --unattended --acceptteeeula --pool Default --agent "TEST123156 AWS EC2 Ubuntu 18.04" --replace --url https://dev.azure.com/Accelize/ --auth PAT --token ***

=> Error The HTTP request timed out after 00:01:40. (Same full log as previously)

Retrying with curl after the config command.

curl https://dev.azure.com/Accelize/ -L

=> Success.

Test with full URL from logs:

curl  https://dev.azure.com/Accelize/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 -L

=> Success

I also run the same test from my full CI process by editing my Ansible playbook, the curl command always works everywhere, but the ./config.sh always fails on Ubuntu 18.04 & CentOS 8 on AWS.

@anatolybolshakov
Copy link
Contributor

@xlz-jgoutin could you please try to increase http timeout/retry count also using agent knobs?

@ghost
Copy link
Author

ghost commented Dec 10, 2020

I rerun the full tests sequences with increased values:
VSTS_HTTP_RETRY: 10
VSTS_HTTP_TIMEOUT: 300

I still get the same error on the same set of OS (But after 50 minutes of a single "./config.sh" run).

Note that:

  • when using something like curl, the response to the URL is instant.
  • The issue is 100% reproducible with the same set of resulting failing/successful OS.

This does not seem to be a network issue.

@ghost
Copy link
Author

ghost commented Dec 10, 2020

Instead of using the latest release, I tried to run my tests (Ubuntu 18.04 only) against pinned versions:

So, the conclusion is that the issue is only with the agent version >= 2.177.0

@ghost ghost changed the title Location.GetConnectionData : The HTTP request timed out Location.GetConnectionData : The HTTP request timed out (With agent 2.177.0) Dec 11, 2020
ghost pushed a commit to Accelize/acid-cli that referenced this issue Dec 11, 2020
ghost pushed a commit to Accelize/acid-cli that referenced this issue Dec 11, 2020
Add option to pin to a specific agent version
Pin version on some tests (Cf microsoft/azure-pipelines-agent#3176)
@ghost
Copy link
Author

ghost commented Dec 11, 2020

I seen "Switch to new HTTP handler by default (#3117)" in the 2.177.0 changelog.

So i tried to run 2.177.1 agent with AZP_AGENT_USE_LEGACY_HTTP=true, and it completes successfully.

The cause of this issue is related to the new HTTP handler.

@anatolybolshakov
Copy link
Contributor

@xlz-jgoutin thanks for investigation here, as per release notes for 2.177.0 - new HTTP handler is used by default, so to use legacy http - you just need to set AZP_AGENT_USE_LEGACY_HTTP=true.
I'm closing it at the moment since it's resolved, please let us know if you have any questions.

@ghost
Copy link
Author

ghost commented Jan 5, 2021

Is there a chance that AZP_AGENT_USE_LEGACY_HTTP=true become deprecated in a near future, or can it be used as a reliable future proof workaround?

Suggestion: the agent may try to fallback itself to the legacy HTTP handler if failing the new HTTP handler.

@TedaLIEz
Copy link

This issue is 100% reproducible even in 2.204.0! It took me hours to find this issue and try to work around it with AZP_AGENT_USE_LEGACY_HTTP=true. Is there an ETA for a fix for this?

@sam-crisp
Copy link

sam-crisp commented Aug 27, 2022

Running Ubuntu 20.04 and could never get service to go past the 'Connecting to the server' error to Azure DevOps Server. Thought it might be SSL related but no logs to indicate. I installed 7 different versions of the agent including the prerelease 2.209.0 to see if anything could help. No dice; same connecting standoff.

Took the suggestion here and installed v2.165.0 and finally got it to connect! But I sent over the first job and the agent updates to 2.181.2 (even though the agent pool is set to NOT auto update).

Once it updates, I'm back to the 'Connection to the server' error.

Very frustrating; I have no way to build on a linux os.


/etc/systemd/system/vsts.agent.acustgint\x2dtfs01.LINUX\x2dBUILD\x2d01.acudevint\x2dadp02.service
● vsts.agent.acustgint\x2dtfs01.LINUX\x2dBUILD\x2d01.acudevint\x2dadp02.service - Azure Pipelines Agent (acustgint-tfs01.LINUX-BUILD-01.acudevint-adp02)
     Loaded: loaded (/etc/systemd/system/vsts.agent.acustgint\x2dtfs01.LINUX\x2dBUILD\x2d01.acudevint\x2dadp02.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-08-27 12:33:06 UTC; 5s ago
   Main PID: 99767 (runsvc.sh)
      Tasks: 9531 (limit: 9530)
     Memory: 1.7G
     CGroup: /system.slice/vsts.agent.acustgint\x2dtfs01.LINUX\x2dBUILD\x2d01.acudevint\x2dadp02.service
             ├─ 5792 ./externals/node/bin/node ./bin/AgentService.js
             ├─ 5799 /home/xxxx/bin/Agent.Listener run --startuptype service
             └─99767 /bin/bash /home/xxxx/vsts-2.165.0/runsvc.sh

Aug 27 12:33:06 acudevint-adp02 systemd[1]: Started Azure Pipelines Agent (acustgint-tfs01.LINUX-BUILD-01.acudev…adp02).
Aug 27 12:33:06 acudevint-adp02 runsvc.sh[99767]: /home/xxxx/vsts-2.165.0/runsvc.sh: fork: retry: Resource tem…ailable
Aug 27 12:33:07 acudevint-adp02 runsvc.sh[99767]: /home/xxxx/vsts-2.165.0/runsvc.sh: fork: retry: Resource tem…ailable
Aug 27 12:33:09 acudevint-adp02 runsvc.sh[99767]: /home/xxxx/vsts-2.165.0/runsvc.sh: fork: retry: Resource tem…ailable
Hint: Some lines were ellipsized, use -l to show in full.
xxxx@xxxx-adp02:~/vsts-2.165.0$ ./run.sh
Scanning for tool capabilities.
Connecting to the server.
2022-08-27 12:33:48Z: Listening for Jobs
Agent update in progress, do not shutdown agent.
Downloading 2.181.2 agent
Waiting for current job finish running.
Generate and execute update script.
Agent will exit shortly for update, should back online within 10 seconds.
xxxx@xxxx-adp02:~/vsts-2.165.0$ renamed '/home/xxxx/vsts-2.165.0/_diag/SelfUpdate-20220827-123429.log' -> '/home/xxxx/vsts-2.165.0/_diag/SelfUpdate-20220827-123429.log.succeed'
Scanning for tool capabilities.
Connecting to the server.```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants