EdgeAgent: Insufficient Timeout to perform polling

[yassineselmi]

Bug description
I'm exposing Portainer Master instance behind Cloudflare Tunnel and the trying to make the remote Edge Agent connects to it.
After troubleshooting, I found that the default request timeout of 10 seconds is not sufficient for high latency environment.

I noticed that the timeout is set to 10 second and cannot be changed from outside.

From edge.go#L107

portainerClient := client.NewPortainerClient(
	// ...
	client.BuildHTTPClient(10, manager.agentOptions), // <=== Timeout hardcoded to 10 seconds
)

Same issue in the healthcheck process:

cli := client.BuildHTTPClient(10, options) // <==Timeout hardcoded to 10 seconds

Expected behavior
A valid response from the Portainer Server when calling the api endpoints /api/endpoints/5/edge/status or /api/status

Portainer Logs

edge-agent_1        | 2022/11/18 09:44PM INF github.com/portainer/agent/cmd/agent/main.go:79 > agent running on Docker platform |
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/cmd/agent/main.go:89 > member_tags="&{AgentPort:9001 EdgeKeySet:false NodeName:de-wks-ys DockerConfiguration:{EngineStatus:1 Leader:false NodeRole:0} KubernetesConfiguration:{}}"
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/docker/docker.go:102 > retrieving IP address from container network | ip_address=172.20.0.3 network_name=portainer-edge_default
edge-agent_1        | 2022/11/18 09:44PM INF github.com/portainer/agent/edge/key.go:144 > edge key loaded from options |
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/cmd/agent/main.go:323 > edge key found in environment. Associating Edge key |
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/edge/edge.go:91 > api_addr=172.20.0.3:9001 edge_id=ID_OF_EDGE=<MY_EDGE-ID> inactivity_timeout=5m insecure_poll=false poll_frequency=5s tunnel_capability=true
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/edge/scheduler/logs.go:26 > logs manager started |
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/edge/poll.go:135 > starting Portainer short-polling client | poll_interval_seconds=5 server_url=https://<cloudflare-tunnel-url-for-portainer>:<port>
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/edge/poll.go:160 > inactivity_timeout=5m0s monitoring_interval_seconds=30
edge-agent_1        | 2022/11/18 09:44PM DBG github.com/portainer/agent/edge/edge.go:261 > Docker runtime configuration check | engine_status=1 leader_node=false
edge-agent_1        | 2022/11/18 09:44PM INF github.com/portainer/agent/edge/registry/server.go:101 > starting registry credential server |
edge-agent_1        | 2022/11/18 09:44PM INF github.com/portainer/agent/http/server.go:100 > starting Agent API server | api_version=2.16.1 server_addr=172.20.0.3 server_port=9001 use_tls=false


edge-agent_1        | 2022/11/18 09:45PM DBG github.com/portainer/agent/edge/client/portainer_edge_client.go:89 > sending agent platform header | header=1
edge-agent_1        | 2022/11/18 09:45PM DBG github.com/portainer/agent/edge/edge.go:261 > Docker runtime configuration check | engine_status=1 leader_node=false
edge-agent_1        | 2022/11/18 09:45PM ERR github.com/portainer/agent/edge/poll.go:142 > an error occured during short poll | error="Get \"https://<cloudflare-tunnel-url-for-portainer>:<port>/api/endpoints/5/edge/status\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Steps to reproduce the issue:

PS: It is hard to reproduce the problem in another environment. Please consider low latency connections.

I wrote a small Go script that calls the same HTTP request (same Endpoint, headers, and timeout of 10 seconds), and I was able to reproduce the issue.

By increasing the timeout to 30 seconds, everything worked fine and I was able to get a valid response from the API (api/endpoints/5/edge/status

Technical details:

• Portainer version: 2.16.1
• Docker version (managed by Portainer): 20.10.17, build 100c701
• Kubernetes version (managed by Portainer): N/A
• Platform (windows/linux): Linux
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): Docker compose
• Browser: N/A
• Use Case (delete as appropriate): Using Portainer at Home, Using Portainer in a Commercial setup.
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context
Add any other context about the problem here.