Stampeding API requests causes stalled workers

[clarafu]

Within our lidar-drills environment (which is running with Algorithm V3 and Lidar), we are putting the concourse deployment under a lot of stress (to put it in perspective we have about double the amount of jobs as wings). When we open the web UI to view the pipelines and click around, it causes a stampede of API requests to various endpoints (for example, GetJobs for the pipeline view or GetBuild from the build page).

These requests build up because there is a new one sent every tick if we haven't received a response back yet as they are all stuck pending. This is becoming a problem now because we have set a dedicated connection pool for the api connections (default to 10 right now) and once it hits that max 10 connections, workers will become stalled because they arn't able to send their heartbeating requests.

Maybe instead of sending a new request every tick, we can just wait for a response after the first request is sent?

Thanks!

[jamieklassen]

I'm confident we can ensure that single client doesn't have multiple in-flight requests to the same endpoint. This is a good web-ui issue.

[clarafu]

That's great! I'm going to add this to the 6.0 milestone since we probably want get this in with the new algorithm.

[jamieklassen]

related: #2776

[zoetian]

Let's not forget to ensure that there is a FetchUser call every 5 seconds on the dashboard. nothing that comes from the /userinfo endpoint will change in any 5 second interval.

[vito]

@pivotal-jamie-klassen I've extracted #4885 to cover the web UI aspect, but there's still a general concern over API requests starving the worker registration API requests of database connections which results in workers stalling.

@concourse/runtime Added this to the Runtime side-road - it's technically API related but I figured it has more to do with worker registration, and putting it in Runtime instead of Core could give us an opportunity to rethink how registration works if we want. (I have no strong suggestion.)

[odormond]

@vito, I noticed that you removed it from the v6.0.0 milestone. As I'm not familiar with your development workflow, I'm wondering if that means it's getting postponed for later or not?

[vito]

@odormond It just means it's not required for v6.0. It doesn't affect the priority; this is still high up in the runtime backlog.

[odormond]

Thanks @vito for the update.
Happy New Year and congratulation for the 5.8.0 release! We're excited by the containerd work and looking forward for a working implementation.