[Synthetics UI] Control the number of concurrent browser runs a Private Location can run

[paulb-elastic]

Heartbeat allows a configuration to control how many browser monitors can run concurrently.

However, in Private Locations this is locked to 2, meaning a fixed number of up to two concurrent browser monitors will run at a time.

Whilst https://github.com/elastic/synthetics-dev/issues/151 will allow a superior load balanced solution, there is still a useful simpler implementation where users can configure how many concurrent browser runs should be allowed, which they set based on the hardware profile of the machine running the Elastic Agent for that Private Location.

ACs:

• Provide an optional configuration parameter for the number of concurrent browser monitors (per Private Location)
• If not set, this will use the default in Heartbeat (which is 2)
• The user can set this to a positive integer larger than zero
  • TBC: do we want to apply a sensible limit to this, even if that’s 100 to stop users setting it to a problematic number?
• The setting can only be changed by users with the same role / permissions as those that can add Private Locations
• The setting can be made when creating a Private Location, or editing it later
• The change will be propagated out to the Private Location reasonably quickly
  • It is expected this will take effect within 2 minutes and be reflected in newly scheduled monitors (i.e. if Heartbeat is in the middle of running monitors, it can wait for those to finish before the new setting takes effect)

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[paulb-elastic]

Speaking with @andrewvc he's thinking it might be a simpler implementation to control this with something like an environment variable as opposed to configuration pushed out through Fleet. He is going to review this and we may end up with a different issue to do that rather than this.

[paulb-elastic]

As per #147081 (comment), I've raised elastic/beats#34290 as an initial/simpler implementation that we will focus on (moving this issue as a potential enhancement, post 1.0)

[angeliski]

Just a comment, it would be nice if we could define this based on the test itself.

We have journeys we want to run concurrently, but others don't make sense. We can achieve that for now if we use separated Private Locations, but it is too much effort just to set the concurrent limit.