[Heartbeat] Limit jobs by type via Env Vars

[andrewvc]

Fixes #34290 by allowing users to control job limits via environment vars like SYNTHETICS_LIMIT_{TYPE}, so, to limit the max number of browser jobs one would use: SYNTHETICS_LIMIT_BROWSER=90 ./heartbeat

This is primarily here to allow users to change the default browser limit in the elastic agent, which is impossible to change otherwise, since there's no way to set a good global config via Fleet.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

mage build && env SYNTHETICS_LIMIT_BROWSER=4 SYNTHETICS_LIMIT_HTTP=200 ./heartbeat -e -E cloud.id=$CLOUD_ID -E cloud.auth=$CLOUD_AUTH

Should yield the following logs:

{"log.level":"info","@timestamp":"2023-01-18T16:33:44.124-0600","log.origin":{"file.name":"scheduler/scheduler.go","file.line":79},"message":"limiting to 4 concurrent jobs for 'browser' type","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-18T16:33:44.124-0600","log.origin":{"file.name":"scheduler/scheduler.go","file.line":79},"message":"limiting to 200 concurrent jobs for 'http' type","service.name":"heartbeat","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrewvc? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-05T20:32:29.384+0000

• Duration: 49 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1921
Skipped	25
Total	1946

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[emilioalvap]

LGTM

[paulb-elastic]

Post FF Testing, setting to 3 (browser) and 150 (HTTP), can see these: