feat(uptime): roll out TaskProducer to EAP producer

[bmckerry]

Followup to #118583

This PR updates the uptime eap_producer to use TaskProducer in tasks.

The reason for this change is that SingletonProducer doesn't guarantee at-least-once delivery when used inside a task. TaskProducer keeps track of any producer futures generated as a part of task execution, and only marks a task as successful if all producer futures succeed.

To ensure a safe rollout, workers will instantiate an instance of the old and new producers, and cut over to the new producers based on a rollout option. This means that task futures won't be properly tracked while the rollout option is <1.0, but since our base state is to just ignore producer futures that's acceptable.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 145685c. Configure here.}

[cursor]

Rollout evaluated per message

Medium Severity

in_random_rollout is invoked inside the loop over trace_items, so each EAP message gets an independent rollout draw. For checks with multiple redirect trace items, one call to produce_eap_uptime_result can mix TaskProducer and SingletonProducer paths, so a task may finish before untracked singleton produces complete.

 

^{Reviewed by Cursor Bugbot for commit 145685c. Configure here.}

[markstory]

It might be better if only one rollout check was done for all of the items. It will be fewer options checks and you will avoid spreading messages across the two producer instances.