DEVPROD-17023 Split large tasks on required variants based on total test runtime

[sleaux]

The heuristic for determining sub-task counts becomes, in priority:

1. The num_tasks specified in Evergreen YAML for the gen task.
2. If the task runs on a required build variant and surpasses a threshold of total test runtime, allocate additional subtasks based on the total test runtime of the suite, up to a maximum.
3. Use the default 5 subtasks.

[sleaux]

This heuristic based approach proved to be more effective or feasible than the alternatives:

1. Trying to use historic task completion times (wall-clock)

   • This more often than not results in using more subtasks in suites where the completion time is only bottle-necked by individual long running test.

2. Trying to predict task completion time (wall-clock) using historic test runtime and predicting the resmoke job count.

   • Even if obtaining the resmoke job count that will be used were inexpensive enough to do, computing the number of subtasks as total runtime / job count is not more effective than the approach taken in this PR. Knowing the job counts at task generation time is not something we could do easily now.

Required patch build in mongodb-mongo-master and honeycomb statistics showing task completion times are shared outside of this PR.