[SPARK-45527][CORE][TESTS][FOLLOW-UP] Reduce the number of test cases in fraction resource calculation

[HyukjinKwon]

What changes were proposed in this pull request?

This PR is a followup of #43494, which reduces the test cases by shuffling taskNum, and taking 5.

Why are the changes needed?

It runts too many test cases, https://github.com/apache/spark/actions/runs/8054862135/job/22000403549 which consumes the limited resources in CI

- SPARK-45527 default rp with task.gpu.amount=1.0 can restrict 1 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.5 can restrict 2 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.3333333333333333 can restrict 3 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.25 can restrict 4 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.2 can restrict 5 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1666666666666666 can restrict 6 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1428571428571428 can restrict 7 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.125 can restrict 8 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1111111111111111 can restrict 9 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1 can restrict 10 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0909090909090909 can restrict 11 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0833333333333333 can restrict 12 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0769230769230769 can restrict 13 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0714285714285714 can restrict 14 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0625 can restrict 16 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0588235294117647 can restrict 17 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0555555555555555 can restrict 18 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0526315789473684 can restrict 19 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.05 can restrict 20 barrier tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=1.0 can restrict 1  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.5 can restrict 2  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.3333333333333333 can restrict 3  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.25 can restrict 4  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.2 can restrict 5  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1666666666666666 can restrict 6  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1428571428571428 can restrict 7  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.125 can restrict 8  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1111111111111111 can restrict 9  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.1 can restrict 10  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0909090909090909 can restrict 11  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0833333333333333 can restrict 12  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0769230769230769 can restrict 13  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0714285714285714 can restrict 14  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0666666666666666 can restrict 15  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0625 can restrict 16  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0588235294117647 can restrict 17  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0555555555555555 can restrict 18  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.0526315789473684 can restrict 19  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=0.05 can restrict 20  tasks run in the same executor
- SPARK-45527 default rp with task.gpu.amount=1.0 can restrict 1 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.5 can restrict 2 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.3333333333333333 can restrict 3 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.25 can restrict 4 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.2 can restrict 5 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.1666666666666666 can restrict 6 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.1428571428571428 can restrict 7 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.125 can restrict 8 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.1111111111111111 can restrict 9 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.1 can restrict 10 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0909090909090909 can restrict 11 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0833333333333333 can restrict 12 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0769230769230769 can restrict 13 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0714285714285714 can restrict 14 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0625 can restrict 16 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0588235294117647 can restrict 17 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0555555555555555 can restrict 18 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.0526315789473684 can restrict 19 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.05 can restrict 20 barrier tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=1.0 can restrict 1  tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.5 can restrict 2  tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.3333333333333333 can restrict 3  tasks run on the different executor
- SPARK-45527 default rp with task.gpu.amount=0.25 can restrict 4  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15  tasks run in the same executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16  tasks run in the same executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17  tasks run in the same executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18  tasks run in the same executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19  tasks run in the same executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20  tasks run in the same executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=1.0 can restrict 1 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.5 can restrict 2 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.3333333333333333 can restrict 3 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.25 can restrict 4 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.2 can restrict 5 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1666666666666666 can restrict 6 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1428571428571428 can restrict 7 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.125 can restrict 8 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1111111111111111 can restrict 9 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1 can restrict 10 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0909090909090909 can restrict 11 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0833333333333333 can restrict 12 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0769230769230769 can restrict 13 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0714285714285714 can restrict 14 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20 barrier tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=1.0 can restrict 1  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.5 can restrict 2  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.3333333333333333 can restrict 3  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.25 can restrict 4  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.2 can restrict 5  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1666666666666666 can restrict 6  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1428571428571428 can restrict 7  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.125 can restrict 8  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1111111111111111 can restrict 9  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1 can restrict 10  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0909090909090909 can restrict 11  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0833333333333333 can restrict 12  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0769230769230769 can restrict 13  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0714285714285714 can restrict 14  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19  tasks run on the different executor
- SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20  tasks run on the different executor
Warning: [766.327s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attribute

Does this PR introduce any user-facing change?

No, test-only.

How was this patch tested?

Manually

Was this patch authored or co-authored using generative AI tooling?

No.

[amaliujia]

Just wondering why we take 5 not just 1 to even cut down the number of tests to run?

[HyukjinKwon]

no specific reason to pick 5. Just for testing enough cases of fraction resource calculation

[dongjoon-hyun]

+1, LGTM. A nice solution! :)

[HyukjinKwon]

Merged to master.

[HyukjinKwon]

I manually ran the tests, and linter to make sure.

[dongjoon-hyun]

Ur, this seems to be a partial fix. Could you reduce these together, @HyukjinKwon ?

 (1 to 20).foreach { taskNum =>
      val gpuTaskAmount = ResourceAmountUtils.toFractionalResource(ONE_ENTIRE_RESOURCE / taskNum)
      test(s"SPARK-45527 TaskResourceProfile with task.gpu.amount=${gpuTaskAmount} can " +

[dongjoon-hyun]

Just for the record, after this commit, the following is the result on master branch. It reduced much but we still have more.

[info] TaskSchedulerImplSuite:
[info] - SPARK-32653: Decommissioned host/executor should be considered as inactive (414 milliseconds)
[info] - Scheduler does not always schedule tasks on the same workers (304 milliseconds)
[info] - Scheduler correctly accounts for multiple CPUs per task (23 milliseconds)
[info] - SPARK-18886 - partial offers (isAllFreeResources = false) reset timer before any resources have been rejected (27 milliseconds)
[info] - SPARK-18886 - delay scheduling timer is reset when it accepts all resources offered when isAllFreeResources = true (21 milliseconds)
[info] - SPARK-18886 - task set with no locality requirements should not starve one with them (21 milliseconds)
[info] - SPARK-18886 - partial resource offers (isAllFreeResources = false) reset time if last full resource offer (isAllResources = true) was accepted as well as any following partial resource offers (21 milliseconds)
[info] - SPARK-18886 - partial resource offers (isAllFreeResources = false) do not reset time if any offer was rejected since last full offer was fully accepted (20 milliseconds)
[info] - Scheduler does not crash when tasks are not serializable (22 milliseconds)
[info] - concurrent attempts for the same stage only have one active taskset (20 milliseconds)
[info] - don't schedule more tasks after a taskset is zombie (21 milliseconds)
[info] - if a zombie attempt finishes, continue scheduling tasks for non-zombie attempts (20 milliseconds)
[info] - tasks are not re-scheduled while executor loss reason is pending (24 milliseconds)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[info] - scheduled tasks obey task and stage excludelist (726 milliseconds)
[info] - scheduled tasks obey node and executor excludelists (47 milliseconds)
[info] - abort stage when all executors are excluded and we cannot acquire new executor (37 milliseconds)
[info] - SPARK-22148 abort timer should kick in when task is completely excluded & no new executor can be acquired (30 milliseconds)
[info] - SPARK-22148 try to acquire a new executor when task is unschedulable with 1 executor (30 milliseconds)
[info] - SPARK-22148 abort timer should clear unschedulableTaskSetToExpiryTime for all TaskSets (48 milliseconds)
[info] - SPARK-22148 Ensure we don't abort the taskSet if we haven't been completely excluded (38 milliseconds)
[info] - SPARK-31418 abort timer should kick in when task is completely excluded &allocation manager could not acquire a new executor before the timeout (25 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 0 (42 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 1 (36 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 2 (35 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 3 (38 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 4 (35 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 5 (33 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 6 (43 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 7 (34 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 8 (33 milliseconds)
[info] - Excluded node for entire task set prevents per-task exclusion checks: iteration 9 (31 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 0 (33 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 1 (31 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 2 (29 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 3 (28 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 4 (30 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 5 (29 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 6 (29 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 7 (30 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 8 (40 milliseconds)
[info] - Excluded executor for entire task set prevents per-task exclusion checks: iteration 9 (32 milliseconds)
[info] - abort stage if executor loss results in unschedulability from previously failed tasks (20 milliseconds)
[info] - don't abort if there is an executor available, though it hasn't had scheduled tasks yet (18 milliseconds)
[info] - SPARK-16106 locality levels updated if executor added to existing host (16 milliseconds)
[info] - scheduler checks for executors that can be expired from excludeOnFailure (14 milliseconds)
[info] - if an executor is lost then the state for its running tasks is cleaned up (SPARK-18553) (14 milliseconds)
[info] - if a task finishes with TaskState.LOST its executor is marked as dead (15 milliseconds)
[info] - Locality should be used for bulk offers even with delay scheduling off (15 milliseconds)
[info] - With delay scheduling off, tasks can be run at any locality level immediately (16 milliseconds)
[info] - TaskScheduler should throw IllegalArgumentException when schedulingMode is not supported (17 milliseconds)
[info] - don't schedule for a barrier taskSet if available slots are less than pending tasks (16 milliseconds)
[info] - don't schedule for a barrier taskSet if available slots are less than pending tasks gpus limiting (17 milliseconds)
[info] - schedule tasks for a barrier taskSet if all tasks can be launched together gpus (20 milliseconds)
[info] - schedule tasks for a barrier taskSet if all tasks can be launched together diff ResourceProfile (14 milliseconds)
[info] - schedule tasks for a barrier taskSet if all tasks can be launched together diff ResourceProfile, but not enough gpus (13 milliseconds)
[info] - schedule tasks for a barrier taskSet if all tasks can be launched together (14 milliseconds)
[info] - SPARK-29263: barrier TaskSet can't schedule when higher prio taskset takes the slots (14 milliseconds)
[info] - killAllTaskAttempts shall kill all the running tasks (16 milliseconds)
[info] - mark taskset for a barrier stage as zombie in case a task fails (17 milliseconds)
[info] - Scheduler correctly accounts for GPUs per task (18 milliseconds)
[info] - Scheduler correctly accounts for GPUs per task with fractional amount (18 milliseconds)
[info] - Scheduler works with multiple ResourceProfiles and gpus (16 milliseconds)
[info] - Scheduler works with task resource profiles (17 milliseconds)
[info] - Calculate available tasks slots for task resource profiles (18 milliseconds)
[info] - scheduler should keep the decommission state where host was decommissioned (21 milliseconds)
[info] - test full decommissioning flow (22 milliseconds)
[info] - SPARK-40979: Keep removed executor info due to decommission (20 milliseconds)
[info] - SPARK-24818: test delay scheduling for barrier TaskSetManager (16 milliseconds)
[info] - SPARK-24818: test resource revert of barrier TaskSetManager (14 milliseconds)
[info] - SPARK-37300: TaskSchedulerImpl should ignore task finished event if its task was finished state (19 milliseconds)
[info] - SPARK-39955: executor lost could fail task set if task is running (18 milliseconds)
[info] - SPARK-39955: executor lost should not fail task set if task is launching (14 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run in the same executor (33 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=1.0 can restrict 1 barrier tasks run in the same executor (21 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0769230769230769 can restrict 13 barrier tasks run in the same executor (23 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.1 can restrict 10 barrier tasks run in the same executor (23 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0714285714285714 can restrict 14 barrier tasks run in the same executor (24 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.1428571428571428 can restrict 7  tasks run in the same executor (23 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=1.0 can restrict 1  tasks run in the same executor (24 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0526315789473684 can restrict 19  tasks run in the same executor (26 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0666666666666666 can restrict 15  tasks run in the same executor (27 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0909090909090909 can restrict 11  tasks run in the same executor (25 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.25 can restrict 4 barrier tasks run on the different executor (21 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.2 can restrict 5 barrier tasks run on the different executor (23 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.05 can restrict 20 barrier tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.1666666666666666 can restrict 6 barrier tasks run on the different executor (23 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.1111111111111111 can restrict 9  tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0588235294117647 can restrict 17  tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.25 can restrict 4  tasks run on the different executor (24 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.0909090909090909 can restrict 11  tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 default rp with task.gpu.amount=0.3333333333333333 can restrict 3  tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=1.0 can restrict 1 barrier tasks run in the same executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.5 can restrict 2 barrier tasks run in the same executor (25 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.3333333333333333 can restrict 3 barrier tasks run in the same executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.25 can restrict 4 barrier tasks run in the same executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.2 can restrict 5 barrier tasks run in the same executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1666666666666666 can restrict 6 barrier tasks run in the same executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1428571428571428 can restrict 7 barrier tasks run in the same executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.125 can restrict 8 barrier tasks run in the same executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1111111111111111 can restrict 9 barrier tasks run in the same executor (31 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1 can restrict 10 barrier tasks run in the same executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0909090909090909 can restrict 11 barrier tasks run in the same executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0833333333333333 can restrict 12 barrier tasks run in the same executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0769230769230769 can restrict 13 barrier tasks run in the same executor (31 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0714285714285714 can restrict 14 barrier tasks run in the same executor (31 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run in the same executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16 barrier tasks run in the same executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17 barrier tasks run in the same executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18 barrier tasks run in the same executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19 barrier tasks run in the same executor (31 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20 barrier tasks run in the same executor (35 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=1.0 can restrict 1  tasks run in the same executor (33 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.5 can restrict 2  tasks run in the same executor (34 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.3333333333333333 can restrict 3  tasks run in the same executor (35 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.25 can restrict 4  tasks run in the same executor (33 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.2 can restrict 5  tasks run in the same executor (34 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1666666666666666 can restrict 6  tasks run in the same executor (39 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1428571428571428 can restrict 7  tasks run in the same executor (36 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.125 can restrict 8  tasks run in the same executor (37 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1111111111111111 can restrict 9  tasks run in the same executor (33 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1 can restrict 10  tasks run in the same executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0909090909090909 can restrict 11  tasks run in the same executor (32 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0833333333333333 can restrict 12  tasks run in the same executor (34 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0769230769230769 can restrict 13  tasks run in the same executor (31 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0714285714285714 can restrict 14  tasks run in the same executor (34 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15  tasks run in the same executor (31 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16  tasks run in the same executor (34 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17  tasks run in the same executor (42 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18  tasks run in the same executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19  tasks run in the same executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20  tasks run in the same executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=1.0 can restrict 1 barrier tasks run on the different executor (24 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.5 can restrict 2 barrier tasks run on the different executor (23 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.3333333333333333 can restrict 3 barrier tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.25 can restrict 4 barrier tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.2 can restrict 5 barrier tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1666666666666666 can restrict 6 barrier tasks run on the different executor (30 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1428571428571428 can restrict 7 barrier tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.125 can restrict 8 barrier tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1111111111111111 can restrict 9 barrier tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1 can restrict 10 barrier tasks run on the different executor (36 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0909090909090909 can restrict 11 barrier tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0833333333333333 can restrict 12 barrier tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0769230769230769 can restrict 13 barrier tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0714285714285714 can restrict 14 barrier tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15 barrier tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16 barrier tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17 barrier tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18 barrier tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19 barrier tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20 barrier tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=1.0 can restrict 1  tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.5 can restrict 2  tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.3333333333333333 can restrict 3  tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.25 can restrict 4  tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.2 can restrict 5  tasks run on the different executor (28 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1666666666666666 can restrict 6  tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1428571428571428 can restrict 7  tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.125 can restrict 8  tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1111111111111111 can restrict 9  tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.1 can restrict 10  tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0909090909090909 can restrict 11  tasks run on the different executor (29 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0833333333333333 can restrict 12  tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0769230769230769 can restrict 13  tasks run on the different executor (27 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0714285714285714 can restrict 14  tasks run on the different executor (35 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0666666666666666 can restrict 15  tasks run on the different executor (24 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0625 can restrict 16  tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0588235294117647 can restrict 17  tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0555555555555555 can restrict 18  tasks run on the different executor (25 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.0526315789473684 can restrict 19  tasks run on the different executor (26 milliseconds)
[info] - SPARK-45527 TaskResourceProfile with task.gpu.amount=0.05 can restrict 20  tasks run on the different executor (24 milliseconds)
[info] - SPARK-45527 TaskResourceProfile: the left multiple gpu resources on 1 executor can assign to other taskset (49 milliseconds)
[info] - SPARK-45527 TaskResourceProfile: the left gpu resources on multiple executors can assign to other taskset (151 milliseconds)
[info] - SPARK-45527 TaskResourceProfile: the left multiple gpu resources on 1 executor can't assign to other taskset due to not enough gpu resource (40 milliseconds)
[info] - SPARK-45527 schedule tasks for a barrier taskSet if all tasks can be launched together (14 milliseconds)
[info] Run completed in 1 minute, 19 seconds.
[info] Total number of tests run: 176
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 176, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 90 s (01:30), completed Feb 26, 2024, 6:35:17 PM

[HyukjinKwon]

ahh .. I am sorry for rushing it. I will make a followup.