pkg/bisect: bisector treats a GCE "out of resources" as a valid run

[tytso]

Describe the bug

The bisector appears to try each trial run 10 times. If one or more of the trials fails to boot due to the GCE cell being out of resources, the bisector treats the failure as one of the valid runs. Consider [1] from the syzbot report[1] INFO: task hung in do_truncate (2):

[1] https://syzkaller.appspot.com/text?tag=Log&x=111aa0cf500000
[2] https://syzkaller.appspot.com/bug?id=d38f8eae55e27aaef60b4748bc77ecb712dba4b9

`testing commit 42a44704367cd18d069c9855cb84090ff90ecd86 with gcc (GCC) 8.1.0
kernel signature: bdc5b8bcd5bcd9488a84e3b04c36951f6c6f44451aa0711461f202e963c4a7e5
run #0: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #1: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #2: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #3: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #4: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #5: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #6: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #7: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #8: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #9: OK

git bisect skip 42a44704367cd18d069c9855cb84090ff90ecd86`

First of all, given the burst of gce ZONE RESOURCE POOL EXHAUSTED errors, I'm guessing that the bisector isn't delaying between runs --- or if it is delaying, it's not backing off aggressively enough. Some kind of exponential backup, with the delay not being counted towards the 8 hour limit, would probably be in order here. Secondly the bisector treated a single "OK" run as a "git bisect skip". It would be better if GCE resource exhausted errors aren't treated as one of the ten runs allowed for each commit test, as well. There are other examples in this run where effectively the bisection was decided with only half the number of valid runs, due to the resource pool exhausted.