num-flaky-test-attempts reset to 0 and renamed to flaky-test-attempts

[rpappalax]

I'm setting a value of 3 for num-flaky-test-attempts in our config file:
https://github.com/mozilla-mobile/fenix/blob/disable-findinpage-test/automation/taskcluster/androidTest/flank-x86.yml#L26
However when we run our tests, it looks like this param is not being handled properly as the config is resetting to the old key name and the value is being reset from 3 to 0 flaky-test-attempts: 0

[task 2019-09-26T21:15:09.268Z] AndroidArgs
[task 2019-09-26T21:15:09.269Z]     gcloud:
[task 2019-09-26T21:15:09.270Z]       results-bucket: fenix_test_artifacts
[task 2019-09-26T21:15:09.271Z]       results-dir: null
[task 2019-09-26T21:15:09.272Z]       record-video: true
[task 2019-09-26T21:15:09.273Z]       timeout: 30m
[task 2019-09-26T21:15:09.274Z]       async: false
[task 2019-09-26T21:15:09.275Z]       results-history-name: null
[task 2019-09-26T21:15:09.276Z]       # Android gcloud
[task 2019-09-26T21:15:09.277Z]       app: /builds/worker/checkouts/src/app/build/outputs/apk/geckoNightly/debug/app-geckoNightly-x86-debug.apk
[task 2019-09-26T21:15:09.277Z]       test: /builds/worker/checkouts/src/app/build/outputs/apk/androidTest/geckoNightly/debug/app-geckoNightly-debug-androidTest.apk
[task 2019-09-26T21:15:09.277Z]       auto-google-login: true
[task 2019-09-26T21:15:09.277Z]       use-orchestrator: true
[task 2019-09-26T21:15:09.277Z]       directories-to-pull:
[task 2019-09-26T21:15:09.278Z]         - /sdcard/screenshots
[task 2019-09-26T21:15:09.278Z]       performance-metrics: true
[task 2019-09-26T21:15:09.278Z]       test-runner-class: null
[task 2019-09-26T21:15:09.278Z]       test-targets:
[task 2019-09-26T21:15:09.278Z] 
[task 2019-09-26T21:15:09.278Z]       device:
[task 2019-09-26T21:15:09.278Z]         - model: Nexus9
[task 2019-09-26T21:15:09.278Z]           version: 25
[task 2019-09-26T21:15:09.279Z]           locale: en
[task 2019-09-26T21:15:09.279Z]           orientation: portrait
[task 2019-09-26T21:15:09.279Z]         - model: Pixel2
[task 2019-09-26T21:15:09.279Z]           version: 28
[task 2019-09-26T21:15:09.279Z]           locale: en
[task 2019-09-26T21:15:09.279Z]           orientation: portrait
[task 2019-09-26T21:15:09.279Z]       flaky-test-attempts: 0
[task 2019-09-26T21:15:09.279Z] 
[task 2019-09-26T21:15:09.279Z]     flank:
[task 2019-09-26T21:15:09.279Z]       max-test-shards: -1
[task 2019-09-26T21:15:09.279Z]       shard-time: -1
[task 2019-09-26T21:15:09.280Z]       repeat-tests: 1
[task 2019-09-26T21:15:09.280Z]       smart-flank-gcs-path: 
[task 2019-09-26T21:15:09.280Z]       smart-flank-disable-upload: false
[task 2019-09-26T21:15:09.280Z]       files-to-download:
[task 2019-09-26T21:15:09.280Z] 
[task 2019-09-26T21:15:09.280Z]       test-targets-always-run:
[task 2019-09-26T21:15:09.280Z] 
[task 2019-09-26T21:15:09.280Z]       disable-sharding: false
[task 2019-09-26T21:15:09.280Z]       project: moz-fenix
[task 2019-09-26T21:15:09.280Z]       local-result-dir: results
[task 2019-09-26T21:15:09.280Z]       # Android Flank Yml
[task 2019-09-26T21:15:09.280Z]       additional-app-test-apks:
[task 2019-09-26T21:15:09.280Z] 

see:
#595

[bootstraponline]

Thanks for the report! I'll try to look into this once I have some time.

[rpappalax]

Thanks for the report! I'll try to look into this once I have some time.

Howdy @bootstraponline, is there any news on when a fix for this might be available? Now that we've switched to x86 devices we're seeing quite a few more flaky tests. Many are just intermittently flaky and this option would be extremely helpful.

[bootstraponline]

hey, sorry for the lack of response.

I've transitioned to a new role that has been consuming all my time. I will try and get this prioritized on the weekend.

[bootstraponline]

Sorry for the delay.

1. I have fixed the Flank CI job so new releases can be published
2. This specific issue should be fixed in Flank v8

https://github.com/TestArmada/flank/releases/tag/v8.0.0

If you're still seeing the issue in Flank v8 let me know and I'll fix it quickly.

[bootstraponline]

I confirmed this is working as expected in Flank v8.0.1.

• https://github.com/TestArmada/flank/releases/tag/v8.0.1

[rpappalax]

works great!

[bootstraponline]

Awesome!