Automatically retry failed MIRI runs to work around intermittent failures

[alamb]

Which issue does this PR close?

Closes #879

Rationale for this change

We are seeing intermittent failures while running MIRI checks on CI that are not reproducible locally and whose symptoms are consistent with miri being killed by the OOM killer (for example, perhaps github has over provisioned their runners so that the upper memory limit is not consistent). See here for details

When we manually rerun the MIRI run it often passes

What changes are included in this PR?

Automatically re-run MIRI up to 5 times looking for a clean run

Are there any user-facing changes?

[codecov-commenter]

Codecov Report

Merging #922 (2ca81ed) into master (bb05b00) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #922   +/-   ##
=======================================
  Coverage   82.30%   82.30%           
=======================================
  Files         168      168           
  Lines       48028    48028           
=======================================
+ Hits        39529    39530    +1     
+ Misses       8499     8498    -1     

Impacted Files	Coverage Δ
parquet/src/encodings/encoding.rs	93.71% <0.00%> (+0.19%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb05b00...2ca81ed. Read the comment docs.

[alamb]

First run passed but also doesn't appear that the retry kicked in.
https://github.com/apache/arrow-rs/runs/4116270188?check_suite_focus=true

Running again

[alamb]

https://github.com/apache/arrow-rs/runs/4117071438?check_suite_focus=true also passed, but no retry kicked in. Will try again

[alamb]

Still passing: https://github.com/apache/arrow-rs/runs/4117509943?check_suite_focus=true

[alamb]

I want to see at least one PR where MIRI doesn't pass and is rerun to success prior to saying this is ready to merge

[alamb]

OOh -- here is a run where MIRI crashed the first time but succeeded the second: https://github.com/apache/arrow-rs/runs/4118067989?check_suite_focus=true 🎉

Will keep rerunning and see how it goes

[alamb]

Another pass: https://github.com/apache/arrow-rs/runs/4118608993?check_suite_focus=true

[alamb]

@houqp and @jimexist I think this approach is looking promising (even though it is not intellectually satisfying)

[alamb]

Another pass https://github.com/apache/arrow-rs/runs/4119629441?check_suite_focus=true