PF-4325 Introduce retry backoff weight

[victormartins]

This PR introduces the retry_back_off_weight setting for the Queue class.
This setting functions like an extra multiplier to allow us to have meaningful timeout values when the retry_delay is too small.

It also adds a utility script that allows us to instrument the CalculateVisibilityTimeout class and observe its behaviour using various settings. This allows us to confidently choose the configurations we need.
The script lives in utilities/plot_visibility_timeout.rb

Eg:

Given a grace period of 20_000s (15min) and a retry delay of 30ms and running a simulation of 259309s (72h).

queue_retry_back_off_grace = 20000
queue_retry_delay = 30

With

queue_back_off_weight = 1

• 24169 total retries.
• 4169 total retries after grace period.
• 125 max timeout in seconds

Retries	Timeout (s)
0 - 20_000	0.03*
20_000 -20_033	0.03*
20_034 - 20_066	1
20_066 - 20_099	2
20_100 - 20_133	3
...
24169	125

(*Here the calculation is actualy zero but we assume 30ms by default.
So we see the timeout grows very slowly over time resulting in many iterations and after 72h we only have a max timeout of 125s.

With

queue_back_off_weight = 100

• 20415 total retries.
• 415 total retries after grace period.
• 1245 max timeout in seconds

From 0 - 20_000 retries we get a timeout of 0.03

Retries	Timeout (s)	ElapsedTime(s)
20001	3	603
20002	6	609
20003	9	618
20004	12	630
20005	15	645
20006	18	663
20007	21	684
20008	24	708
...
20415	1245	259560

With

queue_back_off_weight = 200

• 20294 total retries.
• 294 total retries after grace period.
• 1764 max timeout in seconds

Retries	Timeout (s)	ElapsedTime(s)
20001	6	606
20002	12	618
20003	18	636
20004	24	660
20005	30	690
20006	36	726
20007	42	768
20008	48	816
...
20294	1764	260790

With

queue_back_off_weight = 300

• 20240 total retries.
• 240 total retries after grace period.
• 2160 max timeout in seconds

Retries	Timeout (s)	ElapsedTime(s)
20001	9	609
20002	18	627
20003	27	654
20004	36	690
20005	45	735
20006	54	789
20007	63	852
20008	72	924
...
20240	2160	260880

[vaughanbrittonsage]

Can we make this retry_back_off_weight to keep the naming convention used for retry_back_off_grace.

[victormartins]

sure, just a sec.

[victormartins]

👍 done