PBTS: experimental values to be evaluated in QA experiments

[cason]

Overview

To define good default synchronous parameters for PBTS, collecting some relevant metrics is essential.

Original issue: tendermint/tendermint#7202

Currently planned test cases

1. QA/PBTS: Run 200-nodes test with v0.38.x's saturation point #2460 [Auxiliary run - not part of regular QA report]

• Run on v1
• PBTS enable height: 1
• Run QA 200-node TC, but only with v0.38.x saturation point (r=200, c=2)
• Run the experiment (90 secs each) 4-5 times
  • or just run the experiment for 3-4 mins
• Using latency emulation
• output: PBTS team can look at ProposalTimestampDifference to set default values

2. QA/v1: Run 200-nodes test for final report without latency emulation #2461 [Part of regular QA report]

• Run on v1
• PBTS enable height: 1
• Running saturation discovery
• Duration ~40 mins
• NOT Using latency emulation
• output: QA report, 200-node test, with saturation point report section, and metrics report section (for sat point)
• output: compare metrics between (v0.38.x, bft time) and (v1, PBTS)

3. QA/v1: Run 200-nodes test for final report with latency emulation #2513 [Part of regular QA report]

• Run on v1
• PBTS enable height: 1
• Running saturation discovery (mainly because with latencies the saturation point is likely to be different)
• Duration ~40 mins
• Using latency emulation
• output: baseline for future releases (200-node without latency emulation is now deprecated)
• output: compare metrics between (v1, no-lat-emulation) and (v1, lat-emulation)

4. New test case: X nodes [PBTS specific - not part of the regular QA report]

• Run on v1
• PBTS enable height: 1
• Duration ~Y mins
• Using latency emulation
• No Tx load
• clock_skew enable on: close to 1/3 of voting power
• clock_skew value: +5 seconds (i.e. 5 s in the future)
• output: average (or histogram of) number of rounds to decide, and block latency
• This test case is a DRAFT:
  • Play around with e2e on docker (Daniel volunteered 😄)

[cason]

Some comments regarding the existing metrics:

• QuorumPrevoteDelay and FullPrevoteDelay: is anyone using? What is the value of the data produced by it?
• ProposalTimestampDifference: it is current enabled only when PBTS is enabled.
  • I wonder why not to enable it all the times, as it can provide useful data for chains that are considering to switch to PBTS.
  • I don't see much point on creating two data sets, for timely and for untimely proposals. The bounds for timely proposals are known, determined by consensus params. Moreover, considering the previous item, what value we put when PBTS is disabled?

[sergio-mena]

@cason Regarding your comments to ProposalTimestampDifference. I fully agree with both

[cason]

Ok, I will fix that in #2321

[cason]

This issue mixes experiments to be performed with metrics to be implemented/reviewed.

Should we break the concerns into different issues?

[cason]

Metrics for QA experiments for PBTS were updated by #2479.

[cason]

This issue mixes experiments to be performed with metrics to be implemented/reviewed.

Should we break the concerns into different issues?

Created #2480 to track in production metrics.

[cason]

Are we running scenario 4., namely experiments with clock skew?

[sergio-mena]

@hvanz and I experimented with (a somewhat similar version of) scenario 4, when we were troubleshooting the several problems in our e2e nightlies. In particular, in one testnet where the clock-skewed validators amounted to more that 1/3 of the voting power. When troubleshooting those runs, we could see that the adaptive mechanism we put in place (#2432) is doing its job properly.

I think we can close this issue, as all the other 3 cases are being tracked by @hvanz in the Q1 tracking issue.

[cason]

Ok, closing this issue as experiment 4. was performed and it worked. We didn't have the goal of publishing its results, as it was a proof of concept.