[Testing] Remove non-determinism from E2E Consensus Tests

[Olshansk]

Objective

Avoid using timers or other non-deterministic approaches in E2E tests.

Origin Document

The following comment in #874 and as it relates to https://github.com/pokt-network/pocket/pull/948/files#diff-01dec4121ae8acb7a1f4bb72a6c2104827d2c2d2197eb5f45fa5c032ffba32cdR93:

Goals

• Prevent non-determinism and flakiness from E2E tests
• Create robust tooling to help iterate on E2E tests

Deliverable

• Remove all the developer waits for ... milliseconds in all .feature files
• Introduce alternate mechanisms mechanism to that waiting for a node to reach a certain height include and not limited to:
  • Polling for a specific height (with a max timeout)
  • Polling a health check endpoint (with a max timeout)

Non-goals / Non-deliverables

• Introducing new tests
• Introducing new functionality

General issue deliverables

• Update any relevant local/global README(s)
• Add or update any relevant or supporting mermaid diagrams

Testing Methodology

• All tests: make test_all
• E2E Tests: make test_e2e
• LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md
• k8s LocalNet: verify a k8s LocalNet is still functioning correctly by following the instructions here

---

Creator: @Olshansk
Co-Owners: @0xBigBoss @dylanlott

[Olshansk]

Copying kubectl wait --for=condition=ready pod/full-node-001-pocket-0 here so the idea is not based on this comment: