Rolling out new functionality faster, but reliably

[thanethomson]

High-level, multi-quarter tracking issue for specific work aiming to make the codebase more malleable, while maintaining high standards in terms of QA. This issue will be expanded over time.

The ecosystem benefits from changes to Tendermint relatively slowly. This is partially due to the sheer complexity and entanglement of various parts of the codebase as it has evolved somewhat organically over the years.

• Having clear specifications will allow us to modularize the system better, which we believe will allow us to make changes more easily:
  • Tendermint specification #9321
• Improving our QA will give us greater confidence when making changes
  • QA runner improvements #9580
  • Improve QA infra to test upgrades on e2e/testnets #9937
  • Add CI workflows to give better assurances with every pull request
    • Automatically flag Go API-breaking changes on release branches
    • proto: use a breaking change checker in CI workflow of release branches #8003
    • ci: add Go vulnerability checker to proactively flag vulnerable code #9872
• Better testing for user-facing parts of the system
  • Ensure that all breaking RPC changes are deliberate
• Automation and streamlining of upgrade processes
  • Soft upgrades and version numbers [Tracking Issue]  #5680
  • Add automated upgrade tests to our QA for new major releases

Related:

• Making the system more modular should lend itself to being able to make changes more easily
  • Use case: Consensus engine developers #9878
• Offloading non-critical data will allow us to simplify the node architecture
  • ADR-082: Data Companion Push API #9437