Introduce Clear Module-readiness Criteria

[aaronc]

In preparation for mainnet, I would like us to take a more conservative approach to module readiness. I see two pieces to address:

• introduce a regend flag for enabling experimental modules which we should do in testnets and devnets (but not the final pre-mainnet testnet)
• tag all new/experimental modules as experimental and only enable them when the regend experimental flag is set. This includes: group, authz, feegrant, ecocredit, data and cosmwasm, basically anything that is not already live on the hub
• define "pre-flight" clear gating criteria for module readiness and only move modules out of experimental once we've checked and re-checked this checklist

I have a number of concerns about module readiness that I think we need to address and I think it's more prudent to not rush things.

I propose to prioritize making modules really production ready in the following order:

1. group
2. ecocredit
3. authz & feegrant
4. data
5. cosmwasm

I would propose the following draft pre-flight checklist and welcome other input:

• good unit and integration test coverage for all state machine logic
• internal API audit (at least 1 person)
  • are Msg and Query methods and types well-named and organized?
  • is everything well documented?
• internal state machine audit (at least 2 people)
  • read through MsgServer code and verify correctness upon visual inspection
  • ensure all state machine code which could be confusing is properly commented
  • make sure state machine logic matches Msg method documentation
  • ensure that all state machine edge cases are covered with tests and that test coverage is sufficient
  • assess potential threats for each method including spam attacks and ensure that threats have been addressed sufficiently. This should be done by writing up threat assessment for each method
  • assess potential risks of any new third party dependencies and decide whether a dependency audit is needed
• internal completeness audit (at least 1 person). Verify that the following are properly implemented, ideally with tests:
  • genesis import and export of all state
  • query services
  • CLI methods
• testnet/devnet testing after above internal audits
  • all Msg methods have been tested especially in light of any potential threats identified
  • genesis import and export has been tested
• nice to have (and needed in some cases if threats could be high): official 3rd party audit

[clevinson]

We should split this out into an issue for each module (w/ the checklist), and maybe a central place for it.

[aaronc]

I would also like an SDK upgrade checklist that we run using RCs in testnets or devnets to vet new SDK releases before launching in production.

Maybe something like:

• test migration of existing state to new state in a testnet or devnet
• review CHANGELOG and test all new or changed features in a testnet or devnet (at least 1 person)
• ensure proper unit, integration and simulation test coverage for new or changed features (at least 1 person)

[amaury1093]

This checklist lgtm, thanks @aaronc.

I would explicitly assign 2 people (or 3 even) on testnet/devnet testing after above internal audits, I personally see this step as the most important one, ideally each person with some variation:

• with or without ledger
• REST or CLI

[aaronc]

@clevinson can you update the issue template to take into account @AmauryM's suggest of 2-3 people for testnet testing?