abci: Adapt unsynchronized local client to replicate remote client concurrency

[thanethomson]

After a prior discussion with @cmwaters, we decided that it's probably better in the short-term to offer a less flexible concurrency model than what I introduced in #9660, instead effectively replicating the concurrency model of the remote client (i.e. the socket client) for local clients.

This therefore reverts #9660 and simply creates a new ABCI client creator that does not hold a single lock over all created ABCI client interfaces.

This still, however, extends the E2E tests to randomly select either the local client creator with a global lock, or the new unsynchronized local client creator. This selection only applies to generated E2E tests that use the builtin app.

This is also introduced in such a way that we can still backport it to v0.37 and v0.34 in a non-breaking way.

---

PR checklist

• Tests written/updated, or no tests needed
• CHANGELOG_PENDING.md updated, or no changelog entry needed
• Updated relevant documentation (docs/) and code comments, or no
  documentation updates needed

[thanethomson]

This toy program attempts to demonstrate the problem with the current local client creator (queries can block core consensus functions), as well as how this solution changes the concurrency model: https://go.dev/play/p/umBdTfA5ohF

[sergio-mena]

I like this "unsyc flavor" more as well, since it's somehow consistent with the socket protocol.

[thanethomson]

I've changed the approach here to one where, instead of having a separate manifest option for the builtin proxy mode, we just have an additional ABCIProtocol option of builtin_unsync (in addition to builtin, tcp, etc.).

The first approach I chose unfortunately resulted in all generated testnets whose ABCIProtocol was builtin also happening to have a BuiltinProxyMode of unsync, effectively making all of our nightly E2E tests exclusively run with the unsync option (seems like it was just how the dice rolled unfortunately).

This new approach better balances out the ratio of sync:unsync variations when running the generator:

> grep 'builtin' gen-group0*.toml
gen-group00-0000.toml:abci_protocol = "builtin"
gen-group00-0001.toml:abci_protocol = "builtin_unsync"
gen-group01-0005.toml:abci_protocol = "builtin"
gen-group02-0000.toml:abci_protocol = "builtin"
gen-group02-0001.toml:abci_protocol = "builtin"
gen-group02-0003.toml:abci_protocol = "builtin_unsync"
gen-group02-0004.toml:abci_protocol = "builtin"
gen-group03-0000.toml:abci_protocol = "builtin"
gen-group03-0001.toml:abci_protocol = "builtin"
gen-group04-0000.toml:abci_protocol = "builtin_unsync"
gen-group04-0002.toml:abci_protocol = "builtin_unsync"

I've triggered a manual E2E run here for the newest code in this PR.

[sergio-mena]

Love the new approach