Efficient Consensus State Iteration #125

AdityaSripal · 2021-04-15T23:02:53Z

Description

This approach implements a backward compatible fix for efficient consensus state iteration without changing the host path, which is what the counterparty chain expects as the consensus state key.

Prune Consensus States
~~Enforce monotonic time~~ (will address in subsequent PR)

closes: #XXXX

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer
Review Codecov Report in the comment section below once CI passes

modules/light-clients/07-tendermint/types/update.go

AdityaSripal · 2021-04-16T19:22:09Z

modules/light-clients/07-tendermint/types/update.go

+		}
+		return true
+	}
+	IterateConsensusStateAscending(clientStore, pruneCb)


This will fetch earliest consensus state, we can implement pruning now without breaking changes.

Can similarly test for monotonic time as well

colin-axner

I agree with the solution 👍 Clever fix!!

I do think this isn't a sustainable fix and is definitely a hack to support backwards compatibility. I think we should open a followup issue (or better yet, an ADR!) which explains in detail the sustainable fix so if IBC ever becomes non-backwards compatible we can be ready to swap out this solution for a more robust one that supports all light clients

I left various comments on areas for improvement in the code. I'd mostly like more code documentation explaining that this is a backwards compatible hack. I can see it being very confusing trying to understand why the code is constructed how it is and I think it is good to keep this code isolated so we can easily replace it when we are ready. The more it becomes entangled with existing logic, the harder it will be to replace. So we should have in code comments reminding us to keep it isolated

modules/light-clients/07-tendermint/types/consensus_state.go

modules/light-clients/07-tendermint/types/update.go

modules/light-clients/07-tendermint/types/store.go

modules/light-clients/07-tendermint/types/store_test.go

modules/light-clients/07-tendermint/types/update.go

AdityaSripal · 2021-04-21T20:58:35Z

This PR implements (minimal) consensus state pruning. Future PRs will address monotonic time as well as more complete pruning solution

codecov-commenter · 2021-04-21T21:04:34Z

Codecov Report

Merging #125 (8b9dff1) into main (4e145d8) will increase coverage by 0.04%.
The diff coverage is 81.52%.

@@            Coverage Diff             @@
##             main     #125      +/-   ##
==========================================
+ Coverage   65.22%   65.26%   +0.04%     
==========================================
  Files         127      127              
  Lines        8413     8496      +83     
==========================================
+ Hits         5487     5545      +58     
- Misses       2561     2574      +13     
- Partials      365      377      +12

Impacted Files	Coverage Δ
...odules/light-clients/07-tendermint/types/update.go	`77.65% <73.68%> (-1.01%)`	⬇️
modules/light-clients/07-tendermint/types/store.go	`83.83% <81.25%> (-4.74%)`	⬇️
...light-clients/06-solomachine/types/client_state.go	`66.33% <100.00%> (-4.03%)`	⬇️

colin-axner

LGTM, left some suggestions and nits

modules/light-clients/07-tendermint/types/store.go

colin-axner · 2021-04-22T11:17:40Z

modules/light-clients/07-tendermint/types/store.go

+	csKey := iterator.Value()
+	bz := clientStore.Get(csKey)
+	if bz == nil {
+		return nil, false
+	}
+
+	consensusStateI, err := clienttypes.UnmarshalConsensusState(cdc, bz)
+	if err != nil {
+		return nil, false
+	}
+
+	consensusState, ok := consensusStateI.(*ConsensusState)
+	if !ok {
+		return nil, false
+	}


these errors are also missing codecov, but I believe de duplicating logic from the existing GetConsensusState function could allow reusal of that test to ensure codecov

colin-axner · 2021-04-22T11:20:04Z

modules/light-clients/07-tendermint/types/update.go

+		consState, err := GetConsensusState(clientStore, cdc, height)
+		// this error should never occur
+		if err != nil {
+			pruneError = err


I think maybe we should wrap this error with some message that says "please open an issue". I wouldn't want a relayer to get this issue trying to debug it since really there'd need to be something really wrong

Hmm should I just log and ignore the issue. If this is caught in wild, it would suck that no updates can happen because clientstore can't prune properly. It should still just revert to old updating logic, while we implement a fix since this is a nice-to-have and not critical for IBC functioning

I'm ok with this. How do you plan to log the error?

I could have used logger in context.

I decided against this because logging instead of returning error would make it easy for this to be missed in unit tests. I'd rather just ensure our test coverage makes us confident about this, rather than implementing with mistakes and not catching them until deployment

Sounds like to correct decision to me. There is something severely wrong if this error occurs

modules/light-clients/07-tendermint/types/update_test.go

… into aditya/efficient-consensus

cwgoes · 2021-04-23T10:08:32Z

This seems safe, but the extra layer of indirection is a bit wonky, I think we should try to implement a proper (backwards-incompatible) fix sometime in the future. Maybe it's worth keeping a list of "pending fixes" like this so we don't accumulate too technical debt and then lose track of what was necessary vs. what was merely done to avoid breaking backwards compatibility.

colin-axner · 2021-04-23T11:09:38Z

This seems safe, but the extra layer of indirection is a bit wonky, I think we should try to implement a proper (backwards-incompatible) fix sometime in the future. Maybe it's worth keeping a list of "pending fixes" like this so we don't accumulate too technical debt and then lose track of what was necessary vs. what was merely done to avoid breaking backwards compatibility.

I very much agree. I've added a ibcv2 label as a start to keep track of issues that should be tackled when we are ready to do a backwards-incompatible fix

modules/light-clients/07-tendermint/types/update_test.go

modules/light-clients/07-tendermint/types/store.go

modules/light-clients/07-tendermint/types/update_test.go

lgtm-com · 2021-04-23T16:58:30Z

This pull request introduces 1 alert when merging 30758d0 into db6f316 - view on LGTM.com

new alerts:

1 for Useless assignment to local variable

colin-axner

Great work!! ACK pending CI fix and changelog entry

modules/light-clients/07-tendermint/types/update_test.go

modules/light-clients/07-tendermint/types/update.go

Track gas in queries

* fix: remove agoric-solo; ag-nchainz is in agoric-sdk More fixes for the general nchainz script and agoric.json. * refactor: remove metaprogramming from nchainz; use a config file * feat: factor nchainz run command into chains and clients

…#125) Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.4.0 to 0.4.1. - [Release notes](https://github.com/multiformats/go-multiaddr/releases) - [Commits](multiformats/go-multiaddr@v0.4.0...v0.4.1) --- updated-dependencies: - dependency-name: github.com/multiformats/go-multiaddr dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

start with efficient consensus state lookup

aca247f

AdityaSripal requested review from colin-axner and fedekunze as code owners April 15, 2021 23:02

AdityaSripal marked this pull request as draft April 15, 2021 23:03

writeup pruning logic

24e870c

AdityaSripal commented Apr 16, 2021

View reviewed changes

colin-axner reviewed Apr 19, 2021

View reviewed changes

AdityaSripal added 2 commits April 21, 2021 16:27

fix tests

5600cb1

add documentation

3f5fe15

AdityaSripal requested a review from cwgoes April 21, 2021 20:54

AdityaSripal marked this pull request as ready for review April 21, 2021 20:54

Merge branch 'main' into aditya/efficient-consensus

8b9dff1

colin-axner reviewed Apr 22, 2021

View reviewed changes

This was referenced Apr 22, 2021

Support Efficient Iteration of ConsensusStates in Reverse Chronological Order #64

Open

Prune Expired Tendermint Consensus States #52

Closed

colin-axner added this to the 1.0.0 milestone Apr 22, 2021

AdityaSripal added 2 commits April 22, 2021 18:43

improve byte efficiency

b5e29f3

Merge branch 'aditya/efficient-consensus' of github.com:cosmos/ibc-go…

97e0b30

… into aditya/efficient-consensus

fix tests

ce002c4

colin-axner reviewed Apr 23, 2021

View reviewed changes

modules/light-clients/07-tendermint/types/update_test.go Outdated Show resolved Hide resolved

colin-axner reviewed Apr 23, 2021

View reviewed changes

modules/light-clients/07-tendermint/types/update_test.go Outdated Show resolved Hide resolved

AdityaSripal added 2 commits April 23, 2021 12:17

actually fix tests and bug

317bb64

deduplicate

30758d0

colin-axner reviewed Apr 23, 2021

View reviewed changes

modules/light-clients/07-tendermint/types/store.go Show resolved Hide resolved

colin-axner reviewed Apr 23, 2021

View reviewed changes

modules/light-clients/07-tendermint/types/update_test.go Show resolved Hide resolved

colin-axner approved these changes Apr 23, 2021

View reviewed changes

AdityaSripal added 2 commits April 23, 2021 15:33

fix return

c0f9513

edit changelog

fb33637

AdityaSripal added the automerge label Apr 23, 2021

colin-axner enabled auto-merge (squash) April 23, 2021 19:42

colin-axner merged commit 239301e into main Apr 23, 2021

colin-axner deleted the aditya/efficient-consensus branch April 23, 2021 19:46

colin-axner reviewed Apr 26, 2021

View reviewed changes

modules/light-clients/07-tendermint/types/update.go Show resolved Hide resolved

AdityaSripal mentioned this pull request Apr 28, 2021

Technical Debt Tracker #145

Closed

6 tasks

faddat referenced this pull request in notional-labs/ibc-go Feb 23, 2022

Merge pull request #125 from CosmWasm/track-gas-in-queries

c5dc55f

Track gas in queries

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Efficient Consensus State Iteration #125

Efficient Consensus State Iteration #125

AdityaSripal commented Apr 15, 2021 •

edited

AdityaSripal Apr 16, 2021

colin-axner left a comment •

edited

AdityaSripal commented Apr 21, 2021

codecov-commenter commented Apr 21, 2021

colin-axner left a comment

colin-axner Apr 22, 2021

colin-axner Apr 22, 2021

AdityaSripal Apr 23, 2021

colin-axner Apr 23, 2021

AdityaSripal Apr 23, 2021

colin-axner Apr 23, 2021

cwgoes commented Apr 23, 2021

colin-axner commented Apr 23, 2021

lgtm-com bot commented Apr 23, 2021

colin-axner left a comment •

edited

Efficient Consensus State Iteration #125

Efficient Consensus State Iteration #125

Conversation

AdityaSripal commented Apr 15, 2021 • edited

Description

AdityaSripal Apr 16, 2021

Choose a reason for hiding this comment

colin-axner left a comment • edited

Choose a reason for hiding this comment

AdityaSripal commented Apr 21, 2021

codecov-commenter commented Apr 21, 2021

Codecov Report

colin-axner left a comment

Choose a reason for hiding this comment

colin-axner Apr 22, 2021

Choose a reason for hiding this comment

colin-axner Apr 22, 2021

Choose a reason for hiding this comment

AdityaSripal Apr 23, 2021

Choose a reason for hiding this comment

colin-axner Apr 23, 2021

Choose a reason for hiding this comment

AdityaSripal Apr 23, 2021

Choose a reason for hiding this comment

colin-axner Apr 23, 2021

Choose a reason for hiding this comment

cwgoes commented Apr 23, 2021

colin-axner commented Apr 23, 2021

lgtm-com bot commented Apr 23, 2021

colin-axner left a comment • edited

Choose a reason for hiding this comment

AdityaSripal commented Apr 15, 2021 •

edited

colin-axner left a comment •

edited

colin-axner left a comment •

edited