Removed amino folder #592

greg-szabo · 2020-09-30T02:53:37Z

Removed all amino_types structs and implemented DomainType for the relevant Tendermint structs.

Referenced an issue explaining the need for the change
Updated all relevant documentation in docs
Updated all code comments where relevant
Wrote tests
Updated CHANGELOG.md

This commit fixes the header hashing mechanism to match that of the current Go version. It also adds a test to ensure the happy path works. Signed-off-by: Thane Thomson <connect@thanethomson.com>

Demonstrates that the previous commit (fixing the header hash) fixes #600 when executed against a running Tendermint node. Signed-off-by: Thane Thomson <connect@thanethomson.com>

Signed-off-by: Thane Thomson <connect@thanethomson.com>

Prost seems to encode an empty raw protobuf BlockId as an empty vector of bytes, whereas the Go encoder encodes the same object to [18, 0]. This commit adds a workaround to demonstrate that this is the case when testing against Tendermint v0.34 (otherwise header hashing would fail). Signed-off-by: Thane Thomson <connect@thanethomson.com>

It seems as though the previous commit data was generated by an older version of Tendermint. The new commit fixtures here are generated by Tendermint 0.34.0-99aea7b0 running in a Docker container on my local machine. Signed-off-by: Thane Thomson <connect@thanethomson.com>

Signed-off-by: Thane Thomson <connect@thanethomson.com>

… greg/aminoff3

greg-szabo · 2020-10-02T03:02:49Z

Unfortunately one of the model-based tests (single_step_tests) in light-client started acting up since a merge from master: the commit_hash is invalid. I think those commit hashes might have been created with the old tendermint source code before Thane's fixes.

CHANGELOG.md

tendermint/Cargo.toml

codecov-commenter · 2020-10-05T03:12:00Z

Codecov Report

Merging #592 into master will increase coverage by 1.0%.
The diff coverage is 68.5%.

@@           Coverage Diff            @@
##           master    #592     +/-   ##
========================================
+ Coverage    41.3%   42.4%   +1.0%     
========================================
  Files         163     166      +3     
  Lines       11311   11220     -91     
  Branches     2559    2527     -32     
========================================
+ Hits         4682    4763     +81     
+ Misses       6258    6081    -177     
- Partials      371     376      +5

Impacted Files	Coverage Δ
light-client/src/components/io.rs	`0.0% <ø> (ø)`
light-client/src/components/scheduler.rs	`0.0% <0.0%> (ø)`
light-client/tests/integration.rs	`0.0% <0.0%> (ø)`
light-node/src/commands/initialize.rs	`0.0% <0.0%> (ø)`
rpc/src/client/transport/http.rs	`0.0% <ø> (ø)`
tendermint/src/block.rs	`32.5% <ø> (ø)`
tendermint/src/chain.rs	`0.0% <ø> (ø)`
tendermint/src/error.rs	`75.0% <ø> (ø)`
tendermint/tests/config.rs	`99.3% <ø> (ø)`
tendermint/tests/integration.rs	`0.0% <0.0%> (ø)`
... and 54 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ec241f...665fe72. Read the comment docs.

tomtau · 2020-10-05T05:35:06Z

tendermint/src/amino_types/signature.rs

-pub trait SignableMsg {
-    /// Sign this message as bytes
-    fn sign_bytes<B: BufMut>(
-        &self,
-        chain_id: chain::Id,
-        sign_bytes: &mut B,
-    ) -> Result<bool, EncodeError>;
-
-    /// Set the Ed25519 signature on the underlying message
-    fn set_signature(&mut self, sig: &ed25519::Signature);
-    fn validate(&self) -> Result<(), validate::Error>;
-    fn consensus_state(&self) -> Option<consensus::State>;
-    fn height(&self) -> Option<i64>;
-    fn msg_type(&self) -> Option<SignedMsgType>;
-}


Will there be an equivalent of this trait? Currently, some of the types have to_signable_bytes, so the trait implementations can just use that.
In tmkms, it's used for over different requests: https://github.com/iqlusioninc/tmkms/blob/develop/src/session.rs#L171

Or is it expected that the equivalent of this trait would be moved to tmkms?

I'll be honest, I didn't see the need for a trait here. (Your link shows that this might be incorrect, but I didn't find that specific part of the KMS code before.)
Since the trait is only used in two types: Vote and Proposal, the trait implementation is a lot of extra code for very little benefit. (Unfortunately, traits are not interfaces.)

I'm going to go through that code to see what the expectations are. One option I would like to investigate is a message type enum that includes all types of messages. In that case the code can be replaced with a match message {} and call the functions in there. Since we're talking about two types of structs, this is not that overbearing. (If the number of structs can explode, then maybe the trait is a good call.)

If this doesn't pan out, we can always reintroduce the trait.

Ah, I think I found it: https://github.com/iqlusioninc/tmkms/blob/3e3ae75fee35d1d81258ed61c286b2efb240be40/src/session.rs#L113

You can see that when a request is identified as a Vote or a Proposal, it calls the generic .sign() function to get the signable bytes. But if we already identified the structs, there's no need to generalize over them any more.

Instead of

Request::SignProposal(req) => self.sign(req)?,

it could be something like:

Request::SignProposal(req) => SignProposalRequest::try_from(req)?.to_signable_bytes(),

where the try_from ensures that the incoming data is valid.

This removes both the trait and the generic sign function and we got a clearer understanding of what's happening when calling that code.

Does this way of thinking help? Honest question. I feel it's clearer to me but I understand it's a different concept than previous iterations.

Extra note: I went further into the code and I see that the req is an amino_type (or as we call it in the new version, a Raw type). I have a feeling that it could be upgraded into the domaintype even and all checks of incoming data moved into the read function. The benefits of domain types are that there is a strict input check when creating the domain type and it can happen very early in the process. By the time the .to_signable_bytes() function has to be called, you might have a valid type already with no need to check for invalid data.

I'll try to look more into tmkms, but yes, I think using the domain types may make the code simpler there

I'll just say I'm not a fan of this trait either

romac

Wow, what an undertaking! Thank you so much for putting in all this work.

Just went through the Rust code, everything looks good to me, only left some minor comments.

Let's wait until someone can review the changes in the TLA models before approving.

tendermint/src/block/id.rs

tendermint/src/serializers/custom.rs

testgen/src/header.rs

tendermint/src/error.rs

tendermint/src/proposal.rs

tendermint/src/hash.rs

tendermint/src/proposal/sign_proposal.rs

tendermint/src/public_key.rs

tendermint/src/vote.rs

greg-szabo · 2020-10-06T01:59:36Z

Thanks Romain for the great feedback! Your observations are appreciated, I'll fix the code tomorrow. Most of the suboptimal code is because of late-night coding. Some of the suboptimal code is copy-pasted from the earlier amino_types folder, we can and should fix those too.

I was looking for that transpose function, thanks for pointing it out.

rpc/src/lib.rs

romac · 2020-10-06T19:04:44Z

Great stuff, @greg-szabo!

Now it looks like the model-based tests are failing because tendermint-testgen not found. /cc @andrey-kuprianov

tendermint/src/public_key.rs

Co-authored-by: Tomas Tauber <2410580+tomtau@users.noreply.github.com>

tomtau · 2020-10-08T03:08:36Z

tendermint/src/vote/sign_vote.rs

+}
+
+/// SignedVoteResponse is a response containing a signed vote or an error
+#[derive(Clone, PartialEq)]


Suggested change

#[derive(Clone, PartialEq)]

#[derive(Clone, PartialEq, Debug)]

Thanks, exactly the type of feedback I need to make the lib more usable. Added to the next commit.

tomtau · 2020-10-08T03:09:16Z

tendermint/src/proposal/sign_proposal.rs

+}
+
+/// SignedProposalResponse is response containing a signed proposal or an error
+#[derive(Clone, PartialEq)]


Suggested change

#[derive(Clone, PartialEq)]

#[derive(Clone, PartialEq, Debug)]

Thanks, added to the next commit

tomtau · 2020-10-08T07:31:23Z

tendermint/src/proposal.rs

+        CanonicalProposal::new(self.clone(), chain_id).encode_length_delimited_vec()
+    }
+
+    /// Consensus state from this proposal - This doesn't seem to be used anywhere.


this is used in tmkms via that trait: https://github.com/iqlusioninc/tmkms/blob/develop/src/session.rs#L315

but I guess it could be simpler / clearer without it

Yes, I think so too. It's a similar case to the previous. It seems it's generating a State{} struct and matching whatever the incoming request was. We can easily do that with a
consensus_state = State::from(vote); or consensus_state = State::from(proposal); call. Which tells me that implementing this From trait would be a good idea. I'll check the KMS code a bit further, I'm interested how this state is used.

tomtau · 2020-10-08T07:48:02Z

tendermint/src/chain/id.rs


 /// Maximum length of a `chain::Id` name. Matches `MaxChainIDLen` from:
 /// <https://github.com/tendermint/tendermint/blob/develop/types/genesis.go>
 // TODO: update this when `chain::Id` is derived from a digest output
 pub const MAX_LENGTH: usize = 50;

 /// Chain identifier (e.g. 'gaia-9000')
-#[derive(Copy, Clone)]
-pub struct Id([u8; MAX_LENGTH]);
+#[derive(Clone)]


Suggested change

#[derive(Clone)]

#[derive(Copy, Clone)]

This, unfortunately won't work, because the internally implemented String does not have the Copy trait. If you need bytes, you can use as_bytes() or as_str(). (You can read about the issue with the Copy trait and String in the Rust book. It's a size-in-memory thing.)

Alternatively, we can revert back to using [u8] internally but it seemed to me that String is a better representation with the added functions. Up to the team to decide.

ebuchman

Awesome, thanks Greg! This is just an initial review, haven't gone through it all, but let's merge and continue to address anything else in a follow up

ebuchman · 2020-10-05T13:17:00Z

light-client/Cargo.toml

@@ -35,7 +35,6 @@ contracts = "0.4.0"
 crossbeam-channel = "0.4.2"
 derive_more = "0.99.5"
 futures = "0.3.4"
-prost-amino = "0.6.0"


ebuchman · 2020-10-05T13:18:02Z

light-client/src/components/scheduler.rs

-    (low.value() + (high.value() + 1 - low.value()) / 2).into()
+    (low.value() + (high.value() + 1 - low.value()) / 2)
+        .try_into()
+        .unwrap() // Will panic if midpoint is higher than i64::MAX


Can we label these with something consistent so it's easy to find them in the future if we change how we handle this? Maybe should be an XXX or something easy to find...

Good idea, although in this case it's more of a theoretical possibility than a real one.
You can get the biggest number with high.value()==i64::MAX and midpoint is going to be i64::MAX/2 in that case. So this is an unwrap case that can never fail, unless the developer changes the types without concern to the note.

Maybe it's better to remove it, since it's theoretical only.

ebuchman · 2020-10-05T13:24:00Z

rpc/tests/integration.rs

@@ -208,7 +208,7 @@ fn broadcast_tx_commit_null_data() {
 fn commit() {
    let response = endpoint::commit::Response::from_string(&read_json_fixture("commit")).unwrap();
    let header = response.signed_header.header;
-    assert_eq!(header.chain_id.as_ref(), EXAMPLE_CHAIN);
+    assert_eq!(header.chain_id.as_ref(), "dockerchain");


Why not keep EXAMPLE_CHAIN or else why not make a variable for this?

Uh, I'm not sure, it was completely fine as it was. I don't think it was changed by me, it seems it was changed when the commit.json was regenerated. (The previous one had cosmoshub-2 as the chain ID and the current one is dockerchain.) I don't mind it this way because most other assertions use constants to check validity. But we can revert to a const too.

ebuchman · 2020-10-05T13:28:09Z

tendermint/src/block/height.rs

+    type Error = Error;
+
+    fn try_from(value: i64) -> Result<Self, Self::Error> {
+        Ok(Height(value.try_into().map_err(|_| Kind::NegativeHeight)?))


This catches negative numbers?

Yes, it's a kind of Rust inherent magic: Height is defined as Height(u64). Hence the value's try_from() method is going to try to convert the number to u64 from the original i64. This fails if the value is negative (if it's positive it fits without overflow) so the resultant error is because the height was negative. The TryInto trait's error is not compatible with the defined Self::Error, so it has to be converted.

This can fail miserably if/when we redefine the Height newtype. We can change it to u64::try_from(value) which is more explicit and resolves into the same code.

ebuchman · 2020-10-05T13:39:21Z

tendermint/src/block/id.rs

+    }
+}
+
+impl TryFrom<RawCanonicalBlockId> for Id {


Do we actually need this anywhere? I would have thought we only have to marshal from domain types to the canonical type, but we never need to go back from canonical types, since the canonical types are just for signing - maybe I'm missing something?

We don't use it. There are multiple weak reasons why it's there:

Symmetric implementation of all conversions: Domain::TryFrom(Raw) and Raw::From(Domain). This sets a good precedent for the developer and standard expectations for all types and conversions. It's a weak reason because if we know that we don't use one conversion, we could simply omit it from the code. I kept it because this is a library and I wasn't sure who uses it for what.

The above symmetry is a prerequisite for the domaintype trait. It's a weak argument, because the trait is only available with exactly one raw type, and in this case, it's implemented with the raw BlockId and not the CanonicalBlockId. So, there's no real need for the same symmetric implementation because the DomainType trait doesn't really depend on it.

So it's there for good measure. We can remove it, if unnecessary.

ebuchman · 2020-10-05T13:48:06Z

tendermint/src/chain/id.rs


 /// Maximum length of a `chain::Id` name. Matches `MaxChainIDLen` from:
 /// <https://github.com/tendermint/tendermint/blob/develop/types/genesis.go>
 // TODO: update this when `chain::Id` is derived from a digest output
 pub const MAX_LENGTH: usize = 50;

 /// Chain identifier (e.g. 'gaia-9000')
-#[derive(Copy, Clone)]
-pub struct Id([u8; MAX_LENGTH]);


Do we have any idea why this didn't just use a String before?

Recording it for any outsiders:
We discussed this on Slack and the reason might be because of the Copy trait (which was removed when this became a String). The Copy trait is not available for Strings. But Strings are more convenient to use in the code and the Clone trait is good enough for copying the structure where necessary.

ebuchman · 2020-10-05T13:48:59Z

tendermint/src/consensus/state.rs

            step: 0,
            block_id: None,
        };

        let oldest = State {
            height: block::Height::default(),
-            round: 0,
+            round: block::Round::try_from(0).unwrap(),


Is the try_from/unwrap dance really necessary even with a constant?

LOL, we can change that to Round::default() and get on with life. (Maybe in the next PR.)

The longer answer for the try-unwrap dance for constants is this:
We made it a strict requirement that the domain structs we use are safe or we invite abuse of the structs and the possibility of bad code. If constants can add any value without consequences, invalid structs can be created. A Round is a positive value between 0 and i32::MAX. There's no primitive value that can store this in Rust, hence the safe struct that only accepts and emits values between these two. If There's a Round(pub i32) definition, then I can create rounds with negative numbers. If there's a Round(pub u32) definition, I can create rounds with bigger numbers than i32::MAX. Both create problems when we want to encode to the wire and the current expectation is that encoding to the wire is infallible.

Also, we're only going to deal with constants in our tests. In "real code", we get other variables coming in from the wire. The try-unwrap dance is warranted there. (And it's possibly a try-? dance.)

Obviously, after my long-winded explanation last night, Romain woke up this morning and asked me something that made me realize how overcomplicated my explanation is. (I always learn something from Romain.)

So, we implemented the From trait for Height in #628 and it will come for the Round struct too with types that will definitely fit into the struct. In the case of Round, you can do for example: Round::from(0_u16).

ebuchman · 2020-10-05T13:49:50Z

tendermint/src/error.rs

+    NegativeValidatorIndex,
+
+    /// Wrong hash size in part_set_header
+    #[error("Wrong hash: expected Hash size to be 32 bytes")]


Can/should we parametrize this hard coded 32 value or omit it?

This was copied from the amino_types/error.rs . Yeah, eventually this should be more general. Right now all our hashes that can have invalid length, are supposed to be 32 bytes.

(I might create issues tomorrow from these notes.)

Addressed or follow up in future PRs

melekes

🚀

greg-szabo and others added 17 commits September 29, 2020 22:49

Removed amino traits, block, signature, vote

e243c70

tendermint: Fix header hashing

007bb09

This commit fixes the header hashing mechanism to match that of the current Go version. It also adds a test to ensure the happy path works. Signed-off-by: Thane Thomson <connect@thanethomson.com>

light-client: Resolve #600

f7bd911

Demonstrates that the previous commit (fixing the header hash) fixes #600 when executed against a running Tendermint node. Signed-off-by: Thane Thomson <connect@thanethomson.com>

tendermint: Use shorthand form of method call

0b05bdb

Signed-off-by: Thane Thomson <connect@thanethomson.com>

tendermint: Add header hash check in integration test

4d55b61

Signed-off-by: Thane Thomson <connect@thanethomson.com>

Added Proposal domain type with signing - tests fail for now

535e5a1

tendermint: Make last_block_id encoding workaround more explicit

c66fe9d

Signed-off-by: Thane Thomson <connect@thanethomson.com>

tendermint: Fix empty last_block_id encoding issue

428a369

Signed-off-by: Thane Thomson <connect@thanethomson.com>

Proposal and Vote signable bytes functions

631f5c2

Merge remote-tracking branch 'refs/remotes/origin/greg/aminoff3' into…

da4db48

… greg/aminoff3

PubKeyResponse made into PublicKey

b46bc85

Header.Hash() fixes

90ecfc8

Merge branch 'master' into greg/aminoff3

66a3a54

Merge conflict fixes

8600b54

Proposal test fixes

fcef436

greg-szabo marked this pull request as ready for review October 2, 2020 03:02

greg-szabo linked an issue Oct 2, 2020 that may be closed by this pull request

Types for positive i64 #585

Closed

Updated CHANGELOG

8f9f0ba

greg-szabo commented Oct 2, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

greg-szabo commented Oct 2, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

greg-szabo and others added 3 commits October 2, 2020 10:14

Update CHANGELOG.md

696efd3

Update CHANGELOG.md

661e294

regenerate static model-based files

19dddbb

greg-szabo commented Oct 2, 2020

View reviewed changes

tendermint/Cargo.toml Outdated Show resolved Hide resolved

greg-szabo and others added 3 commits October 2, 2020 19:25

Update tendermint/Cargo.toml

9fade2b

Remvoed amino bech32 encoding

5a63cb9

Updated priv_validator.json test with non-amino public key result

3829909

Removed comments referencing amino.

665fe72

tomtau reviewed Oct 5, 2020

View reviewed changes

romac reviewed Oct 5, 2020

View reviewed changes

ebuchman mentioned this pull request Oct 6, 2020

Test against Stargate/Tendermint 0.34 iqlusioninc/tmkms#144

Closed

thanethomson previously requested changes Oct 6, 2020

View reviewed changes

rpc/src/lib.rs Show resolved Hide resolved

greg-szabo added 3 commits October 6, 2020 12:09

Review fixes #1

f5b1202

Review fixes #2

96b6f7f

Review fixes #3

8cc3d23

tomtau reviewed Oct 7, 2020

View reviewed changes

tendermint/src/public_key.rs Outdated Show resolved Hide resolved

romac mentioned this pull request Oct 7, 2020

light-client: Verification failure #600

Closed

greg-szabo and others added 4 commits October 7, 2020 12:00

Updated tests using testgen (#615)

ff3cb43

Update tendermint/src/public_key.rs

2557cf4

Co-authored-by: Tomas Tauber <2410580+tomtau@users.noreply.github.com>

Merge branch 'master' into greg/aminoff3

54be939

fmt fix

1d84575

tomtau reviewed Oct 8, 2020

View reviewed changes

greg-szabo and others added 2 commits October 8, 2020 15:38

Fixes number four, additional derive traits

1e1255a

Merge branch 'master' into greg/aminoff3

e2d3773

ebuchman approved these changes Oct 8, 2020

View reviewed changes

ebuchman merged commit 96cd36c into master Oct 8, 2020

ebuchman deleted the greg/aminoff3 branch October 8, 2020 22:26

melekes reviewed Oct 12, 2020

View reviewed changes

	#[derive(Clone, PartialEq)]
	#[derive(Clone, PartialEq, Debug)]

Removed amino folder #592

Removed amino folder #592

Conversation

greg-szabo commented Sep 30, 2020 • edited by romac Loading

greg-szabo commented Oct 2, 2020

codecov-commenter commented Oct 5, 2020

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greg-szabo Oct 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romac left a comment

Choose a reason for hiding this comment

greg-szabo commented Oct 6, 2020

romac commented Oct 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greg-szabo Oct 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebuchman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

melekes left a comment

Choose a reason for hiding this comment

greg-szabo commented Sep 30, 2020 •

edited by romac

Loading

greg-szabo Oct 5, 2020 •

edited

Loading

greg-szabo Oct 8, 2020 •

edited

Loading