accounts/abi: fix event tupleUnpack errors + cosmetic cleaning #15452

robert-zaremba · 2017-11-10T02:36:26Z

The fix commits contain fixes for two bugs in event unpacker:

Not ignoring Indexed arguments when computing data offset
Wrong expected slice/array size
Excessive value extraction from the slice element (.Elem method) - which caused panic in tests.

The tests commit contains tests for all cases. For convenience I've added new dependency for doing assertions in tests (github.com/stretchr/testify).

Last commit contains cosmetic cleanup.

GitCop · 2017-11-10T02:36:27Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 5b4992b
Your commit message body contains a line that is longer than 80 characters

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-11-10T02:43:11Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 5b4992b
Your commit message body contains a line that is longer than 80 characters

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

gballet

Please revert your "cosmetic changes" as they clutter the patch. For the same reason, remove the dependency.
Also, explain exactly what tests panic when calling Elem(). Last, is there an associated issue that describe how wrong the array size is? If not, it wouldn't hurt describing what goes wrong as well.

gballet · 2017-11-10T16:27:11Z

accounts/abi/method.go

-		typ   = value.Type()
+		value    = valueOf.Elem()
+		typ      = value.Type()
+		isStruct = true


It would make more sense to set isStruct to false and only set it to true if value.Kind() is a reflect.Struct. If that switch starts supporting more types, the programmer will have to set isStruct = false on each entry (and might forget)

good point. The only additional type which we could support here is map, and this can be easily handled with enum type, which we will define once (performance, checks...)

gballet · 2017-11-10T16:28:24Z

accounts/abi/method.go

 	)
+	switch value.Kind() {
+	case reflect.Struct:


Another reason to initialize isStruct to true would be to not have an empty case here

gballet · 2017-11-10T16:48:30Z

accounts/abi/abi.go

@@ -85,7 +85,7 @@ func (abi ABI) Unpack(v interface{}, name string, output []byte) (err error) {
 	} else if event, ok := abi.Events[name]; ok {
 		unpack = event
 	} else {
-		return fmt.Errorf("abi: could not locate named method or event.")
+		return fmt.Errorf("abi: could not locate named method or event")


Is that change really necessary? I can see the point (pun intended) of not displaying two dots if that error is subsequently printed as a %v followed by another dot. However, it would be preferable not to add that dot over there, and not clutter the diff with unnecessary entries (more on that later)

This was reported by golinit

gballet · 2017-11-10T16:53:45Z

accounts/abi/event.go

 		marshalledValue, err := toGoType((i+j)*32, input.Type, output)
 		if err != nil {
 			return err
 		}
 		reflectValue := reflect.ValueOf(marshalledValue)

-		switch value.Kind() {
-		case reflect.Struct:
+		if isStruct {


The switch made perfect sense here (especially since you are also merely moving it), as other types can be supported in the future.

as explained below - we should check the type before unpacking the arguments in order to validate the array size once and for all. We can do the switch here, but since we already did it it's easier to reuse what we have learned.

The second switch will be optimized by the compiler, and we want to make adding new types easier. Revert back to it, please.

gballet · 2017-11-10T16:55:51Z

accounts/abi/event.go

 						return err
 					}
 				}
 			}
-		case reflect.Slice, reflect.Array:
-			if value.Len() < i {


You could move the checks from line 68 right here, for a reduced diff and better understanding

The slice / array size should be done once, and at the beginning. Not every time in the loop. Also the error message was wrong, because it could fail before reaching the end (reporting the wrong expected size).

Or you could let the loop unroll with a i++ followed by a continue and make the size check at the end. It would mean that you only scan the argument list once, and then that you would report other errors earlier than an error on the length of the array (which is unlikely to occur). Do let me know what you think of this, I might be missing something out here.

Why you want to do the check at the end? We can do both things in fact: to use switch before the unpacking to verify preconditions, and the use switch here as well.

Furthermore if we will not change the size of slice before setting the value we will make a panic when we will go out of the slice / array.
BTW: the previous behaviour (few weeks ago) was a bit better in this case because it was specific to handle the slice/array use case (the tradeof was to have multiple copies of for loops for each destination value kind case). Also it handled the case where the slice was nil, and was appending the values.

gballet · 2017-11-10T17:00:54Z

accounts/abi/event_test.go

 	"strings"
 	"testing"

 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/crypto"
+	"github.com/stretchr/testify/assert"


You're pulling in two libraries that produce a big diff (and code) size, and you're the only one using it. I suggest that you create a different PR to start using those two libraries, so that the content of this PR is only about tupleUnpack. If the community is interested in using testify, you will get upvotes and we'll then include it. This PR does too many different things and we can't include it whole.

OK, will do that.

Testify is a really convenient library when doing tests. I will extract it from this pull request and do another one then.

gballet · 2017-11-10T17:04:34Z

accounts/abi/method.go

@@ -192,6 +194,8 @@ func (m Method) String() string {
 	return fmt.Sprintf("function %v(%v) %sreturns(%v)", m.Name, strings.Join(inputs, ", "), constant, strings.Join(outputs, ", "))
 }

+// Id returns the canonical representation of the method's signature used by the
+// abi definition to identify method name.


identify THE method name

Updated, thanks.

GitCop · 2017-11-10T18:49:28Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 9cb7455
Commits must be prefixed with the package(s) they modify
Commit: 5a20f16
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

robert-zaremba · 2017-11-10T18:55:54Z

accounts/abi/event_test.go

+			[3]byte{'u', 's', 'd'}},
+		jsonEventPledge,
+		"Can unpack Pledge event into slice",
+	}}


All the Slice based tests were failing because of the Elem() call in tupleUnpack:

Can unpack ERC20 Transfer event into slice

Can unpack Pledge event into slice

robert-zaremba · 2017-11-10T19:01:12Z

@gballet Thanks for review. Please look at my answers above.

I've updated isStruct initialization
in the comment I've pointed out which tests are failing because of Elem() call
removed the testify dependency and updated tests accordingly to that
I kept the cosmetic update because it's not that big and solve most of the linter warnings. It was easier for me to do it while working on this package.

Please confirm everything is OK, and I can squash the commits before merging the PR.

robert-zaremba · 2017-11-10T23:28:22Z

@gballet I've squashed the review changes. Each commit has the description explaining the changes (eg the Elem() call issue). It's much easier to review individual commits if you want to see the logical changes. As requested the testify dependency has been removed (with commit).

gballet

We're almost there. If you go back to switch and also add unit tests to cover your changes to methods.go, we should be good to go.

gballet · 2017-11-13T13:17:54Z

accounts/abi/reflect.go

@@ -85,3 +85,10 @@ func set(dst, src reflect.Value, output Argument) error {
 	}
 	return nil
 }
+
+func requireAssignable(dst, src reflect.Value) error {


Please add a comment to explain what that function does, it's obvious to everyone familiar with this PR but it won't be down the road.

gballet · 2017-11-13T13:29:12Z

accounts/abi/method.go

@@ -24,7 +24,7 @@ import (
 	"github.com/ethereum/go-ethereum/crypto"
 )

-// Callable method given a `Name` and whether the method is a constant.
+// Method represents method ABI entry.


There's some information being lost here, as well as an article missing.

comment should start with the object name. The thing about Name and constant is covered in next lines of the comment.
Updating to Method represents a method ...

gballet · 2017-11-13T13:37:44Z

accounts/abi/event.go

 		marshalledValue, err := toGoType((i+j)*32, input.Type, output)
 		if err != nil {
 			return err
 		}
 		reflectValue := reflect.ValueOf(marshalledValue)

-		switch value.Kind() {
-		case reflect.Struct:
+		if isStruct {


The second switch will be optimized by the compiler, and we want to make adding new types easier. Revert back to it, please.

gballet · 2017-11-14T14:07:05Z

accounts/abi/event.go

 						return err
 					}
 				}
 			}
-		case reflect.Slice, reflect.Array:
-			if value.Len() < i {


Or you could let the loop unroll with a i++ followed by a continue and make the size check at the end. It would mean that you only scan the argument list once, and then that you would report other errors earlier than an error on the length of the array (which is unlikely to occur). Do let me know what you think of this, I might be missing something out here.

gballet · 2017-11-14T17:11:24Z

accounts/abi/method.go

 			v := value.Index(i)
-			if v.Kind() != reflect.Ptr && v.Kind() != reflect.Interface {
-				return fmt.Errorf("abi: cannot unmarshal %v in to %v", v.Type(), reflectValue.Type())
+			if err := requireAssignable(v, reflectValue); err != nil {


You're testing your changes in events.go, but not those in methods.go. The code is duplicated, and therefore in the future it could drift apart. So you need to cover that too.

gballet

Forgot

gballet · 2017-11-14T17:31:39Z

accounts/abi/event_test.go

+		expected interface{}
+		jsonLog  []byte
+		name     string
+	}{{


Forgot to ask in my previous review: are you going to test for errors as well? Like, the array boundary check that you made some changes for?

I can add tests for that (my goal was to quickly fix the bugs in this package which I encountered when using it)

robert-zaremba · 2017-11-23T23:40:59Z

@gballet @karalabe I've updated the repo according to the review:

Added tests for Method slice unpack
researched the use-case for Elem: shortly it is there to be able to create a slice with elements we want to unpack to. Elem requires that we provide a slice of pointers into unpack. In my events test-cases I assumed that we we want to unpack directly into slice values. I brought back the Elem call and updated tests.

robert-zaremba

Done, we are using switch again.

robert-zaremba · 2017-11-14T17:25:51Z

accounts/abi/event.go

 		marshalledValue, err := toGoType((i+j)*32, input.Type, output)
 		if err != nil {
 			return err
 		}
 		reflectValue := reflect.ValueOf(marshalledValue)

-		switch value.Kind() {
-		case reflect.Struct:
+		if isStruct {


robert-zaremba · 2017-11-14T17:27:52Z

accounts/abi/event.go

 						return err
 					}
 				}
 			}
-		case reflect.Slice, reflect.Array:
-			if value.Len() < i {


Why you want to do the check at the end? We can do both things in fact: to use switch before the unpacking to verify preconditions, and the use switch here as well.

robert-zaremba · 2017-11-14T17:31:32Z

accounts/abi/method.go

@@ -24,7 +24,7 @@ import (
 	"github.com/ethereum/go-ethereum/crypto"
 )

-// Callable method given a `Name` and whether the method is a constant.
+// Method represents method ABI entry.


comment should start with the object name. The thing about Name and constant is covered in next lines of the comment.
Updating to Method represents a method ...

robert-zaremba · 2017-11-14T17:35:33Z

accounts/abi/event_test.go

+		expected interface{}
+		jsonLog  []byte
+		name     string
+	}{{


I can add tests for that (my goal was to quickly fix the bugs in this package which I encountered when using it)

robert-zaremba · 2017-11-14T17:54:03Z

accounts/abi/event.go

 						return err
 					}
 				}
 			}
-		case reflect.Slice, reflect.Array:
-			if value.Len() < i {


Furthermore if we will not change the size of slice before setting the value we will make a panic when we will go out of the slice / array.
BTW: the previous behaviour (few weeks ago) was a bit better in this case because it was specific to handle the slice/array use case (the tradeof was to have multiple copies of for loops for each destination value kind case). Also it handled the case where the slice was nil, and was appending the values.

robert-zaremba · 2017-11-23T23:43:25Z

Also, since testify is merged I've updated (only) new tests to take advantage of it, which saved around 2 lines of code for each check :)

robert-zaremba · 2017-11-23T23:50:19Z

Into the cleanning commit I've added one more golint complaint fix: unified Method method receiver name: Receiver names should be consistent across a type's methods.

gballet

We still have this issue with the Travis build, but it doesn't seem to be directly related. Let's not block this PR any longer and merge it.

gballet · 2017-11-24T09:33:55Z

accounts/abi/method.go

@@ -24,7 +24,7 @@ import (
 	"github.com/ethereum/go-ethereum/crypto"
 )

-// Callable method given a `Name` and whether the method is a constant.
+// Method represents a method ABI entry.


Your comment still drops information from the comment it replaced.

Not really. Look at the whole comment. I think it's more clear now. The Name is part of it's public interface, and other part of the comment mentions about constant part.

No it definitely drops critical information regarding the ABI as the field "constant" is a core component of the current ABI. This will eventually be changed to "stateMutability" but for now it is needed, especially if the goal is to reduce breaking changes.

Well, the second line tells:

If the method is `Const` no transaction needs to be created for this

I don't want to insist here. I will update to:

Method represents a callable given a `Name` and whether the method is a constant.

VoR0220 · 2017-12-01T16:22:02Z

accounts/abi/event.go

-	j := 0
-	for i := 0; i < len(e.Inputs); i++ {
-		input := e.Inputs[i]
+	i, j := -1, 0


why is i -1 here? Is there any way to get a comment of some kind clarifying the logic here?

Because we are incrementing at the beginning of the of the loop (if the argument is not Indexed) rather then at the end.

That doesn't seem to clarify things for me. So in the case of an event a(uint b, string indexed c, uint d), how does the -1 help read through b? Again, the logic could use some documentation is all I'm saying.

VoR0220 · 2017-12-01T16:23:01Z

accounts/abi/event_test.go

+      "indexed": false, "name": "currency", "type": "bytes3"
+  }],
+  "name": "Pledge",
+  "type": "event"


Needs a test to take into account static array types.

VoR0220 · 2017-12-01T16:25:09Z

accounts/abi/reflect.go

@@ -85,3 +85,12 @@ func set(dst, src reflect.Value, output Argument) error {
 	}
 	return nil
 }
+
+// requireAssignable verifies that `dest` and `src` types are compatible
+// and it's possible to safely assign `src` value to `dest`.


This does not appear to be what it's actually doing.

Right, there is better way to document it. Howe about:

// requireAssignable verifies that dest type is assignable.

That works better.

changed to:

// requireAssignable assures that `dest` is a pointer and it's not an interface.

VoR0220 · 2017-12-01T16:29:27Z

accounts/abi/method.go

-	for i := 0; i < len(method.Outputs); i++ {
-		toUnpack := method.Outputs[i]
-		if toUnpack.Type.T == ArrayTy {
+	for i, o := range m.Outputs {


can we please use a more descriptive name than o?

This variable is used very often, and doesn't have a big context. But If you want I can change to something else.

VoR0220 · 2017-12-01T16:33:55Z

accounts/abi/event.go

 			continue
 		} else if input.Type.T == ArrayTy {
 			// need to move this up because they read sequentially
 			j += input.Type.Size
 		}
+		i++


Okay, I see the logic now, slightly...but this definitely needs documentation because currently it's a bit confusing as to why it's set up this way.

OK, will add some comments.

VoR0220 · 2017-12-12T14:36:25Z

@dshulyak any issues?

dshulyak · 2017-12-12T14:43:06Z

no issues, i will make separate pr with that bugfix or include it into this one #15618

robert-zaremba · 2017-12-12T18:07:48Z

Great! I'm happy that we are finally moving forward. So

I'm removing last commit from this Pull Request (the static array argument fix). For reference it's available in the fix-static-array-event-argument branch
@dshulyak please add your change to this Pull Request (master branch in my repo) - I've just added you as a collaborator.
Should I remove my static-array test (it will fail without the last commit I've just removed from this PR).

Or please advice if you want to do differently?

dshulyak · 2017-12-13T13:06:49Z

@robert-zaremba i would prefer to merge both PRs separately (my change is this one #15618), doesn't matter for me in which order.
and i think you can remove a test for array tuple unpack, cause it is not related to the original bugfix.

robert-zaremba · 2017-12-13T14:32:42Z

@dshulyak OK. Removed.
Just in case, the ported fix + test is in fix-static-array-event-argument branch.

VoR0220 · 2017-12-13T18:53:11Z

@gballet @holiman @fjl @karalabe Can we get these two PRs merged up?

robert-zaremba · 2017-12-19T09:25:39Z

Ping. Any update here?

Event.tupleUnpack doesn't handle correctly Indexed arguments, hence it can't unpack an event with indexed arguments.

+ The event slice unpacker doesn't correctly extract element from the slice. The indexed arguments are not ignored as they should be (the data offset should not include the indexed arguments). + The `Elem()` call in the slice unpack doesn't work. The Slice related tests fails because of that. + the check in the loop are suboptimal and have been extracted out of the loop. + extracted common code from event and method tupleUnpack

+ adding missing comments + small cleanups which won't significantly change function body. + unify Method receiver name

+ Reworked Method Unpack tests into more readable components + Added Method Unpack into slice test

holiman · 2017-12-21T14:10:34Z

Did you just rebase this ?
I was just doing that locally aswell :)

gballet · 2017-12-21T14:16:51Z

yes I did, there seems to be a merge error though

robert-zaremba · 2017-12-21T14:57:55Z

Let me know if you need some help with rebase or squashe. I didn't squash last commits because they were after the approved review.

gballet · 2017-12-21T15:09:19Z

that should be fine now, if tests pass then we're good to merge

robert-zaremba · 2017-12-22T10:41:29Z

It seams that there is an unrelated dependency build (installation) problem for GETH_ARCH=386, MSYS2_ARCH=i686, MSYS2_BITS=32, MSYSTEM=MINGW32

holiman · 2017-12-22T20:00:52Z

Closing due to #15731

robert-zaremba force-pushed the master branch from 7d246c5 to 2dca219 Compare November 10, 2017 02:43

robert-zaremba force-pushed the master branch from 2dca219 to 4dcada2 Compare November 10, 2017 02:46

gballet requested changes Nov 10, 2017

View reviewed changes

robert-zaremba force-pushed the master branch from 5a20f16 to d6232b1 Compare November 10, 2017 18:51

robert-zaremba commented Nov 10, 2017

View reviewed changes

robert-zaremba force-pushed the master branch from 13d96af to e430273 Compare November 10, 2017 23:26

robert-zaremba mentioned this pull request Nov 12, 2017

vendor: add github.com/stretchr/testify test dependency #15457

Merged

fjl added this to the 1.7.3 milestone Nov 13, 2017

gballet requested changes Nov 14, 2017

View reviewed changes

gballet reviewed Nov 14, 2017

View reviewed changes

karalabe modified the milestones: 1.7.3, 1.8.0 Nov 21, 2017

robert-zaremba closed this Nov 23, 2017

robert-zaremba force-pushed the master branch from e430273 to 35801f9 Compare November 23, 2017 23:34

robert-zaremba reopened this Nov 23, 2017

robert-zaremba commented Nov 23, 2017

View reviewed changes

gballet approved these changes Dec 1, 2017

View reviewed changes

VoR0220 reviewed Dec 1, 2017

View reviewed changes

robert-zaremba force-pushed the master branch from 293f550 to b1eaf2b Compare December 12, 2017 18:13

robert-zaremba force-pushed the master branch from b1eaf2b to a938796 Compare December 13, 2017 14:31

dshulyak mentioned this pull request Dec 19, 2017

accounts/abi: add array length after parsing array #15618

Merged

robert-zaremba added 5 commits December 21, 2017 15:00

accounts/abi: adding event unpacker tests

ebb68bd

accounts/abi: fix event tupleUnpack

6d604a8

Event.tupleUnpack doesn't handle correctly Indexed arguments, hence it can't unpack an event with indexed arguments.

accounts/abi: satisfy most of the linter warnings

f0fc23b

+ adding missing comments + small cleanups which won't significantly change function body. + unify Method receiver name

accounts/abi: add Method Unpack tests

882a32a

+ Reworked Method Unpack tests into more readable components + Added Method Unpack into slice test

gballet force-pushed the master branch from a938796 to d57b6be Compare December 21, 2017 14:08

accounts/abi: add unpack into array test

2aeb255

gballet force-pushed the master branch from d57b6be to 2aeb255 Compare December 21, 2017 15:06

holiman added a commit to holiman/go-ethereum that referenced this pull request Dec 21, 2017

accounts/abi: merging of ethereum#15452

b183404

holiman mentioned this pull request Dec 21, 2017

accounts/abi refactor #15731

Merged

holiman added a commit to holiman/go-ethereum that referenced this pull request Dec 22, 2017

accounts/abi: merging of ethereum#15452 + lookup by id

c095c87

holiman closed this Dec 22, 2017

mariameda pushed a commit to NiluPlatform/go-nilu that referenced this pull request Aug 23, 2018

accounts/abi: merging of ethereum#15452 + lookup by id

62b79d2

hatricker pushed a commit to thundercore/thundercore-localchain that referenced this pull request Oct 8, 2018

accounts/abi: merging of ethereum/go-ethereum#15452 + lookup by id

706cdec

accounts/abi: fix event tupleUnpack errors + cosmetic cleaning #15452

accounts/abi: fix event tupleUnpack errors + cosmetic cleaning #15452

Conversation

robert-zaremba commented Nov 10, 2017

GitCop commented Nov 10, 2017

GitCop commented Nov 10, 2017

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitCop commented Nov 10, 2017

Choose a reason for hiding this comment

robert-zaremba commented Nov 10, 2017

robert-zaremba commented Nov 10, 2017

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert-zaremba commented Nov 23, 2017

robert-zaremba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert-zaremba commented Nov 23, 2017

robert-zaremba commented Nov 23, 2017

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VoR0220 commented Dec 12, 2017

dshulyak commented Dec 12, 2017

robert-zaremba commented Dec 12, 2017

dshulyak commented Dec 13, 2017

robert-zaremba commented Dec 13, 2017

VoR0220 commented Dec 13, 2017

robert-zaremba commented Dec 19, 2017

gballet commented Dec 21, 2017 •

edited