CI: ensure msgp generator has been run and is clean

[cce]

Summary

This should ensure make msgp has been run for changes that impact msgp serialization on CI builds.

It looks like #3829 was merged but missed changes to agreement/msgp_gen.go and gci updates from algorand/msgp#14 were not incorporated into #3919.

Test Plan

Tests should fail in the codgen_verification.sh job for the first build on this PR, then should pass after a commit is pushed that updates the msgp-generated files with the latest on master.

[algorandskiy]

go mod tidy to remove the old version?

[cce]

I tried and it cleared out all of go.mod! I guess all of these packages are unused. TBH I'm not really sure long-term whether it is a good idea to keep installing CLI tools this way.

[cce]

I see now that install_buildtools.sh is just grepping go.mod to pull version numbers, so I changed it to a flat text file.

[jannotti]

Does that make it harder to create though? Presumably it was created my some go tool automatically, but the versions file will need to be edited by hand. Could it grep the root go.mod instead? I don't really know what this one is, how it was made, or why it might differ.

[cce]

IMO it is much easier to maintain a simple "versions" file now by hand — I would even be OK to just move the tool versions into install_buildtools.sh directly, as it's the only user. The reason it's not in root go.mod is because these tools are not dependencies of go-algorand. (They don't even really need to be implemented in go — they're just CLI tools to install)

[cce]

@Aharonee @algoidan wanted to makes sure this was as expected?

[Aharonee]

I think I know what happened.
04814e1 (#3829)

This is the commit where I've renamed the TxnRoot to TxnCommitments, and its nested fields.
While doing that I've regenerated msgp, but after noticing the fields in TxnCommitments where no longer in alphabetical order, I swapped them.

	TxnCommitments struct {
		_struct struct{} `codec:",omitempty,omitemptyarray"`

		Sha256Commitment crypto.Digest `codec:"txn256"`
		NativeSha512_256Commitment crypto.Digest `codec:"txn"`

to

	TxnCommitments struct {
		_struct struct{} `codec:",omitempty,omitemptyarray"`

		NativeSha512_256Commitment crypto.Digest `codec:"txn"`
		Sha256Commitment crypto.Digest `codec:"txn256"`

This doesn't break any tests of course, and wasn't caught the the codegen_verification script, but when generating msgp after this change the resulting code will be different (it seems to be generated in a sequential manner) even though its logic is identical.

[algorandskiy]

Any idea why it complains about protocol now?

--- a/agreement/msgp_gen_test.go
+++ b/agreement/msgp_gen_test.go
@@ -10,7 +10,6 @@ import (
 
 	"github.com/algorand/msgp/msgp"
 
-	"github.com/algorand/go-algorand/protocol"
 	"github.com/algorand/go-algorand/test/partitiontest"
 )

[cce]

I am trying to figure it out still, but yes.. something is weird about running make msgp in CircleCI.. even when I ssh into the CircleCI VM and run it manually it works ...

[algorandskiy]

Look, the msgp's master misses it:

writeImportHeader(
	testbuf,
	"github.com/algorand/msgp/msgp", "github.com/algorand/go-algorand/test/partitiontest", "testing")

[algorandskiy]

I think goformat that is called right before flushing the generated files was adding protocol import before and does not add it now.

[cce]

@algorandskiy see in the latest commit, even after downgrading msgp to the old version the protocol package is not detected. However again when I ssh in and rerun msgp manually I can get it to work (and also locally). There is something strange about the environment in CircleCI. But perhaps the easiest thing to do is to just add the protocol import to the msgp generator.

[cce]

Updated msgp to explicitly import missing package: algorand/msgp#17

[algorandskiy]

LGTM but why did scripts/buildtools/go.mod and go.sum go?

[cce]

@algorandskiy we don't have any go code in that directory, so "go mod tidy" stopped working in there (it detected no dependencies) and we are only using go.mod to track a few version numbers used by a single shell script (install_buildtools.sh), so I switched it to use a text file containing those numbers instead.

[codecov]

Codecov Report

Merging #3978 (5c0a6b5) into master (91095fa) will increase coverage by 0.04%.
The diff coverage is 27.27%.

@@            Coverage Diff             @@
##           master    #3978      +/-   ##
==========================================
+ Coverage   49.77%   49.82%   +0.04%     
==========================================
  Files         409      409              
  Lines       69157    69157              
==========================================
+ Hits        34426    34460      +34     
+ Misses      31011    30983      -28     
+ Partials     3720     3714       -6     

Impacted Files	Coverage Δ
crypto/merklearray/msgp_gen.go	38.52% <ø> (ø)
crypto/merklesignature/msgp_gen.go	41.99% <ø> (ø)
daemon/algod/api/spec/v2/msgp_gen.go	20.31% <ø> (ø)
data/account/msgp_gen.go	40.81% <ø> (ø)
data/basics/msgp_gen.go	34.14% <ø> (ø)
data/transactions/msgp_gen.go	40.38% <ø> (ø)
ledger/msgp_gen.go	41.28% <ø> (ø)
agreement/msgp_gen.go	41.08% <27.27%> (ø)
data/bookkeeping/msgp_gen.go	42.60% <27.27%> (ø)
cmd/algoh/blockWatcher.go	77.77% <0.00%> (-3.18%)	⬇️
... and 8 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91095fa...5c0a6b5. Read the comment docs.