Revert upgrade to "hashicorp/go-msgpack v2"

[lgfa29]

After some internal testing we noticed that this changed causes a server upgrade to fail with an error like this:

2023-05-01T16:39:31.542-0400 [ERROR] nomad: failed to start Raft: error="failed to get last log at index 20: Time.UnmarshalBinary: unsupported version"
2023-05-01T16:39:31.542-0400 [ERROR] agent: error starting agent: error="server setup failed: Failed to start Raft: failed to get last log at index 20: Time.UnmarshalBinary: unsupported version"

go-msgpack/v2 was supposed to be the exact same code as v1.1.5 that we were using before, so I'm not sure what is causing this issue. Reverting these changes fixes the problem.

Reverts #16810

[schmichael]

Yikes! Are you able to followup with what went wrong?

[lgfa29]

Yup, I've reopened #16808 while I investigate this further.

[IamTheFij]

That's really strange... It's not just supposed to be the same exact code, but it is the same exact code: hashicorp/go-msgpack@v1.1.5...v2.0.0.

Is there a local proxy you're using aside from the Google one? I could try to verify a test from outside your infra, if that helps rule out some cached package with additional diffs.

[lgfa29]

I'm really not sure what happened. I don't have any proxy configured, but I do have GOPRIVATE=github.com/hashicorp set to help working on some of our private repos, which I believe bypasses any cache.

We decided to revert this because we're approaching a release and this was the quickest fix for now. I will need some time to investigate this further. It may be some incompatibility between the versions used in Nomad and other libraries, like https://github.com/hashicorp/raft-boltdb.

If you have a chance to help us investigate we would very much appreciate 🙂

You can easily test the problem by starting a (non-dev) cluster using the 1.5.3 official binary, run a job, stop the agent, start it with the custom binary.