Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rev of proto-gen-go to ProtoPackageIsVersion3 causing breakage #763

Closed
edwarnicke opened this Issue Nov 27, 2018 · 60 comments

Comments

Projects
None yet
@edwarnicke
Copy link

edwarnicke commented Nov 27, 2018

I am hopeful that there is a simple solution to get around this issue :)

A few hours ago, this commit introduced proto.ProtoPackageIsVersion3

and set const generatedCodeVersion = 3

Our CI system (and I suspect many other peoples) do a:

go get github.com/golang/protobuf/protoc-gen-go

as part of setting up the system, and also do a:

go generate ./...

and check to make sure that regenerated files match the files in the commit (basically, checking for regen on updating the .proto files).

This has been going swimmingly until a few hours ago. In trying to debug this problem, I came to realize that the fact this worked at all is a testament to how good you guys are at backward compatibility 👍

What I discovered digging in is that we have been vendoring github.com/golang/protobuf/ v1.2.0 (because of other dependencies we have), but of course go get pulls the latest from master.

v1.2.0 lacks proto.ProtoPackageIsVersion3, so our generated code suddenly doesn't compile.

So I fell back to using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

in our CI. Unfortunately, this seems to result in some generation differences. If I run it locally, push, and then our CI runs it, the generated code doesn't quite match.

I get diffs like:

-var fileDescriptor_networkservice_d4de7ff5f5769d53 = []byte{
+var fileDescriptor_networkservice_5b085879d0d87bdf = []byte{

I don't precisely consider this a bug on your end, you guys are super good about backward compatibility, and at some point you do have to make the change.

Do you have any ideas about how I might navigate all of this? I suspect lots and lots of folks will shortly have similar issues...

edwarnicke added a commit to haiodo/networkservicemesh that referenced this issue Nov 27, 2018

Temporarily disable regen check for proto till things get sorted out.
See:

golang/protobuf#763

Signed-off-by: Ed Warnicke <hagbard@gmail.com>
@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

@dsnet do you have any insight?

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 27, 2018

I'm not sure what we can do on our end, this seems to be an issue really with the Go toolchain that there isn't an easy way (or at least I don't know of one) to build a binary at a specific revision.

For the time being, you can do something like this:

$ git clone git@github.com:golang/protobuf.git && (cd protobuf && git checkout v1.2.0 && go build -o $BIN_PATH/protoc-gen-go ./protoc-gen-go) && rm -r protobuf

Be sure to have $BIN_PATH be the output path where the binary should end up.

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 27, 2018

I'm going to close this as a duplicate of #751 since the general issue is that the instructions for how to install protoc-gen-go are not ideal. However, the broader issue at hand is golang/go#24250

@dsnet dsnet closed this Nov 27, 2018

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

@dsnet Yeah... I can build the protoc-gen-go from my vendored dir (v1.2.0) but that version (unlike more recent ones) doesn’t seem to always generate the same go code for identical .proto.

At what point did proto-gen-go start generating identical go for identical proto?

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 27, 2018

The output of protoc-gen-go should be deterministic. If it isn't, then that's a bug.

That said, the most likely cause of non-deterministic output is due to how protoc is invoked. The descriptor protos that are embedded in each .pb.go file contains the filepath of the proto file and depending on the -I flags to protoc, it is possible that those change.

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

@dsnet Also, I don’t think is the same as #751 as it doesn’t have anything to do with modules and we are seeing it with go 1.10 (pre modules)

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

@dsnet OK, that’s super helpful to know. Are the determined by the fully qualified path of included proto files? I ask, because we also just started using -I ../.../vendor in our protoc invocations.

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

Which is to say, the fully qualified path of the imported protofiles (things like github.com/golang/protobufs/ptypes/empty/empty.proto) will different depending on where they are built, as we are pulling them in from vendor/ )

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 27, 2018

Errr, sorry, I mispoke. It's not the -I flag. It's the how the proto file to compile was specified when given to protoc.

For example:

$ protoc --go_out=paths=source_relative:. ptypes/any/any.proto
$ mv ptypes/any/any.pb.go ptypes/any/any1.pb.go
$ (cd ptypes && protoc --go_out=paths=source_relative:. any/any.proto)
$ mv ptypes/any/any.pb.go ptypes/any/any2.pb.go 
$ (cd ptypes/any && protoc --go_out=paths=source_relative:. any.proto)
$ mv ptypes/any/any.pb.go ptypes/any/any3.pb.go 
$ cat any*.pb.go | grep RegisterFile
func init() { proto.RegisterFile("ptypes/any/any.proto", fileDescriptor_any_fea4429e4270d3cd) }
func init() { proto.RegisterFile("any/any.proto", fileDescriptor_any_f2863f9fa4d3af49) }
func init() { proto.RegisterFile("any.proto", fileDescriptor_any_0a174ad2e4d7ef58) }

Notice that all three have different values?

This behavior is surprising, but is an artifact of how protobufs were originally designed. That is, they assume a single root where all source files universally lived under in a build system similar to Bazel. It's not great for today's world of independently distributed build processes, but such is the status quo.

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

OK.. so that’s a super helpful hint... let me go look for those differences and see if I can think of a solution.

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

@dsnet OK... so I can't quite square this with my understanding of your explanation:

-	proto.RegisterFile("networkservice.proto", fileDescriptor_networkservice_efee0c6a43fef699)
+	proto.RegisterFile("networkservice.proto", fileDescriptor_networkservice_1d42c3056c917b94)
@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 27, 2018

So clearly something else is feeding into the hash. What versions of protoc-gen-go were used to produce those .pb.go files? Also what versions of the Go toolchain were used to build those binaries?

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 27, 2018

dep has our vendored version of github.com/golang/protobuf/protoc-gen-go/ at version v1.2.0.

we are building protoc-gen-go using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

So I'm going to go out on a limb and say its proto-gen-go version v1.2.0 (I couldn't find any way for the protoc-gen-go binary to output a version, would love to know if there's a trick there :) ).

go version yields go1.10.5 linux/amd64

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 27, 2018

And what are the versions going into the other protoc-gen-go binary being used?

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 28, 2018

The other go version is:
go version go1.11.2 darwin/amd64
(my dev box, where I did the gen I'm pushing to our CI).

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 28, 2018

OK... this is super weird.

I installed a downgraded go version on my laptop:
go version go1.10.5 darwin/amd64

and created a fresh new alternate GOPATH and cloned the PR that is failing our CI into it.
And ran
go generate ./...
(at top level, same as where the CI runs it).

And I am not getting different hashes. So I suspect that its not the location of the GOPATH, or the go version feeding into the hash difference... what else might it be?

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 28, 2018

It's not so much that it depends directly on the Go version, but rather parts of the standard library like compress/gzip, which does not have stable output between versions.

@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 28, 2018

Ah... OK... that makes a lot of sense... hmm... I have now tried with downgrade to the same go version, but on the Mac, and seem to be OK in terms of the output not changing...

This is super weird. I appreciate your patience talking me through some of the ins and outs of how things work it makes it much much easier to figure out how to debug :)

fkautz added a commit to networkservicemesh/networkservicemesh that referenced this issue Nov 28, 2018

NetworkServiceServer/NetworkService client. (#522)
* 1. usage of NetworkServiceDiscovery
2. services registry to do connections to internal services
3. services apiRegistry for starting up network apis.
4. partial test of NSC -> NSMD/Registry
5. NSMD <-> NSMD client -> Request.

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* Overly simple fix to get memif working again.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Switch to using NSM Name rather than URL.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Added Machinery to make testing in multi-node vagrant doable

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Fixed vppagent-dataplane to send the correct srcIP in response to MonitorMechanisms

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Update remote mechaism selections

+ Fix tests

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* + Register discovery inside nsmd-k8s
+ Fixes Public API port.
+ Dst/Src fips.

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* It's Alive!  Alive!

make k8s-check works

1. Fixed the Dataplane to attach vpp to the Pod interface
   Otherwise... we can't send or receive the VXLAN packets
2. Factored out Selector, and implemented RoundRobin Selector
3. We need to set the workspace parameter on the local side of CrossConnects
   that come in remotely.  Did a terrible terrible hack on serviceRegistry
   to make that doable.  I would love to see that done cleaner.
4. nsmd-k8s wasn't correctly returning the NSM.Name ... this was causing
   weird situations where an nsmd was treating itself as remote.  Fixed.
5. Added network_service_endpoint to remote.Connection
6. Setup CI for remote case.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Undo miscommit of kubeadm join command and other CI fixes.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* dep ensure fix

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* #522 Fix tests

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* #522 Add ifconfig dump for ping test failures

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* #522 Fixing tests

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* Fix issues with make k8s-check and vagtant

Signed-off-by: Andrey Sobolev <haiodo@gmail.com>

* Fix machinery to work in both Virtualbox and VMWare

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Fix shellcheck error

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* More shellcheck fixes

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Fix .circleci test for out of date protofiles

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Fixed issue with code generation.

github.com/golang/protobuf/protoc-gen-go installed via go get is from master.
master just introduced breaking changes in protoc-gen-go, that are not in any released
version.

So we
go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

to make sure the protoc-gen-go matches the various things they import.
Sadly, the results in somewhat different code generation than we had before :(

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Make sure we can include the proto from vendor.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Fix include paths so that generation here works with older protoc-gen-go

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Temporarily disable regen check for proto till things get sorted out.

See:

golang/protobuf#763

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Switch to ip addr as ifconfig isn't in all containers

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Experiment with go 1.11

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Get go version in sanity check

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Switch back to go version 1.10

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Add a bit more output for debugging in logs.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Wait for nsc pods to come up before running make k8s-check

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Switch to weave, until we can figure out why Calico blocks VXLAN.

For reasons unclear yet, Calico results in inability for VXLAN
packets to flow between Nodes.  Weave seems to work.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* Response to comments on PR

Signed-off-by: Ed Warnicke <hagbard@gmail.com>

* One more comment response

Signed-off-by: Ed Warnicke <hagbard@gmail.com>
@sjdweb

This comment has been minimized.

Copy link

sjdweb commented Nov 28, 2018

Hi there, this issue hit me this morning. It was fine locally due to having an older version installed, but CI uses clean docker builds.

I switched to installing the vendored protoc-gen-go like @edwarnicke and all is well now!

The actual error I saw was

undefined: proto.ProtoPackageIsVersion3

@dsnet dsnet changed the title Rev of proto-gen-c to ProtoPackageIsVersion3 causing breakage Rev of proto-gen-go to ProtoPackageIsVersion3 causing breakage Nov 28, 2018

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Nov 28, 2018

Hi everyone,

I apologize that 8d0c54c recently broke a lot of people's workflows. The breakage is unfortunate and several people have written to me personally requesting that we issue v1.3.0 of Go protobufs "to resolve the issue". However, I'm going to make the case that the problem at hand is not solved by any form of versioning that we do on golang/protobuf's end. Breakage is still going to happen (unless protoc-gen-go is unreasonably restricted from changing the internal implementation details of protobufs).

Fundamentally, a CI system must be hermetic such that testing a given commit of your project uses a consistent set of dependencies and provides repeatability. This implies that running the CI script on a specific version of your project today, last year, or 10 years from now should produce the exact same test results. CI scripts that hard-code a specific version of the proto package, but continues to obtain protoc-gen-go from HEAD is fundamentally not hermetic. (In fact, if any part of the CI script depends on anything from HEAD, then it is not hermetic). Breakages due to externally changing factors (e.g., 8d0c54c ) is an implicit (and undesirable) property of non-hermetic builds.

Here are things that we can do to help mitigate, but not automatically resolve the issue:

  • Issue #751 should fix README.md to specify how to install a specific version of protoc-gen-go. Unfortunately, this is complicated by the fact that the Go toolchain currently doesn't support directly building at a specific version. See PR #764 for a workaround while golang/go#24250 (@bcmills) is still being addressed.
  • Issue #738 requests that we release pre-compiled binaries of protoc-gen-go, which would help make it easier to download and obtain a specific version of the tool in CI scripts.

Either way, if your CI system broke as a result of 8d0c54c, then some form of change needs to happen to the CI script to ensure hermetic builds. This is not something that we can fix on golang/protobuf's end.

It is unfortunate that a change needs to be made in the user's codebase, but in this situation, I strongly believe it is actually going to put your CI script in a healthier situation.

As an example of a hermetic CI scripts, our own tests for the Go protobuf project hardcodes the exact versions of non-Go package dependencies for protoc and the go toolchain and the exact versions of Go dependencies using a go.mod file. Is it annoying to need to manually increment the versions? Yes, but well worth it in terms of guaranteeing hermetic builds.

dsnet referenced this issue Nov 28, 2018

protoc-gen-go: generate XXX_OneofWrappers instead of XXX_OneofFuncs (#…
…760)

The marshaler, unmarshaler, and sizer functions are unused ever since
the underlying implementation was switched to be table-driven.
Change the function to only return the wrapper structs.

This change:
* enables generated protos to drop dependencies on certain proto types
* reduces the size of generated protos
* simplifies the implementation of oneofs in protoc-gen-go

Updates #708
@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 28, 2018

I'd like to second @dsnet 's point about hermetic CI. I was frankly a bit embarrassed to discover that our CI was vulnerable to this kind of thing. It was vulnerable because we were go getting (which means the version we were getting tracked master). It's a testament to the overall stability of protoc-gen-go that it took us this long to hit any issues.

@dsnet Thanks for taking the time to talk through with me my hash issues... still trying to get to the bottom of them, but your insight helped a lot.

edwarnicke added a commit to edwarnicke/protobuf that referenced this issue Nov 28, 2018

Add note to README on responsible CI usage.
This is in the hopes of easing folks hitting golang#763 in getting
to a proper use of CI.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>
@edwarnicke

This comment has been minimized.

Copy link
Author

edwarnicke commented Nov 28, 2018

@sjdweb

Hi there, this issue hit me this morning. It was fine locally due to having an older version installed, but CI uses clean docker builds.

I switched to installing the vendored protoc-gen-go like @edwarnicke and all is well now!

The actual error I saw was

undefined: proto.ProtoPackageIsVersion3

Do keep an eye out for some of the other issues I've been seeing. I'm still uncertain as to whether the issues I'm hitting from downgrading to v1.2.0 are purely of my own making (most probable), but if downgrading from master to v1.2.0 is part of the cause, best to huddle together for mutual assistance in resolving them :)

@rcgoodfellow

This comment has been minimized.

Copy link

rcgoodfellow commented Nov 28, 2018

I'm also a bit embarrassed that our CI got caught by this too. go get -u github.com/golang/protobuf/protoc-gen-go was just too convenient.

Something that could be helpful coming from the golang/protobuf side is to put binaries at a well known stable location. In the repo release would be great, or something like the k8s release notes binaries section. Forcing users to build the tool is a bit of a complexity leak. go get handled this more or less transparently, until it didn't.

@lispc

This comment has been minimized.

Copy link

lispc commented Dec 13, 2018

I have solve the problem ! I think its because the protoc-gen-go's version not correct.

git clone https://github.com/golang/protobuf
cd ~/protobuf/protoc-gen-go
git chectout tags/v1.2.0 -b v1.2.0
go install

my environment is mac go1.11

spelling error : git chec[k]out tags/v1.2.0 -b v1.2.0

@fertobar

This comment has been minimized.

Copy link

fertobar commented Dec 19, 2018

steps by @lenvs fixed the ProtoPackageIsVersion3 issue, but I'm getting
undefined: utilities.IOReaderFactory with grpc after switch to v1.2.0 .
any help is appreciated

@johanbrandhorst

This comment has been minimized.

Copy link
Member

johanbrandhorst commented Dec 19, 2018

@fertobar I might be wrong but I think that's a grpc-gateway issue. Almost certainly the same problem as here though, you've got one version of your generator and another version of the runtime library. Please make sure you're using the same version of both.

@fertobar

This comment has been minimized.

Copy link

fertobar commented Dec 19, 2018

thanks for the feedback @johanbrandhorst ! I will check that

@fertobar

This comment has been minimized.

Copy link

fertobar commented Dec 19, 2018

I could fix the dependency issue in this way:

dep ensure
go install ./vendor/github.com/golang/protobuf/protoc-gen-go/
go install ./vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/
@nosferatu500

This comment has been minimized.

Copy link

nosferatu500 commented Dec 21, 2018

Thank you @fertobar for research. It worked for me too.

@eldad87

This comment has been minimized.

Copy link

eldad87 commented Dec 21, 2018

Thank you @fertobar.

@sugelalin

This comment has been minimized.

Copy link

sugelalin commented Dec 25, 2018

Thanks @lenvs inspired me! I also solved the problem !
You may do that.

rm -rf vendor/github.com/golang/protobuf
cd vendor/github.com/golang/
git clone https://github.com/golang/protobuf.git
go install

Stay in branch master, don`t checkout to V1.2.0 !!!

@trtg

This comment has been minimized.

Copy link

trtg commented Jan 8, 2019

@johanbrandhorst @edwarnicke @lavoiesl If any of you guys happen to be using go modules- this worked for me:
go get github.com/golang/protobuf@8d0c54c1246661d9a51ca0ba455d22116d485eaa
^^^
that will update your go.mod to use the specific revision needed to avoid problems

@longfellowone

This comment has been minimized.

Copy link

longfellowone commented Jan 9, 2019

@johanbrandhorst @edwarnicke @lavoiesl If any of you guys happen to be using go modules- this worked for me:
go get github.com/golang/protobuf@8d0c54c1246661d9a51ca0ba455d22116d485eaa
^^^
that will update your go.mod to use the specific revision needed to avoid problems

Can confirm this worked for me. Thanks!

@trtg

This comment has been minimized.

Copy link

trtg commented Jan 9, 2019

@longfellowone glad that worked. Go mod is super simple when all your dependencies use semver... But it's still possible with slight deviations like this one (breaking change that technically does not change the API) as I'm painfully learning

@longfellowone

This comment has been minimized.

Copy link

longfellowone commented Jan 9, 2019

@trtg

Not sure what was causing the "undefined: proto.ProtoPackageIsVersion3" error for me. These are the steps I did and your go get is what fixed it.

go get -u github.com/golang/protobuf
protoc --go_out=plugins=grpc:. *.proto

using protoc --version 3.6.1

@dsnet

This comment has been minimized.

Copy link
Member

dsnet commented Jan 9, 2019

I want to note some caveats with #763 (comment) suggestion:

  • The issue that most people are running into is because they are building protoc-gen-go from head, but have a specific version of the proto package pinned.
  • The suggestion above pins the proto package to a more recent version closer to today's head, but doesn't address underlying problem; that is, you are still building protoc-gen-go from head, but have a specific version of proto pinned.
  • The README now contains instructions for how to pin a specific version of protoc-gen-go, which is a more stable path forward, than to still build the binary from head.
  • Go 1.12 will make this much easier, since it will allow installing a Go binary at a specific revision. See golang/go#24250.
@LeoHChen

This comment has been minimized.

Copy link

LeoHChen commented Jan 9, 2019

@johanbrandhorst @edwarnicke @lavoiesl If any of you guys happen to be using go modules- this worked for me:
go get github.com/golang/protobuf@8d0c54c1246661d9a51ca0ba455d22116d485eaa
^^^
that will update your go.mod to use the specific revision needed to avoid problems

works for me. confirmed! thanks @trtg

@xiaozhoz

This comment has been minimized.

Copy link

xiaozhoz commented Jan 15, 2019

problem solved.
This is because proto-go is installed by go get -u github.com/golang/protobuf/protoc-gen-go, which is the newest master branch.
you can switch to tag 1.2.0. reinstall proto-go, reprotoc all files will fix this problem.

harmony-ek added a commit to harmony-one/harmony that referenced this issue Jan 17, 2019

Vendor github.com/golang/protobuf
Peg at commit 8d0c54c1246661d9a51ca0ba455d22116d485eaa in order to avoid
golang/protobuf#763.
@meysam81

This comment has been minimized.

Copy link

meysam81 commented Jan 19, 2019

@johanbrandhorst @edwarnicke @lavoiesl If any of you guys happen to be using go modules- this worked for me:
go get github.com/golang/protobuf@8d0c54c1246661d9a51ca0ba455d22116d485eaa
^^^
that will update your go.mod to use the specific revision needed to avoid problems

Didn't solve for me. Arrrggggghhh, it's frustrating....!

@meysam81

This comment has been minimized.

Copy link

meysam81 commented Jan 19, 2019

even tag v1.2.0 didn't solve it either

@BewareARP

This comment has been minimized.

Copy link

BewareARP commented Jan 19, 2019

even tag v1.2.0 didn't solve it either

did you find the answer yet? Please Help Me

@meysam81

This comment has been minimized.

Copy link

meysam81 commented Jan 20, 2019

even tag v1.2.0 didn't solve it either

did you find the answer yet? Please Help Me

No man nothing so far.

vancluever added a commit to hashicorp/sentinel-sdk that referenced this issue Jan 29, 2019

Add modules
This adds modules support so that we know what versions of our various
tools we are tracking.

Note that there are a lot of indirect dependencies added on part of the
fact we are pinning golang/protobuf at master (go get -u
github.com/golang/protobuf@master) to ensure that we don't run into
golang/protobuf#763.

ramrunner added a commit to CSUNetSec/netsec-protobufs that referenced this issue Feb 2, 2019

@knuckolls

This comment has been minimized.

Copy link

knuckolls commented Feb 18, 2019

In the spirit of @dsnet's comment above about hermetic builds and because i just solved this with go modules in 1.11 I made a repeatable process you can follow so you don't have to waste all the time I did.

TLDR: I pulled the commit from master and checkout & build protobuf-gen-go on that commit, then i did a go get github.com/golang/protobuf/protoc-gen-go@c823c79ea1570fb5ff454033735a8e68575d1d0f in the folder with my existing go.mod file.

Dump of the pertinent files and build process.

go.mod -- after running the above go get command. ^^

module github.com/placeholder/protobuf-go-mod-fix

require github.com/golang/protobuf v1.2.1-0.20190205222052-c823c79ea157 // indirect

Dockerfile.build

FROM golang:1.11.5-alpine3.9 as golang-builder

RUN apk add --no-cache alpine-sdk git protobuf tree

RUN    go get -d -u github.com/golang/protobuf/protoc-gen-go \
    && cd /go/src/github.com/golang/protobuf/protoc-gen-go/ \
    && git checkout c823c79ea1570fb5ff454033735a8e68575d1d0f \
    && go install

ADD . /mnt/src/

RUN    cd /mnt/src/ \
    && tree \
    && protoc -I ./api --go_out=plugins=grpc:api api.proto \
    && go build -o ./protobuf-go-mod-fix \
    && tree \
    && file ./protobuf-go-mod-fix

Then run...

docker build -t placeholder/protobuf-go-mod-fix:latest -f ./Dockerfile.build .
➜  protobuf-go-mod-fix docker build -t placeholder/protobuf-go-mod-fix:latest -f ./Dockerfile.build .
Sending build context to Docker daemon  9.216kB
Step 1/5 : FROM golang:1.11.5-alpine3.9 as golang-builder
 ---> cb1c8647572c
Step 2/5 : RUN apk add --no-cache alpine-sdk git protobuf tree
 ---> Using cache
 ---> b93e8f8c7342
Step 3/5 : RUN    go get -d -u github.com/golang/protobuf/protoc-gen-go     && cd /go/src/github.com/golang/protobuf/protoc-gen-go/     && git checkout c823c79ea1570fb5ff454033735a8e68575d1d0f     && go install
 ---> Using cache
 ---> 16ba8398861a
Step 4/5 : ADD . /mnt/src/
 ---> ed92ca48f2cc
Step 5/5 : RUN    cd /mnt/src/     && tree     && protoc -I ./api --go_out=plugins=grpc:api api.proto     && go build -o ./protobuf-go-mod-fix     && tree     && file ./protobuf-go-mod-fix
 ---> Running in d7dfc8ca2627
.
├── Dockerfile.build
├── api
│   ├── api.proto
│   └── handler.go
├── go.mod
├── go.sum
└── main.go

1 directory, 6 files
go: finding github.com/golang/protobuf v1.2.1-0.20190205222052-c823c79ea157
< snipped .. more of these lines here>

.
├── Dockerfile.build
├── api
│   ├── api.pb.go
│   ├── api.proto
│   └── handler.go
├── go.mod
├── go.sum
├── main.go
└── protobuf-go-mod-fix

1 directory, 8 files
./protobuf-go-mod-fix: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=DoJscSvpyqlOUpOPWgsH/4YDWvLflS9B7ZGPcODhk/xN3P7Yw83ibPvWI20RzE/83WgRHg5an2yiudtBC0b, not stripped
Removing intermediate container d7dfc8ca2627
 ---> 3875635d4d2a
Successfully built 3875635d4d2a
Successfully tagged placeholder/protobuf-go-mod-fix:latest

Hope that helps some of you wayward internet travelers. 👋

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.