Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go_proto_library: missing strict dependencies on WKTs with protoc 3.14 #2721

Closed
jayconrod opened this issue Nov 16, 2020 · 1 comment · Fixed by #2739
Closed

go_proto_library: missing strict dependencies on WKTs with protoc 3.14 #2721

jayconrod opened this issue Nov 16, 2020 · 1 comment · Fixed by #2739
Milestone

Comments

@jayconrod
Copy link
Contributor

What version of rules_go are you using?

v0.24.7

What version of gazelle are you using?

n/a

What version of Bazel are you using?

3.7.0

Does this issue reproduce with the latest releases of all the above?

yes

What operating system and processor architecture are you using?

Darwin amd64

What did you do?

Build pretty much any go_proto_library using protoc 3.14 or later.

For example:

bazel build @go_googleapis//google/api:annotations_go_proto

-- WORKSPACE --
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "207fad3e6689135c5d8713e5a17ba9d1290238f47b9ba545b63d9303406209c6",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.24.7/rules_go-v0.24.7.tar.gz",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.24.7/rules_go-v0.24.7.tar.gz",
    ],
)

http_archive(
    name = "com_google_protobuf",
    sha256 = "bf0e5070b4b99240183b29df78155eee335885e53a8af8683964579c214ad301",
    strip_prefix = "protobuf-3.14.0",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.14.0.zip"],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")

go_rules_dependencies()

go_register_toolchains()

protobuf_deps()
-- BUILD.bazel --

What did you expect to see?

Successful build.

What did you see instead?

ERROR: /private/var/tmp/_bazel_jayconrod/60e122df6a2a16ab6af86d77e9a71ec5/external/go_googleapis/google/api/BUILD.bazel:107:17: GoCompilePkg external/go_googleapis/google/api/annotations_go_proto.a failed (Exit 1): builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix darwin_amd64 -src ... (remaining 57 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix darwin_amd64 -src ... (remaining 57 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
compilepkg: missing strict dependencies:
	/private/var/tmp/_bazel_jayconrod/60e122df6a2a16ab6af86d77e9a71ec5/sandbox/darwin-sandbox/555/execroot/__main__/bazel-out/darwin-fastbuild/bin/external/go_googleapis/google/api/annotations_go_proto_/google.golang.org/genproto/googleapis/api/annotations/annotations.pb.go: import of "google.golang.org/protobuf/types/descriptorpb"
	/private/var/tmp/_bazel_jayconrod/60e122df6a2a16ab6af86d77e9a71ec5/sandbox/darwin-sandbox/555/execroot/__main__/bazel-out/darwin-fastbuild/bin/external/go_googleapis/google/api/annotations_go_proto_/google.golang.org/genproto/googleapis/api/annotations/client.pb.go: import of "google.golang.org/protobuf/types/descriptorpb"
	/private/var/tmp/_bazel_jayconrod/60e122df6a2a16ab6af86d77e9a71ec5/sandbox/darwin-sandbox/555/execroot/__main__/bazel-out/darwin-fastbuild/bin/external/go_googleapis/google/api/annotations_go_proto_/google.golang.org/genproto/googleapis/api/annotations/field_behavior.pb.go: import of "google.golang.org/protobuf/types/descriptorpb"
	/private/var/tmp/_bazel_jayconrod/60e122df6a2a16ab6af86d77e9a71ec5/sandbox/darwin-sandbox/555/execroot/__main__/bazel-out/darwin-fastbuild/bin/external/go_googleapis/google/api/annotations_go_proto_/google.golang.org/genproto/googleapis/api/annotations/resource.pb.go: import of "google.golang.org/protobuf/types/descriptorpb"
No dependencies were provided.
Check that imports in Go sources match importpath attributes in deps.
bazel: Leaving directory `/private/var/tmp/_bazel_jayconrod/60e122df6a2a16ab6af86d77e9a71ec5/execroot/__main__/'
Target @go_googleapis//google/api:annotations_go_proto failed to build

Analysis

From an email with @dsnet:

This is almost certainly caused by the go_package option in each .proto file being updated to reference the new Go protobuf module.

Builds with the native Go toolchain will work fine since the tool knows how to resolve and obtain modules. However, the Bazel support for Go probably needs to be taught about these changed dependencies.

In particular, the following packages have been "moved":
"github.com/golang/protobuf/protoc-gen-go/descriptor" => "google.golang.org/protobuf/types/descriptorpb"
"github.com/golang/protobuf/protoc-gen-go/plugin" => "google.golang.org/protobuf/types/pluginpb"
"github.com/golang/protobuf/ptypes/any" => "google.golang.org/protobuf/types/known/anypb"
"github.com/golang/protobuf/ptypes/duration" => "google.golang.org/protobuf/types/known/durationpb"
"github.com/golang/protobuf/ptypes/empty" => "google.golang.org/protobuf/types/known/emptypb"
"github.com/golang/protobuf/ptypes/struct" => "google.golang.org/protobuf/types/known/structpb"
"github.com/golang/protobuf/ptypes/timestamp" => "google.golang.org/protobuf/types/known/timestamppb"
"github.com/golang/protobuf/ptypes/wrappers" => "google.golang.org/protobuf/types/known/wrapperspb"
"google.golang.org/genproto/protobuf/api" => "google.golang.org/protobuf/types/known/apipb"
"google.golang.org/genproto/protobuf/field_mask" => "google.golang.org/protobuf/types/known/fieldmaskpb"
"google.golang.org/genproto/protobuf/ptype" => "google.golang.org/protobuf/types/known/typepb"
"google.golang.org/genproto/protobuf/source_context" => "google.golang.org/protobuf/types/known/sourcecontextpb"

@kyessenov
Copy link

This appears to be the culprit of why updating protobuf to 3.14 fails in envoyproxy/envoy.

jayconrod pushed a commit to jayconrod/rules_go that referenced this issue Nov 30, 2020
Since protobuf 3.14, the Well Known Types declare APIv2 package paths,
which means generated code that imports them will import APIv2
packages.

This PR is intended to be cherry-picked to the 0.23 and 0.24 release
branches. It may slow down builds, since both APIv1 and APIv2 packages
will be compiled; we can't tell which is needed during analysis.

For bazelbuild#2721
jayconrod pushed a commit to jayconrod/rules_go that referenced this issue Nov 30, 2020
These should not be imported in generated code when protobuf 3.14 or
higher is used.

protobuf 3.14.0 will be the minimum supported version for rules_go 0.25.0.

Fixes bazelbuild#2721
jayconrod pushed a commit that referenced this issue Dec 1, 2020
Since protobuf 3.14, the Well Known Types declare APIv2 package paths,
which means generated code that imports them will import APIv2
packages.

This PR is intended to be cherry-picked to the 0.23 and 0.24 release
branches. It may slow down builds, since both APIv1 and APIv2 packages
will be compiled; we can't tell which is needed during analysis.

For #2721
jayconrod pushed a commit that referenced this issue Dec 2, 2020
Since protobuf 3.14, the Well Known Types declare APIv2 package paths,
which means generated code that imports them will import APIv2
packages.

This PR is intended to be cherry-picked to the 0.23 and 0.24 release
branches. It may slow down builds, since both APIv1 and APIv2 packages
will be compiled; we can't tell which is needed during analysis.

For #2721
jayconrod pushed a commit that referenced this issue Jun 28, 2021
Since protobuf 3.14, the Well Known Types declare APIv2 package paths,
which means generated code that imports them will import APIv2
packages.

Fixes #2721
bdhess added a commit to GoogleCloudPlatform/kms-integrations that referenced this issue Aug 2, 2021
This makes it easier to test cases where the deadline is exceeded.

I think I have an ever better/more flexible approach that we can move to
later, but it's blocked by an open issue in rules-go:

bazelbuild/rules_go#2721

Change-Id: Ie5533a39281cd48138f55a0538d1bfc676552f26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants