New submodules: upgrade before 2024-03-01 to avoid ambiguous import errors with google.golang.org/genproto

[quartzmo]

TL;DR

Upgrade to a version of google.golang.org/genproto at or above version v0.0.0-20230526161137-0005af68ea54 with the following command:

go get google.golang.org/genproto@latest
go mod tidy

Doing this before the 9-month deadline of 2024-03-01 will avoid "ambiguous import" errors.

Background

Many of the packages in the google.golang.org/genproto module represent API surfaces belonging to Google Cloud. Since approximately the v0.0.0-20220921223823-23cae91e6737 release of google.golang.org/genproto, a change in #901 introduced dependencies on virtually every submodule in cloud.google.com/go. This change moved type definitions for Google Cloud client libraries to cloud.google.com/go, greatly improving usability of the latter. To maintain compatibility with existing code, aliases were introduced into google.golang.org/genproto that pointed to the new sources of truth in cloud.google.com/go. This added the dependencies on cloud modules.

Even though most packages in google.golang.org/genproto are Google Cloud, a few packages provide simple “common” types used in projects unrelated to Google Cloud. For users of the simpler packages, this commingled existence resulted in a large number of extra cloud modules appearing in the dependency graph. This can trigger alarms in automatic dependency checks, false positives in vulnerability checks, and adds constraints to minimum version selection when resolving acceptable versions of other dependencies.

New submodules in this repo

To improve usability for consumers that do not depend on Google Cloud modules, packages providing widely used "common" types will be published as submodules. The following packages do not depend on cloud types and are now isolated in new non-cloud submodules:

• google.golang.org/genproto/googleapis/api (Note: this does not include 4 subpackages referencing cloud types, listed below)
• google.golang.org/genproto/googleapis/bytestream
• google.golang.org/genproto/googleapis/rpc

The following subpackages of google.golang.org/genproto/googleapis/api reference cloud types and are now isolated from the api package as new cloud submodules:

• google.golang.org/genproto/googleapis/api/apikeys
• google.golang.org/genproto/googleapis/api/servicecontrol
• google.golang.org/genproto/googleapis/api/servicemanagement
• google.golang.org/genproto/googleapis/api/serviceusage

To ease migration, these submodules initially have a “back-ref” dependency on the google.golang.org/genproto parent. After a 9-month period, this "back-ref" dependency will be removed, making it possible to once again depend on the simpler packages of this project without adding cloud modules to the dependency graph.

Avoiding ambiguous imports for submodules

The root package docs for google.golang.org/genproto state:

IMPORTANT This repository is currently experimental. The structure of the contained packages is subject to change. Please see the original source repositories (listed below) to find out the status of each protocol buffer's associated service.

In addition to this warning, the pseudo version numbering of the module:

Signals that the module is still in development and unstable. This release carries no backward compatibility or stability guarantees.

However, despite these warnings, we know that critical infrastructure depends on many of the packages in google.golang.org/genproto. To avoid "ambiguous import" errors for consumers, we are adding a “back-ref” dependency on the google.golang.org/genproto parent to each new submodule for 9 months to allow a seamless transition.

After the 9-month deadline of 2024-03-01 has passed, the “back-ref” dependency on the google.golang.org/genproto parent will be removed from each new submodule. Client code still using a version of the google.golang.org/genproto parent earlier than v0.0.0-20230525234044-86246a831fb4 may then encounter "ambiguous import" errors when importing packages that have been moved to submodules. Those errors can be resolved by updating to a version at or above version v0.0.0-20230526161137-0005af68ea54 with the following command:

go get google.golang.org/genproto@latest
go mod tidy

References

• #113366
• FAQS — Multi-Module Repositories
• Is it possible to add a module to a multi-module repository?

[smocherla-brex]

We're going to migrate but curious regarding the behavior after 2024-03-01

Client code still using a version of the google.golang.org/genproto parent earlier than v0.0.0-20230525234044-86246a831fb4 may then encounter "ambiguous import" errors when importing packages that have been moved to submodules

If we were using offical Go modules proxy which caches the modules indefinitely (from my understanding), will this still error out (or will it be removed from the cache as well)?

[liggitt]

It isn't a matter of the old module versions being removed or becoming unavailable, they would still be available via module proxy or git sha.

The "ambiguous import" issue happens when some dependencies in your dependency graph refer to a version of genproto prior to the carveout, some things refer to a version after the carveout along with some of the submodules, and dependency resolution can't figure out which module is supposed to be providing the package.

Making sure you require a version of genproto after the carveout makes it unambiguous which module provides that package.