Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/exp: reconsider using single module for all unrelated experimental packages of various maturity levels #37175

dmitshur opened this issue Feb 11, 2020 · 4 comments


Copy link

@dmitshur dmitshur commented Feb 11, 2020

The repository is described in its README as:


This subrepository holds experimental and deprecated (in the old
directory) packages.

The idea for this subrepository originated as the pkg/exp directory
of the main repository, but its presence there made it unavailable
to users of the binary downloads of the Go installation. The
subrepository has therefore been created to make it possible to go get these packages.

Warning: Packages here are experimental and unreliable. Some may
one day be promoted to the main repository or other subrepository,
or they may be modified arbitrarily or even disappear altogether.

In short, code in this subrepository is not subject to the Go 1
compatibility promise. (No subrepo is, but the promise is even more
likely to be violated by go.exp than the others.)

Caveat emptor.

It currently contains various unrelated experimental packages that are not related to each other. Some of them are more experimental and less stable, while others are more stable and less experimental.

Back when GOPATH mode was the only build mode available, there was not a high cost to having many unrelated packages in one repository.

In module mode, having many unrelated packages in one module can contribute to increasing the size of the module graph.

For example, the module requires because it uses cmd/apidiff, which means all transitive module requirements of all packages in are added to anyone's module that uses at least one Google Cloud Client library from

The module mirror ( helps protect users from individual dependencies being unavailable at the origin server, but at this time there are still some users that haven't started using it for various reasons. I expect that number will continue to go down over time. See various user feedback from a recent outage affecting one of dependencies:

It can be argued that perhaps the module should not be requiring (that can still be investigated in a separate issue at, but maybe there can be a better solution in that would make it possible to keep separate packages and their dependencies more isolated.

One possible idea is to move the shiny project into a nested module at, which would remove its dependencies from the main module. It's unclear how scaleable such a solution is or how applicable it would be in the general case, but it would likely help people who want to import cmd/apidiff.

Opening this issue to discuss and look for ways to improve the current situation if possible.

Other related issues:

  • #27900 - cmd/go: 'go mod why' should have an answer for every module in 'go list -m all'
  • #29935 - x/build: reconsider the large number of third-party dependencies
  • #35221 - x/exp: depends on unreliable repository

/cc @bcmills @jayconrod @matloob @jba @dsymonds @robpike

Copy link

@dsymonds dsymonds commented Feb 11, 2020

It seems like cmd/apidiff should at least be moved to x/tools given its stability these days.

But separate go.mod files throughout x/exp seems good too, especially since that repository is prone to having dependencies outside the standard library and other x/ repos.


Copy link

@jayconrod jayconrod commented Feb 11, 2020

I'd argue that packages in should be promoted to their own repos or to other repos like x/tools if people depend on them being stable. cmd/apidiff may fit that description.

@bcmills is working on a change that would reduce the cost of a large module graph (both in load time and stability). Placeholder issue is #36460. So this may not be a justification for splitting modules in the future.

In general, multi-module repos complicate development in a lot of ways, and I think we should avoid that if we can.


Copy link

@rsc rsc commented Feb 12, 2020

Nothing should depend on x/exp. If lots of things need cmd/apidiff, it sounds like it should move.

In general, x/ repos should not depend on non-golang repos. x/exp is an exception because it's not supposed to be important for people (see previous paragraph).

It would be fine to put a go.mod in x/exp/shiny today to contain the dependencies and not pollute the rest of x/exp that people should not be depending on.

In the longer term we need to work out the plan for x/ repos generally. I'm OK with a band-aid in x/exp because, again, people should not be depending on it.

x/tools is a bit of a dumping ground but maybe is the right home for cmd/apidiff at this point. It at least tends dependencies better.

The right fix for today is to put a go.mod in x/exp/shiny.

In the longer term, lazy loading of go.mods will make this situation not bother looking at shiny, because it doesn't matter for the build at hand.


Copy link

@bcmills bcmills commented Feb 12, 2020

To speak specifically to the point about lazy module loading: note that the module dependencies that provide all packages reachable from x/exp are a strict subset of the dependencies found in the module graph:

~/src/$ go list -m all | sort -u v0.0.0-20190408044501-666a987793e9 v0.0.0-20160522181843-27f122750802 v0.0.0-20191125211704-12ad95a8df72 v0.0.0-20191011191535-87dc89f01550 v0.0.0-20190802002840-cff245a6509b v0.0.0-20190719004257-d2bd2a29d028 v0.1.1-0.20191105210325-c90efee705ee v0.0.0-20190620200207-3b0461eec859 v0.0.0-20190423024810-112230192c58 v0.0.0-20190412213103-97732733099d v0.3.0 v0.0.0-20200207183749-b753a1ba74fa v0.0.0-20191011141410-1b5146add898

~/src/$ go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' all | sort -u v0.0.0-20160522181843-27f122750802 v0.0.0-20190802002840-cff245a6509b v0.0.0-20190719004257-d2bd2a29d028 v0.1.1-0.20191105210325-c90efee705ee v0.0.0-20200207183749-b753a1ba74fa v0.0.0-20191011141410-1b5146add898


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants