Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: x/mod: new repo for module mechanics #31761

Open
9 of 13 tasks
rsc opened this issue Apr 30, 2019 · 30 comments
Open
9 of 13 tasks

proposal: x/mod: new repo for module mechanics #31761

rsc opened this issue Apr 30, 2019 · 30 comments

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Apr 30, 2019

For the initial development of module support, we've kept everything in internal directories to make it easier to make changes as our understanding of what the pieces should look like becomes clearer. But now I think the basic pieces have become pretty clear, and we should think about exporting some packages to help other people who want to write tools working directly with module mechanics.

Note that I said "module mechanics". This is about supporting direct manipulation of modules themselves. It is not about supporting general development tools that need to do things like load packages in module mode. That use case, where modules are incidental rather than the focus, should remain in x/tools, specifically x/tools/go/packages. And the specific case of loading packages should still be done by invoking the go command, which will remain the single point of truth for those algorithms.

I propose to create a new repo golang.org/x/mod that will initially contain:

There's an open issue that this proposal addresses (about exposing some of this stuff) but once again I cannot find it.

/cc @bcmills @jayconrod @ianthehat @myitcv @rogpeppe

@rsc rsc added this to the Proposal milestone Apr 30, 2019
@rsc rsc added the Proposal label Apr 30, 2019
@bcmills
Copy link
Member

@bcmills bcmills commented Apr 30, 2019

There's an open issue that this proposal addresses (about exposing some of this stuff) but once again I cannot find it.

That's #28101.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Apr 30, 2019

I propose to create a new repo golang.org/x/mod

Does this proposal include a plan for how the new module, modfile, semver, etc., packages will be kept in sync with the ones in cmd/go/internal/...?

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 30, 2019

I would assume that they'll be vendored in using go mod vendor.

That does imply that we'll have to have a clean (and perhaps smaller) API boundary between golang.org/x/mod and cmd/go/internal, but that's arguably a good thing anyway.

@rsc
Copy link
Contributor Author

@rsc rsc commented Apr 30, 2019

@dmitshur, yes they would be vendored into cmd/vendor using "go mod vendor".
All the ones I've chosen should already have clean API boundaries, and if not, we'd work on that.

@ianthehat
Copy link

@ianthehat ianthehat commented May 2, 2019

I need quite a few of these in order to support mod files in editors. I have an experiment I have been working on that has them copied to an internal directory of x/tools, but it would be much nicer if I could go straight to using this repository instead.

@hyangah
Copy link
Contributor

@hyangah hyangah commented May 3, 2019

Consider pseudo-version related logic as well
e.g. https://go.googlesource.com/go/+/refs/heads/master/src/cmd/go/internal/modfetch/pseudo.go

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented May 3, 2019

The list of packages above looks good. I expect this will be useful for developing gorelease and a lot of other tools.

I notice that none of internal/modload, internal/modfetch, internal/modfetch/coderepo, internal/mvs are in the list. I think that's the right decision for now -- these are all pretty tightly coupled with cmd/go. It might be worth moving these to x/mod after some refactoring though. I think the proxy and discovery projects have run into use cases where query and download functionality would be useful, and the Go command doesn't do quite what they need.

@rsc
Copy link
Contributor Author

@rsc rsc commented May 7, 2019

Sentiment here seems overwhelmingly in favor. Accepting proposal.

@rsc rsc removed this from the Proposal milestone May 7, 2019
@rsc rsc added this to the Go1.13 milestone May 7, 2019
@rsc
Copy link
Contributor Author

@rsc rsc commented May 7, 2019

Obviously x/mod itself is "unreleased" but it will have effects in the main repo, so milestoned to Go 1.13.

@rsc
Copy link
Contributor Author

@rsc rsc commented May 10, 2019

@andybons, I created this repo. Can you take care of adding it to gopherbot, builders, etc.? Thanks.

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176463 mentions this issue: sumdb/note: signed note formats for checksum database

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176464 mentions this issue: sumdb/dirhash: directory tree hash algorithm

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176465 mentions this issue: sumdb/storage: basic storage interface and tests

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176466 mentions this issue: sumdb: client and server protocol for checksum database

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176460 mentions this issue: semver: basic semver parsing package

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176461 mentions this issue: module: basic module.Version type and support

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176462 mentions this issue: sumdb/tlog: transparent log data structures and algorithms

@andybons
Copy link
Member

@andybons andybons commented May 10, 2019

Everything is set up for x/mod (mirroring, gerritbot, golang.org/x/mod, etc.)

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176639 mentions this issue: sumdb/gosumcheck: basic go.sum checker

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176540 mentions this issue: proxy: basic proxy server protocol

@gopherbot
Copy link

@gopherbot gopherbot commented May 10, 2019

Change https://golang.org/cl/176541 mentions this issue: gomodproxy: simple Go module proxy implementation

gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
Copied from cmd/go/internal/semver.

For golang/go#31761.

Change-Id: I59e8a264f49a5d20ba1caa85b59c3177209a6ff5
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176460
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
Copied from cmd/go/internal/module and added Version.String method.

For golang/go#31761.

Change-Id: I2fbe224843378e68f0e5890dbc36766936a7e3ed
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176461
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
Copied from golang.org/x/exp/sumdb/internal/tlog.

For golang/go#31761.

Change-Id: If44ad226195908ebc8757c1ca54f20f2a6adb3ef
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176462
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
Copied from golang.org/x/exp/sumdb/internal/note.

For golang/go#31761.

Change-Id: Ie1feebc86b6d34a6cdd20ee0c0ba24588841d1c3
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176463
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
Copied from cmd/go/internal/dirhash, with additional doc comments.

For golang/go#31761.

Change-Id: Id56c1bbb6f27e69194f383d49b9def6876f948be
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176464
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
Copied from golang.org/x/exp/sumdb/internal/tkv and tkvtest.

For golang/go#31761.

Change-Id: Ib5a411c9b9641d1006a2e721043e8e16bf0f6bea
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176465
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@dmitris
Copy link
Contributor

@dmitris dmitris commented May 28, 2019

I'm looking for a published library with the types / structures listed in the output of go mod help edit - namely Module, GoMod, Require, and Replace, ideally with the String() methods attached? Would they be in the new library (x/mod)?

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented May 28, 2019

@dmitris Yes, that will be the golang.org/x/mod/modfile package. It hasn’t been added to x/mod just yet.

gopherbot pushed a commit to golang/mod that referenced this issue May 31, 2019
Copied from golang.org/x/exp/sumdb/internal/sumweb
with types cleaned up:

	Conn -> Client
	Client -> ClientOps
	Handler -> Server
	Server -> ServerOps

For golang/go#31761.

Change-Id: If0e004e6c9cab69c82de428810d67aba074aa843
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176466
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 31, 2019
Copied from golang.org/x/exp/sumdb/gosumcheck.

For golang/go#31761.

Change-Id: Ia2cf81372b8e2da9b879f3d737de014eb30ea34b
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176639
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@tsal
Copy link

@tsal tsal commented Jun 22, 2019

Do we know when that might happen? This would be very useful for some CI tooling.

@tsal
Copy link

@tsal tsal commented Jun 28, 2019

Oops! I missed that this was added to the Go1.13 milestone back in May!

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2019

Change https://golang.org/cl/202543 mentions this issue: modfile: copy from cmd/go/internal/modfile

gopherbot pushed a commit to golang/mod that referenced this issue Oct 22, 2019
Copied from 4be6b4a73d (CL 202565). No changes other than import paths.

Updates golang/go#34924
Updates golang/go#31761

Change-Id: Ic25cb983f6641045fc24edf76953b06d4aa5cd43
Reviewed-on: https://go-review.googlesource.com/c/mod/+/202543
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2019

Change https://golang.org/cl/202698 mentions this issue: cmd/go: delete internal packages moved to x/mod

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 25, 2019

Change https://golang.org/cl/203537 mentions this issue: cmd/dist: support GOROOT vendoring

gopherbot pushed a commit that referenced this issue Oct 28, 2019
In the second step of make.bash, cmd/dist builds cmd/go by invoking
the compiler, linker, and other tools directly on transitive
dependencies of cmd/go. Essentially, cmd/dist acts as a minimal
version of 'go install' when building go_toolchain.

Until now, cmd/go has had no transitive dependencies in vendor
directories. This changes in CL 202698, where several packages are
deleted and equivalent versions in golang.org/x/mod are used
instead. So this CL adds support to cmd/dist for vendor directories.

Updates #31761

Change-Id: Iab4cdc7e505069a8df296287d16fbaa871944955
Reviewed-on: https://go-review.googlesource.com/c/go/+/203537
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 29, 2019

Change https://golang.org/cl/204138 mentions this issue: sumdb: sync change from cmd/go internal package

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 29, 2019

Change https://golang.org/cl/204103 mentions this issue: cmd/go/internal/modfile: don't use cmd/internal/diff

gopherbot pushed a commit that referenced this issue Oct 29, 2019
This is a partial revert of CL 203218.

cmd/go/internal/modfile is about to be deleted and replaced with
golang.org/x/mod/modfile in CL 202698. cmd/internal/diff is not
visible from golang.org/x/mod/modfile, and it doesn't make sense to
extract it into a new package there.

Updates #31761

Change-Id: I3bbbc4cae81120020e1092c1138524729530b415
Reviewed-on: https://go-review.googlesource.com/c/go/+/204103
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit to golang/mod that referenced this issue Oct 29, 2019
Copy change in CL 203607 from cmd/go/internal/sumdb to
golang.org/x/mod/sumdb. This just removed an unused parameter. These
packages should be identical (except for imports) before
cmd/go/internal/sumdb is deleted in CL 202698.

Updates golang/go#31761

Change-Id: I1ce96d66bc71ef5c9515b68811049c47dce97765
Reviewed-on: https://go-review.googlesource.com/c/mod/+/204138
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Oct 29, 2019
This change deletes several internal packages, replaces imports to
them with the equivalent golang.org/x/mod packages, updates x/mod, and
re-runs 'go mod vendor'.

Packages are replaced as follows:

cmd/go/internal/modfile → golang.org/x/mod/modfile
cmd/go/internal/module → golang.org/x/mod/module
cmd/go/internal/semver → golang.org/x/mod/semver
cmd/go/internal/sumdb → golang.org/x/mod/sumdb
cmd/go/internal/dirhash → golang.org/x/mod/sumdb/dirhash
cmd/go/internal/note → golang.org/x/mod/sumdb/note
cmd/go/internal/tlog → golang.org/x/mod/sumdb/tlog

Updates #31761
Fixes #34924

Change-Id: Ie3bf677bb0be49af969f654a0214243a6547eb57
Reviewed-on: https://go-review.googlesource.com/c/go/+/202698
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet