cmd/go: allow serving module under the subdirectory of git repository

[nhooyr]

If you head to https://github.com/nhooyr/websocket presently, you'll get blasted with a massive root directory listing, mostly due to all the Go files. It's obnoxious.

Compare that to https://github.com/nhooyr/websocket/tree/067b40e227d0d6af9e399c1efe0dd80efae1b79f where the Go module has been moved to the subdirectory ./mod inside the repository.
See nhooyr/websocket#136

So I want to move the Go module to the subdirectory ./mod inside the repository and serve that subdirectory for the nhooyr.io/websocket import path but it doesn't look like there is an easy way to do that.

The go-import meta tag only allows me to specify the import path of the repository, which in this case would make it nhooyr.io/websocket/mod which is nasty. I want to serve the subdirectory directly under nhooyr.io/websocket. Looks like I can do this with the new mod vcs to the go-import tag but then I have to run my own module server which I want to ideally avoid.

Is this something that would be considered or is already possible?

[nhooyr]

For now I'm redirecting go get to https://github.com/nhooyr/websocket-mod where I've moved the module to the root of the repository. Unideal for sure but works well for now.

[nhooyr]

I might also just switch to moving the real code into the subdir mod but have a package in the root that uses type aliases to alias directly to the mod subpkg. Would involve some duplicate code and be harder to maintain though.

[nhooyr]

So unfortunately people were getting confused that the mod path didn't resolve directly to the root of the repo and I tried the type alias thing but in the docs it links directly to the other package instead of inlining the docs so I'd have a decent amount of duplication if I copied the types and translated between them.

[katiehockman]

It's unclear what bug is being raised in this issue and/or what proposal to the go command is being suggested. I'm going to close this bug, but check out https://github.com/golang/go/wiki/Questions as it may provide a good avenue to continue this discussion.

[nhooyr]

Apologies for the confusion. To clarify, I'm trying to serve a subdir of my repository https://github.com/nhooyr/websocket under the import path nhooyr.io/websocket but this doesn't seem feasible with the current tooling without my own module proxy and event then GoDoc wouldn't work as it doesn't support modules yet.

[nejtr0n]

I've got same issue. I have project with migrations, protobuf files and other infrastracture files.
So I decide to put my project under
https://vcs/vendor/project/src/.
But i could not import in go modules.

[jepotter1-archive]

I have the same issue. @katiehockman Please reopen, this is a legitimate issue that a lot of people are experiencing.

A brief proposal to fix the issue would be to add an optional fourth parameter to go-import, so if you wanted to specify a subdirectory you could do something like this:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would import the mod subdirectory of the github.com/nhooyr/websocket git repository.

[katiehockman]

/cc @bcmills @jayconrod

[bcmills]

See previously #33562.

[scribnar]

I have the same issue. @katiehockman Please reopen, this is a legitimate issue that a lot of people are experiencing.

A brief proposal to fix the issue would be to add an optional fourth parameter to go-import, so if you wanted to specify a subdirectory you could do something like this:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would import the mod subdirectory of the github.com/nhooyr/websocket git repository.

The subdirectory rule is not working. Does anyone find it working with "go version go1.13.6 darwin/amd64"?

Thanks

[fishy]

We (Apache Thrift) are bitten by this issue as well.

For Apache Thrift project, the only part that would be imported by others is github.com/apache/thrift/lib/go/thrift. If we put the go.mod file under lib/go/thrift, the importing still works, but that comes with caveats for users importing this library:

1. They have to import github.com/apache/thrift/lib/go/thrift instead of github.com/apache/thrift in their go.mod files
2. They no longer get the git tagged versions. The version of github.com/apache/thrift/lib/go/thrift in their go.mod files will always be v0.0.0-timestamp-commitsha1.

If we move go.mod file to the top level, that comes with the problem that we have test code running in the repo in other directories that would affect the now shared go.mod file, causing more problems (the most notable one is that people only work on the library code cannot run go mod tidy because some of the go code used by other test code in the repo are generated by thrift compiler and they need to run the whole automake process to generate them).

[rsc]

Adding to proposal minutes. The basic problem is to be able to use a meta tag to redirect not just to a Git repo but to a subdirectory within a Git repo where the module root lives. That could be an extra field in the meta tag form, but of course it would not be safe to use until the change trickled to older versions of Go (or you assume those older versions are all using the proxy).

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[FiloSottile]

For Apache Thrift project, the only part that would be imported by others is github.com/apache/thrift/lib/go/thrift. If we put the go.mod file under lib/go/thrift, the importing still works, but that comes with caveats for users importing this library:

1. They have to import github.com/apache/thrift/lib/go/thrift instead of github.com/apache/thrift in their go.mod files

Note that this proposal wouldn't help with github.com paths, as those don't use go-import metadata.

2. They no longer get the git tagged versions. The version of github.com/apache/thrift/lib/go/thrift in their go.mod files will always be v0.0.0-timestamp-commitsha1.

You can make tags for the module by calling the tags lib/go/thrift/v1.2.3. Not ideal, just FYI.

See https://golang.org/ref/mod#vcs-version.

---

In general, I support this proposal. When using go-import metadata there is already indirection between the import path and where the source is stored, and not requiring it to be at the root of a repository makes sense to me.

A question is what the applicable tags would be. I think people would expect and want unprefixed tags to work judging from the described use cases, but since different modules might point to different subdirectories, prefixed tags would probably be more consistent.

[bcmills]

since different modules might point to different subdirectories, prefixed tags would probably be more consistent.

I agree. I think prefixed tags are the only viable option.

[fishy]

While I can see the prefixed tags could be useful in some use cases, for Apache Thrift we would really prefer to be able to use the global git tag.

The reason is that for Apache Thrift the releases are always global: it includes the compiler, and libraries for all the supported languages. So for go library, people really need to use the matching version with the compiler (e.g. always use v0.14.0 compiler with v0.14.0 go library). If we only support prefixed tags, that means whenever we do a release we would need to tag it twice, once with v0.14.0 and again with lib/go/thrift/v0.14.0.

May I suggest that we use the global tag as a "fallback"? e.g. if there's no prefixed tags matching the go.mod path, fallback to use the global tag instead? I can see there could be a potential issue of a project used to not use prefixed tags, so we fallback to global tag, then at some point they started to use prefixed tags, and then we could have two, conflicting v1.0.0. For that issue, I think maybe we can add a suffix to the fallback global tag, similar to how we auto add -incompatible suffix. The suffix could be something like -global, -repo, etc. (so when we use fallback tag, the version tag in user's go.mod file would be v0.14.0-global or something like that).

[bcmills]

@fishy, fallbacks are not viable because they could result in time-dependent meanings of versions for a given repo.

If we start out with only a v0.14.0 tag, then users who require … v0.14.0 will use that tag. But then what happens if a lib/go/thrift/v0.14.0 tag is added, on a different commit, down the road? Proxies will continue to cache and serve the original v0.14.0, but new users with GOPROXY=direct will instead see the nested tag and get checksum errors.

Given that it's straightforward to for module authors to duplicate tags with prefixes if desired, it seems better to avoid introducing that ambiguity.

[aykevl]

Is someone working on this, or is it scheduled?
(I'm not trying to demand an ETA here, just curious. I might try to implement it myself otherwise).

[bcmills]

No one is working on this at the moment, and the freeze is coming up soon. (I think we should try to address it in 1.19, and you're welcome to make a run at an implementation.)

[janpfeifer]

Any news on this proposal ?

I don't see it reflected in the go.dev/ref/mod#vcs-version so I'm assuming not yet, right ?

[bcmills]

No update at the moment, but I'm planning to do some work on the git module fetch path for #47308 and #56881. I'll keep an eye out to see whether I can fit this in easily too.