proposal: cmd/go: allow serving module under the subdirectory of git repository

[nhooyr]

If you head to https://github.com/nhooyr/websocket presently, you'll get blasted with a massive root directory listing, mostly due to all the Go files. It's obnoxious.

Compare that to https://github.com/nhooyr/websocket/tree/067b40e227d0d6af9e399c1efe0dd80efae1b79f where the Go module has been moved to the subdirectory ./mod inside the repository.
See nhooyr/websocket#136

So I want to move the Go module to the subdirectory ./mod inside the repository and serve that subdirectory for the nhooyr.io/websocket import path but it doesn't look like there is an easy way to do that.

The go-import meta tag only allows me to specify the import path of the repository, which in this case would make it nhooyr.io/websocket/mod which is nasty. I want to serve the subdirectory directly under nhooyr.io/websocket. Looks like I can do this with the new mod vcs to the go-import tag but then I have to run my own module server which I want to ideally avoid.

Is this something that would be considered or is already possible?

[nhooyr]

For now I'm redirecting go get to https://github.com/nhooyr/websocket-mod where I've moved the module to the root of the repository. Unideal for sure but works well for now.

[nhooyr]

I might also just switch to moving the real code into the subdir mod but have a package in the root that uses type aliases to alias directly to the mod subpkg. Would involve some duplicate code and be harder to maintain though.

[nhooyr]

So unfortunately people were getting confused that the mod path didn't resolve directly to the root of the repo and I tried the type alias thing but in the docs it links directly to the other package instead of inlining the docs so I'd have a decent amount of duplication if I copied the types and translated between them.

[katiehockman]

It's unclear what bug is being raised in this issue and/or what proposal to the go command is being suggested. I'm going to close this bug, but check out https://github.com/golang/go/wiki/Questions as it may provide a good avenue to continue this discussion.

[nhooyr]

Apologies for the confusion. To clarify, I'm trying to serve a subdir of my repository https://github.com/nhooyr/websocket under the import path nhooyr.io/websocket but this doesn't seem feasible with the current tooling without my own module proxy and event then GoDoc wouldn't work as it doesn't support modules yet.

[nejtr0n]

I've got same issue. I have project with migrations, protobuf files and other infrastracture files.
So I decide to put my project under
https://vcs/vendor/project/src/.
But i could not import in go modules.

[jwinnie]

I have the same issue. @katiehockman Please reopen, this is a legitimate issue that a lot of people are experiencing.

A brief proposal to fix the issue would be to add an optional fourth parameter to go-import, so if you wanted to specify a subdirectory you could do something like this:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would import the mod subdirectory of the github.com/nhooyr/websocket git repository.

[katiehockman]

/cc @bcmills @jayconrod

[bcmills]

See previously #33562.

[scribnar]

I have the same issue. @katiehockman Please reopen, this is a legitimate issue that a lot of people are experiencing.

A brief proposal to fix the issue would be to add an optional fourth parameter to go-import, so if you wanted to specify a subdirectory you could do something like this:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would import the mod subdirectory of the github.com/nhooyr/websocket git repository.

The subdirectory rule is not working. Does anyone find it working with "go version go1.13.6 darwin/amd64"?

Thanks

[fishy]

We (Apache Thrift) are bitten by this issue as well.

For Apache Thrift project, the only part that would be imported by others is github.com/apache/thrift/lib/go/thrift. If we put the go.mod file under lib/go/thrift, the importing still works, but that comes with caveats for users importing this library:

1. They have to import github.com/apache/thrift/lib/go/thrift instead of github.com/apache/thrift in their go.mod files
2. They no longer get the git tagged versions. The version of github.com/apache/thrift/lib/go/thrift in their go.mod files will always be v0.0.0-timestamp-commitsha1.

If we move go.mod file to the top level, that comes with the problem that we have test code running in the repo in other directories that would affect the now shared go.mod file, causing more problems (the most notable one is that people only work on the library code cannot run go mod tidy because some of the go code used by other test code in the repo are generated by thrift compiler and they need to run the whole automake process to generate them).

[rsc]

Adding to proposal minutes. The basic problem is to be able to use a meta tag to redirect not just to a Git repo but to a subdirectory within a Git repo where the module root lives. That could be an extra field in the meta tag form, but of course it would not be safe to use until the change trickled to older versions of Go (or you assume those older versions are all using the proxy).

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[FiloSottile]

For Apache Thrift project, the only part that would be imported by others is github.com/apache/thrift/lib/go/thrift. If we put the go.mod file under lib/go/thrift, the importing still works, but that comes with caveats for users importing this library:

1. They have to import github.com/apache/thrift/lib/go/thrift instead of github.com/apache/thrift in their go.mod files

Note that this proposal wouldn't help with github.com paths, as those don't use go-import metadata.

2. They no longer get the git tagged versions. The version of github.com/apache/thrift/lib/go/thrift in their go.mod files will always be v0.0.0-timestamp-commitsha1.

You can make tags for the module by calling the tags lib/go/thrift/v1.2.3. Not ideal, just FYI.

See https://golang.org/ref/mod#vcs-version.

---

In general, I support this proposal. When using go-import metadata there is already indirection between the import path and where the source is stored, and not requiring it to be at the root of a repository makes sense to me.

A question is what the applicable tags would be. I think people would expect and want unprefixed tags to work judging from the described use cases, but since different modules might point to different subdirectories, prefixed tags would probably be more consistent.

[bcmills]

since different modules might point to different subdirectories, prefixed tags would probably be more consistent.

I agree. I think prefixed tags are the only viable option.

[fishy]

While I can see the prefixed tags could be useful in some use cases, for Apache Thrift we would really prefer to be able to use the global git tag.

The reason is that for Apache Thrift the releases are always global: it includes the compiler, and libraries for all the supported languages. So for go library, people really need to use the matching version with the compiler (e.g. always use v0.14.0 compiler with v0.14.0 go library). If we only support prefixed tags, that means whenever we do a release we would need to tag it twice, once with v0.14.0 and again with lib/go/thrift/v0.14.0.

May I suggest that we use the global tag as a "fallback"? e.g. if there's no prefixed tags matching the go.mod path, fallback to use the global tag instead? I can see there could be a potential issue of a project used to not use prefixed tags, so we fallback to global tag, then at some point they started to use prefixed tags, and then we could have two, conflicting v1.0.0. For that issue, I think maybe we can add a suffix to the fallback global tag, similar to how we auto add -incompatible suffix. The suffix could be something like -global, -repo, etc. (so when we use fallback tag, the version tag in user's go.mod file would be v0.14.0-global or something like that).

[bcmills]

@fishy, fallbacks are not viable because they could result in time-dependent meanings of versions for a given repo.

If we start out with only a v0.14.0 tag, then users who require … v0.14.0 will use that tag. But then what happens if a lib/go/thrift/v0.14.0 tag is added, on a different commit, down the road? Proxies will continue to cache and serve the original v0.14.0, but new users with GOPROXY=direct will instead see the nested tag and get checksum errors.

Given that it's straightforward to for module authors to duplicate tags with prefixes if desired, it seems better to avoid introducing that ambiguity.

[fishy]

@bcmills did you see the part that I suggested to add suffix for fallback global tags to resolve that issue?

tag it twice is "trivial", sure, but it's also some special thing we need to do for go library. thrift is a project supporting tens of languages, and no other language requires this special treatment like go does (if we move forward to only support prefixed tags).

[FiloSottile]

If we start out with only a v0.14.0 tag, then users who require … v0.14.0 will use that tag. But then what happens if a lib/go/thrift/v0.14.0 tag is added, on a different commit, down the road? Proxies will continue to cache and serve the original v0.14.0, but new users with GOPROXY=direct will instead see the nested tag and get checksum errors.

I am not saying I like fallbacks, but this is not actually that much of a blocker. git tags can change, so we can treat adding a prefixed tag for an existing unprefixed tag the same way we treat changing a tag: it's a mistake and it will be blocked by the Checksum Database.

A bigger problem is unprefixed tags leaking into all modules which might not be intended, and inconsistency with the current behavior of modules in subdirectories which we can't change.

[fishy]

A bigger problem is unprefixed tags leaking into all modules which might not be intended, and inconsistency with the current behavior of modules in subdirectories which we can't change.

I think an auto added suffix can mitigate that issue to some degree?

[rsc]

The original proposal was about being able to make a custom domain like nhooyr.io/websocket be able to redirect to a subdirectory of a Git repo. That's still something we can do and it doesn't seem too bad to do it.
Thoughts, @bcmills, @jayconrod, @matloob?

It is definitely the case that we can't make github.com/apache/thrift somehow redirect to github.com/apache/thrift/lib/go/thrift, both due to ambiguity issues and because there is no control over that rule. (Maybe if GitHub let you edit the <meta> tag we could relax the rule and respect the meta tag, but right now the hard-coded rule in cmd/go matches GitHub's hard-coded <meta> tag.)

But is the custom domain version still worth doing?

[jayconrod]

This seems feasible. Two thoughts about remaining issues.

First, what should the <meta> tag look like? Adding another space-separated field was suggested earlier:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would work: the go command ignores tags if they don't have exactly three fields, so older versions of Go would act as if the meta tag weren't present at all. They wouldn't be able to handle the subdirectory anyway, so that's fine. golang.org/x/tools/go/vcs would need to be updated, too.

Second, what should the version tags look like. I agree with @bcmills: we should avoid letting multiple tags represent the same version; it leads to too much confusion. #32945 also introduces to this problem, and if we accepted both, we'd have four possible tags for each version. I'm not really convinced that creating a Go-specific tag is that much of a burden; I'd expect each language would have its own idiosyncrasies, like creating and uploading a release artifact.

We already have the convention that if a module is rooted in a subdirectory of a repository, its version tags must have that subdirectory as a prefix (not including a major version subdirectory if any). So for the module above, the tags would be:

mod/v1.0.0
mod/v2.0.0

If there were another module nhooyr.io/websocket/example in a subdirectory mod/example, those tags would be:

mod/example/v1.0.0
mod/example/v2.0.0

[antichris]

Feel free to yell and throw chairs at me:

• what if some form of aliasing "magic" go.mod could be devised, one that could be placed at the root of the repository to point at the actual root of the module (and/or its actual go.mod), similar to how the replace directive already works for 3rd party modules?

[sturfee-petrl]

I went through all messages .

I would like to clarify that this issue points to this scenario

1. We follow golang "standard" layout

• packages are located under pkg
• services are located under cmd

2. With years we have a lot of modules under pkg and some of them are used very extensively in our ecosystem.
   So the versioning of submodules started to conflict with versions of services cmd as git's tags also are used for versions in our build system and as tags of docker containers.
3. We decided to move most general packages to the separate repository. Till it's clear that one repository for one module would be ideal. We don't want to have 30 additional repositories 😳. From other side to keep one go.mod at the root is totally ok but doesn't seem right .

P.S.

While writing this I was decide that to have the go.mod in the root is not incorrect.
I don't see any benefits with go.mod in every subdirectory in this scenario except of the inner satisfaction 😌 .

P.S.S

Actually if you import one package from the repository with root go.mod then the go.sum of package importer will import all ❗ dependencies from imported package. Maybe it's negligible till it can't affect on nothing except of build time

[rsc]

We follow golang "standard" layout

Apologies, but this is not actually a standard layout. If you look at any of the github.com/golang repos, they have no "pkg" directory. The link https://github.com/golang-standards/project-layout is in the github.com/golang-standards org, which I've never heard of before today. (We do tend to put commands in cmd/, but that's not relevant here.)

It's fine if you want to put Go code in pkg/. It just means that pkg will appear in your import path.

Moving general packages to a separate repo sounds like a reasonable solution.

[rsc]

OK, so it sounds like people are on board with adding a new space-separated field in the metadata, and then having the module version tag include the subdirectory name. So:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

tagged in the repo as mod/v1.0.0. That is, go get nhooyr.io/websocket@v1.0.0 would fetch the same code you'd get from go get github.com/nhooyr/websocket/mod@v1.0.0 (but fetching the latter would reject because the go.mod would say module nhooyr.io/websocket).

Do I have that right?
Does anyone object to this?