Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: allow serving module under the subdirectory of git repository #34055

Open
nhooyr opened this issue Sep 4, 2019 · 26 comments
Open

Comments

@nhooyr
Copy link
Contributor

@nhooyr nhooyr commented Sep 4, 2019

If you head to https://github.com/nhooyr/websocket presently, you'll get blasted with a massive root directory listing, mostly due to all the Go files. It's obnoxious.

Compare that to https://github.com/nhooyr/websocket/tree/067b40e227d0d6af9e399c1efe0dd80efae1b79f where the Go module has been moved to the subdirectory ./mod inside the repository.
See nhooyr/websocket#136

So I want to move the Go module to the subdirectory ./mod inside the repository and serve that subdirectory for the nhooyr.io/websocket import path but it doesn't look like there is an easy way to do that.

The go-import meta tag only allows me to specify the import path of the repository, which in this case would make it nhooyr.io/websocket/mod which is nasty. I want to serve the subdirectory directly under nhooyr.io/websocket. Looks like I can do this with the new mod vcs to the go-import tag but then I have to run my own module server which I want to ideally avoid.

Is this something that would be considered or is already possible?

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Sep 4, 2019

For now I'm redirecting go get to https://github.com/nhooyr/websocket-mod where I've moved the module to the root of the repository. Unideal for sure but works well for now.

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Sep 4, 2019

I might also just switch to moving the real code into the subdir mod but have a package in the root that uses type aliases to alias directly to the mod subpkg. Would involve some duplicate code and be harder to maintain though.

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Sep 4, 2019

So unfortunately people were getting confused that the mod path didn't resolve directly to the root of the repo and I tried the type alias thing but in the docs it links directly to the other package instead of inlining the docs so I'd have a decent amount of duplication if I copied the types and translated between them.

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Sep 4, 2019

It's unclear what bug is being raised in this issue and/or what proposal to the go command is being suggested. I'm going to close this bug, but check out https://github.com/golang/go/wiki/Questions as it may provide a good avenue to continue this discussion.

@nhooyr
Copy link
Contributor Author

@nhooyr nhooyr commented Sep 5, 2019

Apologies for the confusion. To clarify, I'm trying to serve a subdir of my repository https://github.com/nhooyr/websocket under the import path nhooyr.io/websocket but this doesn't seem feasible with the current tooling without my own module proxy and event then GoDoc wouldn't work as it doesn't support modules yet.

@nejtr0n
Copy link

@nejtr0n nejtr0n commented Feb 25, 2020

I've got same issue. I have project with migrations, protobuf files and other infrastracture files.
So I decide to put my project under
https://vcs/vendor/project/src/.
But i could not import in go modules.

@jwinnie
Copy link

@jwinnie jwinnie commented Jul 21, 2020

I have the same issue. @katiehockman Please reopen, this is a legitimate issue that a lot of people are experiencing.

A brief proposal to fix the issue would be to add an optional fourth parameter to go-import, so if you wanted to specify a subdirectory you could do something like this:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would import the mod subdirectory of the github.com/nhooyr/websocket git repository.

@katiehockman katiehockman reopened this Jul 21, 2020
@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jul 21, 2020

@bcmills
Copy link
Member

@bcmills bcmills commented Jul 21, 2020

See previously #33562.

@scribnar
Copy link

@scribnar scribnar commented Sep 24, 2020

I have the same issue. @katiehockman Please reopen, this is a legitimate issue that a lot of people are experiencing.

A brief proposal to fix the issue would be to add an optional fourth parameter to go-import, so if you wanted to specify a subdirectory you could do something like this:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would import the mod subdirectory of the github.com/nhooyr/websocket git repository.

The subdirectory rule is not working. Does anyone find it working with "go version go1.13.6 darwin/amd64"?

Thanks

@fishy
Copy link

@fishy fishy commented Jan 20, 2021

We (Apache Thrift) are bitten by this issue as well.

For Apache Thrift project, the only part that would be imported by others is github.com/apache/thrift/lib/go/thrift. If we put the go.mod file under lib/go/thrift, the importing still works, but that comes with caveats for users importing this library:

  1. They have to import github.com/apache/thrift/lib/go/thrift instead of github.com/apache/thrift in their go.mod files
  2. They no longer get the git tagged versions. The version of github.com/apache/thrift/lib/go/thrift in their go.mod files will always be v0.0.0-timestamp-commitsha1.

If we move go.mod file to the top level, that comes with the problem that we have test code running in the repo in other directories that would affect the now shared go.mod file, causing more problems (the most notable one is that people only work on the library code cannot run go mod tidy because some of the go code used by other test code in the repo are generated by thrift compiler and they need to run the whole automake process to generate them).

@rsc rsc changed the title cmd/go: allow serving module under the subdirectory of git repository proposal: cmd/go: allow serving module under the subdirectory of git repository Feb 2, 2021
@gopherbot gopherbot added the Proposal label Feb 2, 2021
@rsc rsc added this to Incoming in Proposals Feb 2, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Feb 10, 2021

Adding to proposal minutes. The basic problem is to be able to use a meta tag to redirect not just to a Git repo but to a subdirectory within a Git repo where the module root lives. That could be an extra field in the meta tag form, but of course it would not be safe to use until the change trickled to older versions of Go (or you assume those older versions are all using the proxy).

@rsc rsc moved this from Incoming to Active in Proposals Feb 10, 2021
@rsc
Copy link
Contributor

@rsc rsc commented Feb 10, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Feb 11, 2021

For Apache Thrift project, the only part that would be imported by others is github.com/apache/thrift/lib/go/thrift. If we put the go.mod file under lib/go/thrift, the importing still works, but that comes with caveats for users importing this library:

  1. They have to import github.com/apache/thrift/lib/go/thrift instead of github.com/apache/thrift in their go.mod files

Note that this proposal wouldn't help with github.com paths, as those don't use go-import metadata.

  1. They no longer get the git tagged versions. The version of github.com/apache/thrift/lib/go/thrift in their go.mod files will always be v0.0.0-timestamp-commitsha1.

You can make tags for the module by calling the tags lib/go/thrift/v1.2.3. Not ideal, just FYI.

See https://golang.org/ref/mod#vcs-version.


In general, I support this proposal. When using go-import metadata there is already indirection between the import path and where the source is stored, and not requiring it to be at the root of a repository makes sense to me.

A question is what the applicable tags would be. I think people would expect and want unprefixed tags to work judging from the described use cases, but since different modules might point to different subdirectories, prefixed tags would probably be more consistent.

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 11, 2021

since different modules might point to different subdirectories, prefixed tags would probably be more consistent.

I agree. I think prefixed tags are the only viable option.

@fishy
Copy link

@fishy fishy commented Feb 11, 2021

While I can see the prefixed tags could be useful in some use cases, for Apache Thrift we would really prefer to be able to use the global git tag.

The reason is that for Apache Thrift the releases are always global: it includes the compiler, and libraries for all the supported languages. So for go library, people really need to use the matching version with the compiler (e.g. always use v0.14.0 compiler with v0.14.0 go library). If we only support prefixed tags, that means whenever we do a release we would need to tag it twice, once with v0.14.0 and again with lib/go/thrift/v0.14.0.

May I suggest that we use the global tag as a "fallback"? e.g. if there's no prefixed tags matching the go.mod path, fallback to use the global tag instead? I can see there could be a potential issue of a project used to not use prefixed tags, so we fallback to global tag, then at some point they started to use prefixed tags, and then we could have two, conflicting v1.0.0. For that issue, I think maybe we can add a suffix to the fallback global tag, similar to how we auto add -incompatible suffix. The suffix could be something like -global, -repo, etc. (so when we use fallback tag, the version tag in user's go.mod file would be v0.14.0-global or something like that).

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 11, 2021

@fishy, fallbacks are not viable because they could result in time-dependent meanings of versions for a given repo.

If we start out with only a v0.14.0 tag, then users who require … v0.14.0 will use that tag. But then what happens if a lib/go/thrift/v0.14.0 tag is added, on a different commit, down the road? Proxies will continue to cache and serve the original v0.14.0, but new users with GOPROXY=direct will instead see the nested tag and get checksum errors.

Given that it's straightforward to for module authors to duplicate tags with prefixes if desired, it seems better to avoid introducing that ambiguity.

@fishy
Copy link

@fishy fishy commented Feb 11, 2021

@bcmills did you see the part that I suggested to add suffix for fallback global tags to resolve that issue?

tag it twice is "trivial", sure, but it's also some special thing we need to do for go library. thrift is a project supporting tens of languages, and no other language requires this special treatment like go does (if we move forward to only support prefixed tags).

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Feb 11, 2021

If we start out with only a v0.14.0 tag, then users who require … v0.14.0 will use that tag. But then what happens if a lib/go/thrift/v0.14.0 tag is added, on a different commit, down the road? Proxies will continue to cache and serve the original v0.14.0, but new users with GOPROXY=direct will instead see the nested tag and get checksum errors.

I am not saying I like fallbacks, but this is not actually that much of a blocker. git tags can change, so we can treat adding a prefixed tag for an existing unprefixed tag the same way we treat changing a tag: it's a mistake and it will be blocked by the Checksum Database.

A bigger problem is unprefixed tags leaking into all modules which might not be intended, and inconsistency with the current behavior of modules in subdirectories which we can't change.

@fishy
Copy link

@fishy fishy commented Feb 11, 2021

A bigger problem is unprefixed tags leaking into all modules which might not be intended, and inconsistency with the current behavior of modules in subdirectories which we can't change.

I think an auto added suffix can mitigate that issue to some degree?

@rsc
Copy link
Contributor

@rsc rsc commented Feb 17, 2021

The original proposal was about being able to make a custom domain like nhooyr.io/websocket be able to redirect to a subdirectory of a Git repo. That's still something we can do and it doesn't seem too bad to do it.
Thoughts, @bcmills, @jayconrod, @matloob?

It is definitely the case that we can't make github.com/apache/thrift somehow redirect to github.com/apache/thrift/lib/go/thrift, both due to ambiguity issues and because there is no control over that rule. (Maybe if GitHub let you edit the <meta> tag we could relax the rule and respect the meta tag, but right now the hard-coded rule in cmd/go matches GitHub's hard-coded <meta> tag.)

But is the custom domain version still worth doing?

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Feb 17, 2021

This seems feasible. Two thoughts about remaining issues.

First, what should the <meta> tag look like? Adding another space-separated field was suggested earlier:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

This would work: the go command ignores tags if they don't have exactly three fields, so older versions of Go would act as if the meta tag weren't present at all. They wouldn't be able to handle the subdirectory anyway, so that's fine. golang.org/x/tools/go/vcs would need to be updated, too.

Second, what should the version tags look like. I agree with @bcmills: we should avoid letting multiple tags represent the same version; it leads to too much confusion. #32945 also introduces to this problem, and if we accepted both, we'd have four possible tags for each version. I'm not really convinced that creating a Go-specific tag is that much of a burden; I'd expect each language would have its own idiosyncrasies, like creating and uploading a release artifact.

We already have the convention that if a module is rooted in a subdirectory of a repository, its version tags must have that subdirectory as a prefix (not including a major version subdirectory if any). So for the module above, the tags would be:

mod/v1.0.0
mod/v2.0.0

If there were another module nhooyr.io/websocket/example in a subdirectory mod/example, those tags would be:

mod/example/v1.0.0
mod/example/v2.0.0
@antichris
Copy link

@antichris antichris commented Feb 18, 2021

Feel free to yell and throw chairs at me:

  • what if some form of aliasing "magic" go.mod could be devised, one that could be placed at the root of the repository to point at the actual root of the module (and/or its actual go.mod), similar to how the replace directive already works for 3rd party modules?
@sturfee-petrl
Copy link

@sturfee-petrl sturfee-petrl commented Feb 23, 2021

I went through all messages :neckbeard:.

I would like to clarify that this issue points to this scenario

  1. We follow golang "standard" layout
  • packages are located under pkg
  • services are located under cmd
  1. With years we have a lot of modules under pkg and some of them are used very extensively in our ecosystem.
    So the versioning of submodules started to conflict with versions of services cmd as git's tags also are used for versions in our build system and as tags of docker containers.
  2. We decided to move most general packages to the separate repository. Till it's clear that one repository for one module would be ideal. We don't want to have 30 additional repositories 😳. From other side to keep one go.mod at the root is totally ok but doesn't seem right .

P.S.

While writing this I was decide that to have the go.mod in the root is not incorrect.
I don't see any benefits with go.mod in every subdirectory in this scenario except of the inner satisfaction 😌 .

P.S.S

Actually if you import one package from the repository with root go.mod then the go.sum of package importer will import all dependencies from imported package. Maybe it's negligible till it can't affect on nothing except of build time

@rsc
Copy link
Contributor

@rsc rsc commented Feb 24, 2021

We follow golang "standard" layout

Apologies, but this is not actually a standard layout. If you look at any of the github.com/golang repos, they have no "pkg" directory. The link https://github.com/golang-standards/project-layout is in the github.com/golang-standards org, which I've never heard of before today. (We do tend to put commands in cmd/, but that's not relevant here.)

It's fine if you want to put Go code in pkg/. It just means that pkg will appear in your import path.

Moving general packages to a separate repo sounds like a reasonable solution.

@rsc
Copy link
Contributor

@rsc rsc commented Feb 24, 2021

OK, so it sounds like people are on board with adding a new space-separated field in the metadata, and then having the module version tag include the subdirectory name. So:

<meta name="go-import" content="nhooyr.io/websocket git https://github.com/nhooyr/websocket mod">

tagged in the repo as mod/v1.0.0. That is, go get nhooyr.io/websocket@v1.0.0 would fetch the same code you'd get from go get github.com/nhooyr/websocket/mod@v1.0.0 (but fetching the latter would reject because the go.mod would say module nhooyr.io/websocket).

Do I have that right?
Does anyone object to this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet