Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: reserve specific path prefixes for local (user-defined) modules #37641

Open
bcmills opened this issue Mar 3, 2020 · 18 comments
Open
Labels
Projects
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 3, 2020

The go command automatically fetches modules from network hosts, provided that those modules have a path whose first element includes a dot character (.).

First-elements that do not include a dot are, in general, reserved for the standard library and the Go toolchain (see #32819). Some names (std, cmd, all) have a special meaning no matter where they are used. Others (such as archive and debug) are currently standard-library prefixes but not packages, but could be made so in the future (as was done in the case of encoding).

Still others currently have no meaning, but may be assigned a special meaning in the future: for example, I would like to add mod to refer to the main module independent of its path (that would be a separate proposal).

The fact that import paths are required to contain a dot occasionally causes user consternation (#37554, #34592).

I propose that we explicitly reserve the following package paths for user code:

  • for general use:
    • local
      • But not localhost, which we may at some point want to treat as a hostname.
  • for documentation, tutorials, and bug reports:
    • example (along the lines of example.com, because it looks more like “something you should replace” than local does)
  • for tests, especially tests of tools that manipulate modules (and thus need a “clean” module to run in):
  • for module-graph and package-import-graph examples:
    • main (because it is useful to indicate "the main module", and a package main is not importable anyway), and
    • any valid module path that consists of a single letter, because we already use those a lot in cmd/go tests.

CC @bronze1man @DisposaBoy @mvdan @jayconrod @matloob @thepudds

@gopherbot gopherbot added this to the Proposal milestone Mar 3, 2020
@gopherbot gopherbot added the Proposal label Mar 3, 2020
@bcmills bcmills added this to Incoming in Proposals Mar 3, 2020
@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Mar 3, 2020

Wouldn't main fall in the same category as, or perhaps be confused with, mod?

I'm overall in favor of this proposal. I seem to recall that @rogpeppe uses 0.0.0.0 as a module name, because it's a valid hostname and it contains dots, yet it can never be used to reach anything on the internet as it's not a usable IP. However, I have to admit that I too prefer using human-readable names like example or test, and I too have been doing it in tests.

From the tooling perspective, these rules are easy to implement, so that doesn't worry me either. The rules should be well documented, though, and probably alongside #32819.

Do you reckon that any sort of API would be useful for tools here? Similar to https://golang.org/pkg/go/token/#IsKeyword, imagine something like x/mod.IsLocal(path string) bool and x/mod.IsReserved(path string) bool. Their implementations would be pretty trivial, but the big advantage is that if we ever change the rules, we could easily bring the entire ecosystem in sync.

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

Wouldn't main fall in the same category as, or perhaps be confused with, mod?

Maybe? I'd be fine with omitting it from the reserved names.

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

Do you reckon that any sort of API would be useful for tools here?

Maybe? I'm not sure who would use them, though: I don't think we should explicitly reject other dotless paths at this point (we can break them if/when we need the name for something else, but if users are confident we'll never use their name I see little harm in letting it persist).

I guess it could be useful for linters..?

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

(search.IsMetaPackage is what we currently use to enumerate the actually-meaningful non-package paths, but I don't think it covers builtin which is also somewhat special.)

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Mar 4, 2020

I'll point out a cost of implementing this change, since I don't see it mentioned here so far, is that this will increase the amount of special cases. Special cases need to be documented, taught and learned by all Go users. Users who are not aware of the special cases and run into them accidentally may be surprised.

I think it's a good idea to evaluate how often each special case is expected to be used. For example, the package name main is a special case, and it is used very often, and adds more value than cost. The package name documentation is another special case, but it is used very infrequently, and has a higher cost than value (it was kept because of backwards compatibility reasons).

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Mar 4, 2020

What would it mean to reserve these paths? I'm guessing a documented promise to never create std packages with those prefixes?

Should we allow dotless paths using reserved prefixes in the GOPROXY protocol?

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

What would it mean to reserve these paths? I'm guessing a documented promise to never create std packages with those prefixes?

Yep. Specifically, we would apply the Go 1 compatibility policy to the non-existence of these paths in std.

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

Should we allow dotless paths using reserved prefixes in the GOPROXY protocol?

Not at this time. (We could consider it as a future change, but since these paths explicitly do not have a single source of truth, I don't think they'd be a good fit.)

@rsc rsc moved this from Incoming to Active in Proposals Mar 4, 2020
@rsc rsc changed the title proposal: all: reserve specific path prefixes for local (user-defined) modules proposal: cmd/go: reserve specific path prefixes for local (user-defined) modules Mar 4, 2020
@slrz

This comment has been minimized.

Copy link

@slrz slrz commented Mar 4, 2020

  * `example` (along the lines of `example.com`, because it looks more like “something you should replace” than `local` does)

What about jusing using example.com and the other existing DNS zones (example, invalid, test) reserved for test and documentation usage? If we already rely on the DNS, why not use its existing conventions for reserved names?

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Mar 4, 2020

@slrz What you're suggesting is already possible, it does need any work to be completed. Or are you suggesting that as a reason this proposal should not be accepted?

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

@slrz, IANA made the dubious decision to host an actual server on https://example.com.

While I do use those domains in examples myself, I could reasonably understand someone wanting to avoid them on security grounds: if those servers are ever compromised, they could serve go-import tags that point to malicious code.

@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Mar 5, 2020

Please leave reserve vendor for GOPATH , and remove the GOROOT/src/vendor .
I need to change the go source code when i want to upgrade golang version to make my private project works since version 1.13 (#34068 (comment))

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 6, 2020

@bronze1man, vendor has had a special meaning since Go 1.6 (https://golang.org/doc/go1.6#go_command). The ship sailed on reserving that particular prefix a long time ago.

@thepudds

This comment has been minimized.

Copy link

@thepudds thepudds commented Mar 6, 2020

Hi @bcmills, overall, I like the proposal.

Do you think the main use cases could be addressed by picking just one?

For example, maybe just pick local, or some other singular choice? That might help keep the number of special cases smaller.

People could still then use hierarchy underneath local to perhaps address the other ones you listed (e.g., local/test, local/example, local/main, or local/foo and local/bar if desired).

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Mar 6, 2020

@thepudds, just local could suffice. I suggested the others as well in the interest of maintaining semantic appropriateness.

For example, an “example” module in a tutorial is not really “local”. Someone following the tutorial might not understand local as a keyword that needs to be replaced with a non-local path prefix (or, really, any prefix that is not literally local), whereas the word example more clearly indicates a placeholder to be substituted by the reader.

@bronze1man

This comment has been minimized.

Copy link
Contributor

@bronze1man bronze1man commented Mar 7, 2020

Hi @bcmills, I like the proposal.
I think local and all single letter should be suffice.
Please do not make those reserve path prefixes can be relative resolve like vendor or ./x , it will make it difficult to find the actually package location.
GOPATH/src/local/ep and GOPATH/src/m/tmp and GOPATH/src/c/tmp looks local for me. And a single letter should be easy for different usage to end users.
Like:

  • GOPATH/src/m for fast execute package from command line.
  • GOPATH/src/c for project level config
  • GOPATH/src/t for project level test
  • GOPATH/src/l for private library only for this project.
@mewmew

This comment has been minimized.

Copy link
Contributor

@mewmew mewmew commented Mar 18, 2020

Just as a data point, I often use foo as a temporary local module path.

$ mkdir /tmp/foo
$ cd /tmp/foo
$ go mod init foo
$ go get github.com/llir/llvm/ir
$ cat go.mod 
module foo

go 1.14

require github.com/llir/llvm v0.3.0 // indirect

If prefixes were to be reserved, I'd echo @dmitshur's reflection that these special cases would need to be documented, and having many special cases increase the learning cost.

As such having a very clear rule, e.g. local can be use by whoever, where ever (with arbitrary sub-directories; local/foo). All other module paths where the first element does not contain a . character are reserved for the Go standard library/project.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Apr 8, 2020

Let's officially reserve example and test and stop there.
That is, we would commit that we will never add $GOROOT/src/{example,test}
to the main tree: there will not be packages with those names, nor packages
in subdirectories of those names.

Let's not do the single-letter thing. It's fine for us to do that ourselves,
and I certainly do 'echo module m >go.mod' every time I want to play with something,
but internal tests and ephemeral things don't need a long-term guarantee.

For things that need a long-term guarantee, it is reasonable to ask people to
use longer names, and I think example and test pretty much cover the use cases.

I don't understand the need for local, or else I disagree that it's a good use.
For a long-term use, it's better to use a domain name or other prefix you control.
For example inside Google all the paths begin with "google3/" (long story).
It's not a domain name, but it's also not a name that is going to end up in the
standard library and cause a conflict. Most company names have the same property,
so if you wanted to avoid using a domain name for some reason, your company
name is exceedingly unlikely to end up in the standard library unless it is a
generic word. More importantly, another company is exceedingly unlikely to use
the same name for their paths, so there's no problem when you need to start
sharing code with them (or buy them or are bought by them or whatever).
Similar problem happens sometimes with the reserved IP spaces.
(Even if we promise not to use local/, what if you use local/ for all your stuff
and you acquire a company that also uses local/ for all their stuff?
Better to use real prefixes for real code.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.