Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: reserve example/ and test/ as non-stdlib path prefixes #37641

Open
bcmills opened this issue Mar 3, 2020 · 27 comments
Open

cmd/go: reserve example/ and test/ as non-stdlib path prefixes #37641

bcmills opened this issue Mar 3, 2020 · 27 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 3, 2020

The go command automatically fetches modules from network hosts, provided that those modules have a path whose first element includes a dot character (.).

First-elements that do not include a dot are, in general, reserved for the standard library and the Go toolchain (see #32819). Some names (std, cmd, all) have a special meaning no matter where they are used. Others (such as archive and debug) are currently standard-library prefixes but not packages, but could be made so in the future (as was done in the case of encoding).

Still others currently have no meaning, but may be assigned a special meaning in the future: for example, I would like to add mod to refer to the main module independent of its path (that would be a separate proposal).

The fact that import paths are required to contain a dot occasionally causes user consternation (#37554, #34592).

I propose that we explicitly reserve the following package paths for user code:

  • for general use:
    • local
      • But not localhost, which we may at some point want to treat as a hostname.
  • for documentation, tutorials, and bug reports:
    • example (along the lines of example.com, because it looks more like “something you should replace” than local does)
  • for tests, especially tests of tools that manipulate modules (and thus need a “clean” module to run in):
  • for module-graph and package-import-graph examples:
    • main (because it is useful to indicate "the main module", and a package main is not importable anyway), and
    • any valid module path that consists of a single letter, because we already use those a lot in cmd/go tests.

CC @bronze1man @DisposaBoy @mvdan @jayconrod @matloob @thepudds

@gopherbot gopherbot added this to the Proposal milestone Mar 3, 2020
@gopherbot gopherbot added the Proposal label Mar 3, 2020
@bcmills bcmills added this to Incoming in Proposals Mar 3, 2020
@mvdan
Copy link
Member

@mvdan mvdan commented Mar 3, 2020

Wouldn't main fall in the same category as, or perhaps be confused with, mod?

I'm overall in favor of this proposal. I seem to recall that @rogpeppe uses 0.0.0.0 as a module name, because it's a valid hostname and it contains dots, yet it can never be used to reach anything on the internet as it's not a usable IP. However, I have to admit that I too prefer using human-readable names like example or test, and I too have been doing it in tests.

From the tooling perspective, these rules are easy to implement, so that doesn't worry me either. The rules should be well documented, though, and probably alongside #32819.

Do you reckon that any sort of API would be useful for tools here? Similar to https://golang.org/pkg/go/token/#IsKeyword, imagine something like x/mod.IsLocal(path string) bool and x/mod.IsReserved(path string) bool. Their implementations would be pretty trivial, but the big advantage is that if we ever change the rules, we could easily bring the entire ecosystem in sync.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

Wouldn't main fall in the same category as, or perhaps be confused with, mod?

Maybe? I'd be fine with omitting it from the reserved names.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

Do you reckon that any sort of API would be useful for tools here?

Maybe? I'm not sure who would use them, though: I don't think we should explicitly reject other dotless paths at this point (we can break them if/when we need the name for something else, but if users are confident we'll never use their name I see little harm in letting it persist).

I guess it could be useful for linters..?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

(search.IsMetaPackage is what we currently use to enumerate the actually-meaningful non-package paths, but I don't think it covers builtin which is also somewhat special.)

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Mar 4, 2020

I'll point out a cost of implementing this change, since I don't see it mentioned here so far, is that this will increase the amount of special cases. Special cases need to be documented, taught and learned by all Go users. Users who are not aware of the special cases and run into them accidentally may be surprised.

I think it's a good idea to evaluate how often each special case is expected to be used. For example, the package name main is a special case, and it is used very often, and adds more value than cost. The package name documentation is another special case, but it is used very infrequently, and has a higher cost than value (it was kept because of backwards compatibility reasons).

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 4, 2020

What would it mean to reserve these paths? I'm guessing a documented promise to never create std packages with those prefixes?

Should we allow dotless paths using reserved prefixes in the GOPROXY protocol?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

What would it mean to reserve these paths? I'm guessing a documented promise to never create std packages with those prefixes?

Yep. Specifically, we would apply the Go 1 compatibility policy to the non-existence of these paths in std.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

Should we allow dotless paths using reserved prefixes in the GOPROXY protocol?

Not at this time. (We could consider it as a future change, but since these paths explicitly do not have a single source of truth, I don't think they'd be a good fit.)

@rsc rsc moved this from Incoming to Active in Proposals Mar 4, 2020
@rsc rsc changed the title proposal: all: reserve specific path prefixes for local (user-defined) modules proposal: cmd/go: reserve specific path prefixes for local (user-defined) modules Mar 4, 2020
@slrz
Copy link

@slrz slrz commented Mar 4, 2020

  * `example` (along the lines of `example.com`, because it looks more like “something you should replace” than `local` does)

What about jusing using example.com and the other existing DNS zones (example, invalid, test) reserved for test and documentation usage? If we already rely on the DNS, why not use its existing conventions for reserved names?

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Mar 4, 2020

@slrz What you're suggesting is already possible, it does need any work to be completed. Or are you suggesting that as a reason this proposal should not be accepted?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 4, 2020

@slrz, IANA made the dubious decision to host an actual server on https://example.com.

While I do use those domains in examples myself, I could reasonably understand someone wanting to avoid them on security grounds: if those servers are ever compromised, they could serve go-import tags that point to malicious code.

@bronze1man
Copy link
Contributor

@bronze1man bronze1man commented Mar 5, 2020

Please leave reserve vendor for GOPATH , and remove the GOROOT/src/vendor .
I need to change the go source code when i want to upgrade golang version to make my private project works since version 1.13 (#34068 (comment))

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 6, 2020

@bronze1man, vendor has had a special meaning since Go 1.6 (https://golang.org/doc/go1.6#go_command). The ship sailed on reserving that particular prefix a long time ago.

@thepudds
Copy link

@thepudds thepudds commented Mar 6, 2020

Hi @bcmills, overall, I like the proposal.

Do you think the main use cases could be addressed by picking just one?

For example, maybe just pick local, or some other singular choice? That might help keep the number of special cases smaller.

People could still then use hierarchy underneath local to perhaps address the other ones you listed (e.g., local/test, local/example, local/main, or local/foo and local/bar if desired).

@bcmills
Copy link
Member Author

@bcmills bcmills commented Mar 6, 2020

@thepudds, just local could suffice. I suggested the others as well in the interest of maintaining semantic appropriateness.

For example, an “example” module in a tutorial is not really “local”. Someone following the tutorial might not understand local as a keyword that needs to be replaced with a non-local path prefix (or, really, any prefix that is not literally local), whereas the word example more clearly indicates a placeholder to be substituted by the reader.

@bronze1man
Copy link
Contributor

@bronze1man bronze1man commented Mar 7, 2020

Hi @bcmills, I like the proposal.
I think local and all single letter should be suffice.
Please do not make those reserve path prefixes can be relative resolve like vendor or ./x , it will make it difficult to find the actually package location.
GOPATH/src/local/ep and GOPATH/src/m/tmp and GOPATH/src/c/tmp looks local for me. And a single letter should be easy for different usage to end users.
Like:

  • GOPATH/src/m for fast execute package from command line.
  • GOPATH/src/c for project level config
  • GOPATH/src/t for project level test
  • GOPATH/src/l for private library only for this project.
@mewmew
Copy link
Contributor

@mewmew mewmew commented Mar 18, 2020

Just as a data point, I often use foo as a temporary local module path.

$ mkdir /tmp/foo
$ cd /tmp/foo
$ go mod init foo
$ go get github.com/llir/llvm/ir
$ cat go.mod 
module foo

go 1.14

require github.com/llir/llvm v0.3.0 // indirect

If prefixes were to be reserved, I'd echo @dmitshur's reflection that these special cases would need to be documented, and having many special cases increase the learning cost.

As such having a very clear rule, e.g. local can be use by whoever, where ever (with arbitrary sub-directories; local/foo). All other module paths where the first element does not contain a . character are reserved for the Go standard library/project.

@rsc
Copy link
Contributor

@rsc rsc commented Apr 8, 2020

Let's officially reserve example and test and stop there.
That is, we would commit that we will never add $GOROOT/src/{example,test}
to the main tree: there will not be packages with those names, nor packages
in subdirectories of those names.

Let's not do the single-letter thing. It's fine for us to do that ourselves,
and I certainly do 'echo module m >go.mod' every time I want to play with something,
but internal tests and ephemeral things don't need a long-term guarantee.

For things that need a long-term guarantee, it is reasonable to ask people to
use longer names, and I think example and test pretty much cover the use cases.

I don't understand the need for local, or else I disagree that it's a good use.
For a long-term use, it's better to use a domain name or other prefix you control.
For example inside Google all the paths begin with "google3/" (long story).
It's not a domain name, but it's also not a name that is going to end up in the
standard library and cause a conflict. Most company names have the same property,
so if you wanted to avoid using a domain name for some reason, your company
name is exceedingly unlikely to end up in the standard library unless it is a
generic word. More importantly, another company is exceedingly unlikely to use
the same name for their paths, so there's no problem when you need to start
sharing code with them (or buy them or are bought by them or whatever).
Similar problem happens sometimes with the reserved IP spaces.
(Even if we promise not to use local/, what if you use local/ for all your stuff
and you acquire a company that also uses local/ for all their stuff?
Better to use real prefixes for real code.)

@rsc
Copy link
Contributor

@rsc rsc commented Apr 15, 2020

Are there any objections to "Let's officially reserve example and test and stop there."?

/cc @bcmills

@mvdan
Copy link
Member

@mvdan mvdan commented Apr 15, 2020

Reserving test and example sgtm. I also agree that local isn't a clear benefit to my mind, partially because it can be easily confused with localhost.

@rsc one that you didn't include and didn't mention why is main. Could you clarify that bit?

I have to admit that main is a bit niche compared to test and example, though. I can see it being useful for testing cmd/go itself, since making the distinction between the main module and other modules can be very important to keep a test readable. However, one could imagine naming a module test/main, or example/main for an example, and I think we would accomplish the same goal. In a way, it mirrors Go's already present support for tests and examples.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Apr 15, 2020

example and test seem like a reasonable enough start. (If we find that's not sufficient, it's easier to add reserved paths than to remove them.)

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Apr 15, 2020

Are there any objections to "Let's officially reserve example and test and stop there."?

I'm open to being overruled on this, and I don't think this is a very significant change either way, but I would object at this moment because based on my evaluation of this proposal, the value it adds is too low compared to the cost of introducing this change.

The advantages of going with the suggested resolution, as I currently perceive, include:

  • Some modules that don't need to be downloadable from the internet and want to have a very stort module path will be able to shorten the module path by 2 or more letters (e.g., module m.test -> module test or module example.com -> module example).

The costs of going with the suggested resolution, as I currently perceive, include:

  • It creates a change that causes the addition of two Go-specific special cases when it comes to import paths that need to be learned and kept in memory over time.

    These special cases will affect and will need to be kept in mind both by developers and users of Go. Developers will need to remember not to use example or test as a prefix for a new Go standard library package (packages are added infrequently). Users will need to remember that example and test are safe to use, but sample or testing is not. Whenever someone uses this feature in a CL, the reviewers will be forced to learn about it. That means either this feature will never be used in modern practice (similar to the package documentation special case), or eventually all existing and future Go users will need to learn and remember this.

I think there would be a much stronger need for this proposal if the reserved TLDs .example, .test, .invalid, and .localhost did not exist. Given they do exist since 1999¹ and can be used by continuing to use the convention of using a URL for an import path, without introducing Go-specific special cases that need to be taught and learned, I think the incremental benefit offered by reserving "example" and "test" module path prefixes are not justified.

¹ https://tools.ietf.org/html/rfc2606#section-2

Edit: I want to clarify that I have considered this point:

The fact that import paths are required to contain a dot occasionally causes user consternation (#37554, #34592).

I think the proposed solution would benefit a small number of users by a small amount, and incur a small cost on a very large number of users. Because the benefit is so small, the left hand side of the equation doesn't seem to outweigh the right hand side by my current estimates.

@rsc
Copy link
Contributor

@rsc rsc commented Apr 22, 2020

My main hesitation about depending on the TLDs is that example.com was supposed to be the reserved name, and yet example.com has a web page. That means that go module examples using example.com make HTTPS requests. I don't trust the IETF not to create web pages for the other "reserved" TLDs.

The only thing we promise by reserving example/ and test/ is that we will never create those two directories in $GOROOT/src/. That is, there will never be a standard library package named example or example/x or test or test/x.

@rsc
Copy link
Contributor

@rsc rsc commented Apr 22, 2020

(That is, there's no implementation work here, just a promise not to create certain standard library package paths.)

mvdan added a commit to burrowers/garble that referenced this issue Apr 28, 2020
Carefully select a default that will do the right thing when inside a
module, as well as when building ad-hoc packages.

This means we no longer need to look at the compiler's -std flag, which
is nice.

Also replace foo.com/ with test/, as per golang/go#37641.

Fixes #7.
@rsc rsc changed the title proposal: cmd/go: reserve specific path prefixes for local (user-defined) modules proposal: cmd/go: reserve example/ and test/ as non-stdlib path prefixes Apr 29, 2020
@rsc
Copy link
Contributor

@rsc rsc commented Apr 29, 2020

Retitled. Based on the discussion above, this sounds like a likely accept.

@rsc rsc moved this from Active to Likely Accept in Proposals Apr 29, 2020
@rsc
Copy link
Contributor

@rsc rsc commented May 6, 2020

No change in consensus, so accepted. This is a documentation-only change, so it could be done for Go 1.15 still.

@rsc rsc moved this from Likely Accept to Accepted in Proposals May 6, 2020
@rsc rsc modified the milestones: Proposal, Backlog May 6, 2020
@rsc rsc changed the title proposal: cmd/go: reserve example/ and test/ as non-stdlib path prefixes cmd/go: reserve example/ and test/ as non-stdlib path prefixes May 6, 2020
@dmitshur
Copy link
Member

@dmitshur dmitshur commented May 7, 2020

This is a documentation-only change,

Perhaps you meant this implicitly, but I think there should also be a test (or equivalent automated mechanism) that would detect if a future CL accidentally forgets about this promise. We shouldn't rely only on human reviewers for this.

mvdan added a commit to mvdan/gofumpt that referenced this issue Jul 31, 2020
See golang/go#37641, which was accepted a few
months ago.

Packages like internal/foo are also somewhat common, and we can know for
sure that they don't belong in the standard library, as they couldn't be
imported then.

Updates #38.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.