Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: Declare module content in go.files config #52807

Closed
pavelkalinnikov opened this issue May 10, 2022 · 8 comments
Closed

proposal: cmd/go: Declare module content in go.files config #52807

pavelkalinnikov opened this issue May 10, 2022 · 8 comments
Projects
Milestone

Comments

@pavelkalinnikov
Copy link

@pavelkalinnikov pavelkalinnikov commented May 10, 2022

Issue

A typical non-trivial repository with a Go module contains files/directories that don't need to be packaged into the module and distributed with the code, such as:

  • User documentation, which may include large files like images and PDFs.
  • Auto-generated at build time files.
  • Configuration for other tooling, such as CI/CD pipeline for this repository.

It would be ideal to have a fine-grained control on what gets included into the module content. Currently, there are workarounds:

  1. Packages starting with . or _, or named testdata are excluded.
  2. An empty go.mod in a directory cuts this entire directory from the Go module it was in.

Approach (1) is not universal, and requires remembering the exceptions, which is an unnecessary cognitive load.

Approach (2) is more complete, but is less convenient at scale. It also does not allow an easy way to have a consolidated view of the included/excluded paths.

More discussion here: #30058.

Proposal

Introduce a top-level go.files config (next to go.mod and co) which represents the consolidated view of the Go module content. The file consists of include/exclude rules/globs/regexps, in order of their priority. This would be similar in semantics to .gitignore. For example, it can look like this:

# Exclude a bulk of non-Go files.
exclude: ^/docs/
exclude: \.pdf$
exclude: ^/scripts/
exclude: ^/cloudbuild\.yaml$

# Exclude experimental code, except the nearly good one.
include: ^/experimental/nearlygoodcode/
exclude: ^/experimental/

# Example from the issue above.
exclude: ^/node_modules/

# The catch-all default.
include: .*

The current (and the default, if go.files is not present) behaviour would be described by a file like this:

# The exceptions that represent the workaround (1).
exclude: /testdata/
exclude: /[._][^/]*$

# Include any files, except those matching the filters above.
include: .*

This format allows various approaches to controlling the content of the module:

  • default: includes everything but the default exceptions,
  • defensive: author lists all the things they want to include,
  • optimistic: author lists only the things to exclude,
  • hybrid: something in between defensive and optimistic.

Another example:

# Filter the module content to just purely Go files. The repository can have
# any structure.
include: \.go$
include: /go\.mod$
include: /go\.sum$
include: LICENSE
@gopherbot gopherbot added this to the Proposal milestone May 10, 2022
@pavelkalinnikov pavelkalinnikov changed the title proposal: cmd/go: Declare module contents in a top-level go.files config proposal: cmd/go: Declare module contents in go.files config May 10, 2022
@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented May 10, 2022

see also #42965

@pavelkalinnikov pavelkalinnikov changed the title proposal: cmd/go: Declare module contents in go.files config proposal: cmd/go: Declare module content in go.files config May 10, 2022
@seankhliao seankhliao added the GoCommand label May 10, 2022
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 11, 2022

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals May 11, 2022
@rsc
Copy link
Contributor

@rsc rsc commented May 11, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals May 11, 2022
@hherman1
Copy link

@hherman1 hherman1 commented May 11, 2022

This makes go modules substantially less simple for seemingly very minor convenience gains. I guess I have a hard time sympathizing with the problem as you describe it, and would rather protect my simple understanding of go modules. Perhaps there’s something I’m not understanding about the importance of having a consolidated list of excludes?

@pavelkalinnikov
Copy link
Author

@pavelkalinnikov pavelkalinnikov commented May 11, 2022

@hherman1 Could you clarify what your simple understanding of Go modules is?

This mechanism would be optional (default = old behaviour), so the module author could still do the simple thing. In which way would it make Go modules more difficult for you?

One of the gains would be performance: excluding bulky/non-Go clutter from the module saves time and space on the dev machine, module proxy, and all the downstream dependencies (dependent modules, other developers machines, CI workflows).

Potential target auditory for this flexible configuration are the owners of complex repositories who can't necessarily do the simple thing, e.g. put all the Go code in a single directory. This includes projects started before Go modules / semver introduction, who did not account for this in the repository structure.

@hherman1
Copy link

@hherman1 hherman1 commented May 11, 2022

The addition of optional features still affects me even if I don’t personally use them because

  1. Other people use them, so I have to consider the possible existence of these files when trying to interact with other repos
  2. Documentation will have to describe them, making it harder to relearn modules after a sabbatical or teach them to new devs

I prefer uniformity over fragmentation where possible, and I can already foresee the complaints about “go modules are so confusing you have to read globs to figure out where it’s safe and not safe to write code”.

Also, it’s still not entirely clear to me how big the win here is over using the empty go.mod files you referenced.

@rsc
Copy link
Contributor

@rsc rsc commented May 18, 2022

This seems like a duplicate of #42965. Neither seems particularly likely given past discussions.

@rsc rsc moved this from Active to Declined in Proposals May 18, 2022
@rsc
Copy link
Contributor

@rsc rsc commented May 18, 2022

This proposal is a duplicate of a previously discussed proposal, as noted above,
and there is no significant new information to justify reopening the discussion.
The issue has therefore been declined as a duplicate.
— rsc for the proposal review group

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Declined
Development

No branches or pull requests

6 participants