Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Add ignore directive to go.mod #37724

Open
jniedrauer opened this issue Mar 7, 2020 · 8 comments
Open

proposal: Add ignore directive to go.mod #37724

jniedrauer opened this issue Mar 7, 2020 · 8 comments
Labels
Projects
Milestone

Comments

@jniedrauer
Copy link

@jniedrauer jniedrauer commented Mar 7, 2020

x/mod hardcodes the max size of a module, zipped or extracted, at 500M. There is no (sane) way to exclude files from a module. This means that if you have a mixed repo with vendored assets and go source code with a go.mod at the root of the repo, you are effectively restricted to a max repo size of 500M.

#37697 suggests respecting the .gitignore file for x/tools/gopls, however this would not be a good solution to the go module max size restriction.

A solution to both problems would be to add an ignore section to the go.mod file. For example:

ignore (
    assets
)

(As a temporary workaround to the hardcoded module size, you can create an empty go.mod file in any subdirectory you want to exclude, which causes it to be flagged as a submodule and excluded from the module zip.)

@gopherbot gopherbot added this to the Proposal milestone Mar 7, 2020
@gopherbot gopherbot added the Proposal label Mar 7, 2020
@mvdan mvdan added the modules label Mar 7, 2020
@mvdan
Copy link
Member

@mvdan mvdan commented Mar 7, 2020

Have you seen #29987? It seems like dropping a go.mod in the subdirectory would exclude that entire directory sub-tree from the module's archive.

CC @jayconrod @bcmills

@jniedrauer
Copy link
Author

@jniedrauer jniedrauer commented Mar 7, 2020

@mvdan I have. I mentioned this workaround. I consider it a less than ideal hack though. Adding an empty go.mod to non-go source code to exclude it is the kind of behavior that should not be relied on. You also pay a cost when go traverses the entire directory tree. It doesn't exclude directories per se, it excludes individual files that share a prefix with a go.mod file, which means that it does a comparison against every single file. For a sufficiently large number of files, there is a non-trivial performance cost.

@mvdan
Copy link
Member

@mvdan mvdan commented Mar 8, 2020

My bad - I misread part of your original issue.

I still think that an extra go.mod is a reasonable solution, though. If performance is the main worry here, I'm sure we could look at optimizing how the go tool handles this case. I'm generally in favor of exhasuting all options (including optimizing current features/APIs) before adding new features.

@jniedrauer
Copy link
Author

@jniedrauer jniedrauer commented Mar 9, 2020

In most cases I'd agree. But this functionality is needed for a few different tools now (x/tools/gopls and x/mod, at least), and I think the complexity of an explicit configuration option is lower than relying on an unintended side effect of x/mod. Putting an empty go.mod file in with source code that's not related to go is definitely not a pretty or intuitive solution.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 9, 2020

@jniedrauer, the use of go.mod files to defined module boundaries is not an “unintended side effect”. It is an intentional part of the design.

If it is “not a pretty […] solution”, well, that's unfortunate — but we don't generally change Go tooling (or add redundant features) for purely aesthetic reasons.

If it is “not [an] intuitive solution”, that may be a matter of improving documentation and/or user education. (I think it's too early to tell one way or another, given that large projects especially have only been adopting modules for a year or so, and in small projects this sort of repository division is not necessary.)

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Mar 9, 2020

cc @matloob

I somewhat agree that adding a go.mod file in a subdirectory is not an intuitive solution. It reminds me a bit of using a tools.go file to track tool dependencies (#25922).

It's not clear to me that an ignore directive is the right solution though. As a user, I'd rather have a marker file (.goignore or something) in the directory to be ignored to avoid cluttering the go.mod file of a large repository. Having a marker like that makes it obvious why a directory is or is not included without having to look at another location. This is the same idea as adding a go.mod file though, just with a different name.

@jniedrauer
Copy link
Author

@jniedrauer jniedrauer commented Mar 10, 2020

Any mechanism that can be used to exclude (non-go source) directory trees as the intended behavior will solve the problem.

I do think an empty go.mod is a bit of a hack. The presence of a go.mod is supposed to communicate "this is a Go module" but in this case it actually means "this is not a Go module". If a file requires a comment to explain why it exists, then it's not ergonomic. But if it's documented and compatibility is maintained in future versions, then it works for me.

I'm in favor of putting it in go.mod because it's better from a discoverability standpoint. Magic files like dockerignore have bitten me many times in the past. They are not immediately obvious, especially if they start with a . and are hidden from initial view. A central source of truth (go.mod) is desirable. There's also not really a precedent for magic files within the go tooling (with the exception of tools.go as noted above)

@DavidGamba
Copy link

@DavidGamba DavidGamba commented May 25, 2020

Wondering if this ignore concept could be extended to monorepos with vendor/ dirs that are not go code.

I would love to be able to add something in my go.mod so it knows to always run in mod mode.

I would settle for:

ignore (
    assets
    vendor
)
@rsc rsc added this to Incoming in Proposals Jun 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.