Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: [modules + integration] go mod discover, discover a set of unpacked modules within a directory #31299

Open
nim-nim opened this issue Apr 6, 2019 · 8 comments

Comments

@nim-nim
Copy link

commented Apr 6, 2019

This report is part of a series, filled at the request of @mdempsky, focused at making Go modules integrator-friendly.

Please do not close or mark it as duplicate before making sure you’ve read and understood the general context. A lot of work went into identifying problems points precisely.

Needed feature

Go needs an official go mod discover command that processes a directory of unpacked Go code and identifies the valid Go modules within this directory.

Constrains

  • the output should be a machine-readable list of:
    • module path
    • module version
    • filesystem path
  • the command should warn if different filesystem paths declare the same module and version.
  • the feature should also be exposed as a function in the official Go API, returning a go object (map, list, struct…)
  • discovery should not consult a cache or the internet, just the provided directory
  • the directory may include info files next to the corresponding mod files, or not

Motivation

  • existing software baselines have been created in unpacked GOPATH/vendor mode,
  • go allows creating go.mod files in other places than the root of the project
  • go allows nested modules
  • go allows module names that do not match the project url (and mandates them for versions >1)
  • niceties like testdata or vendor mean some go.mod files are not considered as module descriptor but just data, so a basic find won’t work
  • future language evolutions will probably add more twists to the concept of “valid module”

All of this make module discovery a lot of work that should be handled by common tooling.

@nim-nim nim-nim changed the title [modules + integration] unpacked module discovery [modules + integration] unpacked module set discovery Apr 6, 2019

@nim-nim nim-nim changed the title [modules + integration] unpacked module set discovery [modules + integration] go mod discover, unpacked module set discovery Apr 6, 2019

@thepudds thepudds added the modules label Apr 6, 2019

@thepudds thepudds changed the title [modules + integration] go mod discover, unpacked module set discovery cmd/go: [modules + integration] go mod discover, unpacked module set discovery Apr 6, 2019

@nim-nim nim-nim changed the title cmd/go: [modules + integration] go mod discover, unpacked module set discovery cmd/go: [modules + integration] go mod discover, discover unpacked modules within a directory Apr 7, 2019

@nim-nim nim-nim changed the title cmd/go: [modules + integration] go mod discover, discover unpacked modules within a directory cmd/go: [modules + integration] go mod discover, discover a set of unpacked modules within a directory Apr 7, 2019

@mdempsky

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

It would help if you could trim down the redundant boilerplate text from each of your posts into a single sentence + link for more details (e.g., a Gist or meta-issue or something). The added verbosity makes it more difficult to quickly identify the unique details of each individual issue. Thanks.

@mdempsky

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

Go needs an official go mod discover command that processes a directory of unpacked Go code and identifies the valid Go modules within this directory.

What does it mean for a Go module to be "valid" here?

E.g., you suggest go.mod files within testdata and/or vendor subdirectories are not valid. Why is that the case?

@nim-nim

This comment has been minimized.

Copy link
Author

commented Apr 12, 2019

It would help if you could trim down the redundant boilerplate text from each of your posts into a single sentence + link for more details (e.g., a Gist or meta-issue or something). The added verbosity makes it more difficult to quickly identify the unique details of each individual issue. Thanks.

Done. Thanks for the comment. Do ping me if you find more things to improve.

@nim-nim

This comment has been minimized.

Copy link
Author

commented Apr 12, 2019

@mdempsky

Go needs an official go mod discover command that processes a directory of unpacked Go code and identifies the valid Go modules within this directory.

What does it mean for a Go module to be "valid" here?

Short term, probably just that the module descriptor is parse-able by Go without errors. Long term, whatever sanity rules the language wants to impose on its modules.

E.g., you suggest go.mod files within testdata and/or vendor subdirectories are not valid. Why is that the case?

For vendor, that's because the current Go module design seems to ignore vendoring unless passed specific mode switches (I may have misunderstood this part, I didn't look at vendoring closely, our integration best practices call for vendor removal as the first integration step). And if you don't ignore vendoring you have to handle module collisions because the same module will be vendored multiple times in different vendor directories.

For testdata, that's because it is common and expected for projects to ship broken files inside testdata. The broken files can include broken module descriptor files, or module descriptors for a tree of broken packages go mod tidy can make no sense of. I had actually forgotten this part, before testing my toy implementation on real-world Go projects, and see it explode on broken and non-conformant module descriptors inside the testdata of some projects.

Either way ignoring vendor and testdata by default while walking the directory tree is not limitating. You can still point the command directly to the vendor or testdata directory, since it will not have arrived here via a walk the walk protections won't apply.

@mdempsky

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

Short term, probably just that the module descriptor is parse-able by Go without errors.

Today you can check this by running go mod edit -json path/to/go.mod >/dev/null and seeing if it exits 0 or 1.

@nim-nim

This comment has been minimized.

Copy link
Author

commented Apr 15, 2019

@mdempsky you forgot the find part, then the tesdata/vendor magic rules, add them to the mix and you'll find out it's not so simple

@mdempsky

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

you forgot the find part,

There's nothing Go-specific about that, but okay:

find . -name go.mod -print0 | xargs -0 -n1 sh -c 'go mod edit -print "$1" >/dev/null && echo "$1"' --

then the tesdata/vendor magic rules,

You defined "valid" as "just that the module description is parse-able by Go without errors," and I identified a command to distinguish that. But it sounds now like you want something more nuanced than that. It would help if you would clearly and concisely state precisely what you're asking for.

Alternatively, identify some sample Go code repositories and the go.mod files within that should or should not be considered "valid".

@nim-nim

This comment has been minimized.

Copy link
Author

commented Apr 16, 2019

@mdempsky I defined the discovery feature in the first message, you are trying to do bits of it separately, the result is still bits of it. The more nuanced part is just applying all the Go source code rules. There are already quite a lot of them on vendor and testdata. There will be probably more in the future. Will you maintain your shell snippet when the language adds more rules, when it is already taking more than a line?

Your snippet is also not outputting the modules names corresponding to the mod files. That's another thing which that can not be deduced trivially from the go.mod file path (and sometimes when it looks like it can be deduced from the file path, it's still wrong, so deducing is dangerous)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.