Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: add support for new package overlays #29047

Open
ianthehat opened this issue Nov 30, 2018 · 11 comments

Comments

@ianthehat
Copy link

commented Nov 30, 2018

As of cl/151999 we support adding new files in the overlay.
It would be nice if we could use this for the very first file in a package as well, for which go list is going to fundamentally fail.

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 1, 2018

Thanks for opening the issue :) For those wanting a link to the CL above, it's https://golang.org/cl/151999.

The current CL currently extends Overlay map[string][]byte. The one problem with that approach is that the user must know all the files to add beforehand.

For example, I'd like to be able to add some Go files to directories that have a special non-Go file. So if I do Load("./..."), I can't possibly know which files to add beforehand. I could possibly do something like go list -e -f {{.Dir}} on my own, filter the directories containing that special file, then add the custom Go files to the overlay map.

But of course that would hard-code go list and require quite a bit of code. Could it be possible to have some form of overlay that works like a callback, so that I could add Go files on a per-package or per-directory basis? Ideally it would support introducing new packages too.

I know that's perhaps asking for too much, but I want to clarify that the current approach won't quite cut it for some use cases :)

@ianthehat

This comment has been minimized.

Copy link
Author

commented Dec 2, 2018

I have thought about this use case.

The trouble with the callback based overlays is it does not work with an external driver. If you look at the work that Marwan did on a caching version of the go list driver, there is no way to have that separate process cope with a function based overlay in a way that is even slightly efficient, which is why we chose a pure data model that can be handed across a process boundary.

For the moment, you can call Load twice, once in Files mode, and then again in Syntax mode with the constructed overlay.
It's not quite as efficient as it could be, but the functionality is equivalent, and once we have some real use cases we can think about API's that allow the operations people really need to do to become more efficient. Also the Files mode will be very very fast once the new caching code goes in to the go list operation, to the point where it's probably good enough anyway.

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 2, 2018

Thanks for the reply - that indeed does work OK when it comes to adding or changing Go files within existing Go packages. But as far as I know, that can't be used to construct new Go packages easily.

For example, suppose my special file per package is called foo.bar and introduces a foo.go file. With the go list driver, if we load ./... and a directory consists of the files foo.bar main.go, your approach will work just fine. But if a directory only contains foo.bar, I believe that LoadFiles will skip that directory, just like go list ./... would.

I also just realised that it's not possible to list non-package directories with go list -e -f {{.Dir}} ./..., either. I wish that either go list or its go/packages driver had a "find directories" mode, where patterns like ./... would be resolved to lists of "potential packages" before they're discarded for not having any Go files.

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 2, 2018

Also, I do realise that my talking of "Go packages" as "directories" is quite particular to go list, and not necessarily other build systems. So I'd understand if all this was out of scope for the current plans for go/packages. Hopefully I managed to illustrate why I still can't accomplish what I need, though.

@matloob

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2018

@mvdan I'm not sure that go list will be able list non-package directories, so I don't know if this sort of thing will be supported first-class. So solving your problem might require traversing all the relevant directories to find the .bar files and constructing an overlay based on that. I know that's not satisfying, but I don't see a way around it. To make things worse: as you mentioned, it's not clear how we'd make something like this work with alternate build systems

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 4, 2018

Thanks - I agree this is an awkward use case :)

Manually traversing directories can be a tricky solution though, as it essentially means I need to reimplement go list patterns myself. For the simple ./... I could just filepath.Walk, but what about trickier ones like .../foo or .../foo/...?

It's completely fair if this is out of the scope of go/packages, and I do understand I'd be tying my implementation to go list quite a lot - and that's fine. But I'd still need a way for go list to give me directories matching patterns, not just Go packages. This is what I thought go list -e -f {{.Dir}} ../foo would do, which is why I gave that example in my first comment - but it too skips directories with no Go files.

So I should really be filing an issue against go list :) I'd like your thoughts before raising the formal proposal for the feature, though.

@ianthehat

This comment has been minimized.

Copy link
Author

commented Dec 4, 2018

I suspect such an issue would be denied on the assumption that it is the go tools job to deal with go code, so returning information about directories that have no go code is out of scope.
Is there really a use case for this where just saying there must be a doc.go with a valid package statement is not an acceptable solution?

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 4, 2018

I suspect such an issue would be declined too, which is part of the reason why I'm approaching it from the perspective of overlays creating new packages :)

We are indeed forcing users to add doc.go files to each of those directories. We end up with Go-like packages on disk with no actual static Go files. If we force users to add doc.go, that's extra manual work and exposes impelmentation details. If we implement our own go list pattern matching, that will likely lead to lots of added code. I don't see a simple solution.

Perhaps the solution would be a Go library similar to the older https://github.com/kisielk/gotool, but where you'd be able to configure things such as whether a directory should be a package or not. That would still require me to copy some code from go list though, so I'm still hopeful that there's a way to support this in a simple way.

@ianthehat

This comment has been minimized.

Copy link
Author

commented Dec 4, 2018

Creating the doc.go is easy enough to write a small tool for, it does not need to be a manual step, just build a doc.go for any directory that has a .bar file and no .go files before doing any list queries.
Presumably the contents of the doc.go file could easily be guessed from the .bar file.
I guess I still don't really understand your use case properly either, what you are describing is something that would never work with anything else in the go eco system anyway (you can't use any of the normal go tooling on a directory with no go files, including go generate or go get)

@mvdan

This comment has been minimized.

Copy link
Member

commented Jan 11, 2019

Correct. We're hacking our way around these problems with overlays, so I'm moderately happy with the situation. If I come up with a specific proposal for go/packages to improve our use case, I'll file a separate issue.

@beoran

This comment has been minimized.

Copy link

commented Apr 26, 2019

Several other programming languages solve this by having a specific file that can list the files that belong to a package. We already have go.mod, so in the same vein, we could use a go.pkg file that lists patterns of files to ignore and of files to include as go file.

@dmitshur dmitshur changed the title go/packages: add support for new package overlays x/tools/go/packages: add support for new package overlays Jun 17, 2019

@gopherbot gopherbot added this to the Unreleased milestone Jun 17, 2019

@gopherbot gopherbot added the Tools label Sep 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.