go/tools/gopackagesdriver: add automatic target detection #2932

steeve · 2021-08-03T23:55:10Z

What type of PR is this?

Feature

What does this PR do? Why is it needed?

This PR introduces automatic target detection so that no user input
is required after the GOPACKAGESDRIVER setup. This effectively
deprecates the following environment variables:

GOPACKAGESDRIVER_BAZEL_TARGETS
GOPACKAGESDRIVER_BAZEL_QUERY
GOPACKAGESDRIVER_BAZEL_TAG_FILTERS

It works as follows:

for importpath queries, it will bazel query a matching go_library
matching it
for file= queries, it will try to find the matching go_library in the
same package
since it supports multiple queries at the same time, it will union those
queries and query that

Once the go_library targets are found, it will try to build them directly,
which should dramatically speed up compilation times, at the loss of transition
support (which wasn't used as much as I thought it would be).

I may reintroduce it in the future via a user-defined flag (to only build the
part of the graph that needs building).

In any case, toolchain or platforms can be switched with the following
environment variables it configuration transitions are needed:

GOPACKAGESDRIVER_BAZEL_FLAGS which will be passed to bazel invocations
GOPACKAGESDRIVER_BAZEL_QUERY_FLAGS which will be passed to bazel query
invocations
GOPACKAGESDRIVER_BAZEL_QUERY_SCOPE which specifies the scope for importpath queries (since gopls only issues file= queries, so use if you know what you're doing!)
GOPACKAGESDRIVER_BAZEL_BUILD_FLAGS which will be passed to bazel build
invocations

Finally, the driver will not fail in case of a build failure, and even so
uses --keep_going and will return whatever packages that did build.

Which issues(s) does this PR fix?

It should fix most of the issues from #2858.

Other notes for review
I have added a .vscode folder for folks who work on rules_go with it so that gopls now works there too!

linzhp

Can we continue to support GOPACKAGESDRIVER_BAZEL_QUERY etc? This is important for large repos like Uber's Go monorepo. Although it's huge, most users only work on small subset of targets and only need to index their projects and dependencies. It's quite a waste of time for them to wait for hours for package driver to load //...

go/tools/gopackagesdriver/bazel_json_builder.go

steeve · 2021-08-05T14:17:16Z

@linzhp I have updated the PR, it shouldn't load the whole //... for file queries now

steeve · 2021-08-05T14:22:41Z

@linzhp I have added GOPACKAGESDRIVER_BAZEL_QUERY_SCOPE which should control the scope when issuing importpath queries. That said, you should not need it for gopls as it issues file= queries, and I have removed the whole universe loading for those.

go/tools/gopackagesdriver/bazel_json_builder.go

steeve · 2021-08-08T17:45:10Z

@linzhp I have fixed lots of bugs, can you try this version?

steeve · 2021-08-08T20:11:29Z

Here's a summary of the changes:

when called without GOPACKAGESDRIVER_BAZEL_QUERY_SCOPE, return stdlib packages by default (so that vscode doesn't complain)
add a compiled sources output group to ensure all files are generated and present in the sandbox
enable querying by importpath prefix (github.com/bazelbuild/rules_go/...)
fix path for external generated files
fix path for external source files

linzhp

It works now. Commented on other minor issues.

go/tools/gopackagesdriver/bazel.go

go/tools/gopackagesdriver/bazel_json_builder.go

go/tools/gopackagesdriver/main.go

google-cla · 2021-08-17T14:08:34Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

steeve · 2021-08-17T14:09:08Z

@linzhp can you confirm the CLA please?

google-cla · 2021-08-17T15:40:48Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

thinkiny · 2021-08-18T12:32:00Z

can't wait this merge to master

linzhp

Some tests are failing.

It's strange that googlebot thought I didn't have CLA, given I already had several accepted pull requests to this repo.

@googlebot I consent.

go/tools/gopackagesdriver/bazel_json_builder.go

go/tools/gopackagesdriver/main.go

google-cla · 2021-08-18T17:41:07Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

steeve · 2021-08-18T18:02:54Z

The test failure is unrelated (race)

google-cla · 2021-08-18T18:03:08Z

☹️ Sorry, but only Googlers may change the label cla: yes.

steeve · 2021-08-18T18:16:22Z

@linzhp I have added 3512e59 which makes the gopackagesdriver honor the build tags when listing files. Can you try it ?

steeve · 2021-08-18T18:16:35Z

can't wait this merge to master

very soon !

go/tools/gopackagesdriver/bazel_json_builder.go

linzhp · 2021-08-18T18:36:19Z

go/tools/gopackagesdriver/bazel_json_builder.go

+		fp, _ := filepath.Rel(b.bazel.WorkspaceRoot(), filename)
+		filename = fp
+	}
+	return fmt.Sprintf(`kind("go_library", same_pkg_direct_rdeps("%s"))`, filename)


This may return empty result when the filename is a test file like foo_test.go, because no go_library would depend on such file

yeah, test files are not done atm, if you have an idea, i'm all ears

go/tools/gopackagesdriver/bazel_json_builder.go

linzhp · 2021-08-18T18:50:32Z

I have added 3512e59 which makes the gopackagesdriver honor the build tags when listing files. Can you try it ?

How should I try it? It seems to be a separate feature. Can you put it in a separate pull request?

linzhp · 2021-08-18T18:52:40Z

The test failure is unrelated (race)

It failed consistently in recent commits, but not in earlier commits or master. So it may be broken by some recent commit

steeve · 2021-08-18T19:10:05Z

How should I try it? It seems to be a separate feature. Can you put it in a separate pull request?

Just verify that it correctly works as it did before. It's done behind the scenes. Basically gopls complained because all the go files were listed as part of the package, which would break some symbols.

Just check that the packagedriver continues to work on your repo.

steeve · 2021-08-18T19:12:43Z

@linzhp careful:

leaving a comment that contains only @googlebot I consent. in this pull request.

This commit introduces automatic target detection so that no user input is required after the `GOPACKAGESDRIVER` setup. This effectively deprecates the following environment variables: - `GOPACKAGESDRIVER_BAZEL_TARGETS` - `GOPACKAGESDRIVER_BAZEL_QUERY` - `GOPACKAGESDRIVER_BAZEL_TAG_FILTERS` It works as follows: - for `importpath` queries, it will `bazel query` a matching `go_library` matching it - for `file=` queries, it will try to find the matching `go_library` in the same package - since it supports multiple queries at the same time, it will `union` those queries and query that Once the `go_library` targets are found, it will try to build them directly, which should dramatically speed up compilation times, at the loss of transition support (which wasn't used as much as I thought it would be). I may reintroduce it in the future via a user-defined flag (to only build the part of the graph that needs building). In any case, toolchain or platforms can be switched with the following environment variables it configuration transitions are needed: - `GOPACKAGESDRIVER_BAZEL_FLAGS` which will be passed to `bazel` invocations - `GOPACKAGESDRIVER_BAZEL_QUERY_FLAGS` which will be passed to `bazel query` invocations - `GOPACKAGESDRIVER_BAZEL_BUILD_FLAGS` which will be passed to `bazel build` invocations Finally, the driver will not fail in case of a build failure, and even so uses `--keep_going` and will return whatever packages that did build. Signed-off-by: Steeve Morin <steeve@zen.ly>

This allows to use gopls in the project itself. Signed-off-by: Steeve Morin <steeve@zen.ly>

…eries When using `--universe_scope=//...`, the whole `//...` is loaded even though we only use `same_pkg_direct_rdeps`. So remove it and specify the scope in the query, such as `importpath` ones. Signed-off-by: Steeve Morin <steeve@zen.ly>

Add the `GOPACKAGESDRIVER_BAZEL_QUERY_SCOPE` environment variable so that bazel queries are limited the given scope. This should save time when doing `importpath` queries on large monorepos. Signed-off-by: Steeve Morin <steeve@zen.ly>

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

While at it, fetch and parse the whole bazel info data.

…ault When fetching the whole graph, default to returning the stdlib packages so that vscode doesn't complain.

Easier to put them there

Guard against packages with the same prefix by happending `/` to HasPrefix and match the exact name if needed.

While it used to work, this is better.

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

When there is no such go_library, the some function would lead to errors like ERROR: Evaluation of query failed: argument set is empty, with exit code 7. It is unclear whether the query has a bug or there is no matching package. If we don't have the some function, Bazel query will exit normally with empty result. We can report a better error to gopls for this case. Co-authored-by: Zhongpeng Lin <zplin@uber.com>

Signed-off-by: Steeve Morin <steeve@zen.ly>

This is done using a brand new build context that is configured using the regular environment variables. Signed-off-by: Steeve Morin <steeve@zen.ly>

Match a/ab and a/ab/c but not a/abc Signed-off-by: Steeve Morin <steeve@zen.ly>

Signed-off-by: Steeve Morin <steeve@zen.ly>

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

Signed-off-by: Steeve Morin <steeve@zen.ly>

google-cla · 2021-08-21T13:14:33Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

thinkiny · 2021-08-29T02:16:37Z

awesome

google-cla bot added the cla: yes label Aug 3, 2021

linzhp reviewed Aug 5, 2021

View reviewed changes

go/tools/gopackagesdriver/bazel_json_builder.go Outdated Show resolved Hide resolved

linzhp reviewed Aug 6, 2021

View reviewed changes

go/tools/gopackagesdriver/bazel_json_builder.go Outdated Show resolved Hide resolved

linzhp reviewed Aug 6, 2021

View reviewed changes

go/tools/gopackagesdriver/bazel_json_builder.go Outdated Show resolved Hide resolved

linzhp reviewed Aug 9, 2021

View reviewed changes

google-cla bot added cla: no and removed cla: yes labels Aug 17, 2021

linzhp reviewed Aug 18, 2021

View reviewed changes

go/tools/gopackagesdriver/bazel_json_builder.go Outdated Show resolved Hide resolved

go/tools/gopackagesdriver/main.go Show resolved Hide resolved

steeve added cla: yes and removed cla: no labels Aug 18, 2021

google-cla bot removed the cla: yes label Aug 18, 2021

linzhp reviewed Aug 18, 2021

View reviewed changes

google-cla bot added the cla: no label Aug 18, 2021

steeve and others added 24 commits August 21, 2021 15:13

workspace: add vscode configuration

576729d

This allows to use gopls in the project itself. Signed-off-by: Steeve Morin <steeve@zen.ly>

Allow querying external dependencies with importpath

b2b33b3

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

go/tools/gopackagesdriver: fetch output_base from bazel info

646e311

While at it, fetch and parse the whole bazel info data.

go/tools/gopackagesdriver: pull source files and return stdlib by def…

3d2175b

…ault When fetching the whole graph, default to returning the stdlib packages so that vscode doesn't complain.

Fix typo

2071919

Fix formatting

1d4e937

Enable queries on package registry

3c96e2c

Move some utiliy functions to the utils file

6a7a1e6

Easier to put them there

Remove unsued bazel struct members

cd8d554

Don't match only by string prefix when matching /...

a2f3225

Guard against packages with the same prefix by happending `/` to HasPrefix and match the exact name if needed.

Stdlib files are relative to the output_base, not execroot

d3f0ed0

While it used to work, this is better.

Simpler error checking for bazel errors

503ed0e

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

CHeck for empty labels when trying to build a target

42013df

Signed-off-by: Steeve Morin <steeve@zen.ly>

Honor build tags when listing files

ec895a3

This is done using a brand new build context that is configured using the regular environment variables. Signed-off-by: Steeve Morin <steeve@zen.ly>

Properly match full and child importpaths

87c69ed

Match a/ab and a/ab/c but not a/abc Signed-off-by: Steeve Morin <steeve@zen.ly>

Add a comment telling why we exit(0)

64ba983

Signed-off-by: Steeve Morin <steeve@zen.ly>

Better regexp for importpath matching

0def12d

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

Don't append empty bazel queries to bazel build

0703743

Co-authored-by: Zhongpeng Lin <zplin@uber.com>

Make completion on test files work

ddf8e68

Signed-off-by: Steeve Morin <steeve@zen.ly>

Handle tests that don't have embedded libraries

165e959

Signed-off-by: Steeve Morin <steeve@zen.ly>

jayconrod added cla: yes and removed cla: no labels Aug 27, 2021

steeve merged commit 70b8365 into bazelbuild:master Aug 28, 2021

steeve deleted the steeve/gopackagesdriver2 branch August 28, 2021 22:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

go/tools/gopackagesdriver: add automatic target detection #2932

go/tools/gopackagesdriver: add automatic target detection #2932

steeve commented Aug 3, 2021 •

edited

Loading

linzhp left a comment •

edited

Loading

steeve commented Aug 5, 2021

steeve commented Aug 5, 2021

steeve commented Aug 8, 2021

steeve commented Aug 8, 2021

linzhp left a comment

google-cla bot commented Aug 17, 2021

steeve commented Aug 17, 2021

google-cla bot commented Aug 17, 2021

thinkiny commented Aug 18, 2021

linzhp left a comment

google-cla bot commented Aug 18, 2021

steeve commented Aug 18, 2021

google-cla bot commented Aug 18, 2021

steeve commented Aug 18, 2021

steeve commented Aug 18, 2021

linzhp Aug 18, 2021

steeve Aug 18, 2021

linzhp commented Aug 18, 2021 •

edited

Loading

linzhp commented Aug 18, 2021

steeve commented Aug 18, 2021 •

edited

Loading

steeve commented Aug 18, 2021

google-cla bot commented Aug 21, 2021

thinkiny commented Aug 29, 2021

go/tools/gopackagesdriver: add automatic target detection #2932

go/tools/gopackagesdriver: add automatic target detection #2932

Conversation

steeve commented Aug 3, 2021 • edited Loading

linzhp left a comment • edited Loading

Choose a reason for hiding this comment

steeve commented Aug 5, 2021

steeve commented Aug 5, 2021

steeve commented Aug 8, 2021

steeve commented Aug 8, 2021

linzhp left a comment

Choose a reason for hiding this comment

google-cla bot commented Aug 17, 2021

steeve commented Aug 17, 2021

google-cla bot commented Aug 17, 2021

thinkiny commented Aug 18, 2021

linzhp left a comment

Choose a reason for hiding this comment

google-cla bot commented Aug 18, 2021

steeve commented Aug 18, 2021

google-cla bot commented Aug 18, 2021

steeve commented Aug 18, 2021

steeve commented Aug 18, 2021

linzhp Aug 18, 2021

Choose a reason for hiding this comment

steeve Aug 18, 2021

Choose a reason for hiding this comment

linzhp commented Aug 18, 2021 • edited Loading

linzhp commented Aug 18, 2021

steeve commented Aug 18, 2021 • edited Loading

steeve commented Aug 18, 2021

google-cla bot commented Aug 21, 2021

thinkiny commented Aug 29, 2021

steeve commented Aug 3, 2021 •

edited

Loading

linzhp left a comment •

edited

Loading

linzhp commented Aug 18, 2021 •

edited

Loading

steeve commented Aug 18, 2021 •

edited

Loading