Skip to content

Conversation

tgeng
Copy link
Contributor

@tgeng tgeng commented Jun 1, 2023

This PR adds two flags --useCquery and --cqueryCommandOptions which enables using bazel cquery instead of bazel query when computing build graph. This creates more accurate build graph and hence hash computation is more accurate with less false positives.

A e2e test is added to demonstrate this change.

@tgeng tgeng marked this pull request as draft June 1, 2023 11:00
Copy link
Collaborator

@tinder-maxwellelliott tinder-maxwellelliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

} else {
// Unfortunately, cquery does not support streamed_proto yet.
// See https://github.com/bazelbuild/bazel/issues/17743. This poses an issue for large monorepos.
add("proto")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bummer here but seems unavoidable for now

Copy link
Contributor Author

@tgeng tgeng Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah :(. BTW, I realized there is a bug with the first commit so I have pushed another update with one more test covering the basic source code change case.

It seems that the implementation can be cleaner if source code hashing and rule target hashing are done uniformly with a deps query to include everything (rather than separate queries on source and rule targets). But that change would be pretty big so I didn't attempt it.

This PR adds two flags `--useCquery` and `--cqueryCommandOptions` which
enables using `bazel cquery` instead of `bazel query` when computing
build graph. This creates more accurate build graph and hence hash
computation is more accurate with less false positives.

A e2e test is added to demonstrate this change.
@tgeng tgeng marked this pull request as ready for review June 1, 2023 19:59
fix comment
@tinder-maxwellelliott tinder-maxwellelliott merged commit 3098e27 into Tinder:master Jun 2, 2023
queryService.query(repoTargetsQuery.joinToString(" + ") { "'$it'" }))
.distinctBy { it.rule.name }
} else {
val buildTargetsQuery = listOf("//...:all-targets") + fineGrainedHashExternalRepos.map { "@$it//...:all-targets" }
Copy link
Contributor

@honnix honnix Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the reason removing //external:all-targets from query branch? Without those repo generation targets, how would "coarse grained" hashes get computed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#263 is a related change where I removed rule input transformation for query.

@acecilia
Copy link

👋 Just discovered this, thanks!

Would you maybe consider making this the default, given the higher accuracy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants