Change of --output_base causes subsequent builds to fail #10653
Comments
|
Reproduced with Bazel 2.0: |
jin
added
team-OSS
|
I'm seeing something very similar, but even more basic when using the latest pre-release vscode-bazel (which sets --output_base by default). vscode-bazel runs this command:
It fails with the following output: If I remove the --output_base, the query succeeds just fine. If I change the --output_base to --output_user_root, the query succeeds just fine. It's almost like bazel cannot modify things in install_base when outputBase has been specified like that. Perhaps the query is sandboxed and overriding output_base is causing install_base to not be in the sandbox? Just wild-guessing. For reference, I've seen a number of different but very similar errors and they all involve downloading http_archive rules (or similar download rules) for setting up a workspace prior to running a query. I think the intent of output_base was to not disturb the main workspace environment (and thus be able to run concurrently), but maybe the install_base is write-only in that case? For what it's worth the install_base HAD been populated with the downloads prior to running the query with --output_base, so I'm guessing the need to re-download is because changing output_base invalidated something. Maybe that's a clue? |
|
Is anyone actively working on this? This is a pretty annoying bug that affects how e.g. Jenkins build nodes have to be spawned because the cannot handle multiple executors being isolated with |
|
I found a workaround, but I don't know why it works:
I tried setting I hope that helps. |
laurentlb
added
the
team-Core
|
The basic problem here is that the bazel-$WORKSPACE convenience symlink is blindly traversed by Bazel: you can even do My suggestion for a workaround for now is to do cc @mhy1992 |
janakdr
added
P2
|
I found that janakdr's solution works on a simple Bazel project which have very little external dependency. But |
|
Basic question for people experiencing this issue: can the "alternate" output_root invocation just pass |
|
At least for me, neither |
|
Does passing |
|
Not for me. In fact, with that option, I get even more error messages. |
|
The workarounds don't work for me either. Deleting the symlinks did not help, nor did passing in the flags mentioned. I get error messages of the form: I am wondering what state is causing Bazel to get confused even when the symlinks have all been deleted. I also see errors of the form: This suggests that somehow the new output folder is looking for files in the old one through the symlinks. Perhaps I missed deleting some symlinks. Deleting my_old_bazel_output and restarting the bazel server seems to fix the problem but that prevents the ability to have two or more concurrent build folders that the user can switch between. This issue was surprising to me, given the Bazel output directory layout page explicitly mentions:
EDIT: Ignore my comment. I triaged my particular issue to #13601. |
|
For those who are having this failure with the VSCode Bazel Extension with an error along the lines of: For me, this problem was rectified by opening the VSCode Bazel extension settings, finding the "Queries Share Server" checkbox, and ensuring it is ticked (unticked by default). The box has the description: "Use the same Bazel server for queries and builds. By default, vscode-bazel uses a separate server for queries so that they can be executed in parallel with builds. You can enable this setting if running multiple Bazel servers has a negative performance impact on your system, but you may experience degraded performance in Visual Studio Code for operations that require queries." I think the root cause is probably related to this issue, because (I think) to get the separate server the |
|
For those coming from the future: You'll also get this sort of error if the workspace name in your WORKSPACE file differs from the enclosing directory name. e.g. if you have foo/WORKSPACE and that file contains 'workspace(name="bar")' you will get weird and confusing errors. |
|
FYI this is the most commonly reported issue on the vscode bazel extension, and there is no clear workaround for it while using a distinct output base. I initially thought we could pass |
|
Possibly the issue lies here: |
When the output base is changed after the convenience symlinks, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds an additional check to see whether they have the symlink prefix. This will not catch both the symlink prefix and the output base changing, but I think this case is rare. Fixes bazelbuild#10653
The bazel query used to introspect the targets available incorrectly traverses convenience symlinks when the output base has changed. This is a bug in Bazel (see bazelbuild/bazel#10653). A lot of people are hitting this, since it happens with the default setup of bazel and this plugin. Until it is fixed upstream, this works around this issue by enabling the `queriesShareServer` option by default. Workaround for #216 Co-authored-by: Cameron Martin <cameronmartin123@gmail.com>
When the output base is changed after the convenience symlinks have been created, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds some heuristics to determine if a symlink is a convenience symlink: 1. The symlink name has an appropriate suffix. 2. An ancestor of the symlink target at the appropriate level is called `execroot`. 3. The `execroot` directory contains a file called `DO_NOT_BUILD_HERE`. These heuristics should work if both the output base and the symlink prefix change while being quite robust to false positives. Fixes bazelbuild#10653
When the output base is changed after the convenience symlinks have been created, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds some heuristics to determine if a symlink is a convenience symlink: 1. The symlink name has an appropriate suffix. 2. An ancestor of the symlink target at the appropriate level is called `execroot`. 3. The `execroot` directory contains a file called `DO_NOT_BUILD_HERE`. These heuristics should work if both the output base and the symlink prefix change while being quite robust to false positives. Fixes bazelbuild#10653
When the output base is changed after the convenience symlinks have been created, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds some heuristics to determine if a symlink is a convenience symlink: 1. The symlink name has an appropriate suffix. 2. An ancestor of the symlink target at the appropriate level is called `execroot`. 3. The `execroot` directory contains a file called `DO_NOT_BUILD_HERE`. These heuristics should work if both the output base and the symlink prefix change while being quite robust to false positives. Fixes bazelbuild#10653
When the output base is changed after the convenience symlinks have been created, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds some heuristics to determine if a symlink is a convenience symlink: 1. The symlink name has an appropriate suffix. 2. An ancestor of the symlink target at the appropriate level is called `execroot`. 3. The `execroot` directory contains a file called `DO_NOT_BUILD_HERE`. These heuristics should work if both the output base and the symlink prefix change while being quite robust to false positives. Fixes bazelbuild#10653
When the output base is changed after the convenience symlinks have been created, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds some heuristics to determine if a symlink is a convenience symlink: 1. The symlink name has an appropriate suffix. 2. An ancestor of the symlink target at the appropriate level is called `execroot`. 3. The `execroot` directory contains a file called `DO_NOT_BUILD_HERE`. These heuristics should work if both the output base and the symlink prefix change while being quite robust to false positives. This is important for IDE integration, where the [output base is often changed](https://bazel.build/run/scripts#output-base-option) for queries, to prevent concurrent builds from blocking them. An example of this is the vscode-bazel extension. Fixes bazelbuild#10653 Fixes bazelbuild#13951 Closes bazelbuild#21005. PiperOrigin-RevId: 610667735 Change-Id: I1869c9a2063f7f526950e48c0b1ee6efa89fd202
…21505) When the output base is changed after the convenience symlinks have been created, queries such as `//...` try to traverse into them and load them as packages since they are only excluded based on whether the resolved location is in the output base. This adds some heuristics to determine if a symlink is a convenience symlink: 1. The symlink name has an appropriate suffix. 2. An ancestor of the symlink target at the appropriate level is called `execroot`. 3. The `execroot` directory contains a file called `DO_NOT_BUILD_HERE`. These heuristics should work if both the output base and the symlink prefix change while being quite robust to false positives. This is important for IDE integration, where the [output base is often changed](https://bazel.build/run/scripts#output-base-option) for queries, to prevent concurrent builds from blocking them. An example of this is the vscode-bazel extension. Fixes #10653 Fixes #13951 Closes #21005. Commit 8f74ace PiperOrigin-RevId: 610667735 Change-Id: I1869c9a2063f7f526950e48c0b1ee6efa89fd202 Co-authored-by: Cameron Martin <cameronmartin123@gmail.com> Co-authored-by: Yun Peng <pcloudy@google.com>
|
A fix for this issue has been included in Bazel 7.1.0 RC2. Please test out the release candidate and report any issues as soon as possible. |
|
Hi, I found this issue is still not properly handled, it still misses a case when the output_base is inside the workspace |
@LittleCuteBug do you have a reproducible code or repo? |
Description of the problem / feature request:
Change in --output_base parameter causes the build to break with very obscure error messages.
Feature requests: what underlying problem are you trying to solve with this feature?
When build is done using some IDEs (VSCode or IntelliJ) they tend to set their own --output_base, different from the one configured by the user for the command line builds. This should not cause any problems, but unfortunately it appears that after --output_base changes the build is broken.
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Select any simplest Bazel project. Build it with
bazel --output_base=C:/blah1 build ...so it builds successfully. Then try to build it withbazel --output_base=C:/blah2 build ...this time it breaks with the error messages which don't make any sense.What operating system are you running Bazel on?
The problem does not seem to be OS dependent.
What's the output of
bazel info release?2.0.0
Any other information, logs, or outputs that you want to share?
I figured that the problem is must probably caused by the stale "courtesy" symlink in WORKSPACE folder. After the first build Bazel creates "courtesy" symlink such as
bazel-<workspace_name>in the workspace folder and it points inside output_base. When we issue second build command with the different output_base Bazel is smart enough to realize that and spawn second build server process, but unfortunately that stalebazel-<workspace_name>symlink still stays and points inside the old output_base which seems to confuse Bazel. Runningbazel cleanbetween builds or simply deleting of that symlink fixes the problem. It looks like when Bazel discovers the change in startup parameters which warrants spawning new build server it should at the same time remove existing courtesy symlinks as they are not valid anymore and cause the build to fail.The text was updated successfully, but these errors were encountered: