bazel fails with Failed to parse downloader config

[evis]

Description of the bug:

During April, we encounter such a problem three times (on different machines): any bazel command fails locally for some developer with:

% bazel build //...
INFO: Invocation ID: 40c6fcdb-de05-4132-99f6-80ab60643974
ERROR: Failed to parse downloader config: bazel_downloader.cfg

Every time, it works good for everyone else, so, looks like a local problem.

At the end of March, we upgraded Bazel from 6.2.1 to 7.1.1.

Workaround that we found: restart Macbook.

Relevant part of our .bazelrc:

build --experimental_downloader_config=bazel_downloader.cfg
sync --experimental_downloader_config=bazel_downloader.cfg
query --experimental_downloader_config=bazel_downloader.cfg

bazel_downloader.cfg:

rewrite (github.com/.*) https://s3.mds.yandex.net/bazel-build/mirror/$1

block github.com

Any ideas, how to fix it? Or maybe some tips, how to debug the problem next time?

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Mac OS

What is the output of bazel info release?

release 7.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[artem-zinnatullin]

We're seeing this multiple times per day for some reason it happens mostly on macOS CI machines and the issue persists between retries if it bounces onto the same macOS machine.

2024/05/07 13:34:08 Downloading https://releases.bazel.build/8.0.0/rolling/8.0.0-pre.20240128.3/bazel-8.0.0-pre.20240128.3-darwin-arm64...
--
  | WARNING: option '--remote_download_outputs' was expanded to from both option '--remote_download_minimal' (source expanded from --config=ci) and option '--remote_download_toplevel' (source expanded from --config=ci)
  | ERROR: Failed to parse downloader config: tools/bazel/bazel-downloader-config.txt

[KostadinAlmishev]

I thing code fails here, this is called from here.

During bazel build fails with error above (and running local bazel server), I tried to read the file from different program the same way this is done in bazel. Exception didn't occurred:

scala> Files.newBufferedReader(Paths.get("bazel_downloader.cfg"))
val res10: java.io.BufferedReader = java.io.BufferedReader@22bd6b0a

scala> res10.readLine()
val res11: String = rewrite (github.com/.*) https://<example masked>/bazel-build/mirror/$1

scala> res10.readLine()
val res12: String = ""

scala> res10.readLine()
val res13: String = block github.com

scala> res10.readLine()
val res14: String = null

As a result I founded another workaround - killing local bazel server process.

Our code is mounted with fuse filesystem. Sometimes this happens, when developer remounts the system, when local bazel server is running.

Current bazel version: 7.1.1
With previous version 6.2.1 we didn't have this problem.

[meteorcloudy]

/cc @shs96c Can you please take look?

[shs96c]

I can have a look. I'm working on a reproduction so I can debug it better. If anyone has an easy way to get this to occur reliably, that would be very helpful