Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--remote_download_toplevel downloads stuff silently #17417

Closed
jmmv opened this issue Feb 3, 2023 · 2 comments
Closed

--remote_download_toplevel downloads stuff silently #17417

jmmv opened this issue Feb 3, 2023 · 2 comments
Assignees
@coeuvre
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@jmmv
Copy link
Contributor

jmmv commented Feb 3, 2023

Description of the bug:

When building anything on a remote build farm with --remote_download_toplevel, Bazel gets "stuck" after the INFO: Build completed successfully message. When this happen, the network is busy like crazy downloading the request artifacts, which "works" but has a bad UI.

Compare this to building without any --remote_download_* flags, where Bazel will properly print progress messages for every download it executes.

I traced this down to the BwtB downloader code, which enqueues downloads as futures and then... forgets about them until the command shutdown hook waits for their completion without any kind of progress reporting.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

  1. Write a genrule that produces a multi-GB file.
  2. Build the target with --remote_download_toplevel.
  3. Notice how Bazel pauses after completing the command but before returning control to the user. Monitor network activity and notice the download happening in the background.

Which operating system are you running Bazel on?

Doesn't matter.

What is the output of bazel info release?

bazel release 6.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response
@brentleyjones
Copy link
Contributor

@coeuvre

@sgowroji sgowroji added type: bug untriaged team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Feb 4, 2023
@coeuvre coeuvre self-assigned this Feb 6, 2023
@coeuvre coeuvre added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Feb 6, 2023
@coeuvre coeuvre mentioned this issue Feb 27, 2023
Closed
coeuvre added a commit to coeuvre/bazel that referenced this issue Feb 28, 2023
@coeuvre
Report background download for BwoB
e86f932 
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

```
[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```

For downloads after build is completed, the UI looks like:

```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```

Fixes bazelbuild#17417.

Closes bazelbuild#17604.

PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
@coeuvre coeuvre mentioned this issue Feb 28, 2023
Merged
ShreeM01 pushed a commit that referenced this issue Feb 28, 2023
@coeuvre
Report background download for BwoB (#17619)
034a281 
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

```
[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```

For downloads after build is completed, the UI looks like:

```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```

Fixes #17417.

Closes #17604.

PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
@keertk
Copy link
Member

keertk commented Feb 28, 2023

@bazel-io fork 6.1.0

@bazel-io bazel-io mentioned this issue Feb 28, 2023
Closed
fweikert pushed a commit to fweikert/bazel that referenced this issue May 25, 2023
@coeuvre @fweikert
Report background download for BwoB
8076e02 
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.

This PR makes output downloads of toplevel targets visible to the UI.

For downloads during the build, the UI looks like:

```
[2 / 5] 2 actions, 1 running
    [Prepa] Executing genrule //:foo
    Executing genrule //:bar; 2s
    Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```

For downloads after build is completed, the UI looks like:

```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
    Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```

Fixes bazelbuild#17417.

Closes bazelbuild#17604.

PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coeuvre coeuvre

Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Report background download for BwoB coeuvre/bazel
5 participants
@brentleyjones @jmmv @coeuvre @sgowroji @keertk