Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize prefetchInputs. #20557

Closed
wants to merge 3 commits into from
Closed

Optimize prefetchInputs. #20557

wants to merge 3 commits into from

Conversation

coeuvre
Copy link
Member

@coeuvre coeuvre commented Dec 15, 2023

Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads.

This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms.

Fixes #20555.

@coeuvre
Optimize prefetchInputs.
2bb70fd 
Use a pre-allocated array to hold the intermediate transfers to avoid
allocations. Replace some of RxJava code with Futures to avoid RxJava
overheads.

This improves the perfromance of prefetchInputs on a large set of
inputs from ~400ms to ~16ms.

Fixes bazelbuild#20555.
@coeuvre coeuvre requested a review from a team as a code owner December 15, 2023 08:06
@github-actions github-actions bot added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels Dec 15, 2023
@coeuvre coeuvre requested a review from tjgq December 15, 2023 08:08
@brentleyjones
Copy link
Contributor

@bazel-io flag

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Dec 15, 2023
@brentleyjones
Copy link
Contributor

@coeuvre How easy would this be to cherry-pick onto 6.5.0 as well as 7.1.0?

@iancha1992
Copy link
Member

@bazel-io fork 6.5.0

@iancha1992
Copy link
Member

@bazel-io fork 7.1.0

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Dec 15, 2023
This was referenced Dec 15, 2023
Closed
Closed
@coeuvre
Copy link
Member Author

coeuvre commented Dec 18, 2023

@coeuvre How easy would this be to cherry-pick onto 6.5.0 as well as 7.1.0?

Should be straightforward for both branches.

@iancha1992
Copy link
Member

@tjgq is this ready for merge to master?

cc: @bazelbuild/triage

@coeuvre
Copy link
Member Author

coeuvre commented Dec 19, 2023

@iancha1992 I addressed the comments. Should be ready to merge.

@coeuvre coeuvre added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Dec 19, 2023
@copybara-service copybara-service bot closed this in 915fb3e Jan 2, 2024
@github-actions github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Jan 2, 2024
bazel-io pushed a commit to bazel-io/bazel that referenced this pull request Jan 2, 2024
@coeuvre @bazel-io
Optimize prefetchInputs.
4bbeded 
Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads.

This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms.

Fixes bazelbuild#20555.

Closes bazelbuild#20557.

PiperOrigin-RevId: 595226013
Change-Id: If5296fa6b3c0166b95cfca4281255e523724a41f
@bazel-io bazel-io mentioned this pull request Jan 2, 2024
Merged
@coeuvre coeuvre deleted the fix-20555 branch January 4, 2024 05:17
github-merge-queue bot pushed a commit that referenced this pull request Jan 8, 2024
@bazel-io @coeuvre
@Wyverald @iancha1992 @meteorcloudy
[7.1.0] Optimize prefetchInputs. (#20719)
26a733c 
Use a pre-allocated array to hold the intermediate transfers to avoid
allocations. Replace some of RxJava code with Futures to avoid RxJava
overheads.

This improves the perfromance of prefetchInputs on a large set of inputs
from ~400ms to ~16ms.

Fixes #20555.

Closes #20557.

Commit
915fb3e

PiperOrigin-RevId: 595226013
Change-Id: If5296fa6b3c0166b95cfca4281255e523724a41f

Co-authored-by: Chi Wang <chiwang@google.com>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
copybara-service bot pushed a commit that referenced this pull request Jan 17, 2024
@coeuvre @tkiela1
Automated rollback of commit 915fb3e.
162cacd 
*** Reason for rollback ***

Might cause build to hang forever.

b/320630578

*** Original change description ***

Optimize prefetchInputs.

Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads.

This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms.

Fixes #20555.

Closes #20557.

PiperOrigin-RevId: 599135847
Change-Id: Idae6a1c57e634d16091e31e097b16ca97a67e62d
fweikert pushed a commit that referenced this pull request Jan 25, 2024
@coeuvre @fweikert
Automated rollback of commit 915fb3e.
b4343cc 
*** Reason for rollback ***

Might cause build to hang forever.

b/320630578

*** Original change description ***

Optimize prefetchInputs.

Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads.

This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms.

Fixes #20555.

Closes #20557.

PiperOrigin-RevId: 599135847
Change-Id: Idae6a1c57e634d16091e31e097b16ca97a67e62d
copybara-service bot pushed a commit that referenced this pull request Jan 30, 2024
@coeuvre @tkiela1
Roll forward of 915fb3e: Optimize prefetchInputs.
20e89aa 
Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads.

This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms.

Fixes #20555.

Closes #20557.

Automated rollback of commit 162cacd.

*** Reason for rollback ***

unknown commit is the root cause, not this one.

*** Original change description ***

Automated rollback of commit 915fb3e.

*** Reason for rollback ***

Might cause build to hang forever.

b/320630578

*** Original change description ***

Optimize prefetchInputs.

***

PiperOrigin-RevId: 602680109
Change-Id: I357dadd499fecf295e8e87c6fe2892578f3109b1
@coeuvre coeuvre mentioned this pull request Jan 31, 2024
Open
fweikert pushed a commit that referenced this pull request Feb 5, 2024
@coeuvre @fweikert
Automated rollback of commit 915fb3e.
71b68c4 
*** Reason for rollback ***

Might cause build to hang forever.

b/320630578

*** Original change description ***

Optimize prefetchInputs.

Use a pre-allocated array to hold the intermediate transfers to avoid allocations. Replace some of RxJava code with Futures to avoid RxJava overheads.

This improves the perfromance of prefetchInputs on a large set of inputs from ~400ms to ~16ms.

Fixes #20555.

Closes #20557.

PiperOrigin-RevId: 599135847
Change-Id: Idae6a1c57e634d16091e31e097b16ca97a67e62d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tjgq tjgq tjgq approved these changes

Assignees
No one assigned
Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

context.prefetchInputs is slow on large set of inputs
5 participants
@coeuvre @brentleyjones @iancha1992 @tjgq @bazel-io