Windows: Bazel client is slow #5444

laszlocsomor · 2018-06-21T13:02:14Z

Description of the problem / feature request:

This is an umbrella bug for Bazel client performance tuning on Windows.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Bazel on Windows feels a lot slower than on other platforms. Let's find out why and speed it up.

What operating system are you running Bazel on?

Windows 10, v1709

If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

Built from dc986d2

The text was updated successfully, but these errors were encountered:

When extracting embedded binaries, the client now caches which directories it has already created and won't attempt creating them again. This saves some time on all platforms. With n=10 runs, always starting Bazel with a new --output_user_root and shutting down afterwards, the speedup is: - Linux: from XX.X s average and XX.X s median to XX.X s average and XX.X s median - Windows: from 16.3 s average and 17.4 s median to 13.2 s average and 13.2 s median See bazelbuild#5444

When extracting embedded binaries, the client now caches which directories it has already created and won't attempt creating them again. This saves some time on Windows: from 16.3 sec on average down to 13.2 sec. (n=10 runs, always starting Bazel with a new --output_user_root and shutting down afterwards.) On Linux I see only a marginal speedup, not significant enough to claim credit for it. :) See bazelbuild#5444

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stdout. The enable the profiler, set the BAZEL_CLIENT_PROFILE preprocessor definition. See bazelbuild#5444

The Bazel client on Windows is now 50% faster to check the embedded tools than it was before. Results: - Linux: 20 ms -> 6 ms - Windows: 294 ms -> 133 ms Measurements were done with n=10 runs and a hot server, using blaze::GetMillisecondsMonotonic(). Previously the client performed the same tasks multiple times while trying to determine if a path was a good extracted binary. (E.g. converted the path to Windows format multiple times, checked if it was a directory twice, opened the path twice.) Now the client performes these tasks only once, e.g. it converts path once and stats only once. See #5444 Closes #5445. PiperOrigin-RevId: 201913758

When extracting embedded binaries, the client now caches which directories it has already created and won't attempt creating them again. This saves some time on Windows: from 16.3 sec on average down to 13.2 sec. (n=10 runs, always starting Bazel with a new --output_user_root and shutting down afterwards.) On Linux I see only a marginal speedup, not significant enough to claim credit for it. :) See #5444 Closes #5448. PiperOrigin-RevId: 201933181

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stdout. The enable the profiler, set the BAZEL_CLIENT_PROFILE preprocessor definition. See bazelbuild#5444 Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stdout. The enable the profiler: 1. depend on //src/main/cpp/util:profiler 2. use PrintingStopWatch, PrintingStats, and ScopeProfiler objects, as shown in profiler.h's class comments See bazelbuild#5444 Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stdout. Motivation: I recently needed a profiler for PR bazelbuild#5445 and PR bazelbuild#5448, so I'm adding the polished code now. Usage: 1. depend on //src/main/cpp/util:profiler 2. use PrintingStopWatch, PrintingStats, and ScopeProfiler objects, as shown in profiler.h's class documentation See bazelbuild#5444 Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stdout. Motivation: I recently needed a profiler for PR bazelbuild#5445 and PR bazelbuild#5448, so I'm adding the polished code now. Usage: 1. depend on //src/main/cpp/util:profiler 2. use StopWatch, Task, and ScopedTask objects as shown in profiler.h's class documentation See bazelbuild#5444 Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6

Commit bazelbuild@f5043d6 was incorrect in that it cached the file names, not the directory names. This commit fixes that. I verified that the number of calls to ExtractBlazeZipProcessor::Process is greater than the calls to MakeDirectories within (1038 vs. 172 on Linux). See bazelbuild#5444 Change-Id: I314bdc9337c9782a5ceaed7aac785a552b222b1f

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stderr. Motivation: I recently needed a profiler for PR #5445 and PR #5448, so I'm adding the polished code now. Usage: 1. depend on //src/main/cpp/util:profiler 2. use StopWatch, Task, and ScopedTask objects as shown in profiler.h's class documentation See #5444 Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6 Closes #5461. Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6 PiperOrigin-RevId: 202314319

Commit f5043d6 was incorrect in that it cached the file names, not the directory names. This commit fixes that. I verified that the number of calls to ExtractBlazeZipProcessor::Process is greater than the calls to MakeDirectories within (1038 vs. 172 on Linux). See #5444 Change-Id: I314bdc9337c9782a5ceaed7aac785a552b222b1f Closes #5484. Change-Id: I314bdc9337c9782a5ceaed7aac785a552b222b1f PiperOrigin-RevId: 202314400

Use the new blaze_util::threads::ThreadPool class to parallelize disk I/O in writing the extracted embedded binaries to disk. This speeds up the first run of Bazel on Windows (when extracting the embedded binaries). The time spent extracting and writing those binaries drops by 60% on average. Measurements were done on a 12-core Windows machine with HDD and on a 12-core Debian machine with a SSD. Results for blaze.cc::ActuallyExtractData(): - Windows, 1 thread: 12.79s avg, 12.95s median - Windows, 6 threads: 4.68s avg, 4.21s median - Linux, 1 thread: 7.25s avg, 7.24s median - Linux, 6 threads: 7.05s avg, 7.04s median See bazelbuild#5444 Change-Id: If0971f10abff8195d269074a1dd4be6f0f454576

Add a ThreadPool implementation to the Bazel client. The ThreadPool has a fixed number of std::thread objects that each execute tasks from a shared queue. Tasks are std::function objects. The ThreadPool will be used in blaze.cc::ActuallyExtractData, specifically in ExtractBlazeZipProcessor::Process, to parallelize I/O work. Writing extracted binaries to disk from multiple worker threads instead of from a single thread shows a significant speedup when using a HDD instead of a SSD. See bazelbuild#5444 Change-Id: If0971f10abff8195d269074a1dd4be6f0f454576

The Bazel client on Windows now writes extracted binaries to disk in parallel. On all other systems it writes them serially. The Windows-specific code uses the native Threadpool API of Windows, and uses a pool of at least 8 and at most 16 threads. blaze.cc:ActuallyExtractData() speed, before: - Windows: 6.48s (avg) / 6.38s (median) - Linux (Debian): 4.78s (avg) / 4.79s (median) blaze.cc:ActuallyExtractData() speed, after: - Windows (1 thread): 5.77s (avg) / 5.74s (median) - Windows (8-16 threads): 2.05s (avg) / 2.01s (md) See bazelbuild#5444 Change-Id: I7211f3d28eb8b9837352c16ff8df0411d5a9ebe1

The Bazel client on Windows now writes extracted binaries to disk in parallel. On all other systems it writes them serially (as before). This change makes blaze.cc:ActuallyExtractData() about 3x faster when using a HDD. (In previous experiments I saw no speedup with multi-threaded writing on machines with an SSD.) The Windows-specific code uses the native Threadpool API of Windows, creating a pool of at least 8 and at most 16 threads. The OS manages everything about the pool; Bazel submits callbacks and the pool executes them asynchronously. blaze.cc:ActuallyExtractData() speed, before: - Windows: 6.48s (avg) / 6.38s (median) - Linux (Debian): 4.78s (avg) / 4.79s (median) blaze.cc:ActuallyExtractData() speed, after: - Windows (8-16 threads): 2.05s (avg) / 2.01s (md) - Windows (1 thread): 5.77s (avg) / 5.74s (median) See bazelbuild#5444 Change-Id: I7211f3d28eb8b9837352c16ff8df0411d5a9ebe1

The Bazel client on Windows now writes extracted binaries to disk in parallel. On all other systems it writes them serially (as before). This change makes blaze.cc:ActuallyExtractData() about 3x faster when using a HDD. (In previous experiments I saw no speedup with multi-threaded writing on machines with an SSD.) The Windows-specific code uses the native Threadpool API of Windows, creating a pool of at least 8 and at most 16 threads. (This seems to be a good balance between speed and thread count.) The OS manages everything about the pool; Bazel submits callbacks and the pool executes them asynchronously. blaze.cc:ActuallyExtractData() speed, before: - Windows: 6.48s (avg) / 6.38s (median) - Linux (Debian): 4.78s (avg) / 4.79s (median) blaze.cc:ActuallyExtractData() speed, after: - Windows (8-16 threads): 2.05s (avg) / 2.01s (md) - Windows (1 thread): 5.77s (avg) / 5.74s (median) See bazelbuild#5444 Change-Id: I7211f3d28eb8b9837352c16ff8df0411d5a9ebe1

The Bazel client on Windows now writes extracted binaries to disk in parallel. On all other systems it writes them serially (as before). This change makes blaze.cc:ActuallyExtractData() about 3x faster when using a HDD. (In previous experiments I saw no speedup with multi-threaded writing on machines with an SSD.) The Windows-specific code uses the native Threadpool API of Windows, creating a pool of at least 8 and at most 16 threads. (This seems to be a good balance between speed and thread count.) The OS manages everything about the pool; Bazel submits callbacks and the pool executes them asynchronously. blaze.cc:ActuallyExtractData() speed, before: - Windows: 6.48s (avg) / 6.38s (median) - Linux (Debian): 4.78s (avg) / 4.79s (median) blaze.cc:ActuallyExtractData() speed, after: - Windows (8-16 threads): 2.05s (avg) / 2.01s (md) - Windows (1 thread): 5.77s (avg) / 5.74s (median) See #5444 Change-Id: I7211f3d28eb8b9837352c16ff8df0411d5a9ebe1 Closes #5600. Change-Id: I7a74d62a563c92948a4dfa8ad5ac83eae018db10 PiperOrigin-RevId: 204891217

The Bazel client on Windows is now 50% faster to check the embedded tools than it was before. Results: - Linux: 20 ms -> 6 ms - Windows: 294 ms -> 133 ms Measurements were done with n=10 runs and a hot server, using blaze::GetMillisecondsMonotonic(). Previously the client performed the same tasks multiple times while trying to determine if a path was a good extracted binary. (E.g. converted the path to Windows format multiple times, checked if it was a directory twice, opened the path twice.) Now the client performes these tasks only once, e.g. it converts path once and stats only once. See bazelbuild#5444 Closes bazelbuild#5445. PiperOrigin-RevId: 201913758

When extracting embedded binaries, the client now caches which directories it has already created and won't attempt creating them again. This saves some time on Windows: from 16.3 sec on average down to 13.2 sec. (n=10 runs, always starting Bazel with a new --output_user_root and shutting down afterwards.) On Linux I see only a marginal speedup, not significant enough to claim credit for it. :) See bazelbuild#5444 Closes bazelbuild#5448. PiperOrigin-RevId: 201933181

Add a simple profiler that can measure function call counts and durations, and report statistics by printing to stderr. Motivation: I recently needed a profiler for PR bazelbuild#5445 and PR bazelbuild#5448, so I'm adding the polished code now. Usage: 1. depend on //src/main/cpp/util:profiler 2. use StopWatch, Task, and ScopedTask objects as shown in profiler.h's class documentation See bazelbuild#5444 Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6 Closes bazelbuild#5461. Change-Id: I43f0afd124b486c694f451e8455a66ffca8137b6 PiperOrigin-RevId: 202314319

Commit bazelbuild@f5043d6 was incorrect in that it cached the file names, not the directory names. This commit fixes that. I verified that the number of calls to ExtractBlazeZipProcessor::Process is greater than the calls to MakeDirectories within (1038 vs. 172 on Linux). See bazelbuild#5444 Change-Id: I314bdc9337c9782a5ceaed7aac785a552b222b1f Closes bazelbuild#5484. Change-Id: I314bdc9337c9782a5ceaed7aac785a552b222b1f PiperOrigin-RevId: 202314400

The Bazel client on Windows now writes extracted binaries to disk in parallel. On all other systems it writes them serially (as before). This change makes blaze.cc:ActuallyExtractData() about 3x faster when using a HDD. (In previous experiments I saw no speedup with multi-threaded writing on machines with an SSD.) The Windows-specific code uses the native Threadpool API of Windows, creating a pool of at least 8 and at most 16 threads. (This seems to be a good balance between speed and thread count.) The OS manages everything about the pool; Bazel submits callbacks and the pool executes them asynchronously. blaze.cc:ActuallyExtractData() speed, before: - Windows: 6.48s (avg) / 6.38s (median) - Linux (Debian): 4.78s (avg) / 4.79s (median) blaze.cc:ActuallyExtractData() speed, after: - Windows (8-16 threads): 2.05s (avg) / 2.01s (md) - Windows (1 thread): 5.77s (avg) / 5.74s (median) See bazelbuild#5444 Change-Id: I7211f3d28eb8b9837352c16ff8df0411d5a9ebe1 Closes bazelbuild#5600. Change-Id: I7a74d62a563c92948a4dfa8ad5ac83eae018db10 PiperOrigin-RevId: 204891217

The Bazel client on Windows is now 50% faster to check the embedded tools than it was before. Results: - Linux: 20 ms -> 6 ms - Windows: 294 ms -> 133 ms Measurements were done with n=10 runs and a hot server, using blaze::GetMillisecondsMonotonic(). Previously the client performed the same tasks multiple times while trying to determine if a path was a good extracted binary. (E.g. converted the path to Windows format multiple times, checked if it was a directory twice, opened the path twice.) Now the client performes these tasks only once, e.g. it converts path once and stats only once. See bazelbuild#5444 Closes bazelbuild#5445. PiperOrigin-RevId: 201913758

laszlocsomor · 2019-02-28T15:01:21Z

The multi-threaded extraction was a significant speedup. I'm dropping priority to P3.

laszlocsomor · 2019-07-05T08:31:32Z

Thanks to the multi-threaded extraction, the client is faster now than a year ago.
We can reopen this bug if more improvements are needed.

laszlocsomor self-assigned this Jun 21, 2018

laszlocsomor added P1 I'll work on this now. (Assignee required) type: process platform: windows labels Jun 21, 2018

laszlocsomor mentioned this issue Jun 21, 2018

Windows,Bazel client: check embedded tools faster #5445

Closed

laszlocsomor mentioned this issue Jun 21, 2018

Windows,client: extract embedded binaries faster #5448

Closed

laszlocsomor mentioned this issue Jun 25, 2018

Bazel client: add a simple profiler #5461

Closed

laszlocsomor mentioned this issue Jun 27, 2018

Bazel,client: actually cache directory creations #5484

Closed

This was referenced Jun 28, 2018

Bazel client: implement a simple ThreadPool #5487

Closed

Bazel on Windows: improve first-time experience #5254

Closed

laszlocsomor mentioned this issue Jul 16, 2018

Windows,Bazel client: multithreaded file writing #5600

Closed

laszlocsomor added P3 We're not considering working on this, but happy to review a PR. (No assignee) area-Windows Windows-specific issues and feature requests and removed P1 I'll work on this now. (Assignee required) labels Feb 28, 2019

laszlocsomor closed this as completed Jul 5, 2019

philwo added the team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website label Jun 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows: Bazel client is slow #5444

Windows: Bazel client is slow #5444

laszlocsomor commented Jun 21, 2018

laszlocsomor commented Feb 28, 2019

laszlocsomor commented Jul 5, 2019

Windows: Bazel client is slow #5444

Windows: Bazel client is slow #5444

Comments

laszlocsomor commented Jun 21, 2018

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

laszlocsomor commented Feb 28, 2019

laszlocsomor commented Jul 5, 2019

If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.