Slow Build Time with Ninja Build System for ClickHouse

[Zawa-ll]

Company or project name

No response

Question

Hello ClickHouse team,

I am experiencing significant delays while building ClickHouse using the ninja build system. Despite attempting several optimizations, the build process remains quite slow, particularly when running the command ninja clickhouse clickhouse-tests.

Steps to Reproduce:

Configure the build with cmake:
cmake -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX ..
Run the build using ninja:
ninja -j$(nproc) clickhouse clickhouse-tests

Error Logs:
When running the build, it seems to hang or take a particularly long time(nearly 48 hours so far) during certain steps. Here is an example output:

[18/2079] Building CXX object src/CMakeFiles/dbms.dir/Interpreters/InterpreterSystemQuery.cpp.o
FAILED: src/CMakeFiles/dbms.dir/Interpreters/InterpreterSystemQuery.cpp.o
...

Additional Context:
I am currently working on issue #66074 (SYSTEM LOAD PRIMARY KEY). The changes are in place, but due to the extended build times, I am unable to efficiently run the unit tests I added to verify the changes. Any guidance on how to further optimize the build process or potential areas to investigate would be greatly appreciated!

Attempts to Resolve
I have tried the following steps to optimize the build process:

Increased parallel jobs with -j20
Limited build targets
Cleared temporary files
Ensured sufficient disk space is available
Used ccache to speed up recompilations
Clean build

[Algunenano]

• What compiler are you using?
• FAILED: src/CMakeFiles/dbms.dir/Interpreters/InterpreterSystemQuery.cpp.o What was the error?
• Does something odd appears if you run ninja -v
• Do you have enough memory to run as many parallel jobs as you are trying?

[Zawa-ll]

• What compiler are you using?
• FAILED: src/CMakeFiles/dbms.dir/Interpreters/InterpreterSystemQuery.cpp.o What was the error?
• Does something odd appears if you run ninja -v
• Do you have enough memory to run as many parallel jobs as you are trying?

Thank you for your reply, I really appreciate it!
Q1 - I'm using the LLVM compiler for the build process.
Q2 - I accidentally attached the wrong log in my previous message. The log I see when the process is blocked does not include the error message FAILED: src/CMakeFiles/dbms.dir/Interpreters/InterpreterSystemQuery.cpp.o. The build process appears to be getting stuck or taking an unusually long time without generating specific error messages.

[18/2079] Building CXX object src/CMakeFiles/dbms.dir/Interpreters/InterpreterSystemQuery.cpp.o actually I can only see this line when running ninja

Q3 - Running ninja -v does not show any unusual messages, but the build process still experiences significant delays.

I could see log like this being displayed, and nothing get changed even after a long time.
[0/2] /opt/homebrew/Cellar/cmake/3.30.0/bin/cmake -P /Users/lizexu/Desktop/opensource/ClickHouse/build/CMakeFiles/VerifyGlobs.cmake

Q4 - I have checked my system's resources, and there seems to be enough memory available. However, I am encountering significant delays regardless of the number of parallel jobs I configure.

[Algunenano]

What processes are being run? Check ps | grep clang

[Zawa-ll]

What processes are being run? Check ps | grep clang

Thank you for your response! Here is the log displayed when running ps | grep clang

lizexu@Haydn-3 build % ps | grep clang
71563 ttys003 0:00.00 grep clang

[Algunenano]

Check what processes are being run when you are building and it's slow

[Zawa-ll]

I see, a supre long log is displayed when running ps | grep clang while running ninja
It was way too long that I don't think it would be a good idea to paste it here direcly... Can you please take a look? Appreciate it!

_INVOKE_RESULT=1 -DBOOST_ASIO_STANDALONE=1 -DBOOST.txt

[Algunenano]

I see nothing wrong in that log. It's just building multiple objects in parallel.

• Is it progressing or is it always stuck in 18/2079?
• Is the CPU at 100? Many parallel process while the system is doing something else could be an issue
• Is the CPU being throttled because of temperature (laptop)?
• Does it progress (albeit slowly) with just one thread? (ninja -j1)?

[Zawa-ll]

I see nothing wrong in that log. It's just building multiple objects in parallel.

• Is it progressing or is it always stuck in 18/2079?
• Is the CPU at 100? Many parallel process while the system is doing something else could be an issue
• Is the CPU being throttled because of temperature (laptop)?
• Does it progress (albeit slowly) with just one thread? (ninja -j1)?

Thank you for your response:
Q1 - The build process is progressing, but it is extremely slow. It is not like stuck at 18/2079, but each time it increase it took a super long time, though it does eventually move forward.

Q2 - I observed that the CPU usage is high but not necessarily at 100%. The system has enough resources and is not heavily loaded by other processes.

Q3 - I have considered this possibility, but I have not observed any significant thermal throttling that would explain the delay. The CPU temperature appears to be within normal operating ranges.

Q4 - After switching to -j1, the speed did increase. On average, it took about 20 seconds for each step to progress.

Additionally, I am currently using a MacBook for the build. Would switching to a device with higher resources potentially improve the build speed?

[Algunenano]

If it's faster with -j1 I'd say that it's highly likely that you are getting throttled because of temperature or it's a slow disk. Considering it seems a recent MacBook I'd discard the disk and it's most likely a temperature problem.

Would switching to a device with higher resources potentially improve the build speed?

Definitely. A clean build (without ccache) is possible in under 20 minutes with a high-end desktop.

[Zawa-ll]

Thank you for your assistance and insights!
To ensure I'm using the optimal command for running a build without adding unexpected flags, could you please let me know what command is normally used for running a standard build?

[Algunenano]

To ensure I'm using the optimal command for running a build without adding unexpected flags, could you please let me know what command is normally used for running a standard build?

The standard build (in Linux) is simply build and not change flags:

mkdir build
cd build
cmake ..
ninja

But it's totally up to you to know the limits of your system and change if necessary.