Repository navigation
deadlock on mixed-core machine #3122
Unanswered
FlorinAndrei
asked this question in
Q&A
Replies: 1 comment 1 reply
|
It looks more like a bug so I created an issue for this: #3126. P.S. Why are you running CPU-based training on DGX Spark clone? Is it just for testing such configuration? |
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Running CatBoost 1.2.10 on an NVIDIA DGX Spark clone running Ubuntu 24.04, CUDA 13.0
The machine has 20 ARM cores, 10 P-cores and 10 E-cores - like a MacBook, but it runs Linux.
I'm running CatBoost in Docker, the base image is something like python:3.14.6-slim-trixie or a 3.13 version of it. It's at the end of a scikit-learn Pipeline.
Initially I gave it all CPU cores, one thread per core, so 20 threads total. It kept deadlocking during Optuna studies. The container was up, the process was running, but CPU usage was 0 and it was making no progress. I tried
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1but it didn't help.What did help was pinning the CatBoost container onto the P-cores exclusively:
--cpuset-cpus=5-9,15-19. I only get 10 threads, but the P-cores are 3x faster than the E-cores, so it's not a huge loss. When running this way, it does not seem to deadlock anymore.Is it possible that the combination of Linux on ARM, with the heterogeneous cores, is causing a timing issue which leads to deadlock? I bet this combination is not very widespread, so it's probably less tested.
I asked Claude to provide more technical details, and it said this:
https://gist.github.com/FlorinAndrei/6c2b60b4b0f15836df21a4e20c4b038a
Here is a minimal repro:
https://gist.github.com/FlorinAndrei/575a9e1f0d6e62772d7b54fc09bfea69
All reactions