KMeans segfaults on 344-logical-core box: concurrent single-threaded callers exhaust NUM_BUFFERS(256) even with threads pinned to 1; exhaustion path returns NULL instead of terminating

[cakedev0]

Hi, I'm working on scikit-learn and testing/benchmarking it on big machines.

I used Claude to help investigate this and draft the report, I tried to make sure it was relevant and to make it digest.

Summary

sklearn.cluster.KMeans.fit() reliably crashes (segfault or corrupted size vs. prev_size) on a 344-logical-core machine when linked against OpenBLAS's pthreads threading layer. scikit-learn is well aware of BLAS-oversubscription hazards and explicitly guards its Lloyd loop with threadpoolctl, forcing openblas_get_num_threads() == 1 for the whole call. I verified that guard is genuinely active at the moment of the crash. It doesn't help, because the actual trigger doesn't involve OpenBLAS recruiting extra threads per call at all — it's simpler than that:

sklearn's own outer parallelism (one OpenMP worker per logical core — 344 here — in its prange distance-computation loop) means 344 threads hit blas_memory_alloc() at roughly the same time (OpenMP's fork/join barrier), each wanting one buffer for its own single-threaded BLAS call. NUM_BUFFERS (the size of the memory[] allocator table) is 256 on this build. 344 concurrent callers against 256 slots exhausts the table regardless of how many threads any individual call is allowed to use.

Minimal reproducer

/* ob_repro.c — build: gcc -fopenmp -O2 ob_repro.c -o ob_repro -l:libopenblas.so.0
 * run:  ./ob_repro <n_threads> <n_iters> <matrix_size>   e.g. ./ob_repro 344 50 128
 *
 * openblas_set_num_threads(1) mirrors scikit-learn's own
 * threadpool_limits(limits=1, user_api="blas") guard around its Lloyd loop.
 * Crashes anyway: the hazard is *how many threads call in*, not how many
 * threads OpenBLAS recruits per call. */
#include <stdio.h>
#include <stdlib.h>

extern void dgemm_(const char *transa, const char *transb,
                    const int *m, const int *n, const int *k,
                    const double *alpha, const double *a, const int *lda,
                    const double *b, const int *ldb,
                    const double *beta, double *c, const int *ldc);
extern void openblas_set_num_threads(int);

int main(int argc, char **argv) {
    int n_threads = argc > 1 ? atoi(argv[1]) : 344;
    int n_iters   = argc > 2 ? atoi(argv[2]) : 50;
    int m         = argc > 3 ? atoi(argv[3]) : 128;

    openblas_set_num_threads(1);   /* <- pinned, like scikit-learn does */

    char ta = 'N', tb = 'N';
    double alpha = 1.0, beta = 0.0;
    int it;

    for (it = 0; it < n_iters; it++) {
        #pragma omp parallel num_threads(n_threads)
        {
            double *A = malloc(sizeof(double) * m * m);
            double *B = malloc(sizeof(double) * m * m);
            double *C = malloc(sizeof(double) * m * m);
            int i;
            for (i = 0; i < m * m; i++) { A[i] = (double)(i % 97) / 97.0; B[i] = (double)(i % 89) / 89.0; }

            dgemm_(&ta, &tb, &m, &m, &m, &alpha, A, &m, B, &m, &beta, C, &m);

            free(A); free(B); free(C);
        }
    }
    printf("done, no crash (n_threads=%d n_iters=%d m=%d, num_threads pinned to 1)\n", n_threads, n_iters, m);
    return 0;
}

6/6 runs of ./ob_repro 344 50 128 crash (mix of SIGSEGV and SIGABRT with glibc's corrupted size vs. prev_size — real heap corruption, not a clean stop). ./ob_repro 344 50 64 (smaller matrix) never crashes — small enough calls apparently skip the buffer table entirely, so the effect only shows up once individual calls are large enough to need a real workspace buffer.

Every crash is preceded by:

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
Note that your application may still crash, if it is calling OpenBLAS from multiple threads in parallel
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower

Note this is with num_threads already pinned to 1 — the warning's own suggested fixes don't apply to this trigger at all, since the caller-side thread count isn't something either of those settings controls.

Why this crashes instead of erroring cleanly

Table exhaustion should, per the code's own message, terminate cleanly — but it doesn't:

// driver/others/memory.c
terminate:
  ...
  printf("OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.\n");
  printf("This library was built to support a maximum of %d threads - either rebuild OpenBLAS\n", NUM_BUFFERS);
  ...
  return NULL;      // <- not exit(). Caller gets NULL back instead.

OpenBLAS/driver/others/memory.c

Lines 3148 to 3162 in 62bcfb0

	terminate:
	#if (defined(SMP) || defined(USE_LOCKING)) && !defined(USE_OPENMP)
	UNLOCK_COMMAND(&alloc_lock);
	#endif
	printf("OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.\n");
	printf("This library was built to support a maximum of %d threads - either rebuild OpenBLAS\n", NUM_BUFFERS);
	#ifdef USE_OPENMP
	printf("with a larger NUM_THREADS value or set the environment variable OMP_NUM_THREADS to\n");
	#else
	printf("with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to\n");
	#endif
	printf("a sufficiently small number. This error typically occurs when the software that relies on\n");
	printf("OpenBLAS calls BLAS functions from many threads in parallel, or when your computer has more\n");
	printf("cpu cores than what OpenBLAS was configured to handle.\n");
	return NULL;

The only real exit(1) in that function is a different, unrelated branch (mmap retries exhausted):

OpenBLAS/driver/others/memory.c

Lines 2981 to 2987 in 62bcfb0

	if (((BLASLONG) map_address) == -1) {
	base_address = 0UL;
	failcount++;
	if (failcount >10) {
	fprintf(stderr, "OpenBLAS error: Memory allocation still failed after 10 retries, giving up.\n");
	exit(1);
	}

If the internal caller doesn't check for NULL (or can't safely bail out mid-computation), that's a null-pointer write standing in for the clean stop the message promises — consistent with the segfaults/heap-corruption we actually see.

NUM_BUFFERS is MAX_CPU_NUMBER * 2 * MAX_PARALLEL_NUMBER:

OpenBLAS/common.h

Line 192 in 62bcfb0

#define NUM_BUFFERS MAX(50,(MAX_CPU_NUMBER * 2 * MAX_PARALLEL_NUMBER))

128 * 2 * 1 = 256 for this build, matching what's printed.

Related, possibly same root cause

• BLAS memory allocation error in Scikit-learn KMeans & kNN & DBSCAN #3321 / BLAS memory allocation error in KMeans & kNN & DBSCAN scikit-learn/scikit-learn#20539 — same symptom (KMeans/kNN/DBSCAN on a many-core box). OPENBLAS_NUM_THREADS=64 didn't help there either, consistent with this not being about per-call thread count.
• Race condition in memory.c while running dgemm many times from omp parallel region #2444 — a confirmed, partially-fixed race on memory[position].used visibility, but in the fine-grained-locking (USE_OPENMP) path. The plain pthreads path (!USE_OPENMP, used here) takes a single coarse alloc_lock around the whole scan+claim+free cycle, which reads as correctly serialized to me — so I don't think this is literally the same bug, just the same neighborhood of code.
• A secondary, distinct hazard also exists on this build and is worth keeping in mind separately: without pinning num_threads, a single large-enough GEMM called from inside an already-active OpenMP region triggers OpenBLAS's own "Detect OpenMP Loop and this application may hang" nested-dispatch path, which can also corrupt memory. That one is about per-call thread recruitment, and it's what scikit-learn's guard correctly prevents — it just isn't what's causing the scikit-learn crash specifically, since I confirmed num_threads=1 there.

Environment

OpenBLAS 0.3.33 DYNAMIC_ARCH NO_AFFINITY Cooperlake MAX_THREADS=128

• libopenblas-0.3.33-pthreads_h94d23a6_0 (conda-forge)
• CPU: Intel Xeon 6787P — 172 physical / 344 logical cores
• Linux x86_64, kernel 7.0.0-27-generic
• Python 3.12, numpy 2.5.0, scipy 1.18.0, scikit-learn 1.9.0

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 128
         prefix: libopenblas
        version: 0.3.33
threading_layer: pthreads
   architecture: Cooperlake

       user_api: openmp
   internal_api: openmp
    num_threads: 344
         prefix: libgomp

[martin-frbg]

OK, that's a fairly pathological case - normally you'd want to ensure that OpenBLAS was built with a sufficiently high number of thread buffer "slots". I added a fairly wonky emergency allocator for the more common case of a single multi-threaded caller exceeding the compile-time limit, but as you noticed this does not help against an onslaught of independent callers. Returning NULL to nobody in particular is obviously wrong, but I'm not sure if a low-level library should be able to blow up its caller by invoking exit either ?

[martin-frbg]

(The return(NULL) instead of exit() is original GotoBLAS code - hard to tell now with the Frankenstein memory.c that combines the TLS reimplementation with the original version of memory.c. As nobody seems to check that the buffer is actually non-NULL, perhaps
doing an exit() there is actually the best thing to do )

[martin-frbg]

Actually now that we have a means for the user to specify their own "xerbla" error handler, perhaps the best solution would be to call XERBLA with a made-up error code of (say) 999 "memory allocation failure" and have our xerbla do an exit() on this code after printing the message.