fix(hnsw): reject index files whose vector count overflows size_t

[belowzeroff]

What & why

ray_hnsw_load / ray_hnsw_mmap read n_nodes and dim directly from the
on-disk index header and sized the vectors allocation as
n_nodes * dim * sizeof(float) with no overflow check (the vectors block of
hnsw_load_impl in src/store/hnsw.c). A crafted header can make that product
wrap size_t, so ray_sys_alloc under-allocates the buffer while the following
fread(vecs, sizeof(float), vec_count, f) still reads the full (large) element
count — writing past the allocation. That is a heap-overflow write driven
entirely by an untrusted index file (an attacker who can supply a matching
hnsw_header.bin + hnsw_vectors.bin).

The per-layer neighbor arrays a few lines down already carry a divide-based
overflow guard; the vectors block was missing the equivalent. This adds the same
guard in header validation, so an overflowing file is rejected before any
allocation:

if ((uint64_t)hdr.n_nodes > SIZE_MAX / sizeof(float) / (uint64_t)hdr.dim)
    return NULL;

n_nodes and dim are already validated > 0 at that point, so the divide is
safe.

Test

embedding/hnsw_load_overflow_header_rejected saves a valid index, patches the
header's n_nodes/dim to an overflowing pair, and asserts the load is refused.
It lives at the C layer rather than under test/rfl/ because it needs byte-level
manipulation of the binary header, which the behavioural harness can't express.

Full suite under the debug ASan + UBSan build: 3632 of 3633 passed (1 skipped, 0 failed).

Checklist

• PR targets dev (not master)
• Commits follow Conventional Commits (fix:)
• make builds cleanly (no new warnings)
• make test passes; tests added/updated for behaviour changes

[singaraiona]

The regression test does not specifically exercise the new vector-size overflow guard. Only the header is patched: hnsw_levels.bin still contains 3 bytes, while bad_n_nodes makes the loader expect 2^40 bytes. With the guard removed, the loader still returns NULL earlier—either the 1 TiB node-level allocation fails, or the short levels read at hnsw.c:933-936 does—so it never reaches the vectors calculation. Please make the test distinguish this validation path, for example by factoring the header-size validation into a helper that can be tested directly.

[belowzeroff]

Thank you. The header-patch test couldn't reach the vectors guard (the oversized node-level read fails first), so it didn't distinguish the path. I factored the check into ray_hnsw_vec_size_valid(n_nodes, dim) and replaced the test with a direct unit test (ordinary dims, non-positive, an overflowing pair, and the exact SIZE_MAX/sizeof(float) boundary); it fails if the guard logic regresses. Amended in b8d2f5d.