[BUG] cuVS IVFPQ: Occasional CUDA error in raft/util/cudart_utils.hpp or CUDA error 700

[mnorris11]

Describe the bug
Hello Nvidia friends, we are working on benchmarking cuVS IVFPQ. I am getting a CUDA error when trying to add vectors to the index.

Setup details:

• GPU: A100 (80GB RAM)
• 50M vectors with d=256 (it reproduces with generated random float data)
• Vectors are saved to a file with np.lib.format.open_memmap or np.save, it reproduces with both. I was not able to reproduce when generating embeddings --> adding directly without this file IO, so this might be an issue with numpy file loading instead. Let me know if you see the same! For given files, it seems to reproduce at the same place every time (same ith batch when adding it), indicating some file corruption etc, but there are no NaNs in the vectors. Any other things to look for?
• There are confirmed to be no NaN in the vectors (something like the below does not trigger)
• It is nowhere near OOMing. nvidia-smi prints about 8 GB / 80 GB is being used on the GPU when it hits the error for m=128. When doing generated embeddings without file IO, it hits 27 GB / 80 GB for m=256.

    # check for NaN
    for i in range(0, len(arr)):
        if np.isnan(np.sum(arr[i])):
            print(f"FOUND NAN on i: {i}")
            break

The error itself either appears as this from Faiss:

Traceback (most recent call last):
  File "/data/users/mnorris/fbsource/fbcode/scripts/mnorris/run_cuvs.py", line 246, in <module>
    index.add(x)
  File "/home/mnorris/.conda/envs/faiss_gpu_cuvs/lib/python3.12/site-packages/faiss/class_wrappers.py", line 230, in replacement_add
    self.add_c(n, swig_ptr(x))
  File "/home/mnorris/.conda/envs/faiss_gpu_cuvs/lib/python3.12/site-packages/faiss/swigfaiss_avx512.py", line 12216, in add
    return _swigfaiss_avx512.GpuIndex_add(self, arg2, x)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: C++ exception CUDA error encountered at: file=/home/mnorris/.conda/envs/faiss_gpu_cuvs/include/raft/core/interruptible.hpp line=303: 
Faiss assertion 'err__ == cudaSuccess' failed in virtual faiss::gpu::StandardGpuResourcesImpl::~StandardGpuResourcesImpl() at /home/runner/miniconda3/conda-bld/faiss-pkg_1743488319982/work/faiss/gpu/StandardGpuResources.cpp:141; details: CUDA error 700 an illegal memory access was encountered

or this:

Traceback (most recent call last):
  File "/data/users/mnorris/fbsource/fbcode/scripts/mnorris/run_cuvs.py", line 260, in <module>
    index.add(x[i : i + BATCH_SIZE])
  File "/home/mnorris/.conda/envs/faiss_gpu_cuvs/lib/python3.12/site-packages/faiss/class_wrappers.py", line 230, in replacement_add
    self.add_c(n, swig_ptr(x))
  File "/home/mnorris/.conda/envs/faiss_gpu_cuvs/lib/python3.12/site-packages/faiss/swigfaiss_avx512.py", line 12216, in add
    return _swigfaiss_avx512.GpuIndex_add(self, arg2, x)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: C++ exception CUDA error encountered at: file=/home/mnorris/.conda/envs/faiss_gpu_cuvs/include/raft/util/cudart_utils.hpp line=148:

Steps/Code to reproduce bug

1. Install via conda (pasted below)
2. Run below Python code to repro:

import cupy as cp
import faiss
import numpy as np
from numpy.lib.format import open_memmap

BATCH_SIZE = 1_000_000
d=256
batches=50

def gpu_ivfpq(m, cuvs=False):
    index = faiss.index_factory(
        d, f"IVF128,PQ{m}x8", faiss.METRIC_INNER_PRODUCT
    )
    co = faiss.GpuClonerOptions()
    co.use_cuvs = cuvs
    # co.useFloat16 = True  # Error happens regardless of this setting?
    res = faiss.StandardGpuResources()
    gpu_index = faiss.index_cpu_to_gpu(res, 0, index, co)
    return gpu_index

# Generates embeddings and saves to a file. Recommend running this as separate script then loading data later.
nb=50_000_000
rs = np.random.RandomState(1234)
x = rs.normal(size=(nb, d))
fp = np.lib.format.open_memmap(
    FILE_PATH,
    mode="w+",
    shape=(nb, d),
    dtype=np.float32,
)
print(f"len(x): {len(x)}") # sanity check 50M
for i, arr in enumerate(x):
    fp[i] = arr[0]
    if i % BATCH_SIZE == 0:
        print(f"flushing at i: {i}")
        fp.flush()
fp.flush()

from numpy.lib.format import open_memmap

# Can run this part as a separate script since data generation takes a while and you want to iterate
x = open_memmap(
    FILE_PATH,
    mode="r",
    shape=(BATCH_SIZE * batches, d),
    dtype=np.float32,
)
# Run m=256 just to use more memory / stress test to repro the issue. Usually reproduces on m=128.
for m in [128, 256]:
    index = gpu_ivfpq(m, True)

    index.train(x)
    # Added in batches to see where it dies. .add(x) to add all at once also reproduces.
    for i in range(0, len(x), BATCH_SIZE):
        print(f"adding ith batch: {i}")
        index.add(x[i : i + BATCH_SIZE])  # ** dies here **
    print(f"sanity check index.ntotal: {index.ntotal}")

    # Other stuff here omitted like doing the actual benchmark with search() ...

    del index

Expected behavior

No error

Environment details (please complete the following information):

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of RAFT install: [conda]

  conda create -yn faiss_gpu_cuvs
  conda activate faiss_gpu_cuvs
  conda install -y -c pytorch -c rapidsai -c conda-forge pytorch/label/nightly::faiss-gpu-cuvs pytorch pytorch-cuda numpy
  conda install -y -c conda-forge cupy

Additional context
Add any other context about the problem here.

[cjnolet]

Thanks for reporting this @mnorris11. Indeed, this behavior is unexpected.

@tfeher do you have any ideas?

[tarang-jain]

I had seen something similar a while ago. The problem was that the Faiss Gpu resource tries to destroy the streams manually, while cuVS (or RAFT) might still be using the streams somewhere. This was fixed when migrating from cuVS to Faiss, by destroying the raft handle before destroying the streams. I suspect this is along those lines, but will have to dig deeper.

[UPDATE]: This is not causing the problem and had been resolved in Faiss during the original migration to cuVS, so the stream destruction happens correctly after all the raft and cublas handles have been destroyed.

[mnorris11]

I had seen something similar a while ago. The problem was that the Faiss Gpu resource tries to destroy the streams manually, while cuVS (or RAFT) might still be using the streams somewhere. This was fixed when migrating from cuVS to Faiss, by destroying the raft handle before destroying the streams. I suspect this is along those lines, but will have to dig deeper.

Thanks for this input @tarang-jain , let me know if I can help on the Faiss side!

[tarang-jain]

I was able to reproduce this. The problem is in the ivf list resizing that is done here. While resizing the ivf list, the vectors from the old list are copied to a new buffer. Upon inserting some print statements, I saw that the new list size to which it is being resized is about 17 million. With the pq_dim of 256 and 8 bits per code, that means that particular list grows to more than 4 GB. Don't have a fix yet though.

[mnorris11]

I was able to reproduce this. The problem is in the ivf list resizing that is done here. While resizing the ivf list, the vectors from the old list are copied to a new buffer. Upon inserting some print statements, I saw that the new list size to which it is being resized is about 17 million. With the pq_dim of 8 bits, that means that particular list grows to about 4 GB. Don't have a fix yet though.

Hi @tarang-jain , are you able to repro when the memmap file is broken down, say just to 5M * 256d floats per memmap file and they get loaded + added in succession to the cuVS index? If you see that it works without error, I am thinking it can be a valid workaround for now?

[tarang-jain]

@mnorris11 The problem is the mmap saving and loading, so that data loading part seems to be incorrect in your script. I printed the list sizes and it turns out that in your mmap case, all vectors are being assigned to only two clusters, and their sizes are extremely large -- 25 million each. On the other hand, if the mmap is not done, and there is no saving and loading, all clusters are roughly of equal size. I also went ahead and printed one row of the original dataset and it looks like this:

[ 1.05228958e-03  4.56870032e-01 -7.75468890e-01 -1.25466436e+00
 -7.40383680e-01 -1.36453014e+00 -8.49555911e-01  2.62332106e-02
  9.85318053e-01  1.06652510e+00  9.77886087e-01  3.13136497e-01
  6.34781804e-01 -1.80565819e+00  8.82120268e-01  1.54486539e-01
 -4.83973622e-01 -1.30162145e+00  3.20023303e-01  1.30315971e+00
 -1.39826234e+00 -1.64994826e+00 -1.76147284e+00  2.88712326e-01
 -3.65336727e-01  1.66543828e+00  7.01620514e-01  6.57549838e-01
  8.99778264e-01  2.05602835e-01 -1.75524646e-01 -2.44801685e+00
  4.75149637e-01  2.87831378e-01 -1.58707040e+00  2.10915836e+00
 -1.44765146e+00 -8.58670136e-01 -2.07567365e-01 -5.19485413e-01
  7.59121084e-01  3.37100443e-01  3.79927277e-02  1.75561544e-01
 -6.56415320e-01 -9.21117156e-01 -3.13790580e-01 -6.49627890e-01
 -8.13257270e-01 -1.54323107e+00  1.22155053e+00 -1.80178782e+00
  1.89673984e+00 -1.38637969e-01 -1.57950397e-01 -1.20073291e+00
 -2.60741613e+00 -1.41145811e+00  9.25306797e-01 -1.58727290e+00
 -6.09000258e-01  3.29455889e-01 -1.35766111e+00 -1.39767570e+00
 -3.11561910e-01  9.10737546e-01  1.05618024e+00 -2.00208737e+00
  2.62056678e-01  7.07780483e-02 -1.46435530e+00 -1.49198428e-01
  1.88005957e+00 -1.44882305e-01 -5.12991127e-01 -6.33495069e-01
  1.25777167e+00  7.55554192e-01 -5.87353154e-01 -7.38994182e-01
 -1.11536613e+00  3.98209537e-02 -7.52852279e-03  2.43112996e-01
 -6.31188481e-01 -8.97196743e-01  1.30031466e+00  5.01104425e-01
 -7.55762299e-01  2.80874868e-01 -2.89715419e+00  7.09478569e-03
 -7.94342261e-02  9.56421930e-01  4.08440791e-01  1.41845542e+00
 -6.17119815e-01  1.22724918e-01 -2.43976558e+00 -2.47028996e-01
 -3.00593015e-01  1.02717224e+00 -6.24490869e-02  1.59644357e-01
  9.73157234e-01  9.31597428e-01  1.55416427e+00 -3.54396057e-01
  7.06681713e-02  5.68868783e-02 -3.86323678e-01  3.79097441e-01
 -4.30441970e-01 -1.08981953e+00 -1.50678780e+00 -8.67324742e-01
  3.98291018e-01  1.43858576e+00 -6.73503567e-01 -9.97842618e-01
  1.99114227e-01  7.28797505e-01  6.36581970e-01 -7.27479055e-01
 -2.06854838e+00  7.54937442e-01 -1.12428544e+00  1.18803583e+00
 -1.62890587e+00  9.41466971e-01 -6.49887119e-01  1.21705616e+00
 -5.48492527e-02  1.42612038e-01 -1.59105983e+00 -7.88886789e-02
  8.00161931e-01  1.33061549e+00 -7.24904013e-01  1.67865135e-01
 -2.00363585e+00 -1.01788242e+00  1.86644523e-01 -8.61810376e-02
 -6.73842164e-01  2.13434204e+00  2.99910751e-01 -8.17812425e-01
  1.01533553e+00 -2.91435565e-01 -7.30682647e-01 -5.36549153e-01
  1.26618613e+00  3.84317969e-01  2.69747818e-01 -2.36660778e-01
  5.23564372e-03  9.72688378e-01  9.70589529e-01  4.90586064e-01
 -5.21748353e-01  3.79007974e-01 -1.16923798e+00 -2.97419439e-01
 -2.58591669e+00  1.26741217e+00 -2.20440668e-01  1.01560709e+00
 -6.77068765e-01 -1.84480338e+00  5.64777541e-01  1.30435223e+00
  1.57995476e+00  8.68201099e-02  8.72148524e-01  9.82947448e-01
 -8.28214889e-01 -8.76645966e-01 -1.61350668e+00 -2.39323966e-01
  1.02875038e+00 -2.21787982e-01  5.18628695e-02  6.61087194e-01
  1.63494076e+00  2.15292886e+00  3.73568908e-01 -8.51695460e-01
 -1.34373084e+00  9.32823153e-01 -1.42117757e-01  4.08041423e-01
  4.03149318e-01  1.03319154e+00 -7.72448857e-01 -3.86339984e-01
 -5.87904235e-01 -5.75042102e-01 -2.37980567e-02  1.59404045e-01
  1.54119174e+00 -4.19132995e-01 -1.21625778e+00  6.75748691e-01
 -1.61681439e+00 -1.25489007e-01 -1.24888955e+00  9.49524881e-01
  6.96311137e-01 -7.42570843e-01 -7.35049544e-04 -5.36061593e-01
  6.18636999e-01  6.13109896e-01 -1.02898747e+00 -3.85436997e-02
 -1.71706534e+00 -1.00094565e+00  7.19554626e-01 -7.47929623e-02
  9.65686161e-01 -5.76726284e-01 -2.28735985e+00 -1.42047804e+00
  2.59313051e-01 -1.90523404e+00  6.71699873e-02  1.05227177e+00
  1.26052783e-03 -9.79038331e-01  7.73073260e-01 -1.82094227e+00
 -5.79301565e-01  2.81682474e-01 -2.83491080e-01  1.28386671e+00
 -1.27242106e+00 -6.87916487e-01  1.41408222e-01 -9.43049364e-01
  4.57161762e-01  1.62867657e+00  9.21268549e-01 -2.35971237e-01
 -1.00060052e+00  1.61016051e-01  1.63362798e-01 -9.95873399e-01
 -2.69496690e-01 -1.75511470e+00  2.17584617e+00 -6.28268974e-01
 -1.22554635e-01  1.43505422e+00  5.83996857e-01 -1.32980061e+00]

whereas, if I print the same row after loading the mmap file, it looks like:

[0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229]

So the problem is not in the implementation. Since the data is corrupted, when you try to copy 4.3 GB of uint8 values from one ivf list to the new resized ivf list, there might be problems with either new list allocation or the cudaMemcpyAsync call itself (@tfeher might be able to answer that better why that happens). Such a large list size is an abnormality due to bad data and I do not consider it a bug in the implementation.

@cjnolet I believe we can consider this issue solved.

[mnorris11]

@mnorris11 The problem is the mmap saving and loading, so that data loading part seems to be incorrect in your script. I printed the list sizes and it turns out that in your mmap case, all vectors are being assigned to only two clusters, and their sizes are extremely large -- 25 million each. On the other hand, if the mmap is not done, and there is no saving and loading, all clusters are roughly of equal size. I also went ahead and printed one row of the original dataset and it looks like this:

[ 1.05228958e-03  4.56870032e-01 -7.75468890e-01 -1.25466436e+00
 -7.40383680e-01 -1.36453014e+00 -8.49555911e-01  2.62332106e-02
  9.85318053e-01  1.06652510e+00  9.77886087e-01  3.13136497e-01
  6.34781804e-01 -1.80565819e+00  8.82120268e-01  1.54486539e-01
 -4.83973622e-01 -1.30162145e+00  3.20023303e-01  1.30315971e+00
 -1.39826234e+00 -1.64994826e+00 -1.76147284e+00  2.88712326e-01
 -3.65336727e-01  1.66543828e+00  7.01620514e-01  6.57549838e-01
  8.99778264e-01  2.05602835e-01 -1.75524646e-01 -2.44801685e+00
  4.75149637e-01  2.87831378e-01 -1.58707040e+00  2.10915836e+00
 -1.44765146e+00 -8.58670136e-01 -2.07567365e-01 -5.19485413e-01
  7.59121084e-01  3.37100443e-01  3.79927277e-02  1.75561544e-01
 -6.56415320e-01 -9.21117156e-01 -3.13790580e-01 -6.49627890e-01
 -8.13257270e-01 -1.54323107e+00  1.22155053e+00 -1.80178782e+00
  1.89673984e+00 -1.38637969e-01 -1.57950397e-01 -1.20073291e+00
 -2.60741613e+00 -1.41145811e+00  9.25306797e-01 -1.58727290e+00
 -6.09000258e-01  3.29455889e-01 -1.35766111e+00 -1.39767570e+00
 -3.11561910e-01  9.10737546e-01  1.05618024e+00 -2.00208737e+00
  2.62056678e-01  7.07780483e-02 -1.46435530e+00 -1.49198428e-01
  1.88005957e+00 -1.44882305e-01 -5.12991127e-01 -6.33495069e-01
  1.25777167e+00  7.55554192e-01 -5.87353154e-01 -7.38994182e-01
 -1.11536613e+00  3.98209537e-02 -7.52852279e-03  2.43112996e-01
 -6.31188481e-01 -8.97196743e-01  1.30031466e+00  5.01104425e-01
 -7.55762299e-01  2.80874868e-01 -2.89715419e+00  7.09478569e-03
 -7.94342261e-02  9.56421930e-01  4.08440791e-01  1.41845542e+00
 -6.17119815e-01  1.22724918e-01 -2.43976558e+00 -2.47028996e-01
 -3.00593015e-01  1.02717224e+00 -6.24490869e-02  1.59644357e-01
  9.73157234e-01  9.31597428e-01  1.55416427e+00 -3.54396057e-01
  7.06681713e-02  5.68868783e-02 -3.86323678e-01  3.79097441e-01
 -4.30441970e-01 -1.08981953e+00 -1.50678780e+00 -8.67324742e-01
  3.98291018e-01  1.43858576e+00 -6.73503567e-01 -9.97842618e-01
  1.99114227e-01  7.28797505e-01  6.36581970e-01 -7.27479055e-01
 -2.06854838e+00  7.54937442e-01 -1.12428544e+00  1.18803583e+00
 -1.62890587e+00  9.41466971e-01 -6.49887119e-01  1.21705616e+00
 -5.48492527e-02  1.42612038e-01 -1.59105983e+00 -7.88886789e-02
  8.00161931e-01  1.33061549e+00 -7.24904013e-01  1.67865135e-01
 -2.00363585e+00 -1.01788242e+00  1.86644523e-01 -8.61810376e-02
 -6.73842164e-01  2.13434204e+00  2.99910751e-01 -8.17812425e-01
  1.01533553e+00 -2.91435565e-01 -7.30682647e-01 -5.36549153e-01
  1.26618613e+00  3.84317969e-01  2.69747818e-01 -2.36660778e-01
  5.23564372e-03  9.72688378e-01  9.70589529e-01  4.90586064e-01
 -5.21748353e-01  3.79007974e-01 -1.16923798e+00 -2.97419439e-01
 -2.58591669e+00  1.26741217e+00 -2.20440668e-01  1.01560709e+00
 -6.77068765e-01 -1.84480338e+00  5.64777541e-01  1.30435223e+00
  1.57995476e+00  8.68201099e-02  8.72148524e-01  9.82947448e-01
 -8.28214889e-01 -8.76645966e-01 -1.61350668e+00 -2.39323966e-01
  1.02875038e+00 -2.21787982e-01  5.18628695e-02  6.61087194e-01
  1.63494076e+00  2.15292886e+00  3.73568908e-01 -8.51695460e-01
 -1.34373084e+00  9.32823153e-01 -1.42117757e-01  4.08041423e-01
  4.03149318e-01  1.03319154e+00 -7.72448857e-01 -3.86339984e-01
 -5.87904235e-01 -5.75042102e-01 -2.37980567e-02  1.59404045e-01
  1.54119174e+00 -4.19132995e-01 -1.21625778e+00  6.75748691e-01
 -1.61681439e+00 -1.25489007e-01 -1.24888955e+00  9.49524881e-01
  6.96311137e-01 -7.42570843e-01 -7.35049544e-04 -5.36061593e-01
  6.18636999e-01  6.13109896e-01 -1.02898747e+00 -3.85436997e-02
 -1.71706534e+00 -1.00094565e+00  7.19554626e-01 -7.47929623e-02
  9.65686161e-01 -5.76726284e-01 -2.28735985e+00 -1.42047804e+00
  2.59313051e-01 -1.90523404e+00  6.71699873e-02  1.05227177e+00
  1.26052783e-03 -9.79038331e-01  7.73073260e-01 -1.82094227e+00
 -5.79301565e-01  2.81682474e-01 -2.83491080e-01  1.28386671e+00
 -1.27242106e+00 -6.87916487e-01  1.41408222e-01 -9.43049364e-01
  4.57161762e-01  1.62867657e+00  9.21268549e-01 -2.35971237e-01
 -1.00060052e+00  1.61016051e-01  1.63362798e-01 -9.95873399e-01
 -2.69496690e-01 -1.75511470e+00  2.17584617e+00 -6.28268974e-01
 -1.22554635e-01  1.43505422e+00  5.83996857e-01 -1.32980061e+00]

whereas, if I print the same row after loading the mmap file, it looks like:

[0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229 0.00105229
 0.00105229 0.00105229 0.00105229 0.00105229]

So the problem is not in the implementation. Since the data is corrupted, when you try to copy 4.3 GB of uint8 values from one ivf list to the new resized ivf list, there might be problems with either new list allocation or the cudaMemcpyAsync call itself (@tfeher might be able to answer that better why that happens). Such a large list size is an abnormality due to bad data and I do not consider it a bug in the implementation.

@cjnolet I believe we can consider this issue solved.

Enormous thanks for looking into this @tarang-jain !

If the mmap results in corruption, is there a sort of accepted way that we should be loading large amounts of data to train() -> add() to the cuVS index? I am wondering because files are an easy way to pull embeddings from a database to have some reproducible set of database embeddings, then run experiments on those.

[tarang-jain]

If the mmap results in corruption, is there a sort of accepted way that we should be loading large amounts of data to train() -> add() to the cuVS index? I am wondering because files are an easy way to pull embeddings from a database to have some reproducible set of database embeddings, then run experiments on those.

The mmap seems to be correct, except if I change this line (assign the file pointer to arr instead of arr[0]) it works:

for i, arr in enumerate(x):
    fp[i] = arr[0] # change this to fp[i] = arr
    if i % BATCH_SIZE == 0:
        print(f"flushing at i: {i}")
        fp.flush()

Let me know if this solves the issue on your end as well.