Double free/Invalid pointer at end of benchmark run

[aniket-borkar]

I'm seeing either a double free or invalid pointer error every time I complete a benchmark run. Here are the logs from runs:

/mnt/nvme0/aborkar/kvikio/python/benchmarks$ python3 single-node-io.py -d /mnt/nvme0/aborkar/ -t 24 --nruns 3 2>&1 | tee kvikio_local_nvme.log
Roundtrip benchmark
----------------------------------
GPU               | NVIDIA A100-SXM4-80GB (dev #0)
GPU Memory Total  | 80.00 GiB
BAR1 Memory Total | 128.00 GiB
GDS driver        | v2.13
GDS config.json   | /usr/local/cuda-11.8/gds/cufile.json
----------------------------------
nbytes            | 10485760 bytes (10.00 MiB)
4K aligned        | True
pre-reg-buf       | True
diretory          | /mnt/nvme0/aborkar
nthreads          | 24
nruns             | 3
==================================
cufile read       |   1.64 GiB/s ±  5.49 % (1.66 GiB/s, 1.72 GiB/s, 1.54 GiB/s)
cufile write      |   3.31 GiB/s ± 12.42 % (2.88 GiB/s, 3.34 GiB/s, 3.70 GiB/s)
posix read        |   2.32 GiB/s ± 46.37 % (1.12 GiB/s, 2.63 GiB/s, 3.20 GiB/s)
posix write       |   0.95 GiB/s ± 13.13 % (824.43 MiB/s, 1.03 GiB/s, 1.01 GiB/s)
double free or corruption (!prev)

/mnt/nvme0/aborkar/kvikio/python/benchmarks$ python3 single-node-io.py -d /mnt/nvme0/aborkar/ -t 8 --nruns 3 2>&1 | tee kvikio_local_nvme.log
Roundtrip benchmark
----------------------------------
GPU               | NVIDIA A100-SXM4-80GB (dev #0)
GPU Memory Total  | 80.00 GiB
BAR1 Memory Total | 128.00 GiB
GDS driver        | v2.13
GDS config.json   | /usr/local/cuda-11.8/gds/cufile.json
----------------------------------
nbytes            | 10485760 bytes (10.00 MiB)
4K aligned        | True
pre-reg-buf       | True
diretory          | /mnt/nvme0/aborkar
nthreads          | 8
nruns             | 3
==================================
cufile read       |   1.50 GiB/s ±  9.57 % (1.36 GiB/s, 1.48 GiB/s, 1.65 GiB/s)
cufile write      |   3.52 GiB/s ± 12.89 % (3.00 GiB/s, 3.84 GiB/s, 3.72 GiB/s)
posix read        |   2.84 GiB/s ± 50.86 % (1.17 GiB/s, 3.66 GiB/s, 3.69 GiB/s)
posix write       |   0.96 GiB/s ± 15.34 % (814.17 MiB/s, 1.01 GiB/s, 1.08 GiB/s)
free(): invalid pointer

[madsbk]

Sorry for the late reply, just got back from vacation.

Can you try enabling KvikIO's compatibility mode like:

/mnt/nvme0/aborkar/kvikio/python/benchmarks$ KVIKIO_COMPAT_MODE=ON python3 single-node-io.py -d /mnt/nvme0/aborkar/ -t 24 --nruns 3 2>&1 | tee kvikio_local_nvme.log

[madsbk]

Hi @aniket-borkar, did it work with KVIKIO_COMPAT_MODE=ON ?

[aniket-borkar]

@madsbk it does work with KVIKIO_COMPAT_MODE=ON. I'm also not seeing this issue when running on a different system. I'll try rebuilding on the original system and see if the issue comes up again.

[madsbk]

@aniket-borkar please re-open if this is still an issue