GPUdirect

[kanglcn]

Use https://github.com/rapidsai/kvikio when it is mature.

[beckernick]

Hi @kanglcn ! I came across this issue due to the rapidsai/kvikio reference. I work on the RAPIDS team at NVIDIA.

If you're willing to share, I'd love to learn more about what potential KvikIO features and functionality would be important for your use cases.

[kanglcn]

Hi @beckernick . It looks I can't install kvikio with python 3.10.

mamba install -c rapidsai kvikio=23.02 

Looking for: ['kvikio=23.02']

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
pkgs/main/linux-64                                            No change
pkgs/main/noarch                                              No change
pkgs/r/linux-64                                               No change
rapidsai/linux-64                                             No change
rapidsai/noarch                                               No change
pkgs/r/noarch                                                 No change

Pinned packages:
  - python 3.10.*


Could not solve for environment specs
The following packages are incompatible
└─ kvikio 23.02**  is installable with the potential options
   ├─ kvikio 23.02.00 would require
   │  └─ python >=3.8,<3.9.0a0 , which can be installed;
   └─ kvikio 23.02.00 would require
      └─ python >=3.9,<3.10.0a0 , which can be installed.

[beckernick]

We've just released v23.04, which includes Python 3.10 support. Would you be able to give that a test?

[kanglcn]

Thank you @beckernick ! I have successfully installed it.

I am working on satellite image processings. My current workflow is:

• read images from zarr.load;
• convert numpy array to cupy with cp.asarray;
• do my processing with cupy;
• convert cupy array to numpy back with cp.asnumpy;
• save the data with zarr.save.

The data reading and writing is too slow. That is why I am looking for kvikio.

After I install it, I find a problem in reading zarr:

rslc_path = '../../data/rslc.zarr'
rslc_zarr = zarr.open(rslc_path,mode='r')
rslc_cpu = rslc_zarr[:]

The data is successfully load into memory. But when I use kvikio:

rslc_zarr = zarr.open(store=GDSStore('./rslc_gpu.zarr'),mode='r')
rslc_gpu = rslc_zarr[:]

I got an error:

Click me

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[37], line 1
----> 1 rslc_gpu = rslc_zarr[:]

File ~/miniconda3/envs/work/lib/python3.10/site-packages/zarr/core.py:821, in Array.__getitem__(self, selection)
    819     result = self.vindex[selection]
    820 else:
--> 821     result = self.get_basic_selection(pure_selection, fields=fields)
    822 return result

File ~/miniconda3/envs/work/lib/python3.10/site-packages/zarr/core.py:947, in Array.get_basic_selection(self, selection, out, fields)
    944     return self._get_basic_selection_zd(selection=selection, out=out,
    945                                         fields=fields)
    946 else:
--> 947     return self._get_basic_selection_nd(selection=selection, out=out,
    948                                         fields=fields)

File ~/miniconda3/envs/work/lib/python3.10/site-packages/zarr/core.py:990, in Array._get_basic_selection_nd(self, selection, out, fields)
    984 def _get_basic_selection_nd(self, selection, out=None, fields=None):
    985     # implementation of basic selection for array with at least one dimension
    986 
    987     # setup indexer
    988     indexer = BasicIndexer(selection, self)
--> 990     return self._get_selection(indexer=indexer, out=out, fields=fields)

File ~/miniconda3/envs/work/lib/python3.10/site-packages/zarr/core.py:1285, in Array._get_selection(self, indexer, out, fields)
   1275 if (
   1276     not hasattr(self.chunk_store, "getitems") and not (
   1277         hasattr(self.chunk_store, "get_partial_values") and
   (...)
   1280 ) or any(map(lambda x: x == 0, self.shape)):
   1281     # sequentially get one key at a time from storage
   1282     for chunk_coords, chunk_selection, out_selection in indexer:
   1283 
   1284         # load chunk selection into output array
-> 1285         self._chunk_getitem(chunk_coords, chunk_selection, out, out_selection,
   1286                             drop_axes=indexer.drop_axes, fields=fields)
   1287 else:
   1288     # allow storage to get multiple items at once
   1289     lchunk_coords, lchunk_selection, lout_selection = zip(*indexer)

File ~/miniconda3/envs/work/lib/python3.10/site-packages/zarr/core.py:2006, in Array._chunk_getitem(self, chunk_coords, chunk_selection, out, out_selection, drop_axes, fields)
   2003         out[out_selection] = fill_value
   2005 else:
-> 2006     self._process_chunk(out, cdata, chunk_selection, drop_axes,
   2007                         out_is_ndarray, fields, out_selection)

File ~/miniconda3/envs/work/lib/python3.10/site-packages/zarr/core.py:1959, in Array._process_chunk(self, out, cdata, chunk_selection, drop_axes, out_is_ndarray, fields, out_selection, partial_read_decode)
   1956     tmp = np.squeeze(tmp, axis=drop_axes)
   1958 # store selected data in output
-> 1959 out[out_selection] = tmp

File cupy/_core/core.pyx:1473, in cupy._core.core._ndarray_base.__array__()

TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array explicitly.

Can you please help me find out how to correctly use `kvikio`?

[madsbk]

Hi @kanglcn, the GPU array support in Zarr is still in development. we just merged the final piece, which will be included in the next Zarr release v2.15.

We need to make KvikIO use this new Zarr feature and then everything should just work hopefully :)

Are you using compression?
Zarr only comes with CPU compressions but we plan to implement GPU compression using NVCOMP.

[kanglcn]

Thanks @madsbk for letting me know!
I haven't used any compression now. But I definitely will try it if it can help speed up the IO.

Would you consider adding support for dask? i.e. dask.array.to_zarr and dask.array.from_zarr. I am scaling my code with dask. If you have that plan, that will be very helpful to me!

[madsbk]

Yes, the plan is to support dask.