Function to wait on multiple IOFutures

[jakirkham]

Currently one needs to perform each read/write and wait on them individually. Like so

kvikio/python/examples/hello_world.py

Lines 31 to 34 in 7a43759

	future1 = f.pread(d[:50])
	future2 = f.pread(d[50:], file_offset=d[:50].nbytes)
	future1.get() # Wait for first read
	future2.get() # Wait for second read

However one use case is to do a bunch of writes or reads asynchronously. For example ( zarr-developers/zarr-python#1040 ). Doing this would accumulate a list of IOFutures from each operation.

In this case it would be helpful to have an API that is able to wait on that whole list of operations to complete. Perhaps like concurrent.futures.wait.

[spec-benno]

Wouldn't it anyway be a good idea to use the build-in future concept of Python:
https://docs.python.org/3/library/concurrent.futures.html#future-objects

Or what might be even nicer, would be to use Queues from the asyncio library:
https://docs.python.org/3/library/asyncio.html

I'm experiencing the problem that I need to recover data from a FIFO buffer on the GPU and want to write it to a RAID system. I can only have a maximum number of pwrites() at the same time and I need to wait for a process to finish before freeing that FIFO memory part.

[madsbk]

It is not easy to bind C++ futures with Python without excessive locking of Python’s GIL. However, ucxx manages to do this without too much performance overhead thus we should investigate this at some point.

For now, you might get a similar result by calling raw_write() from a Python thread, or maybe in a async function.