Parallel read with MPI

[mengaldo]

Is your feature request related to a problem?

Is it possible to somehow extend xarray to use MPI I/O?

Describe the solution you'd like

We would need to know the offset from where the actual data starts within the file.
Is there a way of retrieving that?
Disclaimer: I am not an expert of NetCDF format - so, apologies if the question is trivial!

Describe alternatives you've considered

No response

Additional context

No response

[dcherian]

Haven't used it but you could consider reading with dask chunks and then using dask-mpi

[mengaldo]

Thank you for the prompt response @dcherian
However, is there a way of knowing the offset from where the actual data starts within the .nc file read by xarray?

[dcherian]

I don't know. I think we just hand off to netcdf4-python to figure it out.

[jhamman]

@mengaldo - not sure if this issue is still of interest to you, but now that we can map internal netcdf chunks to dask chunks automatically (#7948), we should be able to use dask-mpi to support parallel reads as you describe.

Going to close this issue since the conversation died off about a year ago but feel free to reopen if there is more to discuss.