Repository navigation
Towards a more performant and array API agnostic Parcels v4 #3072
Replies: 2 comments
|
Note Jax has full api array support jax-ml/jax#18353 |
|
After spending some more time on this, I think there are really two levels of "array agnostic" we could aim for given the way parcels works internally. Parcels writes particle data in place pretty much everywhere (
So in practice the list of libraries we could actually run on is numpy, cupy, torch and dpnp. The two levels are then:
The obvious advantage of the numpy dispatch behavior is that it is far far fewer code changes while still opening gpu support through cupy (which I've gotten to work in a fork, where I am seeing 30-40x performance over cpu using a single v100, so long as particle counts are greater than 1,000,000). For reference, a rough count of the structured core (grid search, interpolators, kernel loop, particle set) gives around 600 One advantage of the full array api compatable parcels core is the ability to test with |
Uh oh!
There was an error while loading. Please reload this page.
I spent some time this afternoon thinking about what it would actually look like to get parcels running on gpus or simply provide users with more avenues to enhance performance. In reading this back it has turned out longer than I expected, and would be a lot of work, but I think could provide the codebase with a framework that gives users (and us as maintainers implementing built in functionaility) maximum flexibility over the different ways to run parcels, and a lot of control over performance. Very curious to hear thoughts @erikvansebille and @fluidnumericsJoe , my understanding is that a lot of this aligns with what @VeckoTheGecko has previously proposed, so large sections of this are likely just me restating those ideas in one spot :).
Currently, when running parcels code there are two separate levels of "backend", the xarray backend (i.e. whatever data structure you get when you do a
ds[name].data), and the parcels backend (ChunkCachedArray,WindowedArray, dask, etc.).Fundamentally, performance in parcels is tied to two separate things: how long does it take to do a mathematical computation on a set of particles, and how long does it take to retrieve the model data needed for said mathematical computation. The mathematical computation time is negligible if the particle count is low, and the retrieval process is negligible if the device doing the computation can fit the entire model dataset in memory. The two separate backend toggles map pretty directly onto each of these computational bottlenecks with the xarray backend (assuming an api agnostic parcels implementation) relating to the speed of mathematical computations and the parcels backend relating to the speed of data retrieval. Currently in parcels there is limited support for different xarray backends as there is extensive use of
.values, which always returns a numpy array, and the genericnp.func(x). Therefore, it is not possible to use an xarray backend such as cupy, torch, or JAX, that would allow for the gpu acceleration of parcels, and could lead to performance comparable or exceeding v3.There has been previous discussion of moving towards array api alignment in parcels, see #2384 and #2534, along with the associated discussions. It seems to me like one of the biggest hurdles to an api agnostic parcels code is the fact that dask is not inherently array api compliant, and requires its chunks to behave like numpy arrays. As dask backed datasets are the main method we have for working with model data too large to fit in memory, this leads to a roadblock. However, this is a roadblock we have worked through before when dealing with slow performance associated with the dask graph. The solution is the parcels backend structure, which overrides some calls to dask through the utilization of caching methods, however, this layer does not have to be responsible for exclusively caching, it could also provide the bridge between the dask backed data and the array api agnostic parcels core. The idea would be that when calling a parcels backend, a user also specifies the array structure that will be used by parcels in the core, something like
.to_chunk_cached_arrays(numpy)/.to_chunk_cached_arrays(cupy)/.to_cache_chunked_array(torch), (in this framework numpy would likely be the default). In this situation, parcels backends would also need to be written to be written to convert to array api agnostic data structures, but the advantage would be any caching related advancement we make in performance within a parcels backend would be felt by any mathematical array library (whether it is running on cpus or gpus).From a data perspective the structure would look something like this:
xr.Dataset, which (for large computations) will usually be dask backed and lazily loaded from NetCDF or zarrChunkCachedArray,WindowedArray, ...): sits inds[name].variable._data. It decides when chunks are read, how many are held, and what gets evicted. The new part is that after dask has computed a chunk as numpy, the backend converts it to the array library it was given before caching it.xp = array_namespace(x)) and callsxp.func(x)..selkeep working, and the grid keeps a cached copy in the compute library for the search. This avoids the coordinate problem from Exploring GPU acceleration #2534 (coords.update()converts CuPy arrays to NumPy for dimension coordinates pydata/xarray#11251) without needing coordinates on the device inside xarray.The contract between a parcels backend and the core is then pretty small. A backend has to:
.isel()in the compute library (they come out of the grid search, so they are already on the device),ParticleSetknows where to allocate.Note that if the model data can fit in memory than these three requirements are met trivially by xarray backed by whatever array api compatible library the data was already in.
A final thought is that, to my understanding, this approach would allow for GPU support without needing to run CI or testing on GPUs, since we could run with
array_api_strict.All reactions