Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance warning in re-order #73

Closed
rogerkuou opened this issue May 6, 2024 · 1 comment · Fixed by #79
Closed

Performance warning in re-order #73

rogerkuou opened this issue May 6, 2024 · 1 comment · Fixed by #79
Assignees

Comments

@rogerkuou
Copy link
Member

The demo notebook or reordering gives performance warning when calling re-order:

# Time the reordering operation.
time_ordering = %timeit -o stmat.copy().stm.reorder(xlabel="azimuth", ylabel="range")
time_ordering
/storage/miniforge3/envs/mbl_stmtools/lib/python3.11/site-packages/xarray/core/indexing.py:1430: PerformanceWarning: Slicing with an out-of-order index is generating 230 times more chunks
  return self.array[key]

This seem to be inevitable since we are performing reorder. It seems we can

  1. Supress the warning
  2. Check some Dask reordering, e.g. :https://docs.dask.org/en/stable/order.html
@vanlankveldthijs
Copy link
Contributor

As we discussed, this performance would warning probably only apply while the data is split into a lot of small chunks (in this case 230 times as many).

However, inside the stm.reorder(...) function, the data is already re-chunked into larger chunks directly after this call to xarray.Dataset.sortby(...).

So, I will suppress the warning and check whether the function description properly mentions that this could be an expensive operation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants