PR for dask support to ARIAtools - WIP

[mgovorcin]

opening PR as placeholder for developments to include dask support to ARIAtools

So far test on unwrappedPhase, coherence and incidenceAngle, and can see improvement in run time for the 507 scenes/pairs:

• unwrappedPhase/connectedComponents - from 5h to 30min (with 30 jobs)
• coherence - run time 4 min (with 30 jobs). did not test how long would take to loop it by default
• incidence angle - there is a memory leak, I add workaround solution to restart dask client after every loop,, run time is 36 min (with 20 jobs). here we need to be careful with number of jobs as one jobs takes almost 10gb of ram memory

Did not have time to work on parallel export of correction layers, TODO

[cmarshak]

Out of curiousity, why did you choose dask over builtins like Pool/concurrent.futures? I know dask really shines with chunking xarray datasets, though from skimming I see you are just using the high level client which is basically multiprocessing?

Honestly, whatever works. A new popular library that I saw on hacker news recently is this: https://github.com/sybrenjansen/mpire

It may be helpful to look on JPL slack for discussions about dask. Or look at bopen's use of dask in their work: https://github.com/search?q=repo%3Abopen%2Fsarsen%20dask&type=code

Good luck, Marin. Hope to learn something here.

[cmarshak]

Also, if there is time / interest (not now), sharing some interesting way to benchmark code reproducibly: https://github.com/scikit-image/scikit-image/blob/main/CONTRIBUTING.rst#benchmarks

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[cmarshak]

Hey I got a notification that there was notebooks in this PR - make sure to put notebooks in folders that won't be distributed with ARIA-tools if this is only showing how to use the current library.

You don't want to distribute large notebook files if they only serve as documentation.