Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Improving parallelization user guide #5407

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[WIP] Improving parallelization user guide #5407

wants to merge 6 commits into from

Conversation

alexdesiqueira
Copy link
Member

Description

Improves parallelization user guide. Closes #1578.

For reviewers

  • Check that the PR title is short, concise, and will make sense 1 year
    later.
  • Check that new functions are imported in corresponding __init__.py.
  • Check that new features, API changes, and deprecations are mentioned in
    doc/release/release_dev.rst.

@alexdesiqueira alexdesiqueira added the 📄 type: Documentation Updates, fixes and additions to documentation label May 26, 2021
@grlee77 grlee77 mentioned this pull request Jun 4, 2021
Closed
sciunto
sciunto reviewed Jun 11, 2021
View reviewed changes
doc/source/user_guide/tutorial_parallelization.rst
The ``delayed`` call prevents ``load_and_threshold`` from executing immediately
while setting up the parallel processing. Instead, it becomes a task that can
be scheduled. Setting ``n_jobs`` to ``-1`` tells ``joblib`` to use all cores
available.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By experience, I prefer -2, all cores minus one, to do not freeze/overload the computer.

doc/source/user_guide/tutorial_parallelization.rst
a) parallelizing computation and b) reducing the amount of memory
used at any point in time.

``dask`` has tbe
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is a WIP :)

@grlee77
Copy link
Contributor

grlee77 commented Jul 25, 2021

Awhile back I was doing some benchmarking for dask-image and had also looked at apply_parallel for most functions in skimage.filters, skimage.morphology and skimage.restoration. If I recall correctly, most functions in filters and restoration show pretty good acceleration e.g. factor 4-7 or so with 10 cores. The morphology functions showed much less benefit. I think median filtering is one case that comes extremely close to factor 10 for 10 cores.

Maybe this would be something to make a wiki page for:
We could have a table of functions that have been verified to work and the recommended border size needed (if any).

@grlee77 grlee77 mentioned this pull request Oct 18, 2021
Open
@grlee77 grlee77 mentioned this pull request Mar 30, 2022
Open
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sciunto sciunto sciunto left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
📄 type: Documentation Updates, fixes and additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Mention apply_parallel in the user guide
4 participants
@alexdesiqueira @grlee77 @sciunto @stefanv