Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add parallel iterators for analysis #112

Merged
merged 25 commits into from Aug 24, 2021

Conversation

brittonsmith
Copy link
Collaborator

@brittonsmith brittonsmith commented Aug 11, 2021

PR Summary

This adds three new functions for parallel iteration over trees and/or nodes using the yt parallel_objects function.

  • parallel_trees: iterates over a list of trees
  • parallel_tree_nodes: iterates over nodes in a single tree
  • parallel_nodes: iterates over all nodes in a list of trees using multi-level parallelism

These are to be used either with analysis fields or simply for iterating over halos. All analysis fields set in the process are copied back to the root processor so they can be saved. In the cases of parallel_trees and parallel_nodes, results will be saved automatically either at the end of the loop or after a designated number of trees have completed. Because they use yt's parallel_objects underneath, the parallel loops can be configured to customize the size of process groups and to toggle on/off a task queue for each level of parallelism (trees and nodes).

See the API reference in the docs built for this PR for more information.

PR Checklist

I have added docstrings, but do not have time at the moment to update narrative docs or add tests. This will stay as a work in progress until I do. Nonetheless, I thought these useful enough to put here for others to use for the time being.

  • Code passes tests.
  • New features are documented with docstrings and narrative docs.
  • Tests added for fixed bugs or new features.

@brittonsmith brittonsmith merged commit 062f408 into ytree-project:main Aug 24, 2021
@brittonsmith brittonsmith changed the title [WIP] add parallel iterators for analysis add parallel iterators for analysis Aug 24, 2021
@brittonsmith brittonsmith deleted the parallel branch August 24, 2021 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant