Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specialization for handoffs preceeding non-monotone/blocking operators (perf optimization) #166

Open
MingweiSamuel opened this issue Aug 4, 2022 · 3 comments
Labels
core api hydroflow excepting the surface syntax enhancement New feature or request performance scheduler Something with the scheduler (part of the Core API)

Comments

@MingweiSamuel
Copy link
Member

For example, if we have max() or min(), a preceding handoff would only ever need to store a single max or min element, streaming. But currently we only have VecHandoffs, which store everything. So we can be more efficient. This will probably tie in to using lattice types in handoffs.

@MingweiSamuel MingweiSamuel changed the title Non-monotone operators can have bespoke handoffs Some non-monotone operators could have bespoke handoffs as an optimization Aug 4, 2022
@MingweiSamuel MingweiSamuel added the hydroflow syntax Hydroflow's custom surface syntax label Aug 24, 2022
@jhellerstein
Copy link
Contributor

Another example: Sort is preceded by Handoff, but we could have a HeapHandoff or something that would do most of Sort, and save rebuffering.

@MingweiSamuel MingweiSamuel added enhancement New feature or request performance labels Oct 31, 2022
@jhellerstein jhellerstein removed the hydroflow syntax Hydroflow's custom surface syntax label Oct 31, 2022
@MingweiSamuel MingweiSamuel added this to the 0.2 release milestone Dec 5, 2022
@MingweiSamuel MingweiSamuel added the P2 Medium priority label Jan 17, 2023
@MingweiSamuel MingweiSamuel removed this from the 0.2 release milestone Jan 17, 2023
@MingweiSamuel MingweiSamuel added the core api hydroflow excepting the surface syntax label Jan 17, 2023
@MingweiSamuel MingweiSamuel added P1 High priority and removed P2 Medium priority labels Mar 13, 2023
@MingweiSamuel
Copy link
Member Author

MingweiSamuel commented Mar 13, 2023

@zzlk is found room for optimizing the current vec handoff #480

@MingweiSamuel MingweiSamuel changed the title Some non-monotone operators could have bespoke handoffs as an optimization Specialization for handoffs preceeding non-monotone/blocking operators (perf optimization) Mar 16, 2023
@jhellerstein jhellerstein added this to the 0.2 release milestone Apr 24, 2023
@jhellerstein jhellerstein added P2 Medium priority and removed P1 High priority labels Apr 24, 2023
@MingweiSamuel MingweiSamuel added the scheduler Something with the scheduler (part of the Core API) label Apr 27, 2023
@jhellerstein
Copy link
Contributor

Required:

  • StatelessHandoff doesn't own state, it participates in the state of its surrounding components

Future Optimization Rule:

  • Rewrite rules to identify when we can replace Handoff with StatelessHandoff

@jhellerstein jhellerstein added P1 High priority and removed P2 Medium priority labels May 9, 2023
@jhellerstein jhellerstein added P2 Medium priority and removed P1 High priority P2 Medium priority labels Jun 5, 2023
@zzlk zzlk modified the milestones: 0.3 release, 0.4 release Jun 29, 2023
@jhellerstein jhellerstein added the P2 Medium priority label Jul 3, 2023
@jhellerstein jhellerstein removed this from the 0.4 release milestone Jul 21, 2023
@MingweiSamuel MingweiSamuel removed the P2 Medium priority label Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core api hydroflow excepting the surface syntax enhancement New feature or request performance scheduler Something with the scheduler (part of the Core API)
Projects
None yet
Development

No branches or pull requests

3 participants