Skip to content
This repository has been archived by the owner on Jul 3, 2023. It is now read-only.

[idea] Node fusing for speeding up execution on systems like Ray, Dask #188

Closed
skrawcz opened this issue Aug 24, 2022 · 2 comments
Closed
Labels
enhancement New feature or request

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented Aug 24, 2022

Is your feature request related to a problem? Please describe.
For delegating to systems like Ray, it could make sense to "fuse" nodes together to reduce serialization costs.

Describe the solution you'd like
ideas:

  1. We need some concept to augment the DAG.
  2. We then need some pluggable way to change this logic. E.g. heuristics, vs some multi-pass logic.

Describe alternatives you've considered
Ideas:

  1. Make people write larger functions. But we wouldn't do this, since it goes against Hamilton's ideals.
  2. Have people tag functions that could be grouped -- seems like a good backdoor capability to have -- could work well with some more automated solution to override whatever it tries to do.

Additional context
I thought of this idea because people were complaining about Hamilton on Ray being slow.

@elijahbenizzy elijahbenizzy added the enhancement New feature or request label Oct 28, 2022
@elijahbenizzy
Copy link
Collaborator

Third idea -- we can group by module...

@elijahbenizzy
Copy link
Collaborator

We are moving repositories! Please see the new version of this issue at DAGWorks-Inc/hamilton#45. Also, please give us a star/update any of your internal links.

Note that everything else (slack community, pypi packages, etc...) will not change at all.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants