Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Follow up issues for ml_tools #833

Closed
4 tasks done
lgray opened this issue Jun 6, 2023 · 3 comments
Closed
4 tasks done

Follow up issues for ml_tools #833

lgray opened this issue Jun 6, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@lgray
Copy link
Collaborator

lgray commented Jun 6, 2023

List of to-do's for #798 for after it is merged:

  • use empty_if_typetracer / length_one_if_typetracer to remove need user to hint necessary columns
  • use weakref/dask.delayed references to have fast tracing while robustly shipping large networks to processing cluster
  • update documentation once empty_if_typetracer / length_one_typetracer solution available
  • re-enable triton testing when dask_awkward can handle nested dask collections in kwargs (fix: adjustments to callable wrap to deal with typetracers in nested python structures #843)

@yimuchen

@lgray lgray added the enhancement New feature or request label Jun 6, 2023
@yimuchen yimuchen mentioned this issue Jun 7, 2023
11 tasks
@nsmith-
Copy link
Member

nsmith- commented Jun 9, 2023

For the second bullet, let's try to avoid weakref whenever possible.

@lgray
Copy link
Collaborator Author

lgray commented Jun 9, 2023

It's necessary if you want to do column_optimization locally and then also have efficient execution remotely.

We'll figure it out - I don't like the pattern either.

@lgray
Copy link
Collaborator Author

lgray commented Jan 13, 2024

We now have a much better way to include ml models in the task graph that seems to not incur that much serialization.

@lgray lgray closed this as completed Jan 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants