Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not Load lookups on tasks that will not use them #13324

Open
zachjsh opened this issue Nov 7, 2022 · 2 comments
Open

Do not Load lookups on tasks that will not use them #13324

zachjsh opened this issue Nov 7, 2022 · 2 comments

Comments

@zachjsh
Copy link
Contributor

zachjsh commented Nov 7, 2022

Description

At the moment, all druid tasks types attempt to load lookups, even when not needed:

  1. Some task types, have absolutely no need to do this, such as compaction, or kill tasks
  2. native batch ingest: only load lookups that are actually needed by transformSpec or metricsSpec
  3. MSQ: only load lookups that are actually needed by the query

Motivation

Lookups can be large. and should not consume resources on tasks that have no use for them. It's a waste of resources.

@gianm
Copy link
Contributor

gianm commented Nov 21, 2022

Seems like a good idea to me. On the MSQ front, note that worker tasks don't know what lookups they're going to need until they get a work order, so we'll need some way for them to be loaded on demand after a task starts up.

@IgorBerman
Copy link
Contributor

Any idea when this might be pushed to roadmap?
At least disable lookups loads for compaction tasks. I believe compaction tasks might be improved using higher value for maxRowsInMemory but currently the memory might be occupied by lookups(that compaction doesn't need)
I believe if we will scope this only for compaction and kill tasks (i.e. item 1) there wont be any need for design or discussion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants