Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Join spilling support in mixed execution mode #8998

Open
xiaoxmeng opened this issue Mar 7, 2024 · 0 comments
Open

Join spilling support in mixed execution mode #8998

xiaoxmeng opened this issue Mar 7, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@xiaoxmeng
Copy link
Contributor

Description

Spilling is now controlled by spill config regardless of velox task execution mode. The current join spilling implementation has limitation in mixed execution mode in which build side is in un-grouped execution mode while probe side is in grouped execution mode. We don't keep the spilled partitions across different group executions at the probe side. For now, we need to disable spilling if the task is under the grouped execution. We might consider to add to support later on to keep the spilled partition info of the build side around.

cc @mbasmanova @spershin @duanmeng @tanjialiang @Bikram
@xiaoxmeng xiaoxmeng added the enhancement New feature or request label Mar 7, 2024
@xiaoxmeng xiaoxmeng mentioned this issue Mar 7, 2024
Closed
xiaoxmeng added a commit to xiaoxmeng/velox that referenced this issue Mar 7, 2024
@xiaoxmeng
Disable hash join spilling in mixed execution mode
cb19001 
The current hash join spilling implementation can't handle mixed execution mode,
see (facebookincubator#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.
xiaoxmeng added a commit to xiaoxmeng/velox that referenced this issue Mar 7, 2024
@xiaoxmeng
Disable hash join spilling in mixed execution mode
f7d5e4d 
The current hash join spilling implementation can't handle mixed execution mode,
see (facebookincubator#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.
xiaoxmeng added a commit to xiaoxmeng/velox that referenced this issue Mar 8, 2024
@xiaoxmeng
Disable hash join spilling in mixed execution mode
150f449 
The current hash join spilling implementation can't handle mixed execution mode,
see (facebookincubator#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.
xiaoxmeng added a commit to xiaoxmeng/velox that referenced this issue Mar 8, 2024
@xiaoxmeng
Disable hash join spilling in mixed execution mode
54ce1b9 
The current hash join spilling implementation can't handle mixed execution mode,
see (facebookincubator#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.
xiaoxmeng added a commit to xiaoxmeng/velox that referenced this issue Mar 8, 2024
@xiaoxmeng
Disable hash join spilling in mixed execution mode
5669300 
The current hash join spilling implementation can't handle mixed execution mode,
see (facebookincubator#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.
facebook-github-bot pushed a commit that referenced this issue Mar 8, 2024
@xiaoxmeng @facebook-github-bot
Disable hash join spilling in mixed execution mode (#9001)
fdbf239 
Summary:
The current hash join spilling implementation can't handle mixed execution mode,
see (#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.

Pull Request resolved: #9001

Reviewed By: mbasmanova

Differential Revision: D54649972

Pulled By: xiaoxmeng

fbshipit-source-id: 4c4ad851f441df11b412e226794c8ed3a061bed4
Joe-Abraham pushed a commit to Joe-Abraham/velox that referenced this issue Jun 7, 2024
@xiaoxmeng @Joe-Abraham
Disable hash join spilling in mixed execution mode (facebookincubator…
f7af92b 
…#9001)

Summary:
The current hash join spilling implementation can't handle mixed execution mode,
see (facebookincubator#8998) for details. For now,
we disable the join spilling under mixed execution mode. We add method in task to
tell if a task is under mixed execution mode: task is group executed and has one
ungrouped split. The hash build then disables the spilling if the task is under mixed
execution mode.

Pull Request resolved: facebookincubator#9001

Reviewed By: mbasmanova

Differential Revision: D54649972

Pulled By: xiaoxmeng

fbshipit-source-id: 4c4ad851f441df11b412e226794c8ed3a061bed4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@xiaoxmeng