Add configurable final stages to MSQ ingestion queries #16699
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
Area - Batch Ingestion
Area - Querying
Area - MSQ
10 tasks
cryptoe
reviewed
Jul 9, 2024
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/sql/MSQTaskQueryMaker.java
Outdated
Show resolved
Hide resolved
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
...ore/multi-stage-query/src/main/java/org/apache/druid/msq/sql/SegmentMorphFactoryCreator.java
Fixed
Show fixed
Hide fixed
...ore/multi-stage-query/src/main/java/org/apache/druid/msq/sql/SegmentMorphFactoryCreator.java
Fixed
Show fixed
Hide fixed
cryptoe
reviewed
Jul 15, 2024
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java
Outdated
Show resolved
Hide resolved
.../multi-stage-query/src/main/java/org/apache/druid/msq/guice/MSQTerminalStageSpecFactory.java
Dismissed
Show dismissed
Hide dismissed
.../multi-stage-query/src/main/java/org/apache/druid/msq/guice/MSQTerminalStageSpecFactory.java
Dismissed
Show dismissed
Hide dismissed
adarshsanjeev
changed the title
cryptoe
reviewed
Jul 22, 2024
| DataSourceMSQDestination destination = (DataSourceMSQDestination) querySpec.getDestination(); | ||
| TerminalStageSpec terminalStageSpec = destination.getTerminalStageSpec(); | ||
| if (terminalStageSpec instanceof SegmentGenerationStageSpec) { | ||
| return ((SegmentGenerationStageSpec) terminalStageSpec).constructFinalStage(queryId, queryDef, querySpec, jsonMapper); |
There was a problem hiding this comment.
can this be a interface method and we pass query def only ?
There was a problem hiding this comment.
We can inject the jsonMapper.
There was a problem hiding this comment.
We would also need the querySpec for the tuningConfig. Passing the jsonMapper here would also require SegmentGenerationStageSpec to not be a singleton class, and instead require a jsonMapper to be injected.
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Modifies the
DataSourceMSQDestinationto have aTerminalStageSpec. This spec tells the controller how the segments are created. This allows queries to configure how segments are created after the queries have finished running.This will allow queries to add new
TerminalStageSpecwhich allow queries to have other results instead of generating segments.The PR also adds a
SegmentGenerationStageSpecto be used by default, which causes queries to create new segments with the query results.The SegmentGenerationTerminalStageSpec for ingestion is serialized as:This PR should not have a functional impact, and is meant to be used for other features in the future.
Backward Compatibility
The changes are backward compatible. The
DataSourceMSQDestinationcreated has a default value which uses theSegmentGenerationStageSpecby default, since normally all ingestions would require a segment generation stage. If indexers/MM are upgraded first, the stage spec automatically deserializes to segment generation.This PR has: