Ability to skip Incremental Index during query using query context #1957
Conversation
nishantmonu51
commented
Nov 12, 2015
nishantmonu51
commented
- add adds the ability to skip incremental index when querying results from realtime nodes.
- default behaviour is to include incrementalIndex in queries.
|
can you explain the logic behind this PR? |
|
I'm guessing its because this context setting causes the query to only hit persisted segments, which should be faster. If you don't care about absolute-up-to-the-second data you can skip hitting the incremental index which might be heavily hit due to ingestion. @nishantmonu51 can you get some performance metrics around this? |
|
@fjy, @drcrallen, yeah its correct, the idea behind this is to help with query performance when you don't want data upto the latest second and can tolerate delay upto the intermediatePersistPeriod,
|
|
@drcrallen will look into getting some performance numbers. |
|
@fjy tldr; it's the difference between real-time and real²time :) |
| @@ -291,6 +294,7 @@ private Sink getSink(long timestamp) | |||
|
|
|||
| // The realtime plumber always uses SingleElementPartitionChunk | |||
| final Sink theSink = holder.getObject().getChunk(0).getObject(); | |||
| final boolean skipIncrementalSegment = query.getContextValue(SKIP_INCREMENTAL_SEGMENT, false); | |||
There was a problem hiding this comment.
I think this will fail with class cast exception if the value is a string instead of a boolean.
There was a problem hiding this comment.
also, it might be useful to get it once on top and reuse instead of getting for each sink.
There was a problem hiding this comment.
I think we only support Strings for backwards compatibility, do we have to support strings for new properties ?
This PR adds adds the ability to skip incremental index when querying results from realtime nodes. default behaviour is to include incrementalIndex in queries. review comment
nishantmonu51
force-pushed
the
skip-incremental-segment
branch
from
0d15dae
to
60f649d
Compare
|
can you document the new attribute? |
|
@nishantmonu51 @xvrl I don't really understand the purpose of this PR? Is it so you guys can avoid querying the in memory component of Druid's realtime? Why not look into why the in-memory component is slow? |
|
👍 |
|
👍 it's helpful to have to investigate performance problems. Probably should not need to be documented, since it's behavior is dependent on real-time ingestion implementation details. If it proves very useful, we can promote it. |
Ability to skip Incremental Index during query using query context
