0.8.3 backports #2022
Merged
0.8.3 backports #2022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
…ful exception when the same inner aggregator is referenced by multiple types of outer aggregators
few fixes delete intermediate file early better exception handling use static pattern instead of compiling it every time Add retry for transient exceptions remove usage of deprecated method. Add test fix imports fix javadoc review comment. review comment: handle crazy snapshot naming review comments remove default retry count in favour of already present constant review comment make random intermediate and final paths. review comment, use temporaryFolder where possible
In apache#933 the ForkingTaskRunner's logging was changed to buffered from unbuffered. This means that the last few KB of the logs are generally not visible while a task is running, which makes debugging running tasks difficult.
…are. This is done by killing and respawning the jvms rather than reconnecting to existing jvms, for a couple reasons. One is that it lets you restore tasks after server reboots too, and another is that it lets you upgrade all the software on a box at once by just restarting everything. The main changes are, 1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only one that currently implements this. 2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown. ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit ForkingTaskRunner- close output stream to restorable tasks, wait for exit RemoteTaskRunner- do nothing special, we actually don't want to shutdown 3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run. Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with a list of restorable tasks. 4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored task and a zombie old task from stomping on each other.
old names are still valid though so as to be backwards compatible for now
Otherwise we could end up with a metric and a dimension with the same column name.
|
can you please include #1991 as well? |
adds support for "cardinality" aggregator. do not create sketch per event at ingestion time to make realtime ingestion faster
|
@himanshug ok, added |
|
@nishantmonu51 added #1957, but there are a lot of merge conflicts with #1943 so maybe we can do that in a separate PR |
|
#1832 could be included too. |
|
can #1929 be added as well ? |
|
@guobingkun added #1832 |
gianm
force-pushed
the
0.8.3-backports
branch
from
401c907
to
e59b7bd
Compare
|
added #1791 |
|
@nishantmonu51 ok, sounds good. I think this PR is ready to review then. |
|
👍 for me once the build passes, not sure if we can review all the changes in this PR. |
|
@himanshug I mostly just meant the list of patches included :) |
|
mmx#10 is just #1960 |
This PR adds adds the ability to skip incremental index when querying results from realtime nodes. default behaviour is to include incrementalIndex in queries. review comment
add an EventReceiverFirehoseMonitor so that we can monitor how many events have been queued in the EventReceiverFirehose and get a sense about whether the firehose is under too much pressure.
gianm
force-pushed
the
0.8.3-backports
branch
from
e59b7bd
to
57c52a9
Compare
gianm
force-pushed
the
0.8.3-backports
branch
from
57c52a9
to
85cb4dc
Compare
|
👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport:
#1387 Add special handler to allow logger messages during shutdown [Ease of Use]
#1791 EventReceiverFirehoseMonitor
#1799 Support multiple outer aggregators of same type and provide more help…
#1815 Fix Race in jar upload during hadoop indexing
#1832 add examples for duration and period granularities
#1843 "druid.manager.segment" should be "druid.manager.segments
#1854 Fix documentation about lookup
#1867 fixing hadoop test scope dependencies in indexing-hadoop
#1881 Restorable indexing tasks
#1888 forward cancellation request to all brokers, fixes #1802
#1897 complex aggregator based on http://datasketches.github.io
#1900 fix doc - correct default value for
#1917 RemoteTaskActionClient: Fix statusCode check.
#1920 Update curator to 2.9.1
#1929 separate ingestion and query thread pool
#1932 DataSchema: Exclude metric names from dimension list.
#1935 ForkingTaskRunner: Log without buffering.
#1943 Enable caching on intermediate realtime persists
#1954 EC2 autoscaler: avoid hitting aws filter limits
#1957 Ability to skip Incremental Index during query using query context
#1960 optimize index merge
#1991 datasketches module updates
Additional Release Notes updates:
min/max aggregators are deprecated and will be removed in druid-0.9.0 release. please use [long/double]Min/Max aggregators.