[WIP] Druid 0.9.1 Release Notes

[fjy]

Druid 0.9.1 contains hundreds of performance improvements, stability improvements, and bug fixes from over 30 contributors. Major new features include an experimental Kafka Supervisor to support exactly-once consumption from Apache Kafka, support for cluster-wide query-time lookups (QTL), and an improved segment balancing algorithm.

The full list of changes is here: https://github.com/druid-io/druid/issues?q=milestone%3A0.9.1+is%3Aclosed

Updating from 0.9.0

Query time lookups

Query time lookup (QTL) functionality has been substantially reworked in this release. Most users will need to update their configurations and queries.

The druid-namespace-lookup extension is now deprecated, and will be removed in a future version of Druid. Users should migrate to the new druid-lookups-cached-global extension. Both extensions can be loaded simultaneously to simplify migration. For details about migrating, see Transitioning to lookups-cached-global in the documentation.

Other notes

Aside from the QTL changes, please note the following changes:

• The default value for maxRowsInMemory has been set to 75,000 across the board for all forms of ingestion. This is in line with previous defaults for Hadoop tasks and Tranquility-based ingestion. If you were creating realtime index tasks directly (without Tranquility) then this is lower than the previous default of 500,000.
• The /druid/coordinator/v1/datasources/{dataSourceName}?kill=true&interval={myISO8601Interval} REST endpoint is now deprecated. The new /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?kill=true REST endpoint can be used instead.
• The druid.indexer.runner.separateIngestionEndpoint property is now deprecated. If you were using this functionality to isolate event-push requests and query serving requests for realtime tasks, you can accomplish something similar with druid.indexer.server.maxChatRequests.
• For developers of Druid extensions, note that the QueryGranularity constants (ALL, NONE, etc) have been moved to io.druid.granularity.QueryGranularities in Move QueryGranularity static fields to QueryGranularities #2980. Query syntax is not affected.
• Some cache keys have changed their calculation methodology. Deployments with centralized caching (like memcached or redis) can expect a drop in cache hit rate until all nodes are upgraded

Rolling updates

The standard Druid update process described by http://druid.io/docs/0.9.1-rc4/operations/rolling-updates.html should be followed for rolling updates.

Kafka Supervisor

Druid 0.9.1 is the first version to include the experimental Kafka indexing service, utilizing a new Kafka-type indexing task and a supervisor that runs within the Druid overlord. The Kafka indexing service provides an exactly-once ingestion guarantee and does not have the restriction of events requiring timestamps which fall within a window period. More details about this feature are available in the documentation: http://druid.io/docs/0.9.1-rc4/development/extensions-core/kafka-ingestion.html.

Note: The Kafka indexing service uses the Java Kafka consumer that was introduced in Kafka 0.9. As there were protocol changes made in this version, Kafka 0.9 consumers are not compatible with older brokers and you will need to ensure that your Kafka brokers are version 0.9 or better. Details on upgrading to the latest version of Kafka can be found here: http://kafka.apache.org/documentation.html#upgrade

New Features

#2656 Supervisor for KafkaIndexTask
#2602 implement special distinctcount
#2220 Appenderators, DataSource metadata, KafkaIndexTask
#2424 Enabling datasource level authorization in Druid
#2410 statsd-emitter
#1576 [QTL] Query time lookup cluster wide config

Full list: https://github.com/druid-io/druid/issues?q=milestone%3A0.9.1+is%3Aclosed+label%3AFeature

Improvements

#2972 Improved Segment Distrubution (new cost function)
#2931 Optimize filter for timeseries, search, and select queries
#2753 More consistent empty-set filtering behavior on multi-value columns
#2727 BoundFilter optimizations, and related interface changes.
#2711 All Filters should work with FilteredAggregators
#2690 Allow filters to use extraction functions
#2577 Implement native in filter

Full list: https://github.com/druid-io/druid/issues?q=milestone%3A0.9.1+is%3Aclosed+label%3AImprovement

Bug Fixes

Full list: https://github.com/druid-io/druid/issues?q=milestone%3A0.9.1+is%3Aclosed+label%3ABug

Documentation

Full list: https://github.com/druid-io/druid/issues?q=milestone%3A0.9.1+is%3Aclosed+label%3ADocumentation

Thanks to everyone who contributed to this release!
@fjy
@gianm
@jon-wei
@dclim
@acslk
@xvrl
@drcrallen
@nishantmonu51
@pjain1
@himanshug
@b-slim
@rasahner
@guobingkun
@navis
@binlijin
@sirpkt
@skilledmonster
@shekhargulati
@spektom
@javasoze
@du00cs
@mrijke
@kilida
@JonStrabala
@jisookim0513
@erikdubbelboer
@gaodayue
@sascha-coenen
@jaehc
@bjozet
@pdeva
@michaelschiff
@se7entyse7en
@harshjain2
@lizhanhui
@yuppie-flu

[drcrallen]

I need to do a short blurb about the QTL caching extension and incompatible changes. Will write when #2926 is in master

[drcrallen]

• The prior druid-namespace-lookup extension has been moved to the Lookups framework and is now druid-lookups-cached-global. Documentation can be found at https://github.com/druid-io/druid/blob/master/docs/content/development/extensions-core/lookups-cached-global.md
• The prior druid-kafka-extraction-namespace has undergone some changes as well. Updated documentation is available at https://github.com/druid-io/druid/blob/master/docs/content/development/extensions-core/kafka-extraction-namespace.md

[drcrallen]

• A change was made to QueryGranularity to prevent a potential deadlock. A side effect is that most of the static fields are now in QueryGranularities. Please update any extensions accordingly. See Move QueryGranularity static fields to QueryGranularities #2980

[fjy]

@drcrallen added to release notes. Can you edit them?

[drcrallen]

@fjy Yes, and I'll add directly to them if you want :) since you took initiative to write them up I

1. Didn't want to step on your toes with managing the release notes
2. Wanted to make the formatting pretty much done.

But If you want I can just edit the notes directly if I think of any more.

[gianm]

@fjy I added these notes from issues marked "release notes"

• The default value for maxRowsInMemory has been set to 75,000 across the board for all forms of ingestion. This is in line with previous defaults for Hadoop tasks and Tranquility-based ingestion. If you were creating realtime index tasks directly (without Tranquility) then this is lower than the previous default of 500,000.
• The /druid/coordinator/v1/datasources/{dataSourceName}?kill=true&interval={myISO8601Interval} REST endpoint is now deprecated. The new /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?kill=true REST endpoint can be used instead.
• druid.indexer.runner.separateIngestionEndpoint is now deprecated. If you were using this functionality to isolate event-push requests and query serving requests for realtime tasks, you can accomplish something similar with druid.indexer.server.maxChatRequests.
• For developers of Druid extensions, note that the QueryGranularity constants (ALL, NONE, etc) have been moved to io.druid.granularity.QueryGranularities in Move QueryGranularity static fields to QueryGranularities #2980. Query syntax is not affected.

[dclim]

added a short blurb on the Kafka indexing service

[gianm]

added this note to QTL:

• Static configuration via the druid.query.extraction.namespace.lookups property is no longer supported. Instead, lookups can be configured at runtime via dynamic configuration.

[gianm]

updated QTL section to:

Query time lookup (QTL) functionality has been substantially reworked in this release. Most users will need to update their configurations and queries.

The druid-namespace-lookup extension is now deprecated, and will be removed in a future version of Druid. Users should migrate to the new druid-lookups-cached-global extension. Both extensions can be loaded simultaneously to simplify migration. For details about migrating, see Transitioning to lookups-cached-global in the documentation.

[drcrallen]

Added

* Some cache keys have changed their calculation methodology. Deployments with centralized caching (like memcached or redis) can expect a drop in cache hit rate until all nodes are upgraded

[gianm]

https://github.com/druid-io/druid/releases/tag/druid-0.9.1