[WIP] Fili 0.10.0 release notes

[QubitPi]

DRAFT

Highlights

Fili Security Added!

Relase security module for fili data security filters. Created ChainingRequestMapper, and a set of mappers for gatekeeping on security roles and whitelisting dimension filters.

Added by @michael-mclawhorn in #405

DataApiRequestFactory layer

Downstream projects now have more flexibility to construct DataApiRequest by using injectableFactory. An additional constructor for DataApiRequestImpl unpacks the config resources bundle to make it easier to override dictionaries.

Added by @michael-mclawhorn in #603

Make Field Accessor PostAggregation able to reference post aggregations in adddition to aggregations

Druid allows (but does not protect against ordering) post aggregation trees referencing columns that are also post aggregation trees. This makes it possible to send such a query by using a field accessor to reference another query expression. Using this capability may have some risk.

Added by @michael-mclawhorn in #543

Etag Cache

In the more recent versions of druid that are released after February 23rd, 2017. Druid added support for HTTP Etag. By including a If-None-Match header along with a druid query, druid will compute a hash as the etag in a way such that each unique response has a corresponding unique etag, the etag will be included in the header along with the response. In addition, if a query to druid includes the If-None-Match with a etag of the query, druid will check if the etag matches the response of the query, if yes, druid will return a HTTP Status 304 Content Not Modified response to indicate that the response is unchanged and matches the etag received from druid query request header. Otherwise druid will execute the query and respond normally with a new etag attached to the response header.

This new feature is designed by @garyluoex . For more info, visit @garyluoex 's design at #255

More robust Lucene Search Provider and Key Value Store

Lucene Search Provider can re-open in a bug-free way and close more cleanly

Added by @garyluoex in #551 and #521

Extraction Function on selector filter

Update Fili to accommodate the deprecated ExtractionFilter in druid, use selector filter with extraction function instead. Added extraction function on dimensional filter, defaults to extraction function on dimension if it exists.

Added by @garyluoex in #617

More controllable RequestLog

Exposes the LogInfo objects stored in the RequestLog, via RequestLog::retrieveAll making it easier for customers to implement their own scheme for logging the RequestLog

Added by @archolewa in #574

Druid lookup metadata load status check

Fili now supports checking Druid lookup status as one of it's health check. It will be very easy to identify any failed lookups.

Added by @QubitPi in #620

Add ability to use custom rate limiting schemes

While backward compatibility is guaranteed, Fili now allows users to rate limit(with a a new rate limiter) based on different criteria other than the default criteria.

Added by @efronbs in #591

Support Time Format Extraction Function in Fili

Druid TimeFormatExtractionFunction is added to Fili. API users could interact with Druid using TimeFormatExtractionFunction through Fili.

Added by @QubitPi in #611

Dimension load strategy indicator

In order to allow clients to be notified if a dimension's values are browsable and searchable, a storage strategy metadata is added to dimension. A browsable and searchable dimension is denoted by LOADED, whereas the opposite is denoted by NONE. This will be very useful for UI backed by Fili on sending dimension-related queries.

Added by @michael-mclawhorn, @garyluoex and @QubitPi in #575, #589, #558, #578

Query Split Logging

Include metrics in logging to allow for better evaluation of the impact of caching for split queries. There used to be only a binary flag (BardQueryInfo.cached) that is inconsistently set for split queries. Now 3 new metrics are added

1. Number of split queries satisfied by cache
2. Number of split queries actually sent to the fact store. (not satisfied by cache)
3. Number of weight-checked queries

Added by @QubitPi in #537

Configurable Metric Long Name

Logical metric has more config-richness to not just configure metric name, but also metric long name, description, etc. MetricInstance is now created by accepting a LogicalMetricInfo which contains all these fields in addition to metric name.

Added by @QubitPi in #492

Search provider can hot-swap index and key value store can hot-swap store location

LuceneSearchProvider is able to hot swap index by replacing Lucene index by moving the old index directory to a different location, moving new indexes to a new directory with the same old name, and deleting the old index directory in file system.
KeyValueStore is also made to support hot-swapping key value store location

Added by @QubitPi in #522

Uptime Status Metric

A metric showing how long Fili has been running is available.

Added by @mpardesh in #518

Consolidate UI & Non-UI broker configurations

ui_druid_broke and non_ui_druid_broker are not used separately anymore. Instead, a single druid_broker replaces the two. For backwards compatibility, Fili checks if druid_broker is set. If not, Fili uses non_ui_druid_broker and then ui_druid_broker

Added by @mpardesh in #489

Credits

Thanks to everyone who contributed to this release!

@michael-mclawhorn Michael Mclawhorn
@garyluoex Gary Luo
@archolewa Andrew Cholewa
@QubitPi Jiaqi Liu
@asifmansoora Asif Mansoor Amanullah
@efronbs Ben Efron
@deepakb91 Deepak Babu
@tarrantzhang Tarrant Zhang
@kevinhinterlong Kevin Hinterlong
@mpardesh Monica Pardeshi
@colemanProjects Neelan
@onlinecco
@dejan2609 Dejan Stojadinović

[kevinhinterlong]

Addressed in #666