[QTL]Cached lookup module for JDBC connectors. #2819

b-slim · 2016-04-12T12:11:27Z

In This PR introduces a cached Lookup leveraging eviction and various caching strategies.
This Lookup implementation is meant to be user with large Lookup tables that need evictions.

himanshug · 2016-04-20T15:41:11Z

extensions-core/druid-lookups/README.md

@@ -0,0 +1,122 @@
+# Cached Lookup Module


can u move this doc to https://github.com/druid-io/druid/tree/master/docs/content/development/extensions-core and update https://github.com/druid-io/druid/blob/master/docs/content/development/extensions.md

b-slim · 2016-04-21T21:10:40Z

io.druid.indexing.overlord.RemoteTaskRunnerTest
testRunSameAvailabilityGroup(io.druid.indexing.overlord.RemoteTaskRunnerTest)  Time elapsed: 4.541 sec  <<< FAILURE!
java.lang.AssertionError: null
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertTrue(Assert.java:52)
    at io.druid.indexing.overlord.RemoteTaskRunnerTest.testRunSameAvailabilityGroup(RemoteTaskRunnerTest.java:232)

himanshug · 2016-04-22T15:32:44Z

extensions-core/druid-lookups/src/main/java/io/druid/server/lookup/DataFetcher.java

+   * Function to perform a one item lookup.
+   * For instance this can be used by a loading cache to fetch the value in case of the cache doesn't have it
+   *
+   * @param key none null key used to lookup the value


s/none null/non-null

can you check in all places please, it seems repeated in many places.

himanshug · 2016-07-21T18:36:51Z

@b-slim can you please take care of the comments? lets get this merged.

b-slim · 2016-07-22T16:14:42Z

@himanshug sure ASAP.

drcrallen · 2016-08-04T23:51:30Z

@b-slim is this ready for review?

b-slim · 2016-08-16T17:25:05Z

@drcrallen and @himanshug this is ready for review, thanks

fjy · 2016-08-30T17:41:14Z

docs/content/development/extensions-core/druid-lookups.md

@@ -0,0 +1,121 @@
+# Cached Lookup Module


this will not render correctly without

layout: doc_page

thanks fixed !

drcrallen · 2016-08-31T23:29:07Z

Minor formatting changes and I suggest reverting LookupDimensionSpec unless the change is required for this PR and it is fully tested.

drcrallen · 2016-08-31T23:30:01Z

I think this extension will benefit more from battle testing by interested parties rather than further review. Good job @b-slim

gianm · 2016-09-16T19:00:37Z

docs/content/development/extensions-core/druid-lookups.md

+---
+# Cached Lookup Module
+
+**Please note that this is an experimental module and the development/testing still at early stage. Feel free to try it and give us your feedback.**


this would look prettier in a <div class="note info">xxx</div>

gianm · 2016-09-16T19:01:24Z

docs/content/development/extensions-core/druid-lookups.md

+## Architecture
+Generally speaking this module can be divided into two main component, namely, the data fetcher layer and caching layer.
+
+### Data Fetcher layer:


no need for : on headings in general

gianm · 2016-09-16T19:03:05Z

...obal/src/main/java/io/druid/server/lookup/namespace/JDBCExtractionNamespaceCacheFactory.java

@@ -170,5 +170,4 @@ public Timestamp withHandle(Handle handle) throws Exception
    );
    return update.getTime();
  }
-


[nit] no need for this change

weird i don't see this on my IDE

ok i see it now i guess

gianm · 2016-09-16T19:05:06Z

docs/content/development/extensions-core/druid-lookups.md

+## Description
+This module provides a per-lookup caching mechanism for JDBC data sources.
+The main goal of this cache is to speed up the access to a high latency lookup sources and to provide a caching isolation for every lookup source.
+Thus user can define various caching strategies or and implementation per lookup, even if the source is the same.


Could you include something about whether it is intended that users could use this together with lookups-cached-global? Would be good to answer questions like:

Do they conflict with each other or can they be loaded simultaneously?

Does it make sense to run them side by side while migrating from one to the other?

Does it make sense to run them side by side long term?

Could you also include how to set up this extension? Something like:

Make sure to [include](../../operations/including-extensions.html) `druid-lookups-cached-single` as an extension.

good point will add

gianm · 2016-09-16T19:06:55Z

docs/content/development/extensions.md

@@ -29,6 +29,7 @@ Core extensions are maintained by Druid committers.
 |druid-kafka-eight|Kafka ingest firehose (high level consumer).|[link](../development/extensions-core/kafka-eight-firehose.html)|
 |druid-kafka-extraction-namespace|Kafka-based namespaced lookup. Requires namespace lookup extension.|[link](../development/extensions-core/kafka-extraction-namespace.html)|
 |druid-lookups-cached-global|A module for [lookups](../querying/lookups.html) providing a jvm-global eager caching for lookups. It provides JDBC and URI implementations for fetching lookup data.|[link](../development/extensions-core/lookups-cached-global.html)|
+|druid-lookups| Per lookup caching module to support the use cases where a lookup need to be isolated from the global pool of lookups |[link](../development/extensions-core/druid-lookups.html)|


This should be druid-lookups-cached-single to match the actual module name of the extension.

gianm

I didn't review the actual extension code, but just the docs and changes to code outside of the new extension. The comment about LookupDimensionSpec is blocking IMO; other than that I had a few suggestions for the docs.

b-slim · 2016-09-16T19:55:19Z

@gianm thanks fixed the doc and the blocking comment.

drcrallen · 2016-09-16T19:59:17Z

Gah! history got overwritten. ok, assuming only the comments got addressed and nothing else got changed then

b-slim · 2016-09-16T20:03:04Z

@drcrallen yes i did rebase it to revert and clean the dozen of commits messages.

gianm · 2016-09-16T20:31:55Z

👍 on the docs & non-extension changes. As discussed on the last dev sync, let's merge this (after travis) so people can give it a shot.

drcrallen · 2016-09-16T20:45:58Z

BOOM!

gianm · 2016-09-16T20:46:39Z

good job, we made it to 300 comments on this PR :)

b-slim · 2016-09-16T23:41:45Z

oh such great PR 🐹

b-slim force-pushed the new_lookup_module branch 2 times, most recently from a3ff3f2 to 6a926d4 Compare April 19, 2016 20:00

b-slim changed the title ~~[WIP][QTL]Cached lookup module for JDBC connectors.~~ [QTL]Cached lookup module for JDBC connectors. Apr 19, 2016

himanshug reviewed Apr 20, 2016
View reviewed changes

b-slim closed this Apr 21, 2016

b-slim reopened this Apr 21, 2016

himanshug reviewed Apr 22, 2016
View reviewed changes

b-slim force-pushed the new_lookup_module branch from 0785b2f to 7887792 Compare July 29, 2016 20:00

fjy reviewed Aug 30, 2016
View reviewed changes

gianm reviewed Sep 16, 2016

View reviewed changes

Cached lookup module. first cut implementing JDBC cache

a294244

b-slim force-pushed the new_lookup_module branch from d122a29 to a294244 Compare September 16, 2016 19:52

drcrallen merged commit 3175e17 into apache:master Sep 16, 2016

gianm mentioned this pull request Sep 23, 2016

Druid 0.9.2 release notes #3503

Closed

MdeArcayne mentioned this pull request Sep 30, 2016

Namespaced Query Time Lookups might not handle drivers correctly #2345

Closed

b-slim deleted the new_lookup_module branch April 26, 2018 01:28

seoeun25 added a commit to seoeun25/incubator-druid that referenced this pull request Jan 10, 2020

apache#2819 Add testAppliesTo at RuleTest

4a1ec1c

[QTL]Cached lookup module for JDBC connectors. #2819

[QTL]Cached lookup module for JDBC connectors. #2819

Conversation

b-slim commented Apr 12, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b-slim commented Apr 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

himanshug commented Jul 21, 2016

b-slim commented Jul 22, 2016

drcrallen commented Aug 4, 2016

b-slim commented Aug 16, 2016

Choose a reason for hiding this comment

layout: doc_page

Choose a reason for hiding this comment

drcrallen commented Aug 31, 2016

drcrallen commented Aug 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianm Sep 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianm left a comment

Choose a reason for hiding this comment

b-slim commented Sep 16, 2016

drcrallen commented Sep 16, 2016

b-slim commented Sep 16, 2016

gianm commented Sep 16, 2016

drcrallen commented Sep 16, 2016

gianm commented Sep 16, 2016

b-slim commented Sep 16, 2016

b-slim commented Apr 12, 2016 •

edited

Loading

gianm Sep 16, 2016 •

edited

Loading