Show candidate hosts for the given query #2282

navis · 2016-01-18T08:11:10Z

Provide location information for the given query. Need for calculating location of input split.

fjy · 2016-01-18T15:58:39Z

@navis can you provide some more information on why this is feature is required. You can get this information through Druid metrics.

navis · 2016-01-19T05:52:55Z

@fjy I'm thinking of implementing a druid storage handler for hive. marked as 'wip'.

himanshug · 2016-01-19T16:31:09Z

@navis having a hive storage adapter would be good but it appears you are thinking of fetching data from historical nodes (and realtime nodes). IMO it will be much more scalable to read segments from hdfs directly.
I wrote a hadoop InputFormat ( a simple wrapper around druid's DatasourceInputFormat) and a Pig loader that works that way (planning to move InputFormat to druid core) , see

navis · 2016-01-20T02:11:31Z

Because we are supposed to use both of druid(simple select/groupby) and hive(for joins, udfs, etc) for the same dataset, it's highly possible that historical node already loaded segments to be used and we want to reuse it if possible. I think we can add a method to ask the historical node about local temporary directory it used for bypassing druid access.

I don't know how the InputFormat you've mentioned is implemented. Is it support predicates to be pushed to use bitmap index in druid smooth file? If it is, there is not much work to be done for me :).

himanshug · 2016-01-20T02:47:41Z

@navis the InputFormat I refered to is https://github.com/himanshug/druid-hadoop-utils/blob/master/druid-mr/src/main/java/com/yahoo/druid/hadoop/DruidInputFormat.java which is a simple wrapper on https://github.com/druid-io/druid/blob/master/indexing-hadoop/src/main/java/io/druid/indexer/hadoop/DatasourceInputFormat.java .

Regarding pushing predicates and iterating through rows in the segment, you can do that with https://github.com/druid-io/druid/blob/master/server/src/main/java/io/druid/segment/realtime/firehose/IngestSegmentFirehose.java which is used in https://github.com/druid-io/druid/blob/master/indexing-hadoop/src/main/java/io/druid/indexer/hadoop/DatasourceRecordReader.java .

navis · 2016-01-20T03:54:09Z

@himanshug Thanks, I'll check that.

navis · 2016-02-03T08:12:54Z

Anyway, below is the test result in my notebook.

navis@navisui-MacBook-Pro:~/druid$ explain_query.sh timeseries.json
[ {
  "itvl" : "2016-01-28T09:00:00.000+09:00/2016-01-29T09:00:00.000+09:00",
  "ver" : "2016-01-28T23:47:17.046Z",
  "part" : 0,
  "locations" : [ {
    "name" : "localhost:8083",
    "host" : "localhost:8083",
    "maxSize" : 10000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "itvl" : "2016-02-02T09:00:00.000+09:00/2016-02-03T09:00:00.000+09:00",
  "ver" : "2016-02-02T01:03:07.127Z",
  "part" : 0,
  "locations" : [ {
    "name" : "localhost:8100",
    "host" : "localhost:8100",
    "maxSize" : 0,
    "type" : "realtime",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
} ]

navis · 2016-03-22T09:19:28Z

Added get method something like curl "http://localhost:8082/druid/v2/candidates/?datasource=wikipedia&intervals=2013-08-31/2020-09-01".

@himanshug I need know target segments and locations from predicates (or sarg) made in hive. DataSourceInputFormat is a good start point but still need this PR to cover our use case.

navis · 2016-04-25T01:04:33Z

I've made patch integrating druid with hive based on this patch. Can anyone review this? I think this is fairly simple patch.

binlijin · 2016-04-25T07:44:39Z

Looks like the coordinator also have the segments info, so what time is the good to query coordinator and what time is the good to query broker?

binlijin · 2016-04-25T08:46:35Z

server/src/main/java/io/druid/server/BrokerQueryResource.java

+    for (Interval interval : intervals) {
+      for (TimelineObjectHolder<String, ServerSelector> holder : timeline.lookup(interval)) {
+        for (PartitionChunk<ServerSelector> chunk : holder.getObject()) {
+          ServerSelector selector = chunk.getObject();


Size can get from ServerSelector?
ServerSelector.getSegment().getSize()

I cannot understand how I've missed that three months ago. Then we can remove controversial approxSize part safely. Thanks.

binlijin · 2016-04-26T02:04:10Z

Patch looks much clean now, i think it is good.

binlijin · 2016-04-26T02:18:54Z

Need doc for how to use it.

navis · 2016-04-26T08:12:06Z

Fail of AnnouncerTest.testSanity, which seemed definitely not related to this.

xvrl · 2016-04-27T20:14:03Z

@navis this is useful, we built something similar standalone for our #2330 prototype, but we'll use this going forward since I think it makes sense for this to live on a broker.

xvrl · 2016-04-27T20:53:28Z

server/src/main/java/io/druid/query/LocatedSegmentDescriptor.java

+
+  @JsonCreator
+  public LocatedSegmentDescriptor(
+      @JsonProperty("itvl") Interval interval,


for public APIs we should probably use fully qualified names instead of abbreviations.

this class is extending SegmentDescriptor and it uses "itvl" as json property name. should we create separated class?

SegmentDescriptor is not exposed to the user so it's less of a problem. However, all our classes that are exposed to the user, use either interval or intervals so we should keep that consistent.

b-slim · 2016-09-20T16:10:02Z

server/src/main/java/io/druid/server/ClientInfoResource.java

@@ -300,6 +303,24 @@ public int compare(Interval o1, Interval o2)
    return metrics;
  }

+  @GET
+  @Path("/{dataSourceName}/candidates/intervals/{intervals}/numCandidates/{numCandidates}")


i guess after the changed in #2424 by @pjain1 we need to add Auth logic ?

Adding @ResourceFilters(DatasourceResourceFilter.class) to the endpoint will add the auth logic. So the endpoint will look like this -

@GET @Path("/{dataSourceName}/candidates/intervals/{intervals}/numCandidates/{numCandidates}") @Produces(MediaType.APPLICATION_JSON) @ResourceFilters(DatasourceResourceFilter.class) public Iterable<LocatedSegmentDescriptor> getQueryTargets( @PathParam("dataSourceName") String datasource, @PathParam("intervals") String intervals, @PathParam("numCandidates") String numCandidates, @Context final HttpServletRequest req ) throws IOException

If this can be added then it's great otherwise I can do a follow up PR to add it.

b-slim · 2016-09-20T16:11:19Z

server/src/main/java/io/druid/client/ServerViewUtil.java

+{
+  public static final int DEFAULT_NUM_CANDIDATES = 5;
+
+  public static List<LocatedSegmentDescriptor> getTargetLocations(


can we add couple of UTs to this core method ?

b-slim · 2016-09-20T16:12:27Z

server/src/main/java/io/druid/client/ServerViewUtil.java

+ */
+public class ServerViewUtil
+{
+  public static final int DEFAULT_NUM_CANDIDATES = 5;


minor: @navis someone my want to have all the server by default but maybe be you have a specific use case in mind ?

not really. I'll change default behavior to get all servers.

b-slim · 2016-09-20T16:13:29Z

docs/content/design/broker.md

@@ -59,7 +59,17 @@ Returns the dimensions of the datasource.

 Returns the metrics of the datasource.

+* `/druid/v2/datasources/{dataSourceName}/candidates/intervals/{comma-separated-intervals-in-ISO8601-format}/numCandidates/{numCandidates}`
+
+Returns segment information lists including server locations for the given datasource and intervals.


if we elect to go with a short list of 5 server if numCanditates is absent, we should add this to the doc i guess.

now default returns all servers. updated doc anyway.

b-slim · 2016-09-20T16:16:15Z

server/src/main/java/io/druid/query/LocatedSegmentDescriptor.java

+  private final List<DruidServerMetadata> locations;
+
+  @JsonCreator
+  public LocatedSegmentDescriptor(


some ser/desr test will be nice thought.

b-slim

@navis i left same small comments

b-slim · 2016-09-20T16:25:03Z

@navis i meant some not same sorry !

navis · 2016-09-21T03:00:25Z

@b-slim Addressed comments

nishantmonu51 · 2016-09-21T10:11:30Z

server/src/main/java/io/druid/server/ClientInfoResource.java

+    List<Interval> intervalList = Lists.newArrayList();
+    for (String interval : intervals.split(",")) {
+      intervalList.add(Interval.parse(interval.trim()));
+    }


When Intervals are specified as PathParam do we need to do any handling for '/' ?
e.g similar thing is done in DataSourcesResource.deleteDataSourceSpecificInterval

i guess we need to have something like this https://github.com/druid-io/druid/blob/0d745ee1207df4990ab92082ec33c75f5777757e/server/src/main/java/io/druid/server/http/DatasourcesResource.java#L346

changed to query-param for intervals/numCandidates. it looks better than before, I think.

nishantmonu51 · 2016-09-21T10:15:43Z

looks good to me after #2282 (comment)

navis · 2016-09-22T02:27:28Z

failed io.druid.server.lookup.PollingLookupTest which seemed not related to this.

drcrallen · 2016-09-22T03:25:14Z

@navis #3480

nishantmonu51

👍

nishantmonu51 · 2016-09-22T08:20:59Z

closing & reopening for travis

nishantmonu51 · 2016-09-22T08:25:02Z

All review comments seem handled, changed milestone to 0.9.2.

b-slim

👍

b-slim · 2016-09-22T17:53:02Z

@pjain1 this is in can you do the follow up PR for 0.9.2 ?

* Show candidate hosts for the given query * Added test cases & minor changes to address comments * Changed path-param to query-pram for intervals/numCandidates

navis changed the title ~~Show candidate hosts for the given query~~ [WIP] Show candidate hosts for the given query Jan 19, 2016

fjy added the Discuss label Jan 19, 2016

navis force-pushed the show-query-segment-host branch from 7b9084b to 8f1a2df Compare February 3, 2016 08:10

navis force-pushed the show-query-segment-host branch from 8f1a2df to 8fd3aec Compare March 22, 2016 09:15

navis changed the title ~~[WIP] Show candidate hosts for the given query~~ Show candidate hosts for the given query Mar 22, 2016

navis force-pushed the show-query-segment-host branch from 8fd3aec to 1485d67 Compare March 23, 2016 01:01

navis force-pushed the show-query-segment-host branch 2 times, most recently from d087707 to a59b9a5 Compare April 21, 2016 00:56

navis mentioned this pull request Apr 25, 2016

[WIP] druid-hive integration #2880

Closed

binlijin reviewed Apr 25, 2016
View reviewed changes

navis force-pushed the show-query-segment-host branch from a59b9a5 to 5f60c24 Compare April 25, 2016 23:52

navis force-pushed the show-query-segment-host branch from 5f60c24 to 4fd4120 Compare April 26, 2016 06:46

xvrl reviewed Apr 27, 2016
View reviewed changes

b-slim reviewed Sep 20, 2016

View reviewed changes

b-slim added the Feature label Sep 20, 2016

b-slim added this to the 0.9.3 milestone Sep 20, 2016

b-slim mentioned this pull request Sep 20, 2016

Exposing segment location via coordinator #3474

Closed

Added test cases & minor changes to address comments

4e20900

nishantmonu51 reviewed Sep 21, 2016

View reviewed changes

Changed path-param to query-pram for intervals/numCandidates

7c9bbb2

nishantmonu51 approved these changes Sep 22, 2016

View reviewed changes

nishantmonu51 closed this Sep 22, 2016

nishantmonu51 reopened this Sep 22, 2016

nishantmonu51 modified the milestones: 0.9.2, 0.9.3 Sep 22, 2016

b-slim approved these changes Sep 22, 2016

View reviewed changes

b-slim merged commit 49c0fe0 into apache:master Sep 22, 2016

pjain1 mentioned this pull request Sep 25, 2016

secure BrokerQueryResource endpoints #3506

Merged

seoeun25 pushed a commit to seoeun25/incubator-druid that referenced this pull request Jan 10, 2020

apache#2282 Use buffered stream for zipping

6570c2e

Show candidate hosts for the given query #2282

Show candidate hosts for the given query #2282

Conversation

navis commented Jan 18, 2016

fjy commented Jan 18, 2016

navis commented Jan 19, 2016

himanshug commented Jan 19, 2016

navis commented Jan 20, 2016

himanshug commented Jan 20, 2016

navis commented Jan 20, 2016

navis commented Feb 3, 2016

navis commented Mar 22, 2016

navis commented Apr 25, 2016

binlijin commented Apr 25, 2016 • edited

binlijin Apr 25, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

binlijin commented Apr 26, 2016 • edited

binlijin commented Apr 26, 2016

navis commented Apr 26, 2016

xvrl commented Apr 27, 2016

Choose a reason for hiding this comment

navis May 2, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b-slim left a comment

Choose a reason for hiding this comment

b-slim commented Sep 20, 2016

navis commented Sep 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nishantmonu51 commented Sep 21, 2016

navis commented Sep 22, 2016

drcrallen commented Sep 22, 2016

nishantmonu51 left a comment

Choose a reason for hiding this comment

nishantmonu51 commented Sep 22, 2016

nishantmonu51 commented Sep 22, 2016

b-slim left a comment

Choose a reason for hiding this comment

b-slim commented Sep 22, 2016

binlijin commented Apr 25, 2016 •

edited

binlijin Apr 25, 2016 •

edited

binlijin commented Apr 26, 2016 •

edited

navis May 2, 2016 •

edited