Plumb task peon host/ports back out to the overlord. #2419

gianm · 2016-02-09T01:51:59Z

The intent is to get rid of the need for Curator service discovery to find tasks.

Motivation
Curator based service discovery is annoying because it needs ZK, and also because it doesn't clean up after itself when a service goes away, requiring hacks like this in tranquility: https://github.com/druid-io/tranquility/blob/v0.7.2/core/src/main/scala/com/metamx/tranquility/druid/DruidBeamMaker.scala#L229. This should also make life easier for the ingestion supervisors needed by #2220, as they will likely run on the overlord and will benefit from the overlord knowing where tasks are.

Intended usage
Tranquility would use this by implementing a resolver (similar to the DiscoResolver for Curator discovery) that polls the overlord's runningTasks endpoint instead of watching ZK.

Overlord-based ingestion supervisors (like the kafka one implied by #2220) would probably register a listener directly with the TaskRunner.

Implementation

Add TaskLocation class to represent the peon that tasks are running on
Add TaskRunner listeners that make it possible for callers to be notified when tasks move around (this happens when they get assigned to a peon for the first time, or potentially on restore)
Add getLocation to TaskRunnerWorkItem, mostly for the overlord servlet
Rework WorkerTaskMonitor to do management out of a single thread so it can handle status and location updates more simply.

fjy · 2016-02-09T02:04:38Z

indexing-service/src/main/java/io/druid/indexing/overlord/TaskRunnerListener.java

+
+import io.druid.indexing.common.TaskLocation;
+
+public interface TaskRunnerListener


can u add some comments about what this does and how to extend it?

why not have the listener take an executor?

It works better when the runner takes it

drcrallen · 2016-02-09T04:01:20Z

I do want to review this but it will take a bit to chew through.

drcrallen · 2016-02-09T04:01:42Z

indexing-service/src/main/java/io/druid/indexing/common/TaskLocation.java

+
+import java.util.Objects;
+
+public class TaskLocation


can this just use Worker?

It's not really semantically a worker- one worker is going to have many tasks at many taskLocations (all at different ports from the parent worker).

Then can they be DruidNodes?

I guess chatPort makes that a no?

is it possible to simply propagate DruidNode and have chatPort discoverable from the node data exposed at DruidNode?

What about just using HostAndPort? (I'm trying to minimize the number of items that are added to the code)

fwiw, in #2242 I am making DruidServerMetadata the source of truth for Druid server's metadata. I think it's reasonable to make DruidServerMetadata to contain a minimum set of metadata(e.g., host, port, name, etc), and then let Worker, TaskLocation, DruidServer extend it.

hostAndPort doesn't have jackson annotations. We could register a jackson module for it, I suppose (or maybe it's already part of the GuavaModule?). I am also ok with replacing this with the stuff from #2242 when that pr is ready.

it looks like HostAndPort serde is included in the GuavaModule, although it uses the ToStringSerializer, which will be kind of a pain for people to deserialize that aren't linking guava in their app. so I am leaning towards keeping TaskLocation as a thing and potentially changing it after #2242.

drcrallen · 2016-02-09T04:05:10Z

FYI, this also uses similar functionality to the remote task runner replacement in #2246

fjy · 2016-02-09T23:58:10Z

indexing-service/src/main/java/io/druid/indexing/worker/WorkerTaskMonitor.java

+    public void handle()
+    {
+      if (running.containsKey(task.getId())) {
+        log.warn(


what happens if a task finishes in this block of code?

ah, should be fine if unannounceTask is idempotent

fjy · 2016-02-10T00:08:30Z

👍 even if my comments not addressed

himanshug · 2016-02-10T22:13:47Z

indexing-service/src/main/java/io/druid/indexing/worker/WorkerTaskMonitor.java

+
+      try {
+        log.info("Updating task [%s] announcement to with location [%s]", taskId, location);
+        workerCuratorCoordinator.updateAnnouncement(


nit: you could skip this announcement if details.location was already same as location.

himanshug · 2016-02-10T22:17:43Z

@gianm i see that this PR allows you to know host and port of peons from runningTask HTTP endpoint at overlord. so, in tranquility, you would use same to find the tasks . however realtime indexing task open a separate chat handler port which you do not obtain from the "TaskLocation" . are you planning to continue using service discovery for finding the chat handler port or should the TaskLocation be updated to have some kind of metadata object inside it too so that it can have additional information like chat handler port?

gianm · 2016-02-10T22:38:46Z

@himanshug that's a good point, I was thinking that the chat handler was still on the same port as the main servlet, but that's not true anymore since the separateIngestionEndpoint was added.

ideally this should be part of the location as well. will adjust that.

gianm · 2016-02-10T23:24:25Z

@himanshug reopened with chatPort included.

gianm · 2016-02-24T22:43:55Z

@guobingkun a task should only have one location at a time, if it is running in two places then the RTR should kill the one it doesn't like

drcrallen · 2016-02-24T22:46:08Z

indexing-service/src/main/java/io/druid/indexing/overlord/TaskRunnerUtils.java

+              @Override
+              public void run()
+              {
+                listener.lhs.locationChanged(taskId, location);


there is no guarantee of execution order or completion here (nor error reporting on error?)

For example, if an over-burdened executor is used that does not have a FIFO queue, location changes can be processed in no particular order compared to the call to notifyLocationChanged.

added comments to registerListener on TaskRunner clarifying the intended usage.

- Add TaskLocation class - Add registerListener to TaskRunner - Add getLocation to TaskRunnerWorkItem - Implement location tracking in existing TaskRunners - Rework WorkerTaskMonitor to do management out of a single thread so it can handle status and location updates more simply.

drcrallen · 2016-02-24T23:16:10Z

Cool, 👍 but suggest removing #2419 (diff)

gianm · 2016-02-25T00:54:25Z

I think this failed due to #2430

(https://travis-ci.org/druid-io/druid/builds/111621322)

testSessionKilled(io.druid.curator.announcement.AnnouncerTest)  Time elapsed: 61.322 sec  <<< ERROR!
java.lang.Exception: test timed out after 60000 milliseconds
    at java.lang.Thread.sleep(Native Method)
    at io.druid.curator.announcement.AnnouncerTest.testSessionKilled(AnnouncerTest.java:176)

Plumb task peon host/ports back out to the overlord.

fjy reviewed Feb 9, 2016
View reviewed changes

gianm force-pushed the task-hostports branch 4 times, most recently from 999af33 to 8d882ae Compare February 9, 2016 03:26

drcrallen reviewed Feb 9, 2016
View reviewed changes

fjy added this to the 0.9.1 milestone Feb 9, 2016

gianm added the Improvement label Feb 9, 2016

fjy reviewed Feb 9, 2016
View reviewed changes

gianm force-pushed the task-hostports branch from 8d882ae to ad38fad Compare February 10, 2016 01:19

himanshug reviewed Feb 10, 2016
View reviewed changes

gianm closed this Feb 10, 2016

gianm reopened this Feb 10, 2016

gianm force-pushed the task-hostports branch from ad38fad to 007891d Compare February 10, 2016 23:19

gianm force-pushed the task-hostports branch 2 times, most recently from b4aa259 to ab9ee21 Compare February 11, 2016 00:44

gianm closed this Feb 11, 2016

gianm reopened this Feb 11, 2016

gianm force-pushed the task-hostports branch from ab9ee21 to 06d4b5c Compare February 11, 2016 01:04

gianm force-pushed the task-hostports branch 2 times, most recently from 0c6298e to 0a5956b Compare February 24, 2016 22:38

gianm force-pushed the task-hostports branch from 0a5956b to 7086e06 Compare February 24, 2016 22:44

drcrallen reviewed Feb 24, 2016
View reviewed changes

gianm force-pushed the task-hostports branch 3 times, most recently from 2acbfef to e18f9ad Compare February 24, 2016 23:12

gianm force-pushed the task-hostports branch from e18f9ad to cf0bc90 Compare February 24, 2016 23:13

gianm closed this Feb 25, 2016

gianm reopened this Feb 25, 2016

fjy added a commit that referenced this pull request Feb 26, 2016

Merge pull request #2419 from gianm/task-hostports

143e85e

Plumb task peon host/ports back out to the overlord.

fjy merged commit 143e85e into apache:master Feb 26, 2016

dclim mentioned this pull request Mar 15, 2016

Supervisor for KafkaIndexTask #2656

Merged

seoeun25 pushed a commit to seoeun25/incubator-druid that referenced this pull request Jan 10, 2020

apache#2419 Separate value and logic in Aggregator

408f1b6

This was referenced Dec 23, 2020

[Snyk] Security upgrade axios from 0.19.0 to 0.21.1 Accedian/incubator-druid#44

Open

[Snyk] Security upgrade axios from 0.18.0 to 0.21.1 ajesse11x/incubator-druid#118

Open

[Snyk] Security upgrade axios from 0.19.0 to 0.21.1 hixio-mh/druid#11

Closed

snyk-bot mentioned this pull request Jan 13, 2022

[Snyk] Security upgrade axios from 0.19.0 to 0.20.0 Accedian/incubator-druid#738

Open

snyk-bot mentioned this pull request Feb 10, 2022

[Snyk] Security upgrade axios from 0.19.0 to 0.20.0 Accedian/incubator-druid#917

Open

gianm deleted the task-hostports branch September 23, 2022 19:28

ajesse11x mentioned this pull request Dec 20, 2023

[Snyk] Security upgrade axios from 0.18.0 to 0.21.1 ajesse11x/incubator-druid#949

Open

ajesse11x mentioned this pull request Jan 2, 2024

[Snyk] Security upgrade axios from 0.18.0 to 0.21.1 ajesse11x/incubator-druid#955

Open

ajesse11x mentioned this pull request Mar 15, 2024

[Snyk] Security upgrade axios from 0.18.0 to 0.21.1 ajesse11x/incubator-druid#998

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plumb task peon host/ports back out to the overlord. #2419

Plumb task peon host/ports back out to the overlord. #2419

gianm commented Feb 9, 2016

fjy Feb 9, 2016

fjy Feb 9, 2016

gianm Feb 9, 2016

drcrallen commented Feb 9, 2016

drcrallen Feb 9, 2016

gianm Feb 9, 2016

drcrallen Feb 16, 2016

drcrallen Feb 16, 2016

drcrallen Feb 16, 2016

drcrallen Feb 24, 2016

guobingkun Feb 24, 2016

gianm Feb 24, 2016

gianm Feb 24, 2016

drcrallen Feb 24, 2016

drcrallen commented Feb 9, 2016

fjy Feb 9, 2016

fjy Feb 9, 2016

fjy commented Feb 10, 2016

himanshug Feb 10, 2016

himanshug commented Feb 10, 2016

gianm commented Feb 10, 2016

gianm commented Feb 10, 2016

gianm commented Feb 24, 2016

drcrallen Feb 24, 2016

gianm Feb 24, 2016

drcrallen commented Feb 24, 2016

gianm commented Feb 25, 2016


		import io.druid.indexing.common.TaskLocation;

		public interface TaskRunnerListener

Plumb task peon host/ports back out to the overlord. #2419

Plumb task peon host/ports back out to the overlord. #2419

Conversation

gianm commented Feb 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drcrallen commented Feb 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drcrallen commented Feb 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjy commented Feb 10, 2016

Choose a reason for hiding this comment

himanshug commented Feb 10, 2016

gianm commented Feb 10, 2016

gianm commented Feb 10, 2016

gianm commented Feb 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drcrallen commented Feb 24, 2016

gianm commented Feb 25, 2016