Overlord High Availability causing loss of ingestion tasks

[giom-l]

Affected Version

Druid 0.15.1

Description

I have a cluster running on AWS that is partly composed by 2 overlords.
As stated in documentation, 1 is leader and all requests should be forwarded to him by the other overlord.

Test run :

Ensure 2 overlords are properly running, with logging debug mode activated on both nodes.
Identify the leader with API endpoint /druid/indexer/v1/leader
Send an ingestion spec to the non leader node.
Watching the UI to see if the task is running

Issue :
When I ran the test, no tasks appear running.
When analyzing the overlords logs, here is what I found

On the non leader node, I find that it forwarded the request to the leader :

DEBUG [qtp1472031375-88] org.apache.druid.server.http.RedirectFilter - Forwarding request to [http://leader-overlord:8090/druid/indexer/v1/task]

On the leader node, the log indicate that the requests arrives, as I can find a request with the following header X-Forwarded-For: non-leader-overlord-ip

But then, I can also find some errors :
an NPE :

2019-10-04T09:20:10,389 ERROR [qtp1472031375-90] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMeth
java.lang.NullPointerException
at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating]

And then a 500 error

2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMMIT for /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57}
500 Server Error HTTP/1.1
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1

Here is the full stack trace

2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.ChannelEndPoint - filled 392 SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=1/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=0,c=false,a=IDLE,uri=null,age=0}
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - HttpConnection@64c6429c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=0,c=false,a=IDLE,uri=null,age=0}<-SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=0/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=0,c=false,a=IDLE,uri=null,age=0} parse HeapByteBuffer@ad74436[p=0,l=392,c=8192,r=392]={<<<POST /druid/index...cation/json\r\n\r\n>>>ation/json\r\nAccep...: {} \n }\x00\x00\x00} {}
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - parseNext s=START HeapByteBuffer@ad74436[p=0,l=392,c=8192,r=392]={<<<POST /druid/index...cation/json\r\n\r\n>>>ation/json\r\nAccep...: {} \n }\x00\x00\x00}
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - START --> SPACE1
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - SPACE1 --> URI
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - URI --> SPACE2
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - SPACE2 --> REQUEST_VERSION
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - REQUEST_VERSION --> HEADER
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_NAME
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,355 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_NAME
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Length: 0 --> IN_VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Length: 0 --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> IN_VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:null --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:accept: */* --> IN_VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:accept: */* --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/json --> IN_VALUE
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER:Content-Type: application/json --> FIELD
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - HEADER --> END
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - REQUEST for //overlord-loadbalancer-url/druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0}
POST //overlord-loadbalancer-url/druid/indexer/v1/task HTTP/1.1
X-Forwarded-For: 172.29.182.70
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: overlord-loadbalancer-url
X-Amzn-Trace-Id: Root=1-5d970eca-416a917c2bee601c58bbbaa0
Content-Length: 0
Authorization: Basic dGVzdDpNMlEwTWpka05ERTROamxtT0RSbE1qZG1PVGRoTWpFeg==
User-Agent: curl/7.61.1
Accept: */*
Content-Type: application/json
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} onContentComplete
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} onRequestComplete
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpInput - HttpInputOverHTTP@8985515[c=0,q=0,[0]=null,s=STREAM] addContent EOF
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0}<-SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=START}]=>HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0}<-SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OP
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} handle //overlord-loadbalancer-url/druid/indexer/v1/task
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - handling HttpChannelState@57964b85{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false}
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0} action DISPATCH
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.Server - REQUEST POST /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=0}
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHandler - org.eclipse.jetty.server.handler.gzip.GzipHandler@48d739f handle Request(POST //overlord-loadbalancer-url/druid/indexer/v1/task)@2049e2fc in null
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - scope null||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE}
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - context=||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE}
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - sessionHandler=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - session=null
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - servlet |/druid/indexer/v1/task|null -> default@5c13d641==org.eclipse.jetty.servlet.DefaultServlet,jsp=null,order=-1,inst=true
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - chain=org.apache.druid.server.security.SecuritySanityCheckFilter-2a0b901c->org.apache.druid.server.security.AuthenticationWrappingFilter-7b96de8d->org.apache.druid.server.security.AllowOptionsResourceFilter-5c3d4f05->org.apache.druid.server.security.PreResponseAuthorizationCheckFilter-7e61e25c->org.apache.d
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.SecuritySanityCheckFilter-2a0b901c
2019-10-04T09:20:10,356 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.AuthenticationWrappingFilter-7b96de8d
2019-10-04T09:20:10,387 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.AllowOptionsResourceFilter-5c3d4f05
2019-10-04T09:20:10,387 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.security.PreResponseAuthorizationCheckFilter-7e61e25c
2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter org.apache.druid.server.http.RedirectFilter-7d30007d
2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - call filter com.google.inject.servlet.GuiceFilter-382d71c7
2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onEof HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=IDLE w=false}
2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onEof HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false}
2019-10-04T09:20:10,388 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onEof HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false}
2019-10-04T09:20:10,389 ERROR [qtp1472031375-90] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMeth
java.lang.NullPointerException
at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.3.jar:1.19.3]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.3.jar:1.19.3]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.3.jar:1.19.3]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.apache.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:71) [druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) [druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) [druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:189) [druid-basic-security-0.15.1-incubating.jar:0.15.1-incubating]
at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) [druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86) [druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
2019-10-04T09:20:10,391 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - FinalSession=null old_session_manager=null this=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1
2019-10-04T09:20:10,392 WARN [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - /druid/indexer/v1/task
java.lang.NullPointerException at org.apache.druid.indexing.overlord.http.OverlordResource.taskPost(OverlordResource.java:153) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMeth
2019-10-04T09:20:10,394 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onError HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false} {}
2019-10-04T09:20:10,394 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@57964b85{s=THROWN a=NOT_ASYNC i=true r=READY w=false}
2019-10-04T09:20:10,394 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=38} action ERROR_DISPATCH
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.Server - ERROR POST /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=false,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=40}
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHandler - org.eclipse.jetty.server.handler.gzip.GzipHandler@48d739f handle Request(POST //overlord-loadbalancer-url/druid/indexer/v1/task)@2049e2fc in null
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHandler - org.eclipse.jetty.server.handler.gzip.GzipHandler@48d739f excluded by dispatcherType ERROR
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - scope null||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE}
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.ContextHandler - context=||/druid/indexer/v1/task @ o.e.j.s.ServletContextHandler@19fec3d6{/,[jar:file:/opt/druid-0.15.1/lib/druid-console-0.15.1-incubating.jar!/org/apache/druid/console],AVAILABLE}
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - sessionHandler=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - session=null
2019-10-04T09:20:10,396 DEBUG [qtp1472031375-90] org.eclipse.jetty.servlet.ServletHandler - servlet |/druid/indexer/v1/task|null -> default@5c13d641==org.eclipse.jetty.servlet.DefaultServlet,jsp=null,order=-1,inst=true
2019-10-04T09:20:10,412 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor - org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@d26694a exclude by status 500
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - sendResponse info=null content=HeapByteBuffer@5d1ddd15[p=0,l=6488,c=32768,r=6488]={<<<<html>\n<head>\n<me.../body>\n</html>\n>>>:-1},"dataSource"...ation":41536,"l} complete=true committing=true callback=Blocker@7b265756{null}
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMMIT for /druid/indexer/v1/task on HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57}
500 Server Error HTTP/1.1
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: NEED_HEADER (null,[p=0,l=6488,c=32768,r=6488],true)@START
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpGenerator - generateHeaders HTTP/1.1{s=500,h=2,cl=-1} last=true content=HeapByteBuffer@5d1ddd15[p=0,l=6488,c=32768,r=6488]={<<<<html>\n<head>\n<me.../body>\n</html>\n>>>:-1},"dataSource"...ation":41536,"l}
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpGenerator - Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpGenerator - EOF_CONTENT
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: FLUSH ([p=0,l=197,c=8192,r=197],[p=0,l=6488,c=32768,r=6488],true)@COMPLETING
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - write: WriteFlusher@7ad026a{IDLE}->null [HeapByteBuffer@ad74436[p=0,l=197,c=8192,r=197]={<<<HTTP/1.1 500 Serv....v20180503)\r\n\r\n>>>ca-416a917c2bee60...: {} \n }\x00\x00\x00},HeapByteBuffer@5d1ddd15[p=0,l=6488,c=32768,r=6488]={<<<<html>\n<head>\n<me.../body>\n</html>\n>>>:-1},"dataSource"...ation":41536,"l}]
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - update WriteFlusher@7ad026a{WRITING}->null:IDLE-->WRITING
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.ChannelEndPoint - flushed 6685 SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=W,to=58/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=COMPLETING}]=>HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//druid-overlord.emea
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - Flushed=true written=6685 remaining=0 WriteFlusher@7ad026a{WRITING}->null
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.WriteFlusher - update WriteFlusher@7ad026a{IDLE}->null:WRITING-->IDLE
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: SHUTDOWN_OUT ([p=197,l=197,c=8192,r=0],[p=6488,l=6488,c=32768,r=0],true)@END
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpConnection - org.eclipse.jetty.server.HttpConnection$SendCallback@41aa5ca4[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@39ff5026] generate: DONE ([p=197,l=197,c=8192,r=0],[p=6488,l=6488,c=32768,r=0],true)@END
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.io.AbstractEndPoint - shutdownOutput SocketChannelEndPoint@3a62407a{/172.29.182.74:41862<->/overlord-leader-ip:8090,OPEN,fill=-,flush=-,to=0/300000}{io=0/0,kio=0,kro=1}->HttpConnection@64c6429c[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4c8fafee{s=END}]=>HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//druid-overlord.emea.iods
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.session - FinalSession=null old_session_manager=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1 this=org.eclipse.jetty.server.session.SessionHandler1384626783==dftMaxIdleSec=-1
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.Server - handled=true async=false committed=true on HttpChannelOverHttp@7529c4ca{r=1,c=true,a=DISPATCHED,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57}
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@57964b85{s=DISPATCHED a=NOT_ASYNC i=true r=READY w=false}
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=true,a=COMPLETING,uri=//overlord-loadbalancer-url/druid/indexer/v1/task,age=57} action COMPLETE
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - onComplete HttpChannelState@57964b85{s=COMPLETING a=NOT_ASYNC i=false r=READY w=false}
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMPLETE for /druid/indexer/v1/task written=6488
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - COMPLETE for /druid/indexer/v1/task written=6488
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.apache.druid.jetty.RequestLog - 172.29.182.74 POST //overlord-loadbalancer-url/druid/indexer/v1/task HTTP/1.1
2019-10-04T09:20:10,413 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannelState - recycle HttpChannelState@57964b85{s=COMPLETED a=NOT_ASYNC i=false r=READY w=false}
2019-10-04T09:20:10,414 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - close HttpParser{s=END,0 of 0}
2019-10-04T09:20:10,414 DEBUG [qtp1472031375-90] org.eclipse.jetty.http.HttpParser - END --> CLOSE
2019-10-04T09:20:10,414 DEBUG [qtp1472031375-90] org.eclipse.jetty.server.HttpChannel - HttpChannelOverHttp@7529c4ca{r=1,c=false,a=IDLE,uri=null,age=0} handle exit, result COMPLETE

[zhaolihe]

have the same question, how to config

[zhaolihe]

because only one Overlord is active when doing HA on Overlord. so when post task to the Overlord of standby, the Overlord will return empty taskid, and no running task at the time.
we can post the same task to the other Overlord which is active, it will return a taskid, and have a running task at the time.
but how to get the active overlord before post task?

[zhaolihe]

but how to get the active overlord before post task?

using http://ha_ip:8888/druid/indexer/v1/task to commit task,it will auto to the active overlord

[giom-l]

Hi, Unless I'm wrong, the port 8888 is the router port. Is your router colocated with your overlord ?

[zhaolihe]

Is your router colocated with your overlord ?

yes, the service of router and the service of overlord are setup at the same machine. but do test using different machine, it is working too

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.