2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[2] to offset[1,532,313,909]. 2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[18] to offset[1,532,760,392]. 2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[6] to offset[1,530,328,715]. 2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[22] to offset[1,532,039,133]. 2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[10] to offset[1,530,276,477]. 2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[26] to offset[1,533,853,406]. 2018-07-09T13:35:48,792 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[14] to offset[1,533,109,749]. 2018-07-09T13:35:48,793 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[30] to offset[1,533,942,419]. 2018-07-09T13:35:48,794 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' 2018-07-09T13:35:48,874 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton" 2018-07-09T13:35:48,876 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" 2018-07-09T13:35:48,883 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton" 2018-07-09T13:35:49,181 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider 2018-07-09T13:35:49,181 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm]: SegmentAllocateAction{dataSource='player-events-historical', timestamp=2018-07-09T13:35:40.000Z, queryGranularity={type=period, period=PT30M, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_player-events-historical_68d16cc2aa7e32e_6', previousSegmentId='null'} 2018-07-09T13:35:49,192 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider 2018-07-09T13:35:49,196 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm] to overlord[http://ip-172-30-32-65.node.us-west-2.consul:8090/druid/indexer/v1/action]: SegmentAllocateAction{dataSource='player-events-historical', timestamp=2018-07-09T13:35:40.000Z, queryGranularity={type=period, period=PT30M, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_player-events-historical_68d16cc2aa7e32e_6', previousSegmentId='null'} 2018-07-09T13:35:49,198 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider 2018-07-09T13:35:49,200 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider 2018-07-09T13:35:49,203 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider 2018-07-09T13:35:49,205 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Undefined" 2018-07-09T13:35:49,209 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,241 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@2de9ca6{/,null,AVAILABLE} 2018-07-09T13:35:49,243 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,244 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,244 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,244 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,244 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,247 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,247 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,248 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,248 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,250 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,252 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,255 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,256 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,257 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,257 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,258 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,258 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,259 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,264 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:35:49,264 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@22309bfa{HTTP/1.1,[http/1.1]}{0.0.0.0:8101} 2018-07-09T13:35:49,264 INFO [main] org.eclipse.jetty.server.Server - Started @4758ms 2018-07-09T13:35:49,265 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.query.lookup.LookupReferencesManager.start()] on object[io.druid.query.lookup.LookupReferencesManager@4cffd3fd]. 2018-07-09T13:35:49,266 INFO [main] io.druid.query.lookup.LookupReferencesManager - Started lookup factory references manager 2018-07-09T13:35:49,266 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.start()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@1142d377]. 2018-07-09T13:35:49,284 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Announcing start time on [/druid/listeners/lookups/__default/ip-172-30-32-52.node.us-west-2.consul:8101] 2018-07-09T13:35:49,285 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.start()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@3ebc955b]. 2018-07-09T13:35:49,285 INFO [main] io.druid.server.coordination.AbstractDataSegmentAnnouncer - Announcing self[DruidServerMetadata{name='ip-172-30-32-52.node.us-west-2.consul:8101', host='ip-172-30-32-52.node.us-west-2.consul:8101', maxSize=0, tier='_default_tier', type='indexer-executor', priority='0'}] at [/druid/announcements/ip-172-30-32-52.node.us-west-2.consul:8101] 2018-07-09T13:36:39,576 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - New segment[player-events-historical_2018-07-09T13:00:00.000Z_2018-07-09T14:00:00.000Z_2018-07-09T13:19:31.611Z_7] for sequenceName[index_kafka_player-events-historical_68d16cc2aa7e32e_6]. 2018-07-09T13:36:39,606 INFO [task-runner-0-priority-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[player-events-historical_2018-07-09T13:00:00.000Z_2018-07-09T14:00:00.000Z_2018-07-09T13:19:31.611Z_7] at new path[/druid/segments/ip-172-30-32-52.node.us-west-2.consul:8101/ip-172-30-32-52.node.us-west-2.consul:8101_indexer-executor__default_tier_2018-07-09T13:36:39.604Z_aa8a4800efa847f3ba5ee5d18473b0ea0] 2018-07-09T13:36:39,807 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm]: SegmentAllocateAction{dataSource='player-events-historical', timestamp=2018-07-09T13:35:40.000Z, queryGranularity={type=period, period=PT30M, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_player-events-historical_68d16cc2aa7e32e_22', previousSegmentId='null'} 2018-07-09T13:36:39,808 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm] to overlord[http://ip-172-30-32-65.node.us-west-2.consul:8090/druid/indexer/v1/action]: SegmentAllocateAction{dataSource='player-events-historical', timestamp=2018-07-09T13:35:40.000Z, queryGranularity={type=period, period=PT30M, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_player-events-historical_68d16cc2aa7e32e_22', previousSegmentId='null'} 2018-07-09T13:36:48,414 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:37:48,431 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:38:48,441 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:39:48,451 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:40:48,463 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:41:48,477 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:42:48,490 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:43:48,511 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:44:48,530 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:45:48,539 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:46:48,549 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:47:48,562 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:48:48,571 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:49:48,581 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:50:48,592 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:51:39,910 WARN [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Exception submitting action for task[index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm] org.jboss.netty.handler.timeout.ReadTimeoutException at org.jboss.netty.handler.timeout.ReadTimeoutHandler.(ReadTimeoutHandler.java:84) ~[netty-3.10.4.Final.jar:?] at com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:182) ~[http-client-1.0.6.jar:?] at com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14) ~[http-client-1.0.6.jar:?] at io.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:102) [druid-indexing-service-0.10.0.jar:0.10.0] at io.druid.indexing.appenderator.ActionBasedSegmentAllocator.allocate(ActionBasedSegmentAllocator.java:52) [druid-indexing-service-0.10.0.jar:0.10.0] at io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.getSegment(FiniteAppenderatorDriver.java:347) [druid-server-0.10.0.jar:0.10.0] at io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.add(FiniteAppenderatorDriver.java:201) [druid-server-0.10.0.jar:0.10.0] at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:425) [druid-kafka-indexing-service-0.10.0.jar:0.10.0] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.0.jar:0.10.0] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.0.jar:0.10.0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] 2018-07-09T13:51:39,915 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Will try again in [PT3.821S]. 2018-07-09T13:51:43,736 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm] to overlord[http://ip-172-30-32-65.node.us-west-2.consul:8090/druid/indexer/v1/action]: SegmentAllocateAction{dataSource='player-events-historical', timestamp=2018-07-09T13:35:40.000Z, queryGranularity={type=period, period=PT30M, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_player-events-historical_68d16cc2aa7e32e_22', previousSegmentId='null'} 2018-07-09T13:51:43,736 INFO [task-runner-0-priority-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://ip-172-30-32-65.node.us-west-2.consul:8090 2018-07-09T13:51:48,604 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:52:48,613 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:53:48,623 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:54:48,632 INFO [HttpPostEmitter-1-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://json-push.service.us-west-2.consul:7000 2018-07-09T13:55:28,024 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - New segment[player-events-historical_2018-07-09T13:00:00.000Z_2018-07-09T14:00:00.000Z_2018-07-09T13:19:31.611Z_18] for sequenceName[index_kafka_player-events-historical_68d16cc2aa7e32e_22]. 2018-07-09T13:55:28,031 INFO [task-runner-0-priority-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[player-events-historical_2018-07-09T13:00:00.000Z_2018-07-09T14:00:00.000Z_2018-07-09T13:19:31.611Z_18] at existing path[/druid/segments/ip-172-30-32-52.node.us-west-2.consul:8101/ip-172-30-32-52.node.us-west-2.consul:8101_indexer-executor__default_tier_2018-07-09T13:36:39.604Z_aa8a4800efa847f3ba5ee5d18473b0ea0] 2018-07-09T13:55:28,042 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Submitting persist runnable for dataSource[player-events-historical] 2018-07-09T13:55:28,044 INFO [player-events-historical-incremental-persist] io.druid.segment.realtime.appenderator.AppenderatorImpl - Segment[player-events-historical_2018-07-09T13:00:00.000Z_2018-07-09T14:00:00.000Z_2018-07-09T13:19:31.611Z_18], persisting Hydrant[FireHydrant{index=io.druid.segment.incremental.OnheapIncrementalIndex@cfe3f06, queryable=io.druid.segment.ReferenceCountingSegment@239c0e9, count=0}] 2018-07-09T13:55:28,048 INFO [player-events-historical-incremental-persist] io.druid.segment.IndexMerger - Starting persist for interval[2018-07-09T13:00:00.000Z/2018-07-09T14:00:00.000Z], rows[1] 2018-07-09T13:55:28,058 INFO [player-events-historical-incremental-persist] io.druid.segment.IndexMergerV9 - Start making v9 index files, outDir:var/druid/task/index_kafka_player-events-historical_68d16cc2aa7e32e_bgncekcm/work/persist/player-events-historical_2018-07-09T13:00:00.000Z_2018-07-09T14:00:00.000Z_2018-07-09T13:19:31.611Z_18/0 2018-07-09T13:55:28,061 INFO [player-events-historical-incremental-persist] io.druid.segment.IndexMergerV9 - Completed version.bin in 3 millis. 2018-07-09T13:55:28,062 INFO [player-events-historical-incremental-persist] io.druid.segment.IndexMergerV9 - Completed factory.json in 1 millis