New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Druid 0.10.1, incomplete #233

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@leventov
Member

leventov commented Jun 16, 2017

No description provided.

@leventov leventov changed the title from Druid 0.10.1, work in progress to Druid 0.10.1 Jun 30, 2017

@leventov leventov changed the title from Druid 0.10.1 to Update to Druid 0.10.1, incomplete Jun 30, 2017

@leventov

This comment has been minimized.

Member

leventov commented Jun 30, 2017

This PR is incomplete but I probably not going to work on it soon, so you are welcome to continue this work if you need it, please announce this with a comment to this PR if you do.

@Igosuki

This comment has been minimized.

Igosuki commented Jul 31, 2017

Is this the official PR to update to druid 0.10.0 ?

@leventov

This comment has been minimized.

Member

leventov commented Jul 31, 2017

@Igosuki what do you mean by "official"? It's just a PR that I started to support Druid 0.10.1 as a dependency of Tranquility

@Igosuki

This comment has been minimized.

Igosuki commented Jul 31, 2017

@leventov ok I meant it as, if I help you finish it, it's not gonna get overwritten by any core committer who already was working on it.

@leventov

This comment has been minimized.

Member

leventov commented Jul 31, 2017

@Igosuki I don't plan to continue work on this PR anytime soon so go ahead.

@Igosuki

This comment has been minimized.

Igosuki commented Aug 1, 2017

@leventov Alright, thanks !

@Igosuki

This comment has been minimized.

Igosuki commented Aug 3, 2017

After updating to 0.10.1-rc2, it seems the default Guice bindings in druid force me to add a lot of modules to the Tranquility Guicer, have you come across these problems ? Do I have to implement some noops for these modules for tranquility ?
Here are the modules I'm forced to add after updating :

+      new DruidProcessingConfigModule,
+      new DruidProcessingModule,
        new QueryableModule,
+      new QueryRunnerFactoryModule,
+      classOf[EmitterModule],
@Igosuki

This comment has been minimized.

Igosuki commented Aug 25, 2017

@Gauravshah

This comment has been minimized.

Gauravshah commented Aug 27, 2017

@Igosuki Thanks. can you help me get started with test cases I would like to contribute.
I am running 'sbt test' but getting disco error with null as requested host. I have local druid 0.10.1 with zookeeper. Should I add some configuration for test case for it to discover my local zookeeper ?

@Gauravshah

This comment has been minimized.

Gauravshah commented Aug 29, 2017

backtrace:

[info]   com.metamx.common.ISE: Failed test
[info]   at com.metamx.tranquility.test.DirectDruidTest$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(DirectDruidTest.scala:142)
[info]   at com.metamx.tranquility.test.DirectDruidTest$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(DirectDruidTest.scala:122)
[info]   at com.metamx.tranquility.test.common.DruidIntegrationSuite$$anonfun$withDruidStack$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(DruidIntegrationSuite.scala:213)
[info]   at com.metamx.tranquility.test.common.DruidIntegrationSuite$$anonfun$withDruidStack$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(DruidIntegrationSuite.scala:212)
[info]   at com.metamx.tranquility.test.common.DruidIntegrationSuite$$anonfun$withOverlord$1.apply(DruidIntegrationSuite.scala:195)
[info]   at com.metamx.common.scala.control$$anonfun$retryOnErrors$1.apply(control.scala:42)
[info]   at com.metamx.common.scala.control$$anonfun$retryOnErrors$1.apply(control.scala:41)
[info]   at com.metamx.common.scala.control$$anonfun$withBackoff$1.apply(control.scala:53)
[info]   at com.metamx.common.scala.control$$anonfun$withBackoff$1.apply(control.scala:53)
[info]   at com.metamx.common.scala.control$.untilSome(control.scala:28)
[info]   ...
[info]   Cause: java.lang.IllegalStateException: Failed to create merged beam: druid:tranquility:indexer/xxx
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:379)
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:375)
[info]   at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.k(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:117)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:98)
[info]   at com.twitter.util.Promise$$anon$1.run(Promise.scala:421)
[info]   at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
[info]   at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
[info]   at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)
[info]   ...
[info]   Cause: java.lang.IllegalStateException: Failed to save new beam for identifier[druid:tranquility:indexer/xxx] timestamp[2017-08-29T04:00:00.000Z]
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:285)
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:282)
[info]   at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1119)
[info]   at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1117)
[info]   at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.k(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:117)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:98)
[info]   at com.twitter.util.Promise$$anon$1.run(Promise.scala:421)
[info]   at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
[info]   ...
[info]   Cause: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!druid:tranquility:indexer, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available
[info]   ...
Reporter completed abruptly with an exception after receiving event: TestFailed(Ordinal(0, 105),Failed test,DirectDruidTest,com.metamx.tranquility.test.DirectDruidTest,Some(com.metamx.tranquility.test.DirectDruidTest),Druid standalone,Druid standalone,Vector(),Some(com.metamx.common.ISE: Failed test),Some(108631),Some(IndentedText(- Druid standalone,Druid standalone,0)),Some(SeeStackDepthException),Some(com.metamx.tranquility.test.DirectDruidTest),None,pool-1-thread-1-ScalaTest-running-DirectDruidTest,1503979593710).
java.io.NotSerializableException: com.twitter.finagle.context.RemoteInfo$NotAvailable$
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at org.scalatest.tools.SocketReporter.apply(SocketReporter.scala:31)
	at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:240)
	at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:239)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:239)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-5" java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.twitter.finagle.context.RemoteInfo$NotAvailable$
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1539)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:552)
	at java.lang.Throwable.readObject(Throwable.java:914)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
@gianm

This comment has been minimized.

Member

gianm commented Aug 29, 2017

@Gauravshah the tests should be completely self-contained; this might be an issue with the tests. In particular this line seems suspicious:

java.io.NotSerializableException: com.twitter.finagle.context.RemoteInfo$NotAvailable$

Maybe it's related to a finagle version change?

@Gauravshah

This comment has been minimized.

Gauravshah commented Oct 9, 2017

@Igosuki were you able to move forward ?
I had spent sometime but was not able to figure out the java.io.NotSerializableException

@Igosuki

This comment has been minimized.

Igosuki commented Jan 18, 2018

@Gauravshah I ended up pausing that work, because 0.12 is coming out soon, and the main signature types and required modules completely changed.

Plus it seems the strategy is to move away from tranquility towards kafka indexing service, will come back to that later.

I am currently running the latest Tranquility with avro extensions against 0.10.x and it works fine because the index specs haven't changed.

@Gauravshah

This comment has been minimized.

Gauravshah commented Jan 18, 2018

thanks @Igosuki , we are using kinesis so wondering how would kafka indexing service would work in that world.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment