Druid 0.12.1 release notes

[jihoonson]

DRAFT

Druid 0.12.1 contains stability improvements and bug fixes from 10 contributors. Major improvements include:

• Large performance improvements for coordinator's loadstatus API
• More memory limiting for HttpPostEmitter
• Fix several issues of Kerberos Authentication
• Fix SQLMetadataSegmentManager to allow successive start and stop
• Fix default interval handling in SegmentMetadataQuery
• Support HTTP OPTIONS request
• Fix a bug of different segments of the same segment id in Kafka indexing
  The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+milestone%3A0.12.1

Documentation for this release is at: http://druid.io/docs/0.12.1-rc1

Highlights

Large performance improvements for coordinator's loadstatus API

The loadstatus API of Coordinators returns the percentage of segments actually loaded in the cluster versus segments that should be loaded in the cluster. The performance of this API has greatly been improved.

Added by @jon-wei in #5632.

More memory limiting for HttpPostEmitter

Druid now can limit the amount of memory used by HttpPostEmitter to 10% of the available JVM heap, thereby avoiding OutOfMemory errors from buffered events.

Added by @jon-wei in #5300.

Fix several issues of Kerberos Authentication

There were some bugs in Kerberos authentication like authentication failure without cookies or broken authentication when router is used. See #5596, #5706, and #5766 for more details.

Added by @nishantmonu51 in #5596.
Added by @b-slim in #5706.
Added by @jon-wei in #5766.

Fix SQLMetadataSegmentManager to allow successive start and stop

Coordinators could be stuck if it loses leadership while starting. This bug has been fixed now.

Added by @jihoonson in #5554.

Fix default interval handling in SegmentMetadataQuery

SegmentMetadataQuery is supposed to use the interval of druid.query.segmentMetadata.defaultHistory if the interval is not specified, but it queried all segments instead which incurs an unexpected performance hit. SegmentMetadataQuery now respects the defaultHistory option again.

Added by @gianm in #5489.

Support HTTP OPTIONS request

Druid now supports the HTTP OPTIONS request by fixing its auth handling.

Added by @jon-wei in #5615.

Fix a bug of different segments of the same segment id in Kafka indexing

Kafka indexing service allowed retrying tasks to overwrite the segments in deep storage written by the previous failed tasks. However, this caused another bug that the same segment ID could have different data on historicals and in deep storage. This bug has been fixed now by using unique segment paths for each Kafka index tasks.

Added by @dclim in #5692.

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+milestone%3A0.12.1

Updating from 0.12.0 and earlier

0.12.1 is a minor release and compatible with 0.12.0. If you're updating from an earlier version than 0.12.0, please see release notes of the relevant intermediate versions for additional notes.

Credits

Thanks to everyone who contributed to this release!

@dclim
@gianm
@JeKuOrdina
@jihoonson
@jon-wei
@leventov
@niketh
@nishantmonu51
@pdeva

[chimmy869]

I had install 0.12.1 but coordinator start failed..seams like mysql connect failed...

2018-05-17T02:33:40,655 WARN [main] io.druid.java.util.common.RetryUtils - Failed on try 1, retrying in 757ms.
org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not create connection to database server.)
at org.skife.jdbi.v2.DBI.open(DBI.java:230) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279) ~[jdbi-2.63.1.jar:2.63.1]
at io.druid.metadata.SQLMetadataConnector$2.call(SQLMetadataConnector.java:135) ~[druid-server-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) [java-util-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) [java-util-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:139) [druid-server-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:148) [druid-server-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:189) [druid-server-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.metadata.SQLMetadataConnector.createConfigTable(SQLMetadataConnector.java:307) [druid-server-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.metadata.SQLMetadataConnector.createConfigTable(SQLMetadataConnector.java:561) [druid-server-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.guice.JacksonConfigManagerModule$1.start(JacksonConfigManagerModule.java:58) [druid-common-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:311) [java-util-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134) [druid-api-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.cli.ServerRunnable.run(ServerRunnable.java:50) [druid-services-0.12.1-rc2.jar:0.12.1-rc2]
at io.druid.cli.Main.main(Main.java:116) [druid-services-0.12.1-rc2.jar:0.12.1-rc2]
Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (Could not create connection to database server.)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2152) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 15 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_73]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_73]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_73]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_73]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[?:?]
at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[?:?]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) ~[?:?]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896) ~[?:?]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885) ~[?:?]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2332) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2085) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:795) ~[?:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:44) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_73]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_73]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_73]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_73]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) ~[?:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327) ~[?:?]
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 15 more
Caused by: java.lang.NullPointerException
at com.mysql.jdbc.Buffer.(Buffer.java:50) ~[?:?]
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1677) ~[?:?]
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2286) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2085) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:795) ~[?:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:44) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_73]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_73]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_73]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_73]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) ~[?:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327) ~[?:?]
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]
... 15 more

[jihoonson]

Hi @chimmy869, sorry for late response.

The error looks that Druid can't connect to metastore. Would you check your configurations?

[jihoonson]

Druid 0.12.1 is available now (https://github.com/druid-io/druid/releases/tag/druid-0.12.1).