Incompatible event loop type with native transport enabled on linux

[anuranjit]

We are seeing high CPU Usage in vertx with eventloop jstack stuck on "at sun.nio.ch.EPollArrayWrapper.epollWait". So we were trying to use native transport
on linux using "setPreferNativeTransport()" but we are getting following error.

       java.lang.IllegalStateException: incompatible event loop type: 
          io.netty.channel.epoll.EpollEventLoop
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:469)
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
	at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331)
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
	at com.github.mauricio.async.db.mysql.codec.MySQLConnectionHandler.connect(MySQLConnectionHandler.scala:87)
	at com.github.mauricio.async.db.mysql.MySQLConnection.connect(MySQLConnection.scala:84)
	at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createConnection(AsyncConnectionPool.java:63)
	at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createOrWaitForAvailableConnection(AsyncConnectionPool.java:84)
	at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.take(AsyncConnectionPool.java:93)
	at io.vertx.ext.asyncsql.impl.BaseSQLClient.getConnection(BaseSQLClient.java:68)
	at io.vertx.ext.asyncsql.impl.AsyncSQLClientImpl.getConnection(AsyncSQLClientImpl.java:54)
	at io.vertx.ext.asyncsql.impl.ClientWrapper.getConnection(ClientWrapper.java:52)
	at io.vertx.ext.sql.SQLClient.queryWithParams(SQLClient.java:109)
	at io.vertx.reactivex.ext.sql.SQLClient.queryWithParams(SQLClient.java:218)
	at io.vertx.reactivex.ext.sql.SQLClient.lambda$rxQueryWithParams$6(SQLClient.java:231)
	at io.vertx.reactivex.core.impl.AsyncResultSingle.subscribeActual(AsyncResultSingle.java:42)

We are using vertx-mysql-postgresql-client and it seems to not support native transport. Has anyone seen this problem ?

We are using
Vertx: 3.5.3
vertx-mysql-postgresql-client: 3.5.3

[vietj]

Hi this CPU issue sometimes happens with Netty, you might want to look at your kernel version and in the existing (closed) Netty issues https://github.com/netty/netty/issues?q=is%3Aissue+epollwait+is%3Aclosed <https://github.com/netty/netty/issues?q=is:issue+epollwait+is:closed> concerning the mysql driver, I think it is because the underlying wrapped driver is not able to use a native transport. if you are using Postgresql you could try to use the reactive-pg-client that works with native transport. Julien

…

On 3 Jan 2019, at 15:47, Anuranjit ***@***.***> wrote: We are seeing high CPU Usage in vertx with eventloop jstack stuck on "at sun.nio.ch.EPollArrayWrapper.epollWait". So we were trying to use native transport on linux using "setPreferNativeTransport()" but we are getting following error. java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.epoll.EpollEventLoop at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:469) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331) at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163) at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145) at com.github.mauricio.async.db.mysql.codec.MySQLConnectionHandler.connect(MySQLConnectionHandler.scala:87) at com.github.mauricio.async.db.mysql.MySQLConnection.connect(MySQLConnection.scala:84) at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createConnection(AsyncConnectionPool.java:63) at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createOrWaitForAvailableConnection(AsyncConnectionPool.java:84) at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.take(AsyncConnectionPool.java:93) at io.vertx.ext.asyncsql.impl.BaseSQLClient.getConnection(BaseSQLClient.java:68) at io.vertx.ext.asyncsql.impl.AsyncSQLClientImpl.getConnection(AsyncSQLClientImpl.java:54) at io.vertx.ext.asyncsql.impl.ClientWrapper.getConnection(ClientWrapper.java:52) at io.vertx.ext.sql.SQLClient.queryWithParams(SQLClient.java:109) at io.vertx.reactivex.ext.sql.SQLClient.queryWithParams(SQLClient.java:218) at io.vertx.reactivex.ext.sql.SQLClient.lambda$rxQueryWithParams$6(SQLClient.java:231) at io.vertx.reactivex.core.impl.AsyncResultSingle.subscribeActual(AsyncResultSingle.java:42) We are using vertx-mysql-postgresql-client and it seems to not support native transport. Has anyone seen this problem ? We are using Vertx: 3.5.3 vertx-mysql-postgresql-client: 3.5.3 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#135>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANxiowjZyDsIpG0NxZyqurTG6wZhNjxks5u_hf8gaJpZM4ZoNXK>.

[oshai]

Native transport is not supported in the driver at the moment. However, from a first look lt looks pretty straightforward to add that: https://netty.io/wiki/native-transports.html.
If you want I can do that.

[vietj]

it means you need to modify and recompile the scala driver I think @oshai or can it be done in the vertx mysql-pg client instead ?

[oshai]

I mean only in the new flavour of jasync.

[vietj]

@oshai ok sounds good too

[oshai]

I created version 0.8.60 of jasync with native epoll support

[andy-yx-chen]

0.8.60 does not pass the test because of the failure of the following stack

[main] INFO 🐳 [postgres:9.6.8] - Container postgres:9.6.8 started
Creating configuration for localhost:32823
failed to create connection
java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.nio.NioEventLoop
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:469)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
        at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:333)
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
        at com.github.jasync.sql.db.postgresql.codec.PostgreSQLConnectionHandler.connect(PostgreSQLConnectionHandler.kt:94)
        at com.github.jasync.sql.db.postgresql.PostgreSQLConnection.connect(PostgreSQLConnection.kt:108)
        at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createAndConnect(AsyncConnectionPool.java:98)
        at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createConnection(AsyncConnectionPool.java:72)
        at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createOrWaitForAvailableConnection(AsyncConnectionPool.java:128)

[andy-yx-chen]

I know what is the issue, will try to see if we are able to integrate, ideally, vertx and jasync should use the same eventloop group

[vietj]

@andy-yx-chen the same channel implementation too

[andy-yx-chen]

@vietj , yes, you are right, I will try to make the changes in jasync-sql and update it back

[andy-yx-chen]

@oshai I created a pr in jasync to infer SocketChannel type from the EventLoopGroup, which allows the user to decide which event loop group to use instead of forcing users to create eventloop groups from NettyUtils jasync-sql/jasync-sql#69

[oshai]

@andy-yx-chen I released version 0.8.61 with this fix.

[oshai]

Update: I created version 0.8.62 with the native dependencies as optional, so I recommend using that one.

[andy-yx-chen]

@oshai thanks, this is good and bad, good thing, more flexible, bad thing, it may hide the issue in jasync-sql/jasync-sql#69, I am now integrating the change, will add test cases for native support as well