netty out of direct memory #425

ioilala · 2017-02-18T04:07:32Z

my pushy version:0.8.1, jdk 1.8.0_66,netty 4.1.5.Final
I get such exception as follows after I use pushy to send IOS push notifycations for several months in my production environment. We send about 30,000,000 notifications every time, and several times per week.

io.netty.util.internal.outofdirectmemoryerror: failed to allocate 16777216 byte(s) of direct memory (used: 17163092255, max: 17179869184),io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 17163092255, max: 17179869184),failed to allocate 16777216 byte(s) of direct memory (used: 17163092255, max: 17179869184),

Do I have to upgrade to the newest version of pushy(0.9.2) to solve this problem?

jchambers · 2017-02-20T16:39:54Z

This is the first I've heard of the problem, and am not certain that upgrading to 0.9.2 would necessarily help.

To check the math, it looks like you're exhausting about 16Gb of direct memory after "several months." At 30,000,000 notifications per patch, an estimated 3 batches per week, four weeks per month, and (let's guess) 3 months until memory exhaustion, it sounds like you're sending somewhere around a billion notifications before you run out of memory. In other words, we're losing something like 16 bytes per notification.

I'm not entirely sure what could be going wrong here, but will see if I can reproduce the problem locally. To be clear, are you using alpn-boot, or are you using netty-tcnative?

kadar2012 · 2017-02-24T07:15:28Z

my pushy version is 0.9.1, jdk 1.8 ,netty 4.1.8.Final,
i am getting below issue, i can only able to send 2028 count of successful IOS push notification. after that i am getting below error,
using alpn-boot-8.1.10.v20161026-2.jar i am running this in local.

can sugges me how to solve this.

[23 Feb 2017 17:29:42]  WARN  io.netty.channel.AbstractChannel                   - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x00afe7a9]
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33)
	at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847)
	at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475)
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
	at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:85)
	at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:330)
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:126)
	at com.relayrides.pushy.apns.ApnsClient.connect(ApnsClient.java:457)
	at com.relayrides.pushy.apns.ApnsClient.connect(ApnsClient.java:418)
	at sg.gov.hdb.pushy.BF08PushUnsubPayload.send(BF08PushUnsubPayload.java:135)
	at sg.gov.hdb.pushy.BF08PushUnsubPayload.main(BF08PushUnsubPayload.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)

jchambers · 2017-02-24T14:35:43Z

@kadar2012 It sounds like you're having a different problem. At your convenience, please open a separate ticket. We'll need some more information to be able to help you, too; when you create that new ticket, can you share some of the code you're using to create ApnsClient instances and send notifications?

Thanks!

ioilala · 2017-02-26T13:24:17Z

@jchambers Hi, there, I'm using alpn-boot 8.1.6.v20151105. I agree with your analysis and wish you can find something helpful with your local experiments. By the way, I have upgrade to version 0.9.2. If there is any thing new about this stuff, I will reply here.

kadar2012 · 2017-02-27T01:21:47Z

@jchambers i have open a new ticket. thanks in advance.

ioilala · 2017-02-28T06:39:13Z

I encounter this same issue just one day after I upgraded to v0.9.2. Moreover, I start to dount that this issue is caused by netty itself, because the inner exception throws from netty code. My netty version is 4.1.6.Final

io.netty.util.internal.outofdirectmemoryerror: failed to allocate 16777216 byte(s) of direct memory (used: 17163092255, max: 17179869184),io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 17163092255, max: 17179869184),failed to allocate 16777216 byte(s) of direct memory (used: 17163092255, max: 17179869184)

ioilala · 2017-03-20T13:28:59Z

Could you please tell me how can I locate the exact place of memory leak? I find memory leak proof in the logs of our production environment as follows:

ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

But after I add -Dio.netty.leakDetection.level=advanced . I still cannot get any more infomation about this issue. I wonder if there is any switch in the code of pushy where I can turn on to locate the exact place of memory leak. Thanks for you attention~

jchambers · 2017-03-23T15:58:22Z

I'm not entirely sure, honestly. Maybe the Netty folks could provide some advice?

The latest update to Netty has some improvements around leak detection, and I plan on updating Pushy's Netty version shortly. Pardon the delay here, by the way; I've just returned from a few weeks of vacation out of the country and am still trying to catch up on everything.

jchambers · 2017-04-01T16:04:42Z

I can only find two places where we're allocating our own direct memory:

When we're writing a message payload.
When we're sending a PING frame due to inactivity.

In both cases, I'm confident we're releasing the buffers we're allocating.

Everything else is either a heap buffer or managed by something deeper inside Netty. I'm worried that this might be an upstream problem. Still, we'll upgrade to the latest version (#440 should be merged shortly), and hopefully that will either (a) solve the problem or (b) give us the tools to figure out where the problem is.

jchambers · 2017-04-04T01:13:06Z

@ioilala While I don't necessarily expect it to solve the problem, we did just release Pushy v0.9.3, which uses the latest version of Netty. Would you mind giving that a try and letting us know if the problem persists? Thanks!

jchambers · 2017-05-10T22:11:30Z

I couldn't reproduce this locally, but may be testing the wrong things. @ioilala to eliminate some possible differences between our environments:

Are you using TLS-based authentication or token-based authentication?
Are you using netty-tcnative or alpn-boot?

My other hypothesis is that something is happening when notifications get rejected (in which case, we'd be losing a larger amount of memory less frequently, and it just so happens to amortize out to about 16 bytes/notification).

Thanks!

jchambers · 2017-05-24T16:49:03Z

I strongly suspect this issue may have been resolved by moving from finalizer-based direct memory recovery to reference-counted direct memory recovery (by way of using reference-counted SSL providers) in v0.10. If you wouldn't mind upgrading and letting us know if the problem persists, we'd certainly appreciate it.

If we don't hear from you in a few weeks, we'll close this issue, but will happily re-open it if you get back to us later.

Thanks!

jchambers · 2017-06-19T17:42:43Z

Closing due to lack of information. @ioilala if you believe this problem persists, please leave a comment here and we'll happily reopen this issue. Thank you!

yanggz01 · 2017-10-14T15:41:37Z

i have the same question in v0.11.0 in my production environment and try to find the reason. There is any method to fix it now?

jchambers · 2017-10-14T17:17:47Z

i have the same question in v0.11.0…

@yanggz01 Try 0.11.3; 0.11.0 does not have reference-counted SSL providers (which we think fix this problem), but 0.11.1 and newer do (and if you're going to 0.11.1, you may as well go all the way to 0.11.3).

yanggz01 · 2017-10-16T06:41:54Z

@jchambers OK,i try 0.11.3 and watch it's perform, thanks.

yanggz01 · 2017-10-22T04:37:25Z

@jchambers i am so sad to tell you that 0.11.3 also throw the issue today in my production environment.

jchambers · 2017-10-22T15:01:24Z

Bummer. Thanks for letting me know.

Is your situation comparable to the one reported by @ioilala (i.e. this happens after something like a billion notifications)?

jchambers · 2017-10-22T20:21:25Z

…also, can you tell me as much as possible about your environment? Which JVM are you using? OS? Are you using netty-tcnative?

Thank you!

yanggz01 · 2017-10-23T01:46:26Z

@jchambers The count of notifications not beyond million and my OS is on virtual cloud hosting, i use maven, netty-tcnative is netty-tcnative-boringssl-static-2.0.6.Final.jar. and

java version:
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

OS:
Linux iZu159mtqr4Z 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

yanggz01 · 2017-10-23T01:48:05Z

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658527, max: 268435456)

jchambers · 2017-10-26T18:39:19Z

Hm. That sounds like a much steeper loss rate (~256 bytes/notification assuming ~1,000,000 notifications) than what @ioilala saw. Will continue to investigate. Thanks for the additional details.

jchambers · 2017-11-12T16:00:43Z

@yanggz01 A new version of Netty just came out with a very interesting fix: netty/netty#7338

I haven't been able to reproduce the problem locally (though I admit I haven't had a ton of time to devote to the problem in the past couple weeks). The experiments I DID run showed that there's definitely no leak under steady-state, error-free conditions.

The users most affected by this issue seem to be in China, and we've observed very high rates of packet loss from China (particularly Beijing) to the production APNs servers. That could mean that the thing that's going wrong is some bad cleanup when something fails unexpectedly, and that's exactly what's fixed in the latest version of Netty. If you'd like to give #551 a try, I'd love to hear if it helps your problem.

Thank you!

yanggz01 · 2017-11-13T02:05:52Z

@jchambers I am very thanks for your reply. I guess maybe the issue is due to the
-XX:MaxDirectMemorySize is too low i set. I set it from 128M to 4G and the last two weeks this issue is not reproduce. Thanks very much!

jchambers · 2017-11-21T16:20:34Z

I set it from 128M to 4G and the last two weeks this issue is not reproduce.

Well, my suspicion is that the problem is still there (since we haven't actually shipped any changes yet), but just takes much longer to run out of memory. After we ship 0.12, I think it should be safe to go back to 128M. If you get a chance to upgrade and make the change, please let us know. 0.12 should be out shortly.

Thanks!

qiankunli · 2017-11-23T01:01:19Z

I tried to use Netty 4.1.17/netty-tcnative 2.0.7. it may be unused in china. the detail as follows.

qiankunli · 2017-11-23T07:00:51Z

the IOExcepiton stack for 0.11.3

java.io.IOException: Stream closed before a reply was received
	at com.turo.pushy.apns.ApnsClientHandler.<clinit>(ApnsClientHandler.java:75)
	at com.turo.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:123)
	at com.turo.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:86)
	at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.buildFromCodec(AbstractHttp2ConnectionHandlerBuilder.java:420)
	at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.buildFromConnection(AbstractHttp2ConnectionHandlerBuilder.java:413)
	at io.netty.handler.codec.http2.AbstractHttp2ConnectionHandlerBuilder.build(AbstractHttp2ConnectionHandlerBuilder.java:381)
	at com.turo.pushy.apns.ApnsClientHandler$ApnsClientHandlerBuilder.build(ApnsClientHandler.java:130)
	at com.turo.pushy.apns.ApnsChannelFactory$1$2.configurePipeline(ApnsChannelFactory.java:128)
	at io.netty.handler.ssl.ApplicationProtocolNegotiationHandler.userEventTriggered(ApplicationProtocolNegotiationHandler.java:92)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
	at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1480)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1318)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1170)

I think the error may be occurred on

ApnsClient.sendNotification
channelPool.acquire()
channelFactory.create() and fail

so ,I think the pushy may need to do :

channelPool support min capacity, create a little connection before ApnsClient can send notification
it is necessary to support retry mechanism on a lossy connection，looking forward very much。

Excuse me for my poor english.

jchambers · 2017-12-01T23:29:41Z

@qiankunli Thanks for the additional details! Just to make sure we're understanding each other, it sounds like you're saying that you're still seeing OutOfDirectMemoryError even after upgrading to Netty 4.1.17. Is that correct?

If so, I'll try to write some tests to simulate lots of connection failures and see if I can reproduce the problem that way. Thank you again for the additional details—this is very helpful!

klopfdreh · 2018-03-26T06:07:08Z

Hi @jchambers - I also joined as a user and I am wondering if there are any updates regarding this topic?

jchambers · 2018-03-26T13:10:54Z

Updates will be posted as they become available. Contributions—even if just of research—are always welcome.

jchambers · 2018-05-12T15:38:25Z

Friends, is anybody still seeing this issue in v0.13 or newer? I spent a long time trying to reproduce the problem with every flavor of bad network conditions I could think of, but haven't had much luck.

Thanks!

klopfdreh · 2018-05-12T16:40:38Z

No error like this anymore, since we updated to the newest version of pushy.

jchambers · 2018-05-12T17:59:32Z

@klopfdreh Glad to hear it! To be clear though, you did affirmatively have this problem with older versions of Pushy?

klopfdreh · 2018-05-12T18:28:58Z

To be honest we had OOM Exceptions in one of the containers running pushy, but I don’t have any Stacktraces available anymore to have a look if this is exactly the issue mentioned here.

We updated to 0.12.1 and even with high load everything is running fine, now.

I would consider this bug to be closed.

jchambers · 2018-05-13T02:21:57Z

Okay; I'm going to close this for now, but if anybody is still seeing the issue in 0.13 or newer, please post a comment here and we'll reopen it. Thanks, everybody!

Ossic · 2018-08-06T03:06:51Z

Hi, @jchambers ,my pushy is 0.13.3, jdk1.8.0_131,netty 4.1.15
I get the same exception as is mentioned firstly in this page when I use pushy to send many ios notifications frequently in my production environment. We send about 4,000,000 notifications every time, and twice a day.

2018-08-05 18:14:35,055 WARN [nioEventLoopGroup-16-1] i.n.c.DefaultChannelPipeline 1d1900000164fddb0cb70a029aba193e - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.IllegalArgumentException: promise already done: DefaultChannelPromise@62c54d33(failure: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 520093983, max: 536870912))
    at io.netty.channel.AbstractChannelHandlerContext.isNotValidPromise(AbstractChannelHandlerContext.java:910) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:714) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.ssl.SslHandler.finishWrap(SslHandler.java:868) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:854) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:771) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:752) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:201) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:526) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.timeout.IdleStateHandler.channelReadComplete(IdleStateHandler.java:295) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:1236) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1439) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:971) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:168) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:647) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.25.Final.jar:4.1.25.Final]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

By the way, I'm not entirely sure if I am using alpn-boot, or using netty-tcnative.

jchambers · 2018-08-06T03:44:59Z

We send about 4,000,000 notifications every time, and twice a day.

And how long does it take to run out of memory?

I'm not entirely sure if I am using alpn-boot, or using netty-tcnative.

Pushy should log what it's using for an SSL provider when you construct your ApnsClient instance. Do you see one of these lines in your logs? https://github.com/relayrides/pushy/blob/master/pushy/src/main/java/com/turo/pushy/apns/ApnsClientBuilder.java#L519-L525

If you didn't deliberately add alpn-boot as a -javaagent argument, I don't think it's something we need to worry about.

jchambers added the bug label Feb 20, 2017

jchambers mentioned this issue Jun 9, 2017

Memory leak? #478

Closed

jchambers closed this as completed Jun 19, 2017

jchambers reopened this Oct 22, 2017

jchambers mentioned this issue Nov 12, 2017

Update to Netty 4.1.17 #551

Merged

flozano mentioned this issue Apr 11, 2018

Drop ALPN in favor of direct protocol negotiation #598

Merged

jchambers closed this as completed May 13, 2018

This was referenced Jan 14, 2019

Possible memory leak #669

Closed

Maybe direct memory leak in some extreme situations? #659

Closed

Clear unattached promises when channels close #670

Merged

netty out of direct memory #425

netty out of direct memory #425

Comments

ioilala commented Feb 18, 2017 • edited Loading

jchambers commented Feb 20, 2017

kadar2012 commented Feb 24, 2017 • edited by jchambers Loading

jchambers commented Feb 24, 2017

ioilala commented Feb 26, 2017 • edited Loading

kadar2012 commented Feb 27, 2017

ioilala commented Feb 28, 2017

ioilala commented Mar 20, 2017

jchambers commented Mar 23, 2017

jchambers commented Apr 1, 2017

jchambers commented Apr 4, 2017

jchambers commented May 10, 2017

jchambers commented May 24, 2017

jchambers commented Jun 19, 2017

yanggz01 commented Oct 14, 2017

jchambers commented Oct 14, 2017 • edited Loading

yanggz01 commented Oct 16, 2017

yanggz01 commented Oct 22, 2017

jchambers commented Oct 22, 2017

jchambers commented Oct 22, 2017

yanggz01 commented Oct 23, 2017 • edited Loading

yanggz01 commented Oct 23, 2017

jchambers commented Oct 26, 2017

jchambers commented Nov 12, 2017 • edited Loading

yanggz01 commented Nov 13, 2017

jchambers commented Nov 21, 2017

qiankunli commented Nov 23, 2017

qiankunli commented Nov 23, 2017 • edited by jchambers Loading

jchambers commented Dec 1, 2017

klopfdreh commented Mar 26, 2018

jchambers commented Mar 26, 2018

jchambers commented May 12, 2018

klopfdreh commented May 12, 2018

jchambers commented May 12, 2018

klopfdreh commented May 12, 2018

jchambers commented May 13, 2018

Ossic commented Aug 6, 2018 • edited Loading

jchambers commented Aug 6, 2018

ioilala commented Feb 18, 2017 •

edited

Loading

kadar2012 commented Feb 24, 2017 •

edited by jchambers

Loading

ioilala commented Feb 26, 2017 •

edited

Loading

jchambers commented Oct 14, 2017 •

edited

Loading

yanggz01 commented Oct 23, 2017 •

edited

Loading

jchambers commented Nov 12, 2017 •

edited

Loading

qiankunli commented Nov 23, 2017 •

edited by jchambers

Loading

Ossic commented Aug 6, 2018 •

edited

Loading