New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DefaultChannelHandlerContext uses too much memory #920

Closed
shijinkui opened this Issue Jan 10, 2013 · 18 comments

Comments

Projects
None yet
5 participants
@shijinkui

keepalive, a connection gen 5 DefaultChannelHandlerContext object, and the DefaultChannelHandlerContext object instance can not be gc.

before a request recieved:

QQ20130110-16

after a request recieved:
QQ20130110-15

question:

  1. DefaultChannelPipeline hold two DefaultChannelHandlerContext object, why evry thread hold five
  2. evry request gen a DefaultChannelHandlerContext?

this is my program run a moment, the heap dump

3062837 instances of class java.lang.Object 
2625047 instances of class java.util.Collections$UnmodifiableSet 
2624969 instances of class java.util.RegularEnumSet 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$1 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$10 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$2 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$3 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$4 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$5 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$6 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$7 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$8 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$9 

...
437495 instances of class io.netty.channel.DefaultChannelPipeline 
437495 instances of class io.netty.channel.DefaultChannelPipeline$HeadHandler 

a obj, References to this object: DefaultChannelHandlerContext


Class 0x50619a828

class io.netty.channel.DefaultChannelHandlerContext$2

Superclass:

class java.lang.Object
Loader Details

ClassLoader:

sun.misc.Launcher$AppClassLoader@0x51cadeb00 (122 bytes)
Signers:

<null>
Protection Domain:

java.security.ProtectionDomain@0x51cadc850 (58 bytes)
Subclasses:

Instance Data Members:

this$0 (L)
Static Data Members:

Instances

Exclude subclasses
Include subclasses
References summary by Type

References summary by type
References to this object:

io.netty.channel.DefaultChannelHandlerContext$2@0x58f8200a8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5452dbb18 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x52e31a3d8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x537ff0d68 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58f2e7ab8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x53fc8eac0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x589d51758 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x580ae9200 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x531477618 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x574904f88 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x598fa1b70 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5459c9750 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5949953c8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x52a1c8e88 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x56bbab320 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x571bdd478 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59b9432e0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x553279fa0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a17d8f90 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59e6951f8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58f1a73d0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x545d15750 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x522546790 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x558d77408 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x524d39490 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x537903c50 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5556813b0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x55db0ca50 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x525557328 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x56ecf9798 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a2ed9970 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x551f4a308 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58276fa80 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a7832ed0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58f5e1958 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x54e558238 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x567545c40 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58fdbea18 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a08b77c0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x51f059000 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a78f3550 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a1a52778 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x526ad1de0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x529b9f6a8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a3fbb908 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59240d840 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a3127350 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x582c94c60 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59e441d48 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a8d84eb8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a07c6750 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x553a859a0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x541a592d8 (24 bytes) : ??

is it normal? @trustin @gresrun @normanmaurer

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 10, 2013

Member

@shijinkui it is normal.. the DefaultChannelHandlerContext can only GC'ed once the Channel is GC'ed..

Member

normanmaurer commented Jan 10, 2013

@shijinkui it is normal.. the DefaultChannelHandlerContext can only GC'ed once the Channel is GC'ed..

@shijinkui

This comment has been minimized.

Show comment
Hide comment
@shijinkui

shijinkui Jan 11, 2013

i think one request gen 10x4 DefaultChannelHandlerContext, it's not normal.

this is the echo example, two request from deffirent jvm instance.

EchoServer started, no request.

QQ20130111-15

a client request arrive.
QQ20130111-16

the second request arrive.
QQ20130111-17

the DefaultChannelHandlerContext increase: requst * 4 * 10.
when the keepalive request is more, the DefaultChannelHandlerContext is too large, old Gen cannot be gc.

i think one request gen 10x4 DefaultChannelHandlerContext, it's not normal.

this is the echo example, two request from deffirent jvm instance.

EchoServer started, no request.

QQ20130111-15

a client request arrive.
QQ20130111-16

the second request arrive.
QQ20130111-17

the DefaultChannelHandlerContext increase: requst * 4 * 10.
when the keepalive request is more, the DefaultChannelHandlerContext is too large, old Gen cannot be gc.

@shijinkui

This comment has been minimized.

Show comment
Hide comment
@shijinkui

shijinkui Jan 11, 2013

master @normanmaurer @trustin , i test echoclient:

use 1 thread to process event, see the below screenshot


b.group(new NioEventLoopGroup(1))
             .channel(NioSocketChannel.class)
             .option(ChannelOption.TCP_NODELAY, true)
             .remoteAddress(new InetSocketAddress(host, port))
             .handler(new ChannelInitializer<SocketChannel>() {
                 @Override
                 public void initChannel(SocketChannel ch) throws Exception {
                     ch.pipeline().addLast(
                             new LoggingHandler(LogLevel.INFO),
                             new EchoClientHandler(firstMessageSize));
                 }
             });

picture1:
QQ20130111-18

picture2:
QQ20130111-19

protected MultithreadEventExecutorGroup(int nThreads, ThreadFactory threadFactory, Object... args)
line 70, really only one NioEventLoop is created, but the number of DEFAULT_POOL_SIZE DefaultChannelHandlerContext where is created.

children = new SingleThreadEventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
                children[i] = newChild(threadFactory, scheduler, args);
                success = true;
            } catch (Exception e) {
                throw new EventLoopException("failed to create a child event loop", e);
            } finally {
                if (!success) {
                    for (int j = 0; j < i; j ++) {
                        children[j].shutdown();
                    }
                }
            }
        }

ps: my english is pool :(

Thanks

master @normanmaurer @trustin , i test echoclient:

use 1 thread to process event, see the below screenshot


b.group(new NioEventLoopGroup(1))
             .channel(NioSocketChannel.class)
             .option(ChannelOption.TCP_NODELAY, true)
             .remoteAddress(new InetSocketAddress(host, port))
             .handler(new ChannelInitializer<SocketChannel>() {
                 @Override
                 public void initChannel(SocketChannel ch) throws Exception {
                     ch.pipeline().addLast(
                             new LoggingHandler(LogLevel.INFO),
                             new EchoClientHandler(firstMessageSize));
                 }
             });

picture1:
QQ20130111-18

picture2:
QQ20130111-19

protected MultithreadEventExecutorGroup(int nThreads, ThreadFactory threadFactory, Object... args)
line 70, really only one NioEventLoop is created, but the number of DEFAULT_POOL_SIZE DefaultChannelHandlerContext where is created.

children = new SingleThreadEventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
                children[i] = newChild(threadFactory, scheduler, args);
                success = true;
            } catch (Exception e) {
                throw new EventLoopException("failed to create a child event loop", e);
            } finally {
                if (!success) {
                    for (int j = 0; j < i; j ++) {
                        children[j].shutdown();
                    }
                }
            }
        }

ps: my english is pool :(

Thanks

@trustin

This comment has been minimized.

Show comment
Hide comment
@trustin

trustin Jan 14, 2013

Member

They are anonymous Runnables which are used to trigger an event. According to your heap dump, a ChannelHandlerContext takes 1152 bytes. Let's assume we have 10 handlers in a pipeline and a server handles 1,000,000 connections, then the memory consumption will be: 1,000,000 * 10 * 1152 = whooping 10.7 GiB.

I agree we need to reduce memory consumption.

Member

trustin commented Jan 14, 2013

They are anonymous Runnables which are used to trigger an event. According to your heap dump, a ChannelHandlerContext takes 1152 bytes. Let's assume we have 10 handlers in a pipeline and a server handles 1,000,000 connections, then the memory consumption will be: 1,000,000 * 10 * 1152 = whooping 10.7 GiB.

I agree we need to reduce memory consumption.

@trustin trustin reopened this Jan 14, 2013

@ghost ghost assigned trustin Jan 14, 2013

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 14, 2013

Member

@trustin where do you see it is 1152 bytes ?

Member

normanmaurer commented Jan 14, 2013

@trustin where do you see it is 1152 bytes ?

@shijinkui

This comment has been minimized.

Show comment
Hide comment
@shijinkui

shijinkui Jan 15, 2013

@normanmaurer @trustin : i have a question, the ChannelHandlerContext how be created. i trace the code, always can't find out where create ten ChannelHandlerContext fixedly.

  1. DefaultChannelPipeline.addLast is first to be executed.
  2. SingleThreadEventExecutor.execute is second to be executed, to add task.

is NioEventLoop listening the network port, add task event to the queue? and then create the ChannelHandlerContext?
where is the number 10 is setted?

i read netty code several days, always can't be clear to the above question. if u have time, please tell me.
Thanks.
💯

QQ20130115-2

QQ20130115-1

@normanmaurer @trustin : i have a question, the ChannelHandlerContext how be created. i trace the code, always can't find out where create ten ChannelHandlerContext fixedly.

  1. DefaultChannelPipeline.addLast is first to be executed.
  2. SingleThreadEventExecutor.execute is second to be executed, to add task.

is NioEventLoop listening the network port, add task event to the queue? and then create the ChannelHandlerContext?
where is the number 10 is setted?

i read netty code several days, always can't be clear to the above question. if u have time, please tell me.
Thanks.
💯

QQ20130115-2

QQ20130115-1

@trustin

This comment has been minimized.

Show comment
Hide comment
@trustin

trustin Jan 15, 2013

Member

@normanmaurer 512 + 64 * 10?

Member

trustin commented Jan 15, 2013

@normanmaurer 512 + 64 * 10?

@trustin

This comment has been minimized.

Show comment
Hide comment
@trustin

trustin Jan 15, 2013

Member

@shijinkui, they are anonymous Runnnable instances in DefaultChannelHandlerContext. I'll create them lazily

Member

trustin commented Jan 15, 2013

@shijinkui, they are anonymous Runnnable instances in DefaultChannelHandlerContext. I'll create them lazily

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 15, 2013

Member

@trustin lol I did not see the 64 .. Do you think it will really help to create them lazily as once they are created they will take the same amount of space again. We could maybe just create a new on demand everytime. This will put more pressure on the GC but only for stuff that is "NOT" run in the EventLoop Thread. WDYT ?

Member

normanmaurer commented Jan 15, 2013

@trustin lol I did not see the 64 .. Do you think it will really help to create them lazily as once they are created they will take the same amount of space again. We could maybe just create a new on demand everytime. This will put more pressure on the GC but only for stuff that is "NOT" run in the EventLoop Thread. WDYT ?

@trustin

This comment has been minimized.

Show comment
Hide comment
@trustin

trustin Jan 15, 2013

Member

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.

Member

trustin commented Jan 15, 2013

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 15, 2013

Member

Yeah… But I think we can still cache them if we detect one is needed.

Am 15.01.2013 um 08:16 schrieb Trustin Lee notifications@github.com:

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.


Reply to this email directly or view it on GitHub.

Member

normanmaurer commented Jan 15, 2013

Yeah… But I think we can still cache them if we detect one is needed.

Am 15.01.2013 um 08:16 schrieb Trustin Lee notifications@github.com:

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.


Reply to this email directly or view it on GitHub.

@normanmaurer

This comment has been minimized.

Show comment
Hide comment
@normanmaurer

normanmaurer Jan 15, 2013

Member

Working on it....

Am 15.01.2013 um 08:16 schrieb Trustin Lee notifications@github.com:

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.


Reply to this email directly or view it on GitHub.

Member

normanmaurer commented Jan 15, 2013

Working on it....

Am 15.01.2013 um 08:16 schrieb Trustin Lee notifications@github.com:

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.


Reply to this email directly or view it on GitHub.

@ghost ghost assigned normanmaurer Jan 15, 2013

normanmaurer pushed a commit that referenced this issue Jan 15, 2013

Norman Maurer
[#920] Reduce memory usage of DefaultChannelHandlerContext by only in…
…it Runnables for Tasks if really neccessary

trustin added a commit that referenced this issue Jan 15, 2013

Overhaul pipeline implementation for clarity and memory efficiency
This pull request cleans up our pipeline implementation by moving most
inter-context traversal code to DefaultChannelHandlerContext.
Previously, outbound traversal was done in DefaultChannelPipeline while
inbound traversal was done in DefaultChannelHandlerContext.

Also, to address the memory inefficiency issue raised in #920, all
runnables are lazily instantiated.

@ghost ghost assigned trustin Jan 15, 2013

@trustin

This comment has been minimized.

Show comment
Hide comment
@trustin

trustin Jan 15, 2013

Member

506474f should fix the problem.

Member

trustin commented Jan 15, 2013

506474f should fix the problem.

@trustin trustin closed this Jan 15, 2013

@trustin

This comment has been minimized.

Show comment
Hide comment
@trustin

trustin Jan 15, 2013

Member

@shijinkui, could you let us know if my change helps your problem?

Member

trustin commented Jan 15, 2013

@shijinkui, could you let us know if my change helps your problem?

@shijinkui

This comment has been minimized.

Show comment
Hide comment
@shijinkui

shijinkui Jan 16, 2013

@trustin @normanmaurer , i works normal, good job 💯 . instance num is 10~11. let me run in product several days, observe the changes.

@trustin @normanmaurer , i works normal, good job 💯 . instance num is 10~11. let me run in product several days, observe the changes.

@shijinkui

This comment has been minimized.

Show comment
Hide comment
@shijinkui

shijinkui Jan 21, 2013

Class Instance Count Total Size
class io.netty.channel.DefaultChannelHandlerContext 2953703 640953551
class io.netty.channel.DefaultChannelHandlerContext$14 584856 9357696
class io.netty.channel.DefaultChannelHandlerContext$9 113279 1812464
class io.netty.channel.DefaultChannelHandlerContext$7 113279 1812464
class io.netty.channel.DefaultChannelHandlerContext$25 113279 906232
class io.netty.channel.DefaultChannelHandlerContext$10 0 0
class io.netty.channel.DefaultChannelHandlerContext$15 0 0
Class Instance Count Total Size
class io.netty.channel.DefaultChannelHandlerContext 2953703 640953551
class io.netty.channel.DefaultChannelHandlerContext$14 584856 9357696
class io.netty.channel.DefaultChannelHandlerContext$9 113279 1812464
class io.netty.channel.DefaultChannelHandlerContext$7 113279 1812464
class io.netty.channel.DefaultChannelHandlerContext$25 113279 906232
class io.netty.channel.DefaultChannelHandlerContext$10 0 0
class io.netty.channel.DefaultChannelHandlerContext$15 0 0
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jan 21, 2013

@shijinkui what to you use to measure / benchmark this?

ghost commented Jan 21, 2013

@shijinkui what to you use to measure / benchmark this?

normanmaurer pushed a commit that referenced this issue Jan 30, 2013

Norman Maurer
[#995] Replace AtomicReference usage with AtomicReferenceFieldUpdater
This will safe as an example 2gb mem when have 10 DefaultHandlerContext instances per connection and the connection count is 1000000.
Also kind of related to [#920]
@qiaodaimadelaowang

This comment has been minimized.

Show comment
Hide comment
Contributor

qiaodaimadelaowang commented Nov 4, 2016

@carrot-garden I think it is VisualVM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment