Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DefaultChannelHandlerContext uses too much memory #920

Closed
shijinkui opened this issue Jan 10, 2013 · 18 comments
Closed

DefaultChannelHandlerContext uses too much memory #920

shijinkui opened this issue Jan 10, 2013 · 18 comments
Assignees
Labels
Milestone

Comments

@shijinkui
Copy link

@shijinkui shijinkui commented Jan 10, 2013

keepalive, a connection gen 5 DefaultChannelHandlerContext object, and the DefaultChannelHandlerContext object instance can not be gc.

before a request recieved:

QQ20130110-16

after a request recieved:
QQ20130110-15

question:

  1. DefaultChannelPipeline hold two DefaultChannelHandlerContext object, why evry thread hold five
  2. evry request gen a DefaultChannelHandlerContext?

this is my program run a moment, the heap dump

3062837 instances of class java.lang.Object 
2625047 instances of class java.util.Collections$UnmodifiableSet 
2624969 instances of class java.util.RegularEnumSet 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$1 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$10 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$2 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$3 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$4 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$5 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$6 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$7 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$8 
2624968 instances of class io.netty.channel.DefaultChannelHandlerContext$9 

...
437495 instances of class io.netty.channel.DefaultChannelPipeline 
437495 instances of class io.netty.channel.DefaultChannelPipeline$HeadHandler 

a obj, References to this object: DefaultChannelHandlerContext


Class 0x50619a828

class io.netty.channel.DefaultChannelHandlerContext$2

Superclass:

class java.lang.Object
Loader Details

ClassLoader:

sun.misc.Launcher$AppClassLoader@0x51cadeb00 (122 bytes)
Signers:

<null>
Protection Domain:

java.security.ProtectionDomain@0x51cadc850 (58 bytes)
Subclasses:

Instance Data Members:

this$0 (L)
Static Data Members:

Instances

Exclude subclasses
Include subclasses
References summary by Type

References summary by type
References to this object:

io.netty.channel.DefaultChannelHandlerContext$2@0x58f8200a8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5452dbb18 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x52e31a3d8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x537ff0d68 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58f2e7ab8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x53fc8eac0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x589d51758 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x580ae9200 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x531477618 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x574904f88 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x598fa1b70 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5459c9750 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5949953c8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x52a1c8e88 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x56bbab320 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x571bdd478 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59b9432e0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x553279fa0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a17d8f90 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59e6951f8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58f1a73d0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x545d15750 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x522546790 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x558d77408 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x524d39490 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x537903c50 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5556813b0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x55db0ca50 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x525557328 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x56ecf9798 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a2ed9970 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x551f4a308 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58276fa80 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a7832ed0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58f5e1958 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x54e558238 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x567545c40 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x58fdbea18 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a08b77c0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x51f059000 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a78f3550 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a1a52778 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x526ad1de0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x529b9f6a8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a3fbb908 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59240d840 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a3127350 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x582c94c60 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x59e441d48 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a8d84eb8 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x5a07c6750 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x553a859a0 (24 bytes) : ??
io.netty.channel.DefaultChannelHandlerContext$2@0x541a592d8 (24 bytes) : ??

is it normal? @trustin @gresrun @normanmaurer

@normanmaurer
Copy link
Member

@normanmaurer normanmaurer commented Jan 10, 2013

@shijinkui it is normal.. the DefaultChannelHandlerContext can only GC'ed once the Channel is GC'ed..

@shijinkui
Copy link
Author

@shijinkui shijinkui commented Jan 11, 2013

i think one request gen 10x4 DefaultChannelHandlerContext, it's not normal.

this is the echo example, two request from deffirent jvm instance.

EchoServer started, no request.

QQ20130111-15

a client request arrive.
QQ20130111-16

the second request arrive.
QQ20130111-17

the DefaultChannelHandlerContext increase: requst * 4 * 10.
when the keepalive request is more, the DefaultChannelHandlerContext is too large, old Gen cannot be gc.

@shijinkui
Copy link
Author

@shijinkui shijinkui commented Jan 11, 2013

master @normanmaurer @trustin , i test echoclient:

use 1 thread to process event, see the below screenshot


b.group(new NioEventLoopGroup(1))
             .channel(NioSocketChannel.class)
             .option(ChannelOption.TCP_NODELAY, true)
             .remoteAddress(new InetSocketAddress(host, port))
             .handler(new ChannelInitializer<SocketChannel>() {
                 @Override
                 public void initChannel(SocketChannel ch) throws Exception {
                     ch.pipeline().addLast(
                             new LoggingHandler(LogLevel.INFO),
                             new EchoClientHandler(firstMessageSize));
                 }
             });

picture1:
QQ20130111-18

picture2:
QQ20130111-19

protected MultithreadEventExecutorGroup(int nThreads, ThreadFactory threadFactory, Object... args)
line 70, really only one NioEventLoop is created, but the number of DEFAULT_POOL_SIZE DefaultChannelHandlerContext where is created.

children = new SingleThreadEventExecutor[nThreads];
        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
                children[i] = newChild(threadFactory, scheduler, args);
                success = true;
            } catch (Exception e) {
                throw new EventLoopException("failed to create a child event loop", e);
            } finally {
                if (!success) {
                    for (int j = 0; j < i; j ++) {
                        children[j].shutdown();
                    }
                }
            }
        }

ps: my english is pool :(

Thanks

@trustin
Copy link
Member

@trustin trustin commented Jan 14, 2013

They are anonymous Runnables which are used to trigger an event. According to your heap dump, a ChannelHandlerContext takes 1152 bytes. Let's assume we have 10 handlers in a pipeline and a server handles 1,000,000 connections, then the memory consumption will be: 1,000,000 * 10 * 1152 = whooping 10.7 GiB.

I agree we need to reduce memory consumption.

@trustin trustin reopened this Jan 14, 2013
@ghost ghost assigned trustin Jan 14, 2013
@normanmaurer
Copy link
Member

@normanmaurer normanmaurer commented Jan 14, 2013

@trustin where do you see it is 1152 bytes ?

@shijinkui
Copy link
Author

@shijinkui shijinkui commented Jan 15, 2013

@normanmaurer @trustin : i have a question, the ChannelHandlerContext how be created. i trace the code, always can't find out where create ten ChannelHandlerContext fixedly.

  1. DefaultChannelPipeline.addLast is first to be executed.
  2. SingleThreadEventExecutor.execute is second to be executed, to add task.

is NioEventLoop listening the network port, add task event to the queue? and then create the ChannelHandlerContext?
where is the number 10 is setted?

i read netty code several days, always can't be clear to the above question. if u have time, please tell me.
Thanks.
💯

QQ20130115-2

QQ20130115-1

@trustin
Copy link
Member

@trustin trustin commented Jan 15, 2013

@normanmaurer 512 + 64 * 10?

@trustin
Copy link
Member

@trustin trustin commented Jan 15, 2013

@shijinkui, they are anonymous Runnnable instances in DefaultChannelHandlerContext. I'll create them lazily

@normanmaurer
Copy link
Member

@normanmaurer normanmaurer commented Jan 15, 2013

@trustin lol I did not see the 64 .. Do you think it will really help to create them lazily as once they are created they will take the same amount of space again. We could maybe just create a new on demand everytime. This will put more pressure on the GC but only for stuff that is "NOT" run in the EventLoop Thread. WDYT ?

@trustin
Copy link
Member

@trustin trustin commented Jan 15, 2013

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.

@normanmaurer
Copy link
Member

@normanmaurer normanmaurer commented Jan 15, 2013

Yeah… But I think we can still cache them if we detect one is needed.

Am 15.01.2013 um 08:16 schrieb Trustin Lee notifications@github.com:

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.


Reply to this email directly or view it on GitHub.

@normanmaurer
Copy link
Member

@normanmaurer normanmaurer commented Jan 15, 2013

Working on it....

Am 15.01.2013 um 08:16 schrieb Trustin Lee notifications@github.com:

@normanmaurer Actually, we should create them only when a handler has to trigger an event to the handler whose executor is different. Otherwise we can just invoke the next handler directly. DefaultChannelHandlerContext currently always uses runnable for both cases.


Reply to this email directly or view it on GitHub.

@ghost ghost assigned normanmaurer Jan 15, 2013
normanmaurer pushed a commit that referenced this issue Jan 15, 2013
…it Runnables for Tasks if really neccessary
trustin added a commit that referenced this issue Jan 15, 2013
This pull request cleans up our pipeline implementation by moving most
inter-context traversal code to DefaultChannelHandlerContext.
Previously, outbound traversal was done in DefaultChannelPipeline while
inbound traversal was done in DefaultChannelHandlerContext.

Also, to address the memory inefficiency issue raised in #920, all
runnables are lazily instantiated.
@ghost ghost assigned trustin Jan 15, 2013
@trustin
Copy link
Member

@trustin trustin commented Jan 15, 2013

506474f should fix the problem.

@trustin trustin closed this Jan 15, 2013
@trustin
Copy link
Member

@trustin trustin commented Jan 15, 2013

@shijinkui, could you let us know if my change helps your problem?

@shijinkui
Copy link
Author

@shijinkui shijinkui commented Jan 16, 2013

@trustin @normanmaurer , i works normal, good job 💯 . instance num is 10~11. let me run in product several days, observe the changes.

@shijinkui
Copy link
Author

@shijinkui shijinkui commented Jan 21, 2013

Class Instance Count Total Size
class io.netty.channel.DefaultChannelHandlerContext 2953703 640953551
class io.netty.channel.DefaultChannelHandlerContext$14 584856 9357696
class io.netty.channel.DefaultChannelHandlerContext$9 113279 1812464
class io.netty.channel.DefaultChannelHandlerContext$7 113279 1812464
class io.netty.channel.DefaultChannelHandlerContext$25 113279 906232
class io.netty.channel.DefaultChannelHandlerContext$10 0 0
class io.netty.channel.DefaultChannelHandlerContext$15 0 0
@ghost
Copy link

@ghost ghost commented Jan 21, 2013

@shijinkui what to you use to measure / benchmark this?

normanmaurer pushed a commit that referenced this issue Jan 30, 2013
This will safe as an example 2gb mem when have 10 DefaultHandlerContext instances per connection and the connection count is 1000000.
Also kind of related to [#920]
@qiaodaimadelaowang
Copy link
Contributor

@qiaodaimadelaowang qiaodaimadelaowang commented Nov 4, 2016

@carrot-garden I think it is VisualVM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.