Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

ConnectDisconnect test failed due to a leak on w3wp.exe #1801

Closed
jcondex opened this Issue · 4 comments

3 participants

@jcondex

There is a leak on the server side of the ConnectDisconnect (WebSockets) stress test that made the test fail after it used 3+ GB of memory.

This failure happened using the Stop variation only (neither Abort or Disconnect)

I have copied the dump file here:
\\indigofs\CommonShare\jconde\SignalR\Bugs\1801

0:080> !dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
<CUT>
000007fea71acae0   120276     17319744 Microsoft.Owin.Host.SystemWeb.OwinCallContext
00000037823092e0   129839     18552462      Free
000007fea71ae638   120263     21166288 Microsoft.AspNet.SignalR.Hubs.HubDispatcher
000007fea6881330   240527     23095248 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Object, mscorlib]][]
000007fea6556008   985478     23651472 System.Object
000007fea67bb788   375048     27003456 System.Runtime.Remoting.Messaging.LogicalCallContext
000007fea67ba900   375933     27067176 System.Threading.ExecutionContext
000007fea67578c0   726482     29026572 System.Int32[]
000007fea66c4e38   366149     29291920 System.Collections.Hashtable
000007fea72189f0   380384     30430720 System.Threading.Tasks.Task`1[[System.Object, mscorlib]]
000007fea6c109e8   479742     30703488 System.Collections.Generic.HashSet`1[[System.String, mscorlib]]
000007fea683ea50   966487     30927584 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
000007fea728a060   488797     31283008 System.Func`1[[System.Threading.Tasks.Task, mscorlib]]
000007fea6eca6e0   360361     33614600 System.Collections.Generic.HashSet`1+Slot[[System.String, mscorlib]][]
000007fea76fdba8   119822     34508736 Microsoft.AspNet.SignalR.Transports.WebSocketTransport
000007fea6fdc0b8   120276     34639488 System.Web.Hosting.IIS7WorkerRequest
000007fea71ad580   120276     36563904 Microsoft.Owin.Host.SystemWeb.CallEnvironment.AspNetDictionary
000007fea6fd5220   120276     40412736 System.Web.HttpResponse
000007fea6b37b18   120276     46185984 System.Web.HttpRequest
000007fea677fa18  1214908     48596320 System.Collections.ArrayList
000007fea6a8e698   120276     50997024 System.Web.HttpContext
000007fea65f2a18   723862     57908960 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]]
000007fea65abec8   260472     58532972 System.Char[]
000007fea6758ab0   366592     70303632 System.Collections.Hashtable+bucket[]
000007fea655b9b0  2826559    188726416 System.Object[]
000007fea65a46c8  4768320    697466582 System.String
0:080> !heapstat
Heap             Gen0         Gen1         Gen2          LOH
Heap0        12851240       816568    327888808       597240
Heap1        10556784       636840    333492048           24
Heap2        13051320       580992    265988320      2293824
Heap3        13736592       583072    298487832           24
Heap4        12831984       778856    297564856            0
Heap5        16142224       831272    304137496            0
Heap6        12746424       915712    309014056            0
Heap7        15519016       609504    299245160            0
Total        50195936      2617472   1225857008      2891112

Free space:                                                 Percentage
Heap0          192384           24      2266328          600SOH:  0% LOH:  0%
Heap1           91248           24      2275776           24SOH:  0% LOH:100%
Heap2          490480           24      1704968           56SOH:  0% LOH:  0%
Heap3           44712           24      1928216           24SOH:  0% LOH:100%
Heap4          653416           24      1885744            0SOH:  0% LOH:-2147483648%
Heap5          414464           24      2022024            0SOH:  0% LOH:-2147483648%
Heap6          389424           24      2058992            0SOH:  0% LOH:-2147483648%
Heap7          391248           24      2001160            0SOH:  0% LOH:-2147483648%
Total          818824           96      8175288          704
0:080> !address -summary


Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    290      7fb`a0e13000 (   7.983 Tb)           99.79%
<unknown>                              1269        4`50f8d000 (  17.265 Gb)  98.74%    0.21%
Image                                   936        0`07393000 ( 115.574 Mb)   0.65%    0.00%
Heap                                     74        0`044a2000 (  68.633 Mb)   0.38%    0.00%
Stack                                   244        0`027c0000 (  39.750 Mb)   0.22%    0.00%
Other                                     7        0`001b8000 (   1.719 Mb)   0.01%    0.00%
TEB                                      81        0`000a2000 ( 648.000 kb)   0.00%    0.00%
PEB                                       1        0`00001000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            1091        4`52bf7000 (  17.293 Gb)  98.89%    0.21%
MEM_IMAGE                              1488        0`0b904000 ( 185.016 Mb)   1.03%    0.00%
MEM_MAPPED                               31        0`00cd7000 (  12.840 Mb)   0.07%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                292      7fb`a0e1e000 (   7.983 Tb)           99.79%
MEM_RESERVE                             723        3`a15eb000 (  14.521 Gb)  83.04%    0.18%
MEM_COMMIT                             1887        0`bdbe7000 (   2.965 Gb)  16.95%    0.04%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          787        0`b1371000 (   2.769 Gb)  15.84%    0.03%
PAGE_READONLY                           746        0`0922e000 ( 146.180 Mb)   0.82%    0.00%
PAGE_EXECUTE_READ                       118        0`02d06000 (  45.023 Mb)   0.25%    0.00%
PAGE_EXECUTE_READWRITE                  100        0`00738000 (   7.219 Mb)   0.04%    0.00%
PAGE_READWRITE|PAGE_GUARD                81        0`00177000 (   1.465 Mb)   0.01%    0.00%
PAGE_WRITECOPY                           51        0`0008e000 ( 568.000 kb)   0.00%    0.00%
PAGE_EXECUTE                              2        0`00003000 (  12.000 kb)   0.00%    0.00%
PAGE_NOACCESS                             2        0`00002000 (   8.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                     3b`d9250000      7bc`26534000 (   7.735 Tb)
<unknown>                                38`94a62000        0`6e1ae000 (   1.720 Gb)
Image                                   7ff`0ba8b000        0`00a6a000 (  10.414 Mb)
Heap                                     3b`cc524000        0`00961000 (   9.379 Mb)
Stack                                    37`80b30000        0`0007b000 ( 492.000 kb)
Other                                    37`807b0000        0`00181000 (   1.504 Mb)
TEB                                     7f7`ff784000        0`00002000 (   8.000 kb)
PEB                                     7f7`ff9ff000        0`00001000 (   4.000 kb)
@davidfowl
Owner

@jcondex So this only happens with websockets?

@davidfowl davidfowl was assigned
@halter73 halter73 was assigned
@jcondex

This happens using ServerSentEvents too

@jcondex jcondex closed this
@jcondex jcondex reopened this
@jcondex

The offending variation is SendingToHub() :

        [StressVariation]
        public void SendingToHub()
        {
            HubConnection connection = Helpers.CreateHubConnection(Properties.HubUrl);

            IHubProxy hubProxy = connection.CreateHubProxy(HubName);

            IClientTransport transport = Helpers.GetTransport();

            Helpers.Retry(
                15,
                TimeSpan.FromSeconds(3),
                () =>
                {
                    connection.Start(transport).Wait();
                },
                transport.Name);

            Log.Verbose("SendingToHub [{0}] - ConnectionToken: {1}", transport.Name, connection.ConnectionToken);

            Payload payload = Payload.Random();

            hubProxy.Invoke("RecivePayload", payload);

            string op = Helpers.AbortStopOrDisconnect(connection, AbortStopDisconnect);
        }

Hub:

        public void ReceivePayload(Payload payload)
        {
            Assert.IsFalse(payload.IsNull);
        }

Payload is a byte[] of size random.Next(128) + 1

@halter73 halter73 referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@davidfowl
Owner

@jcondex can you verify? It's been fixed.

@jcondex jcondex was assigned
@jcondex jcondex closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.