Memory leak in SignalR 1.1 on .NET 4.0 on self host #2413

Closed
bborad1 opened this Issue Aug 9, 2013 · 59 comments

Comments

Projects
None yet
3 participants
@bborad1

bborad1 commented Aug 9, 2013

Hi

I am running SignalR 1.1.0.0 and it looks like it has some kind of memory leak issue as memory keep growing but cannot figure out where. I have copy and past all code in a test application just to be sure that it not something else in my other project. You can download the complete project from below link.

https://www.hightail.com/download/bWJyTGs3TERtMEw1SE1UQw

Any help would be appreciated.

Thanks

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

Try updating to 1.1.3

Member

davidfowl commented Aug 9, 2013

Try updating to 1.1.3

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

Will it work with .Net 4

bborad1 commented Aug 9, 2013

Will it work with .Net 4

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

Just try to update but came up with error "Could not install Microsoft.Owin.Security 1.1.0-beta2". I think its because I am using .Net 4.

Any suggestion? Please.

bborad1 commented Aug 9, 2013

Just try to update but came up with error "Could not install Microsoft.Owin.Security 1.1.0-beta2". I think its because I am using .Net 4.

Any suggestion? Please.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

1.1.3 not the prerelease

Member

davidfowl commented Aug 9, 2013

1.1.3 not the prerelease

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

how do I do that?

bborad1 commented Aug 9, 2013

how do I do that?

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

I'm trying to understand this sample. What is it trying to do? Does memory go up forever? or does it stop growing at some point?

Member

davidfowl commented Aug 9, 2013

I'm trying to understand this sample. What is it trying to do? Does memory go up forever? or does it stop growing at some point?

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

I think it is 1.1.3. When I search for SignalR 1.1.3, NuGet shows that as already installed and before you send me the link I tried to install it from NuGet console and it came with same message application already has reference to Microsoft.AspNet.SignalR 1.1.3.

I checked Version property of the references and it shows Version 1.1.0.0


In the sample application I am sending messages (data ) to all client every 1/0.5 second. In real scenario it will be only sending data every 2 or 10 minutes but just to speed up the issue I am sending it ever 1 second.

Memory does go up forever until I run out of memory and pc starts stop responding.

bborad1 commented Aug 9, 2013

I think it is 1.1.3. When I search for SignalR 1.1.3, NuGet shows that as already installed and before you send me the link I tried to install it from NuGet console and it came with same message application already has reference to Microsoft.AspNet.SignalR 1.1.3.

I checked Version property of the references and it shows Version 1.1.0.0


In the sample application I am sending messages (data ) to all client every 1/0.5 second. In real scenario it will be only sending data every 2 or 10 minutes but just to speed up the issue I am sending it ever 1 second.

Memory does go up forever until I run out of memory and pc starts stop responding.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

Ok I'll take a look.

Member

davidfowl commented Aug 9, 2013

Ok I'll take a look.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

Your sample doesn't compile/

Member

davidfowl commented Aug 9, 2013

Your sample doesn't compile/

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

Some more detail about the application:

There are 5 type of Hub that client can connect to. They all send different messages, in test app I am just send dummy string.

don't worry about the Timer and RestartSignalR method. I was just going to test if I can restart the SignalR so that it can be GC and this might fix the problem for sort time.

bborad1 commented Aug 9, 2013

Some more detail about the application:

There are 5 type of Hub that client can connect to. They all send different messages, in test app I am just send dummy string.

don't worry about the Timer and RestartSignalR method. I was just going to test if I can restart the SignalR so that it can be GC and this might fix the problem for sort time.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

Sorry, Just remove IDisposable from HubConnector

bborad1 commented Aug 9, 2013

Sorry, Just remove IDisposable from HubConnector

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

You also are missing the client. Can you provide the full application along with the steps you used to see the memory leak?

Member

davidfowl commented Aug 9, 2013

You also are missing the client. Can you provide the full application along with the steps you used to see the memory leak?

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

Send an updated sample with specific instructions on what you're observing and what you expect along with a fully runnable sample.

Member

davidfowl commented Aug 9, 2013

Send an updated sample with specific instructions on what you're observing and what you expect along with a fully runnable sample.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

Here's what I am doing

  1. Compile test app
  2. Run .exe from bin
  3. Connect few client
  4. Watch the memory growing in Task Manager and in ANTS Memory Profiler (from Redgate )

I have updated both project and you can download it from below links

SignalR SelfHost Test App
https://www.hightail.com/download/bWJyTGt5SWV6NE85TE1UQw

SignalR Client App
https://www.hightail.com/download/bWJyTGt5SWV3TGdQWWNUQw

I just noticed that my client in my real project has reference to signalR-1.0.1.js. I have updated in above test app to 1.1.3

bborad1 commented Aug 9, 2013

Here's what I am doing

  1. Compile test app
  2. Run .exe from bin
  3. Connect few client
  4. Watch the memory growing in Task Manager and in ANTS Memory Profiler (from Redgate )

I have updated both project and you can download it from below links

SignalR SelfHost Test App
https://www.hightail.com/download/bWJyTGt5SWV6NE85TE1UQw

SignalR Client App
https://www.hightail.com/download/bWJyTGt5SWV3TGdQWWNUQw

I just noticed that my client in my real project has reference to signalR-1.0.1.js. I have updated in above test app to 1.1.3

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

In step 3? how many clients is a few clients? It should show up with 1 client right? Also, can't you put the client and server in a single download in a single solution?

Member

davidfowl commented Aug 9, 2013

In step 3? how many clients is a few clients? It should show up with 1 client right? Also, can't you put the client and server in a single download in a single solution?

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

2-3 clients.
Yes, should show up with 1 client as well.
I had it as separate projects but yes, could have added in one download.

bborad1 commented Aug 9, 2013

2-3 clients.
Yes, should show up with 1 client as well.
I had it as separate projects but yes, could have added in one download.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

I can send you the profiler result if you can open it. its in .amp7results format (ANTS Memory Profiler 7). Please let me know if this can help you.

bborad1 commented Aug 9, 2013

I can send you the profiler result if you can open it. its in .amp7results format (ANTS Memory Profiler 7). Please let me know if this can help you.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

I'd rather run the sample myself and profile it.

Member

davidfowl commented Aug 9, 2013

I'd rather run the sample myself and profile it.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 9, 2013

this image might help you finding problem. Shows the live instances of CancellationTokenSource class in memory.
signalrmemoryleak

bborad1 commented Aug 9, 2013

this image might help you finding problem. Shows the live instances of CancellationTokenSource class in memory.
signalrmemoryleak

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 9, 2013

Member

Thanks. I still need to run the sample.

Member

davidfowl commented Aug 9, 2013

Thanks. I still need to run the sample.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 11, 2013

Member

So how long should I wait for this thing to run out of memory?

Member

davidfowl commented Aug 11, 2013

So how long should I wait for this thing to run out of memory?

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 11, 2013

Member

I think you're missing some steps. I just don't see any memory issues.

Member

davidfowl commented Aug 11, 2013

I think you're missing some steps. I just don't see any memory issues.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 11, 2013

Normally it should show up in about 2-3 hour. Do you see any issue in way I am broadcasting messages from the server? Any coding issue that could be causing problem over time.

bborad1 commented Aug 11, 2013

Normally it should show up in about 2-3 hour. Do you see any issue in way I am broadcasting messages from the server? Any coding issue that could be causing problem over time.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 12, 2013

Any luck?

bborad1 commented Aug 12, 2013

Any luck?

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 12, 2013

Member

Nope. Memory look stable to me. I see the date updating every second and the memory not growing infinitely. Not sure what I'm doing differently to you.

Member

davidfowl commented Aug 12, 2013

Nope. Memory look stable to me. I see the date updating every second and the memory not growing infinitely. Not sure what I'm doing differently to you.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 12, 2013

Huuu, strange
I am gone leave it running on different PC overnight and will see tomorrow where it is.

bborad1 commented Aug 12, 2013

Huuu, strange
I am gone leave it running on different PC overnight and will see tomorrow where it is.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 12, 2013

it's still grows forever. here's screenshot of when application started
image

here's screenshot after running for a night
signalr_runningon163_nextday

bborad1 commented Aug 12, 2013

it's still grows forever. here's screenshot of when application started
image

here's screenshot after running for a night
signalr_runningon163_nextday

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 12, 2013

can't see option to create dump on XP.

bborad1 commented Aug 12, 2013

can't see option to create dump on XP.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 13, 2013

Member

So running overnight you only used up 144 megs? How is that a memory leak? How many hours and how many connections is that? Does it grow until you get an out of memory exception? Do I need to run my program overnight to see the OOM exception?

I'm running on .NET 4.5 so I'm not sure if that's the difference.

Member

davidfowl commented Aug 13, 2013

So running overnight you only used up 144 megs? How is that a memory leak? How many hours and how many connections is that? Does it grow until you get an out of memory exception? Do I need to run my program overnight to see the OOM exception?

I'm running on .NET 4.5 so I'm not sure if that's the difference.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 13, 2013

About 16 hours with 3 clients connected. As it keep growing, I assume it will eventually run out of memory.
It should show grown in about 4-5 hour might not be huge but should be able see the difference.
I am running .Net 4, could this be the problem?

Notice another thing, if I don't have any clients connected memory stays stable, doesn't decrease but stays where it is.

I have another instance running on Win7 with 1 client connected since this morning and memory has gone from ~11MB to ~33MB (in about 5 hour).

I will also run another instance on different VM with .Net 4.5 installed.

bborad1 commented Aug 13, 2013

About 16 hours with 3 clients connected. As it keep growing, I assume it will eventually run out of memory.
It should show grown in about 4-5 hour might not be huge but should be able see the difference.
I am running .Net 4, could this be the problem?

Notice another thing, if I don't have any clients connected memory stays stable, doesn't decrease but stays where it is.

I have another instance running on Win7 with 1 client connected since this morning and memory has gone from ~11MB to ~33MB (in about 5 hour).

I will also run another instance on different VM with .Net 4.5 installed.

@ghost ghost assigned jcondex Aug 13, 2013

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 13, 2013

Member

Ok @jcondex see if you can get a repro? It looks like we might have a slow leak with cancellation tokens in some scenarios. I didn't see memory growing when I ran the example but let me know what you find.

Member

davidfowl commented Aug 13, 2013

Ok @jcondex see if you can get a repro? It looks like we might have a slow leak with cancellation tokens in some scenarios. I didn't see memory growing when I ran the example but let me know what you find.

@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 13, 2013

Contributor

I think I also did something wrong because this did not reproduce for me.

I let the repro running overnight and the Windows Task Manager constantly reported a memory usage of ~16MB

The way I started the repro was by running SignalRSelfHost_Console.exe on an administrative console window and I then opened 4 tabs in Internet Explorer hitting the endpoint. I verified that the repro was working by seeing multiple alert boxes in IE and then a perpetually updating legend similar to: "1111111111111111 -33 1 13/08/2013 07:48:55 A1 ".

I left the repro running for ~9 hours:

0:025> .time
Debug session time: Tue Aug 13 08:00:41.145 2013 (UTC - 7:00)
System Uptime: 6 days 17:40:25.006
Process Uptime: 0 days 8:43:11.782
  Kernel time: 0 days 0:01:07.953
  User time: 0 days 0:08:09.921

Here is the detailed memory usage:

0:025> !address -summary

Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    100          74279000 (   1.815 Gb)           90.75%
Image                                   535           5418000 (  84.094 Mb)  44.38%    4.11%
<unknown>                               225           2b33000 (  43.199 Mb)  22.80%    2.11%
MappedFile                               21           1fca000 (  31.789 Mb)  16.78%    1.55%
Stack                                    78           1880000 (  24.500 Mb)  12.93%    1.20%
Heap                                     20            58f000 (   5.559 Mb)   2.93%    0.27%
Other                                     7             38000 ( 224.000 kb)   0.12%    0.01%
TEB                                      26             1a000 ( 104.000 kb)   0.05%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_IMAGE                               557           54c9000 (  84.785 Mb)  44.75%    4.14%
MEM_PRIVATE                             330           48ae000 (  72.680 Mb)  38.36%    3.55%
MEM_MAPPED                               26           2000000 (  32.000 Mb)  16.89%    1.56%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                100          74279000 (   1.815 Gb)           90.75%
MEM_COMMIT                              768           6e0d000 ( 110.051 Mb)  58.09%    5.37%
MEM_RESERVE                             145           4f6a000 (  79.414 Mb)  41.91%    3.88%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ                        56           43a0000 (  67.625 Mb)  35.69%    3.30%
PAGE_READONLY                           168           121c000 (  18.109 Mb)   9.56%    0.88%
PAGE_READWRITE                          294            f52000 (  15.320 Mb)   8.09%    0.75%
PAGE_WRITECOPY                          110            6e8000 (   6.906 Mb)   3.65%    0.34%
PAGE_EXECUTE_WRITECOPY                   31             e6000 ( 920.000 kb)   0.47%    0.04%
PAGE_EXECUTE_READWRITE                   57             af000 ( 700.000 kb)   0.36%    0.03%
PAGE_READWRITE|PAGE_GUARD                52             82000 ( 520.000 kb)   0.27%    0.02%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         6da0000          4f840000 (   1.242 Gb)
Image                                       73b64000            c8a000 (  12.539 Mb)
<unknown>                                    30e2000            fde000 (  15.867 Mb)
MappedFile                                    c51000           129f000 (  18.621 Mb)
Stack                                        1fc0000             fd000 (1012.000 kb)
Heap                                         69b9000            1e6000 (   1.898 Mb)
Other                                       7f210000             23000 ( 140.000 kb)
TEB                                         7f066000              1000 (   4.000 kb)
PEB                                         7f23b000              1000 (   4.000 kb)

So far, everything looks good and no leaks present. Let's take this a step further and look at the CancellationTokens:

0:025> !dumpheap -type CancellationToken -stat -live
Statistics:
      MT    Count    TotalSize Class Name
003b7c68        1           20 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.UInt64, mscorlib],[System.Lazy`1[[System.Threading.CancellationToken, mscorlib]], mscorlib]]
04c42c64        1           32 System.Func`1[[System.Threading.CancellationToken, mscorlib]]
003b76dc        1           36 System.Collections.Concurrent.ConcurrentDictionary`2[[System.UInt64, mscorlib],[System.Lazy`1[[System.Threading.CancellationToken, mscorlib]], mscorlib]]
04c42ab4       13          156 System.Lazy`1+Boxed[[System.Threading.CancellationToken, mscorlib]]
73ca34d4       21          252 System.Threading.CancellationToken
003b7048       13          260 System.Lazy`1[[System.Threading.CancellationToken, mscorlib]]
73c5ab98        1          284 System.Object[]
003b7c0c       13          364 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.UInt64, mscorlib],[System.Lazy`1[[System.Threading.CancellationToken, mscorlib]], mscorlib]]
01f7dd34       20          400 Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource
01f7dc60       60         1200 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper
01f7dbb8       60         1440 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+DiposeCancellationState
73ca20ec       59         2360 System.Threading.CancellationTokenSource
Total 263 objects

Here things look quite normal too: When I kicked off the repro ~9 hours ago, the CancellationTokenSource cout was exactly at 60. Many hours later, we're still happy at 59.

Contributor

jcondex commented Aug 13, 2013

I think I also did something wrong because this did not reproduce for me.

I let the repro running overnight and the Windows Task Manager constantly reported a memory usage of ~16MB

The way I started the repro was by running SignalRSelfHost_Console.exe on an administrative console window and I then opened 4 tabs in Internet Explorer hitting the endpoint. I verified that the repro was working by seeing multiple alert boxes in IE and then a perpetually updating legend similar to: "1111111111111111 -33 1 13/08/2013 07:48:55 A1 ".

I left the repro running for ~9 hours:

0:025> .time
Debug session time: Tue Aug 13 08:00:41.145 2013 (UTC - 7:00)
System Uptime: 6 days 17:40:25.006
Process Uptime: 0 days 8:43:11.782
  Kernel time: 0 days 0:01:07.953
  User time: 0 days 0:08:09.921

Here is the detailed memory usage:

0:025> !address -summary

Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    100          74279000 (   1.815 Gb)           90.75%
Image                                   535           5418000 (  84.094 Mb)  44.38%    4.11%
<unknown>                               225           2b33000 (  43.199 Mb)  22.80%    2.11%
MappedFile                               21           1fca000 (  31.789 Mb)  16.78%    1.55%
Stack                                    78           1880000 (  24.500 Mb)  12.93%    1.20%
Heap                                     20            58f000 (   5.559 Mb)   2.93%    0.27%
Other                                     7             38000 ( 224.000 kb)   0.12%    0.01%
TEB                                      26             1a000 ( 104.000 kb)   0.05%    0.00%
PEB                                       1              1000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_IMAGE                               557           54c9000 (  84.785 Mb)  44.75%    4.14%
MEM_PRIVATE                             330           48ae000 (  72.680 Mb)  38.36%    3.55%
MEM_MAPPED                               26           2000000 (  32.000 Mb)  16.89%    1.56%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                100          74279000 (   1.815 Gb)           90.75%
MEM_COMMIT                              768           6e0d000 ( 110.051 Mb)  58.09%    5.37%
MEM_RESERVE                             145           4f6a000 (  79.414 Mb)  41.91%    3.88%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ                        56           43a0000 (  67.625 Mb)  35.69%    3.30%
PAGE_READONLY                           168           121c000 (  18.109 Mb)   9.56%    0.88%
PAGE_READWRITE                          294            f52000 (  15.320 Mb)   8.09%    0.75%
PAGE_WRITECOPY                          110            6e8000 (   6.906 Mb)   3.65%    0.34%
PAGE_EXECUTE_WRITECOPY                   31             e6000 ( 920.000 kb)   0.47%    0.04%
PAGE_EXECUTE_READWRITE                   57             af000 ( 700.000 kb)   0.36%    0.03%
PAGE_READWRITE|PAGE_GUARD                52             82000 ( 520.000 kb)   0.27%    0.02%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                         6da0000          4f840000 (   1.242 Gb)
Image                                       73b64000            c8a000 (  12.539 Mb)
<unknown>                                    30e2000            fde000 (  15.867 Mb)
MappedFile                                    c51000           129f000 (  18.621 Mb)
Stack                                        1fc0000             fd000 (1012.000 kb)
Heap                                         69b9000            1e6000 (   1.898 Mb)
Other                                       7f210000             23000 ( 140.000 kb)
TEB                                         7f066000              1000 (   4.000 kb)
PEB                                         7f23b000              1000 (   4.000 kb)

So far, everything looks good and no leaks present. Let's take this a step further and look at the CancellationTokens:

0:025> !dumpheap -type CancellationToken -stat -live
Statistics:
      MT    Count    TotalSize Class Name
003b7c68        1           20 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.UInt64, mscorlib],[System.Lazy`1[[System.Threading.CancellationToken, mscorlib]], mscorlib]]
04c42c64        1           32 System.Func`1[[System.Threading.CancellationToken, mscorlib]]
003b76dc        1           36 System.Collections.Concurrent.ConcurrentDictionary`2[[System.UInt64, mscorlib],[System.Lazy`1[[System.Threading.CancellationToken, mscorlib]], mscorlib]]
04c42ab4       13          156 System.Lazy`1+Boxed[[System.Threading.CancellationToken, mscorlib]]
73ca34d4       21          252 System.Threading.CancellationToken
003b7048       13          260 System.Lazy`1[[System.Threading.CancellationToken, mscorlib]]
73c5ab98        1          284 System.Object[]
003b7c0c       13          364 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.UInt64, mscorlib],[System.Lazy`1[[System.Threading.CancellationToken, mscorlib]], mscorlib]]
01f7dd34       20          400 Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource
01f7dc60       60         1200 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper
01f7dbb8       60         1440 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+DiposeCancellationState
73ca20ec       59         2360 System.Threading.CancellationTokenSource
Total 263 objects

Here things look quite normal too: When I kicked off the repro ~9 hours ago, the CancellationTokenSource cout was exactly at 60. Many hours later, we're still happy at 59.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 13, 2013

Member

@bborad1 What transport are you using? Did you get a chance to try it on .NET 4.5?

Member

davidfowl commented Aug 13, 2013

@bborad1 What transport are you using? Did you get a chance to try it on .NET 4.5?

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 15, 2013

I am not configuring any particular transport. How do I check what transport it's using?

I have created a new VM with .Net 4.5 and have started the application, will update the status by end of the day.

Below is the link to dump file from .Net 4 (on win 7). When started memory was at ~30MB and this morning it was at ~ 300MB.

https://www.hightail.com/download/bWJybUpSSU9tMEs5TE5Vag

Also found another interesting thing, I had the same application running on a XP with .Net 4 for the same time period and when I checked memory on that, it was same as what it was when I started it. Is it OS that's causing this?

bborad1 commented Aug 15, 2013

I am not configuring any particular transport. How do I check what transport it's using?

I have created a new VM with .Net 4.5 and have started the application, will update the status by end of the day.

Below is the link to dump file from .Net 4 (on win 7). When started memory was at ~30MB and this morning it was at ~ 300MB.

https://www.hightail.com/download/bWJybUpSSU9tMEs5TE5Vag

Also found another interesting thing, I had the same application running on a XP with .Net 4 for the same time period and when I checked memory on that, it was same as what it was when I started it. Is it OS that's causing this?

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 15, 2013

On side note, noticed another thing while updating existing project to SignalR 1.1.3

With previous version I just referenced signalr project into my main project and it was working but after updating to 1.1.3 it stopped working. It was throwing "server factory could not be located for the given input" error. On search I found that issue was that it wasn't copying "Microsoft.Owin.Host.HttpListener.dll" in my main projects output directory. To fix the issue I change the output directory of my signalr project to output directory of my main project.

Is this because of something changed in 1.1.3 or could be something else in my project/solution?

If it is because something is changed in 1.1.3 then should add this as new post as this might be helpful for others. Solution was found from Katana project from CodePlex.

bborad1 commented Aug 15, 2013

On side note, noticed another thing while updating existing project to SignalR 1.1.3

With previous version I just referenced signalr project into my main project and it was working but after updating to 1.1.3 it stopped working. It was throwing "server factory could not be located for the given input" error. On search I found that issue was that it wasn't copying "Microsoft.Owin.Host.HttpListener.dll" in my main projects output directory. To fix the issue I change the output directory of my signalr project to output directory of my main project.

Is this because of something changed in 1.1.3 or could be something else in my project/solution?

If it is because something is changed in 1.1.3 then should add this as new post as this might be helpful for others. Solution was found from Katana project from CodePlex.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 15, 2013

Member

Nope nothing changed in 1.1.3, we only made 2 client fixes. Katana is a separate component.

Member

davidfowl commented Aug 15, 2013

Nope nothing changed in 1.1.3, we only made 2 client fixes. Katana is a separate component.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 15, 2013

Ok, don't what happened then but in previous version I didn't need to do this (copying Host.HttpListener.dll). In regards to Katana project , I meant solution was found from their forum, couldn't find anything in SignalR forum/discussion (at least could get anything in google search list).

BTW, it's still running fine (with 1 client) on .Net 4.5 (Win 7). Have just started 2 more clients.

bborad1 commented Aug 15, 2013

Ok, don't what happened then but in previous version I didn't need to do this (copying Host.HttpListener.dll). In regards to Katana project , I meant solution was found from their forum, couldn't find anything in SignalR forum/discussion (at least could get anything in google search list).

BTW, it's still running fine (with 1 client) on .Net 4.5 (Win 7). Have just started 2 more clients.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 15, 2013

Sorry, might not, has ~5MB increase (with 1 client). Will see more how it goes overnight with 3 clients.

On Win 7 with .Net 4.5

bborad1 commented Aug 15, 2013

Sorry, might not, has ~5MB increase (with 1 client). Will see more how it goes overnight with 3 clients.

On Win 7 with .Net 4.5

@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 15, 2013

Contributor

Thanks a lot for the dump file. Now I need you to do me another favor: Can you please zip and upload the following files:

clr.dll
sos.dll
mscordacwks.dll
mscorwks.dll

You can find all these in: C:\Windows\Microsoft.NET\Framework\v4.0.XXX

(don't worry if you don't have mscorwks.dll).

Also, for sakes of convenience, the copy under C:\Windows\Microsoft.NET\Framework64\v4.0.XXX might also come in handy in case you ever provide a dump for x64.

Thanks!

P.S. substitute XXX for what ever version your system has (probably v4.0.30319)

Contributor

jcondex commented Aug 15, 2013

Thanks a lot for the dump file. Now I need you to do me another favor: Can you please zip and upload the following files:

clr.dll
sos.dll
mscordacwks.dll
mscorwks.dll

You can find all these in: C:\Windows\Microsoft.NET\Framework\v4.0.XXX

(don't worry if you don't have mscorwks.dll).

Also, for sakes of convenience, the copy under C:\Windows\Microsoft.NET\Framework64\v4.0.XXX might also come in handy in case you ever provide a dump for x64.

Thanks!

P.S. substitute XXX for what ever version your system has (probably v4.0.30319)

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 15, 2013

Unfortunately I have installed .Net 4.5 on this PC. Will it have updated those files or it wouldn't have made any difference to these files? (i have v4.0.30319 folder in framework).

bborad1 commented Aug 15, 2013

Unfortunately I have installed .Net 4.5 on this PC. Will it have updated those files or it wouldn't have made any difference to these files? (i have v4.0.30319 folder in framework).

@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 15, 2013

Contributor

It won't work anymore :( 4.5 is an in-place replacement for 4.0.

Is there any chance you can re-reproduce and re-send the dump file and include the files that I mentioned above ?

Thanks a lot, and I apologize for the inconvenience!

Contributor

jcondex commented Aug 15, 2013

It won't work anymore :( 4.5 is an in-place replacement for 4.0.

Is there any chance you can re-reproduce and re-send the dump file and include the files that I mentioned above ?

Thanks a lot, and I apologize for the inconvenience!

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 15, 2013

Sure, I am going to run it on .Net 4.5 and .Net 4 overnight, so should be able to send the dump and .dlls tomorrow morning, for both version of .Net.

bborad1 commented Aug 15, 2013

Sure, I am going to run it on .Net 4.5 and .Net 4 overnight, so should be able to send the dump and .dlls tomorrow morning, for both version of .Net.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 16, 2013

Member

@bborad1 any word?

Member

davidfowl commented Aug 16, 2013

@bborad1 any word?

@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 16, 2013

Contributor

I'm still unable to reproduce this. I tried all 3 transports for periods exceeding 15 hours and all works as expected for me.

In all 3 runs, I didn't manage to see the CancellationTokenSource count greater than 31.

Contributor

jcondex commented Aug 16, 2013

I'm still unable to reproduce this. I tried all 3 transports for periods exceeding 15 hours and all works as expected for me.

In all 3 runs, I didn't manage to see the CancellationTokenSource count greater than 31.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 16, 2013

Uploading dump file onto the FTP server. will send the link soon.

Issue still exist on .Net4.0 but seems to be working fine on .Net4.5.

Its really getting confusing. To make sure that its not something on VM that is causing the issue, I updated .Net 4.0 to .Net4.5 on same VM and now it seems to be fine with .Net4.5. Point to .Net 4.0 but then I have the application running fine on XP with .Net 4.0.

bborad1 commented Aug 16, 2013

Uploading dump file onto the FTP server. will send the link soon.

Issue still exist on .Net4.0 but seems to be working fine on .Net4.5.

Its really getting confusing. To make sure that its not something on VM that is causing the issue, I updated .Net 4.0 to .Net4.5 on same VM and now it seems to be fine with .Net4.5. Point to .Net 4.0 but then I have the application running fine on XP with .Net 4.0.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 16, 2013

Here's dump file for the above test

https://www.hightail.com/download/bWJvTkZpVnNRWUpsQXNUQw

There are two files for each test, one when started and and one when ended(today).

File names containing On18_Net40 : are from Win7 with .Net4.0
File names containing On145_Net45 : are from Win7 with .Net4.5

bborad1 commented Aug 16, 2013

Here's dump file for the above test

https://www.hightail.com/download/bWJvTkZpVnNRWUpsQXNUQw

There are two files for each test, one when started and and one when ended(today).

File names containing On18_Net40 : are from Win7 with .Net4.0
File names containing On145_Net45 : are from Win7 with .Net4.5

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 17, 2013

Member

So this is only on 4.0? @jcondex can you confirm? Moving this out of the 2.0 milestone (since that only runs on 4.5)

Member

davidfowl commented Aug 17, 2013

So this is only on 4.0? @jcondex can you confirm? Moving this out of the 2.0 milestone (since that only runs on 4.5)

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 18, 2013

But keep note that it runs fine on XP with .Net 4.0

bborad1 commented Aug 18, 2013

But keep note that it runs fine on XP with .Net 4.0

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 18, 2013

Member

@bborad1 so tell me if this is correct:

  • It repros on .NET 4.0 on server
  • It doesn't repro on .NET 4.5 (no matter what the OS is), and it also doesn't repro on .NET 4.0 on XP
Member

davidfowl commented Aug 18, 2013

@bborad1 so tell me if this is correct:

  • It repros on .NET 4.0 on server
  • It doesn't repro on .NET 4.5 (no matter what the OS is), and it also doesn't repro on .NET 4.0 on XP
@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 18, 2013

It is confirmed that It repros on .Net 4.0 on Windows 7 (have tested on two different PC)

In doesn't repro on .Net 4.0 on Windows XP and .Net 4.5 on Windows 7 but its only been tested on 1 PC. But to confirm both these points I will have to do probably 3 more tests with different PCs/VMs, .Net 4.5 on XP , .Net 4.5 on Windows 7, .Net 4.0 on XP.

I will start these test and will update you with the result soon, probably in next 2 days.

bborad1 commented Aug 18, 2013

It is confirmed that It repros on .Net 4.0 on Windows 7 (have tested on two different PC)

In doesn't repro on .Net 4.0 on Windows XP and .Net 4.5 on Windows 7 but its only been tested on 1 PC. But to confirm both these points I will have to do probably 3 more tests with different PCs/VMs, .Net 4.5 on XP , .Net 4.5 on Windows 7, .Net 4.0 on XP.

I will start these test and will update you with the result soon, probably in next 2 days.

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 20, 2013

Ok, did 1 test with .Net 4.0 on another XP machine and it worked fine. Unfortunately, cannot install .Net 4.5 on XP so can test with 4.5 on XP.

This confirms another point, It works fine with .Net 4 on XP but not on Windows 7.

Just waiting to get another VM ready with 4.5 to confirm last point whether it works fine with 4.5 or not.

bborad1 commented Aug 20, 2013

Ok, did 1 test with .Net 4.0 on another XP machine and it worked fine. Unfortunately, cannot install .Net 4.5 on XP so can test with 4.5 on XP.

This confirms another point, It works fine with .Net 4 on XP but not on Windows 7.

Just waiting to get another VM ready with 4.5 to confirm last point whether it works fine with 4.5 or not.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 20, 2013

Member

I'm pretty sure it's fine on .NET 4.5 since @jcondex and I can't reproduce the problem running your project.

Member

davidfowl commented Aug 20, 2013

I'm pretty sure it's fine on .NET 4.5 since @jcondex and I can't reproduce the problem running your project.

@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 20, 2013

Contributor

I'm now testing this using a clean Win2008 R2 machine. I should have an update in a few hours.

For sake of convenience, I replaced the client with this:

using System.Linq;
using System.Text;
using Microsoft.AspNet.SignalR.Client.Hubs;
using System.Threading;

namespace ConsoleApplication1
{
    class Program
    {
        public static ManualResetEvent stopMre = new ManualResetEvent(false);

        public static void Run(string hubName)
        {
            new Thread(() =>
                {
                    HubConnection connection = new HubConnection("http://localhost:8080/chatroom");
                    IHubProxy hubProxy = connection.CreateHubProxy(hubName);

                    hubProxy.On<object>(
                        "addMessage",
                        (tag) =>
                        {
                            Console.WriteLine("HubName<{0}>: {1}", hubName, tag);
                        });

                    connection.Start().Wait();
                    Console.WriteLine("connected.  HubName: {0}", hubName);

                    stopMre.WaitOne();

                    connection.Stop();
                }).Start();
        }

        static void Main(string[] args)
        {
            Run("TagHub");
            Run("AssetMovementsHub");
            Run("LogHub");
            Run("SensorOutputHub");
            Run("ServerTimeHub");

            Console.ReadLine();
            stopMre.Set();
        }
    }
}
Contributor

jcondex commented Aug 20, 2013

I'm now testing this using a clean Win2008 R2 machine. I should have an update in a few hours.

For sake of convenience, I replaced the client with this:

using System.Linq;
using System.Text;
using Microsoft.AspNet.SignalR.Client.Hubs;
using System.Threading;

namespace ConsoleApplication1
{
    class Program
    {
        public static ManualResetEvent stopMre = new ManualResetEvent(false);

        public static void Run(string hubName)
        {
            new Thread(() =>
                {
                    HubConnection connection = new HubConnection("http://localhost:8080/chatroom");
                    IHubProxy hubProxy = connection.CreateHubProxy(hubName);

                    hubProxy.On<object>(
                        "addMessage",
                        (tag) =>
                        {
                            Console.WriteLine("HubName<{0}>: {1}", hubName, tag);
                        });

                    connection.Start().Wait();
                    Console.WriteLine("connected.  HubName: {0}", hubName);

                    stopMre.WaitOne();

                    connection.Stop();
                }).Start();
        }

        static void Main(string[] args)
        {
            Run("TagHub");
            Run("AssetMovementsHub");
            Run("LogHub");
            Run("SensorOutputHub");
            Run("ServerTimeHub");

            Console.ReadLine();
            stopMre.Set();
        }
    }
}
@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 20, 2013

Contributor

I finally was able to reproduce it!

0:015> !dumpheap -stat
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
<CUT>
0087b2f0      396        12672 Microsoft.AspNet.SignalR.Infrastructure.UrlDecoder+DecoderHelper
006f5370      227        13620 System.Threading.Tasks.Task`1[[System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]], mscorlib]]
6e644f40      966        15456 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
6f7f0328      681        16344 System.Collections.Generic.List`1[[System.Threading.Tasks.Task+TaskContinuation, mscorlib]]
6f03f504      328        17056 System.Signature
6f0454d8      322        18032 System.Collections.Hashtable
006f3dac      305        18300 System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]]
6f03f334      285        19380 System.Reflection.RuntimeParameterInfo
6e676314       61        20740 System.Net.HttpListenerResponse
6f7dcfa0      670        21440 System.Action`1[[System.Threading.Tasks.Task, mscorlib]]
6f0401cc      449        26940 System.Reflection.RuntimeMethodInfo
6eff6180      251        27444 System.SByte[]
6f0443a8     1265        30360 System.Collections.ArrayList
6f7c2468      682        40872 System.Threading.Tasks.Task+TaskContinuation[]
6f02f64c     1406        44992 System.Action`1[[System.Object, mscorlib]]
6f017c04     1269        45684 System.Threading.Tasks.Task+ContingentProperties
6eff7bcc     1104        48576 System.Threading.Tasks.Task
6f0455d8      389        64728 System.Collections.Hashtable+bucket[]
6f042a94      827       160036 System.Int32[]
6f03f79c     7933       190392 System.RuntimeType
00876b04    11414       639184 Microsoft.AspNet.SignalR.Messaging.Message
6f041ee4     1044       643012 System.Char[]
6eff6d84     5184       670828 System.Object[]
6f03fb08    12853      1008152 System.String
00870de8    63141      1262820 Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource
00870d14    63263      1265260 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper
6f044aa0    12796      1678220 System.Byte[]
6f0147dc    63326      1773128 System.Threading.CancellationCallbackInfo
6f03381c    63235      2276460 System.Threading.CancellationTokenSource
6f03b5c4    63335      2280060 System.Threading.ExecutionContext
00536e20    46069     12447700      Free
0:018> !mdt 071c3b5c 
071c3b5c (System.Threading.CancellationTokenSource)
    m_kernelEvent:NULL (System.Threading.ManualResetEvent)
    m_registeredCallbacksLists:NULL (System.Object[])
    m_state:0x3 (System.Int32)
    m_threadIDExecutingCallbacks:0xb (System.Int32)
    m_disposed:true (System.Boolean)
    m_linkingRegistrations:NULL (System.Collections.Generic.List`1[[System.Threading.CancellationTokenRegistration, mscorlib]])
    m_executingCallback:NULL (System.Threading.CancellationCallbackInfo)

0:018> !mdt 024342d8 
024342d8 (System.Threading.CancellationTokenSource)
    m_kernelEvent:NULL (System.Threading.ManualResetEvent)
    m_registeredCallbacksLists:NULL (System.Object[])
    m_state:0x3 (System.Int32)
    m_threadIDExecutingCallbacks:0xb (System.Int32)
    m_disposed:true (System.Boolean)
    m_linkingRegistrations:NULL (System.Collections.Generic.List`1[[System.Threading.CancellationTokenRegistration, mscorlib]])
    m_executingCallback:NULL (System.Threading.CancellationCallbackInfo)
0:018> !gcroot 024342d8 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 5b4
Scan Thread 2 OSTHread 1298
Scan Thread 5 OSTHread fb0
Scan Thread 7 OSTHread 13e0
Scan Thread 6 OSTHread 13f0
Scan Thread 8 OSTHread 7d8
ESP:56bf31c:Root:  023505c0(Microsoft.AspNet.SignalR.Hubs.ClientProxy)->
  0235057c(System.Func`4[[System.String, mscorlib],[Microsoft.AspNet.SignalR.Hubs.ClientHubInvocation, Microsoft.AspNet.SignalR.Core],[System.Collections.Generic.IList`1[[System.String, mscorlib]], mscorlib],[System.Threading.Tasks.Task, mscorlib]])->
  023226dc(Microsoft.AspNet.SignalR.Infrastructure.ConnectionManager+<>c__DisplayClass1)->
  023236c8(Microsoft.AspNet.SignalR.Infrastructure.Connection)->
  0232278c(Microsoft.AspNet.SignalR.Messaging.MessageBus)->
  02322fc8(Microsoft.AspNet.SignalR.Messaging.TopicLookup)->
  02322fd8(System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[Microsoft.AspNet.SignalR.Messaging.Topic, Microsoft.AspNet.SignalR.Core]])->
  023d1a08(System.Object[])->
  023e5210(System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, mscorlib],[Microsoft.AspNet.SignalR.Messaging.Topic, Microsoft.AspNet.SignalR.Core]])->
  023e4d40(Microsoft.AspNet.SignalR.Messaging.Topic)->
  023e4d68(System.Collections.Generic.List`1[[Microsoft.AspNet.SignalR.Messaging.ISubscription, Microsoft.AspNet.SignalR.Core]])->
  023e5c4c(System.Object[])->
  023e4800(Microsoft.AspNet.SignalR.Messaging.DefaultSubscription)->
  023e47ec(Microsoft.AspNet.SignalR.Infrastructure.Connection+ReceiveContext)->
  023e47b4(Microsoft.AspNet.SignalR.Transports.ForeverTransport+MessageContext)->
  023e47a4(Microsoft.AspNet.SignalR.Transports.ForeverTransport+RequestLifetime)->
  023e4480(Microsoft.AspNet.SignalR.Transports.HttpRequestLifeTime)->
  023e449c(System.Threading.Tasks.TaskCompletionSource`1[[System.Object, mscorlib]])->
  023e44a8(System.Threading.Tasks.Task`1[[System.Object, mscorlib]])->
  023e5e78(System.Threading.Tasks.Task+ContingentProperties)->
  023e5e9c(System.Collections.Generic.List`1[[System.Threading.Tasks.Task+TaskContinuation, mscorlib]])->
  023e5eb4(System.Threading.Tasks.Task+TaskContinuation[])->
  023e601c(System.Threading.Tasks.Task)->
  023e5ffc(System.Action`1[[System.Object, mscorlib]])->
  023e5fec(System.Threading.Tasks.Task+<>c__DisplayClassb)->
  023e5fcc(System.Action`1[[System.Threading.Tasks.Task, mscorlib]])->
  023e5f68(System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass45`2[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener],[System.Threading.Tasks.Task, mscorlib]])->
  023e5f48(System.Func`2[[System.Threading.Tasks.Task, mscorlib],[System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]], mscorlib]])->
  023e5f3c(System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass32`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]])->
  023e5f1c(System.Func`1[[System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]], mscorlib]])->
  023e5f10(System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass2e)->
  023e5ef0(System.Func`1[[System.Threading.Tasks.Task, mscorlib]])->
  023e28b4(Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerResponse)->
  023e2494(Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment)->
  0230f754(System.Threading.CancellationTokenSource)->
  0231eb78(System.Object[])->
  0231eb8c(System.Threading.SparselyPopulatedArray`1[[System.Threading.CancellationCallbackInfo, mscorlib]])->
  0241ba0c(System.Threading.SparselyPopulatedArrayFragment`1[[System.Threading.CancellationCallbackInfo, mscorlib]])->
  0241ba24(System.Object[])->
  02434334(System.Threading.CancellationCallbackInfo)->
  024342fc(Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper)->
  024342c4(Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource)->
  024342d8(System.Threading.CancellationTokenSource)

Contributor

jcondex commented Aug 20, 2013

I finally was able to reproduce it!

0:015> !dumpheap -stat
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
<CUT>
0087b2f0      396        12672 Microsoft.AspNet.SignalR.Infrastructure.UrlDecoder+DecoderHelper
006f5370      227        13620 System.Threading.Tasks.Task`1[[System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]], mscorlib]]
6e644f40      966        15456 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
6f7f0328      681        16344 System.Collections.Generic.List`1[[System.Threading.Tasks.Task+TaskContinuation, mscorlib]]
6f03f504      328        17056 System.Signature
6f0454d8      322        18032 System.Collections.Hashtable
006f3dac      305        18300 System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]]
6f03f334      285        19380 System.Reflection.RuntimeParameterInfo
6e676314       61        20740 System.Net.HttpListenerResponse
6f7dcfa0      670        21440 System.Action`1[[System.Threading.Tasks.Task, mscorlib]]
6f0401cc      449        26940 System.Reflection.RuntimeMethodInfo
6eff6180      251        27444 System.SByte[]
6f0443a8     1265        30360 System.Collections.ArrayList
6f7c2468      682        40872 System.Threading.Tasks.Task+TaskContinuation[]
6f02f64c     1406        44992 System.Action`1[[System.Object, mscorlib]]
6f017c04     1269        45684 System.Threading.Tasks.Task+ContingentProperties
6eff7bcc     1104        48576 System.Threading.Tasks.Task
6f0455d8      389        64728 System.Collections.Hashtable+bucket[]
6f042a94      827       160036 System.Int32[]
6f03f79c     7933       190392 System.RuntimeType
00876b04    11414       639184 Microsoft.AspNet.SignalR.Messaging.Message
6f041ee4     1044       643012 System.Char[]
6eff6d84     5184       670828 System.Object[]
6f03fb08    12853      1008152 System.String
00870de8    63141      1262820 Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource
00870d14    63263      1265260 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper
6f044aa0    12796      1678220 System.Byte[]
6f0147dc    63326      1773128 System.Threading.CancellationCallbackInfo
6f03381c    63235      2276460 System.Threading.CancellationTokenSource
6f03b5c4    63335      2280060 System.Threading.ExecutionContext
00536e20    46069     12447700      Free
0:018> !mdt 071c3b5c 
071c3b5c (System.Threading.CancellationTokenSource)
    m_kernelEvent:NULL (System.Threading.ManualResetEvent)
    m_registeredCallbacksLists:NULL (System.Object[])
    m_state:0x3 (System.Int32)
    m_threadIDExecutingCallbacks:0xb (System.Int32)
    m_disposed:true (System.Boolean)
    m_linkingRegistrations:NULL (System.Collections.Generic.List`1[[System.Threading.CancellationTokenRegistration, mscorlib]])
    m_executingCallback:NULL (System.Threading.CancellationCallbackInfo)

0:018> !mdt 024342d8 
024342d8 (System.Threading.CancellationTokenSource)
    m_kernelEvent:NULL (System.Threading.ManualResetEvent)
    m_registeredCallbacksLists:NULL (System.Object[])
    m_state:0x3 (System.Int32)
    m_threadIDExecutingCallbacks:0xb (System.Int32)
    m_disposed:true (System.Boolean)
    m_linkingRegistrations:NULL (System.Collections.Generic.List`1[[System.Threading.CancellationTokenRegistration, mscorlib]])
    m_executingCallback:NULL (System.Threading.CancellationCallbackInfo)
0:018> !gcroot 024342d8 
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 5b4
Scan Thread 2 OSTHread 1298
Scan Thread 5 OSTHread fb0
Scan Thread 7 OSTHread 13e0
Scan Thread 6 OSTHread 13f0
Scan Thread 8 OSTHread 7d8
ESP:56bf31c:Root:  023505c0(Microsoft.AspNet.SignalR.Hubs.ClientProxy)->
  0235057c(System.Func`4[[System.String, mscorlib],[Microsoft.AspNet.SignalR.Hubs.ClientHubInvocation, Microsoft.AspNet.SignalR.Core],[System.Collections.Generic.IList`1[[System.String, mscorlib]], mscorlib],[System.Threading.Tasks.Task, mscorlib]])->
  023226dc(Microsoft.AspNet.SignalR.Infrastructure.ConnectionManager+<>c__DisplayClass1)->
  023236c8(Microsoft.AspNet.SignalR.Infrastructure.Connection)->
  0232278c(Microsoft.AspNet.SignalR.Messaging.MessageBus)->
  02322fc8(Microsoft.AspNet.SignalR.Messaging.TopicLookup)->
  02322fd8(System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[Microsoft.AspNet.SignalR.Messaging.Topic, Microsoft.AspNet.SignalR.Core]])->
  023d1a08(System.Object[])->
  023e5210(System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, mscorlib],[Microsoft.AspNet.SignalR.Messaging.Topic, Microsoft.AspNet.SignalR.Core]])->
  023e4d40(Microsoft.AspNet.SignalR.Messaging.Topic)->
  023e4d68(System.Collections.Generic.List`1[[Microsoft.AspNet.SignalR.Messaging.ISubscription, Microsoft.AspNet.SignalR.Core]])->
  023e5c4c(System.Object[])->
  023e4800(Microsoft.AspNet.SignalR.Messaging.DefaultSubscription)->
  023e47ec(Microsoft.AspNet.SignalR.Infrastructure.Connection+ReceiveContext)->
  023e47b4(Microsoft.AspNet.SignalR.Transports.ForeverTransport+MessageContext)->
  023e47a4(Microsoft.AspNet.SignalR.Transports.ForeverTransport+RequestLifetime)->
  023e4480(Microsoft.AspNet.SignalR.Transports.HttpRequestLifeTime)->
  023e449c(System.Threading.Tasks.TaskCompletionSource`1[[System.Object, mscorlib]])->
  023e44a8(System.Threading.Tasks.Task`1[[System.Object, mscorlib]])->
  023e5e78(System.Threading.Tasks.Task+ContingentProperties)->
  023e5e9c(System.Collections.Generic.List`1[[System.Threading.Tasks.Task+TaskContinuation, mscorlib]])->
  023e5eb4(System.Threading.Tasks.Task+TaskContinuation[])->
  023e601c(System.Threading.Tasks.Task)->
  023e5ffc(System.Action`1[[System.Object, mscorlib]])->
  023e5fec(System.Threading.Tasks.Task+<>c__DisplayClassb)->
  023e5fcc(System.Action`1[[System.Threading.Tasks.Task, mscorlib]])->
  023e5f68(System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass45`2[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener],[System.Threading.Tasks.Task, mscorlib]])->
  023e5f48(System.Func`2[[System.Threading.Tasks.Task, mscorlib],[System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]], mscorlib]])->
  023e5f3c(System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass32`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]])->
  023e5f1c(System.Func`1[[System.Threading.Tasks.Task`1[[System.Threading.Tasks.TaskHelpersExtensions+AsyncVoid, Microsoft.Owin.Host.HttpListener]], mscorlib]])->
  023e5f10(System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass2e)->
  023e5ef0(System.Func`1[[System.Threading.Tasks.Task, mscorlib]])->
  023e28b4(Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerResponse)->
  023e2494(Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment)->
  0230f754(System.Threading.CancellationTokenSource)->
  0231eb78(System.Object[])->
  0231eb8c(System.Threading.SparselyPopulatedArray`1[[System.Threading.CancellationCallbackInfo, mscorlib]])->
  0241ba0c(System.Threading.SparselyPopulatedArrayFragment`1[[System.Threading.CancellationCallbackInfo, mscorlib]])->
  0241ba24(System.Object[])->
  02434334(System.Threading.CancellationCallbackInfo)->
  024342fc(Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper)->
  024342c4(Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource)->
  024342d8(System.Threading.CancellationTokenSource)

@jcondex

This comment has been minimized.

Show comment Hide comment
@jcondex

jcondex Aug 21, 2013

Contributor

This problem only happens using SelfHost.

I changed the app to use WebHost and this is the result after ~4 hrs of running:

0:016> !dumpheap -type CancellationToken -stat
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
70a949a0       10          120 System.Threading.CancellationToken
04d731d0       10          200 Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource
04635ed4       10          320 System.Func`5[[System.String, mscorlib],[System.Int64, mscorlib],[System.Nullable`1[[System.Int64, mscorlib]], mscorlib],[System.Threading.CancellationToken, mscorlib],[System.Threading.Tasks.Task, mscorlib]]
04d75850       30          600 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper
70a9381c       23          828 System.Threading.CancellationTokenSource
04d757a8       30          840 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+DiposeCancellationState
Total 113 objects
0:016> .time
Debug session time: Wed Aug 21 11:50:40.142 2013 (UTC - 7:00)
System Uptime: 0 days 7:50:20.264
Process Uptime: 0 days 3:37:35.122
  Kernel time: 0 days 0:00:02.671
  User time: 0 days 0:00:13.125

Contributor

jcondex commented Aug 21, 2013

This problem only happens using SelfHost.

I changed the app to use WebHost and this is the result after ~4 hrs of running:

0:016> !dumpheap -type CancellationToken -stat
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
70a949a0       10          120 System.Threading.CancellationToken
04d731d0       10          200 Microsoft.AspNet.SignalR.Infrastructure.SafeCancellationTokenSource
04635ed4       10          320 System.Func`5[[System.String, mscorlib],[System.Int64, mscorlib],[System.Nullable`1[[System.Int64, mscorlib]], mscorlib],[System.Threading.CancellationToken, mscorlib],[System.Threading.Tasks.Task, mscorlib]]
04d75850       30          600 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+CancellationCallbackWrapper
70a9381c       23          828 System.Threading.CancellationTokenSource
04d757a8       30          840 Microsoft.AspNet.SignalR.Infrastructure.CancellationTokenExtensions+DiposeCancellationState
Total 113 objects
0:016> .time
Debug session time: Wed Aug 21 11:50:40.142 2013 (UTC - 7:00)
System Uptime: 0 days 7:50:20.264
Process Uptime: 0 days 3:37:35.122
  Kernel time: 0 days 0:00:02.671
  User time: 0 days 0:00:13.125

@bborad1

This comment has been minimized.

Show comment Hide comment
@bborad1

bborad1 Aug 22, 2013

Ok, I can now confirm the last point as well. I does seem to work fine with .Net 4.5 on Windows 7.

So, could it be something in .Net 4.0 that might not be working as expected when using SelfHost except when running on XP?

As longs as it works fine with .Net 4.5, that's ok for now, we can just upgrade client's server with .Net 4.5.

bborad1 commented Aug 22, 2013

Ok, I can now confirm the last point as well. I does seem to work fine with .Net 4.5 on Windows 7.

So, could it be something in .Net 4.0 that might not be working as expected when using SelfHost except when running on XP?

As longs as it works fine with .Net 4.5, that's ok for now, we can just upgrade client's server with .Net 4.5.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Aug 22, 2013

Member

@bborad1 we already figured that out. @jcondex has reproduced the issue and has a potential fix for it. Thanks for the details. We'll update this issue when the bug is resolved.

Member

davidfowl commented Aug 22, 2013

@bborad1 we already figured that out. @jcondex has reproduced the issue and has a potential fix for it. Thanks for the details. We'll update this issue when the bug is resolved.

@davidfowl

This comment has been minimized.

Show comment Hide comment
@davidfowl

davidfowl Oct 15, 2013

Member

This was fixed in a42875e

Member

davidfowl commented Oct 15, 2013

This was fixed in a42875e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment