New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MapSignalR hangs in the creation of the Performance Counters #3414

Closed
corneliutusnea opened this Issue Feb 23, 2015 · 33 comments

Comments

Projects
None yet
@corneliutusnea

corneliutusnea commented Feb 23, 2015

Hi,
With the 2.2.0 build I started to have a problem in the app.MapSignalR() which it gets stuck in creating the Performance Counters. It's hard to reproduce but I get it about 3-4 times a day.

This is the callstack:

    System.dll!System.Diagnostics.PerformanceCounterLib.GetPerformanceCounterLib(string machineName, System.Globalization.CultureInfo culture)  Unknown
    System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown
    System.dll!System.Diagnostics.PerformanceCounter.InitializeImpl()   Unknown
>   System.dll!System.Diagnostics.PerformanceCounter.PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)  Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.SetCounterProperties()  Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.Initialize(string instanceName, System.Threading.CancellationToken hostShutdownToken)   Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Hosting.HostDependencyResolverExtensions.InitializeHost(Microsoft.AspNet.SignalR.IDependencyResolver resolver, string instanceName, System.Threading.CancellationToken hostShutdownToken)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.UseSignalRMiddleware<Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware>(Owin.IAppBuilder builder, object[] args)   Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR.AnonymousMethod__0(Owin.IAppBuilder subApp)    Unknown
    Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, Microsoft.Owin.PathString pathMatch, System.Action<Owin.IAppBuilder> configuration) Unknown
    Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, string pathMatch, System.Action<Owin.IAppBuilder> configuration)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, string path, Microsoft.AspNet.SignalR.HubConfiguration configuration)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder)  Unknown
    SnowSparkle.OneSaas.dll!SnowSparkle.OneSaas.SignalRConfig.Configuration(Owin.IAppBuilder app) Line 22   C#
    [Native to Managed Transition]  
    Microsoft.Owin.Host.SystemWeb.dll!Owin.Loader.DefaultLoader.MakeDelegate.AnonymousMethod__b(Owin.IAppBuilder builder)   Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Owin.Loader.DefaultLoader.LoadImplementation.AnonymousMethod__0(Owin.IAppBuilder builder) Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint.AnonymousMethod__0(Owin.IAppBuilder builder) Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(System.Action<Owin.IAppBuilder> startup)  Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(System.Action<Owin.IAppBuilder> startup)  Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()    Unknown
    mscorlib.dll!System.Threading.LazyInitializer.EnsureInitializedCore<System.__Canon>(ref System.__Canon target, ref bool initialized, ref object syncLock, System.Func<System.__Canon> valueFactory) Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(System.Web.HttpApplication context) Unknown
    System.Web.dll!System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(System.IntPtr appContext, System.Web.HttpContext context, System.Reflection.MethodInfo[] handlers)  Unknown
    System.Web.dll!System.Web.HttpApplication.InitSpecial(System.Web.HttpApplicationState state, System.Reflection.MethodInfo[] handlers, System.IntPtr appContext, System.Web.HttpContext context) Unknown
    System.Web.dll!System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(System.IntPtr appContext, System.Web.HttpContext context)    Unknown
    System.Web.dll!System.Web.Hosting.PipelineRuntime.InitializeApplication(System.IntPtr appContext)   Unknown
    [AppDomain Transition]  

Repro steps:
Run a website in IISExpress. Save the web.config and reload the site several times.
About 4-5 tries later of recycling the site it hangs.

I tried debugging the disassembled code and all I could see is the code jumping indefinitely between those two functions.

>   System.dll!System.Diagnostics.PerformanceCounterLib.GetPerformanceCounterLib(string machineName, System.Globalization.CultureInfo culture)  Unknown
>   System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown

This looks to me like a by-product of this bug:
http://connect.microsoft.com/VisualStudio/feedback/details/864245/multithreaded-bug-in-system-diagnostics-performancecounter

I'd like to have a config value to disable the creation of Performance Counters so I can avoid this problem if possible.

@JonasSyrstad

This comment has been minimized.

Show comment
Hide comment
@JonasSyrstad

JonasSyrstad Feb 23, 2015

This helps on my machine, I found the solution for this when I had similar hangs with the MemoryCache implementation. Running it on a separate thread somehow bypasses the PerfCounter problem.
Hope this helps :)

var task=Task.Run(()=>app.MapSignalR());
            task.Wait(300);
//try again if it fails just to be sure ;)
            if (task.IsCanceled)  Task.Run(() => app.MapSignalR()).Wait(300);

JonasSyrstad commented Feb 23, 2015

This helps on my machine, I found the solution for this when I had similar hangs with the MemoryCache implementation. Running it on a separate thread somehow bypasses the PerfCounter problem.
Hope this helps :)

var task=Task.Run(()=>app.MapSignalR());
            task.Wait(300);
//try again if it fails just to be sure ;)
            if (task.IsCanceled)  Task.Run(() => app.MapSignalR()).Wait(300);
@corneliutusnea

This comment has been minimized.

Show comment
Hide comment
@corneliutusnea

corneliutusnea Feb 23, 2015

For now I found another solution. I implemented a dummy no-op IPerformanceCounterManager and registered it in the global host just before the SignalR gets mapped. This makes SR use my own performance counter manager:

var tempCounterManager = new TempPerformanceCounterManager();
GlobalHost.DependencyResolver.Register(typeof (IPerformanceCounterManager), () => tempCounterManager);

[....]

public class TempPerformanceCounterManager : IPerformanceCounterManager
{
    private readonly static PropertyInfo[] _counterProperties = GetCounterPropertyInfo();
    private readonly static IPerformanceCounter _noOpCounter = new NoOpPerformanceCounter();

    public TempPerformanceCounterManager()
    {
        foreach (var property in _counterProperties)
        {
            property.SetValue(this, new NoOpPerformanceCounter(), null);
        }
    }

    public void Initialize(string instanceName, CancellationToken hostShutdownToken)
    {
    }

    public IPerformanceCounter LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)
    {
        return _noOpCounter;
    }

    internal static PropertyInfo[] GetCounterPropertyInfo()
    {
        return typeof(TempPerformanceCounterManager)
            .GetProperties()
            .Where(p => p.PropertyType == typeof(IPerformanceCounter))
            .ToArray();
    }
    public IPerformanceCounter ConnectionsConnected { get; set; }
    public IPerformanceCounter ConnectionsReconnected { get; set; }
    public IPerformanceCounter ConnectionsDisconnected { get; set; }
    public IPerformanceCounter ConnectionsCurrentForeverFrame { get; private set; }
    public IPerformanceCounter ConnectionsCurrentLongPolling { get; private set; }
    public IPerformanceCounter ConnectionsCurrentServerSentEvents { get; private set; }
    public IPerformanceCounter ConnectionsCurrentWebSockets { get; private set; }
    public IPerformanceCounter ConnectionsCurrent { get; private set; }
    public IPerformanceCounter ConnectionMessagesReceivedTotal { get; private set; }
    public IPerformanceCounter ConnectionMessagesSentTotal { get; private set; }
    public IPerformanceCounter ConnectionMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter ConnectionMessagesSentPerSec { get; private set; }
    public IPerformanceCounter MessageBusMessagesReceivedTotal { get; private set; }
    public IPerformanceCounter MessageBusMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter ScaleoutMessageBusMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter MessageBusMessagesPublishedTotal { get; private set; }
    public IPerformanceCounter MessageBusMessagesPublishedPerSec { get; private set; }
    public IPerformanceCounter MessageBusSubscribersCurrent { get; private set; }
    public IPerformanceCounter MessageBusSubscribersTotal { get; private set; }
    public IPerformanceCounter MessageBusSubscribersPerSec { get; private set; }
    public IPerformanceCounter MessageBusAllocatedWorkers { get; private set; }
    public IPerformanceCounter MessageBusBusyWorkers { get; private set; }
    public IPerformanceCounter MessageBusTopicsCurrent { get; private set; }
    public IPerformanceCounter ErrorsAllTotal { get; private set; }
    public IPerformanceCounter ErrorsAllPerSec { get; private set; }
    public IPerformanceCounter ErrorsHubResolutionTotal { get; private set; }
    public IPerformanceCounter ErrorsHubResolutionPerSec { get; private set; }
    public IPerformanceCounter ErrorsHubInvocationTotal { get; private set; }
    public IPerformanceCounter ErrorsHubInvocationPerSec { get; private set; }
    public IPerformanceCounter ErrorsTransportTotal { get; private set; }
    public IPerformanceCounter ErrorsTransportPerSec { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountTotal { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountOpen { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountBuffering { get; private set; }
    public IPerformanceCounter ScaleoutErrorsTotal { get; private set; }
    public IPerformanceCounter ScaleoutErrorsPerSec { get; private set; }
    public IPerformanceCounter ScaleoutSendQueueLength { get; private set; }
}

internal class NoOpPerformanceCounter : IPerformanceCounter
{
    public string CounterName
    {
        get
        {
            return GetType().Name;
        }
    }
    public long Decrement()
    {
        return 0;
    }
    public long Increment()
    {
        return 0;
    }
    public long IncrementBy(long value)
    {
        return 0;
    }
    public long RawValue
    {
        get { return 0; }
        set { }
    }
    public void Close()
    {
    }
    public void RemoveInstance()
    {
    }
    public CounterSample NextSample()
    {
        return CounterSample.Empty;
    }
}

corneliutusnea commented Feb 23, 2015

For now I found another solution. I implemented a dummy no-op IPerformanceCounterManager and registered it in the global host just before the SignalR gets mapped. This makes SR use my own performance counter manager:

var tempCounterManager = new TempPerformanceCounterManager();
GlobalHost.DependencyResolver.Register(typeof (IPerformanceCounterManager), () => tempCounterManager);

[....]

public class TempPerformanceCounterManager : IPerformanceCounterManager
{
    private readonly static PropertyInfo[] _counterProperties = GetCounterPropertyInfo();
    private readonly static IPerformanceCounter _noOpCounter = new NoOpPerformanceCounter();

    public TempPerformanceCounterManager()
    {
        foreach (var property in _counterProperties)
        {
            property.SetValue(this, new NoOpPerformanceCounter(), null);
        }
    }

    public void Initialize(string instanceName, CancellationToken hostShutdownToken)
    {
    }

    public IPerformanceCounter LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)
    {
        return _noOpCounter;
    }

    internal static PropertyInfo[] GetCounterPropertyInfo()
    {
        return typeof(TempPerformanceCounterManager)
            .GetProperties()
            .Where(p => p.PropertyType == typeof(IPerformanceCounter))
            .ToArray();
    }
    public IPerformanceCounter ConnectionsConnected { get; set; }
    public IPerformanceCounter ConnectionsReconnected { get; set; }
    public IPerformanceCounter ConnectionsDisconnected { get; set; }
    public IPerformanceCounter ConnectionsCurrentForeverFrame { get; private set; }
    public IPerformanceCounter ConnectionsCurrentLongPolling { get; private set; }
    public IPerformanceCounter ConnectionsCurrentServerSentEvents { get; private set; }
    public IPerformanceCounter ConnectionsCurrentWebSockets { get; private set; }
    public IPerformanceCounter ConnectionsCurrent { get; private set; }
    public IPerformanceCounter ConnectionMessagesReceivedTotal { get; private set; }
    public IPerformanceCounter ConnectionMessagesSentTotal { get; private set; }
    public IPerformanceCounter ConnectionMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter ConnectionMessagesSentPerSec { get; private set; }
    public IPerformanceCounter MessageBusMessagesReceivedTotal { get; private set; }
    public IPerformanceCounter MessageBusMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter ScaleoutMessageBusMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter MessageBusMessagesPublishedTotal { get; private set; }
    public IPerformanceCounter MessageBusMessagesPublishedPerSec { get; private set; }
    public IPerformanceCounter MessageBusSubscribersCurrent { get; private set; }
    public IPerformanceCounter MessageBusSubscribersTotal { get; private set; }
    public IPerformanceCounter MessageBusSubscribersPerSec { get; private set; }
    public IPerformanceCounter MessageBusAllocatedWorkers { get; private set; }
    public IPerformanceCounter MessageBusBusyWorkers { get; private set; }
    public IPerformanceCounter MessageBusTopicsCurrent { get; private set; }
    public IPerformanceCounter ErrorsAllTotal { get; private set; }
    public IPerformanceCounter ErrorsAllPerSec { get; private set; }
    public IPerformanceCounter ErrorsHubResolutionTotal { get; private set; }
    public IPerformanceCounter ErrorsHubResolutionPerSec { get; private set; }
    public IPerformanceCounter ErrorsHubInvocationTotal { get; private set; }
    public IPerformanceCounter ErrorsHubInvocationPerSec { get; private set; }
    public IPerformanceCounter ErrorsTransportTotal { get; private set; }
    public IPerformanceCounter ErrorsTransportPerSec { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountTotal { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountOpen { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountBuffering { get; private set; }
    public IPerformanceCounter ScaleoutErrorsTotal { get; private set; }
    public IPerformanceCounter ScaleoutErrorsPerSec { get; private set; }
    public IPerformanceCounter ScaleoutSendQueueLength { get; private set; }
}

internal class NoOpPerformanceCounter : IPerformanceCounter
{
    public string CounterName
    {
        get
        {
            return GetType().Name;
        }
    }
    public long Decrement()
    {
        return 0;
    }
    public long Increment()
    {
        return 0;
    }
    public long IncrementBy(long value)
    {
        return 0;
    }
    public long RawValue
    {
        get { return 0; }
        set { }
    }
    public void Close()
    {
    }
    public void RemoveInstance()
    {
    }
    public CounterSample NextSample()
    {
        return CounterSample.Empty;
    }
}
@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Feb 23, 2015

Contributor

Are you able to tell what the values of categoryName, counterName and instanceName were when it got stuck? Also, do you know if this happens for the same performance counter or different ones?

PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)

Contributor

moozzyk commented Feb 23, 2015

Are you able to tell what the values of categoryName, counterName and instanceName were when it got stuck? Also, do you know if this happens for the same performance counter or different ones?

PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)

@alexandrereyes

This comment has been minimized.

Show comment
Hide comment
@alexandrereyes

alexandrereyes Jul 24, 2015

The same problem occurs randomly on Visual Studio 2015, I think is because the new diagnostic tool

alexandrereyes commented Jul 24, 2015

The same problem occurs randomly on Visual Studio 2015, I think is because the new diagnostic tool

@sharpdevel

This comment has been minimized.

Show comment
Hide comment
@sharpdevel

sharpdevel Aug 12, 2015

I discovered the same behaviour on my development machine as well as on my Windows Azure Cloud Service machine. In both cases I am using the full IIS. On my development machine the cpu consumption of the w3wp.exe rises and never lowers again. The website then gets very unresponsive. It only seems to happen when there are existing SignalR connections and the AppPool and thereby the w3wp.exe are restarted. I attached my Visual Studio 2015 to the w3wp process and see that one thread hangs on the kine that call MapSignalR. The call stack tells me, that the method called some external code but i can't see any name or hint what it is doing.

sharpdevel commented Aug 12, 2015

I discovered the same behaviour on my development machine as well as on my Windows Azure Cloud Service machine. In both cases I am using the full IIS. On my development machine the cpu consumption of the w3wp.exe rises and never lowers again. The website then gets very unresponsive. It only seems to happen when there are existing SignalR connections and the AppPool and thereby the w3wp.exe are restarted. I attached my Visual Studio 2015 to the w3wp process and see that one thread hangs on the kine that call MapSignalR. The call stack tells me, that the method called some external code but i can't see any name or hint what it is doing.

@smourier

This comment has been minimized.

Show comment
Hide comment
@smourier

smourier Aug 20, 2015

I had the same hang issue with my app (not related to SignalR at all) in perfcounter lib (I was using MemoryCache but that's the same story as MemoryCache uses perf counters at initialization - as others in this thread noted). There is a strange piece of code in the private System.dll's method PerformanceCounterLib:IsCustomCategory(string, string) that can possibly loop forever scanning the registry if the current culture/LCID is not 9 -> english. Here is what I did to fix the issue:

        CultureInfo ci = Thread.CurrentThread.CurrentCulture;
        var lcid = CultureInfo.CurrentCulture.Parent.LCID;
        if (lcid != 9)
        {
            Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfo(1033);
        }
        var myCache = new MemoryCache(MyCachePrefix, MyCacheConfig);
        if (lcid != 9)
        {
            Thread.CurrentThread.CurrentCulture = ci;
        }

smourier commented Aug 20, 2015

I had the same hang issue with my app (not related to SignalR at all) in perfcounter lib (I was using MemoryCache but that's the same story as MemoryCache uses perf counters at initialization - as others in this thread noted). There is a strange piece of code in the private System.dll's method PerformanceCounterLib:IsCustomCategory(string, string) that can possibly loop forever scanning the registry if the current culture/LCID is not 9 -> english. Here is what I did to fix the issue:

        CultureInfo ci = Thread.CurrentThread.CurrentCulture;
        var lcid = CultureInfo.CurrentCulture.Parent.LCID;
        if (lcid != 9)
        {
            Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfo(1033);
        }
        var myCache = new MemoryCache(MyCachePrefix, MyCacheConfig);
        if (lcid != 9)
        {
            Thread.CurrentThread.CurrentCulture = ci;
        }
@alisabzevari

This comment has been minimized.

Show comment
Hide comment
@alisabzevari

alisabzevari Sep 19, 2015

I have the same problem in 2.2.0, Windows 10, VS 2015 and IIS Express. It happens after recycling IIS App Pool. This is my call stack:

[Managed to Native Transition]  
mscorlib.dll!Microsoft.Win32.RegistryKey.OpenSubKey(string name, bool writable) Unknown
mscorlib.dll!Microsoft.Win32.RegistryKey.OpenSubKey(string name)    Unknown
System.dll!System.Diagnostics.PerformanceCounterLib.FindCustomCategory(string category, out System.Diagnostics.PerformanceCounterCategoryType categoryType) Unknown
System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown
System.dll!System.Diagnostics.PerformanceCounter.InitializeImpl()   Unknown
System.dll!System.Diagnostics.PerformanceCounter.PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)  Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.SetCounterProperties()  Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.Initialize(string instanceName, System.Threading.CancellationToken hostShutdownToken)   Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Hosting.HostDependencyResolverExtensions.InitializeHost(Microsoft.AspNet.SignalR.IDependencyResolver resolver, string instanceName, System.Threading.CancellationToken hostShutdownToken)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.UseSignalRMiddleware<Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware>(Owin.IAppBuilder builder, object[] args)   Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR.AnonymousMethod__0(Owin.IAppBuilder subApp)    Unknown
Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, Microsoft.Owin.PathString pathMatch, System.Action<Owin.IAppBuilder> configuration) Unknown
Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, string pathMatch, System.Action<Owin.IAppBuilder> configuration)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, string path, Microsoft.AspNet.SignalR.HubConfiguration configuration)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, Microsoft.AspNet.SignalR.HubConfiguration configuration) Unknown
...

I don't know how to find out what it is trying to do with registry but it is always waiting there. I had suspected that it was related to permissions and tried with running VS as administrator; but it didn't solve the problem.

alisabzevari commented Sep 19, 2015

I have the same problem in 2.2.0, Windows 10, VS 2015 and IIS Express. It happens after recycling IIS App Pool. This is my call stack:

[Managed to Native Transition]  
mscorlib.dll!Microsoft.Win32.RegistryKey.OpenSubKey(string name, bool writable) Unknown
mscorlib.dll!Microsoft.Win32.RegistryKey.OpenSubKey(string name)    Unknown
System.dll!System.Diagnostics.PerformanceCounterLib.FindCustomCategory(string category, out System.Diagnostics.PerformanceCounterCategoryType categoryType) Unknown
System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown
System.dll!System.Diagnostics.PerformanceCounter.InitializeImpl()   Unknown
System.dll!System.Diagnostics.PerformanceCounter.PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)  Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.SetCounterProperties()  Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.Initialize(string instanceName, System.Threading.CancellationToken hostShutdownToken)   Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Hosting.HostDependencyResolverExtensions.InitializeHost(Microsoft.AspNet.SignalR.IDependencyResolver resolver, string instanceName, System.Threading.CancellationToken hostShutdownToken)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.UseSignalRMiddleware<Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware>(Owin.IAppBuilder builder, object[] args)   Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR.AnonymousMethod__0(Owin.IAppBuilder subApp)    Unknown
Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, Microsoft.Owin.PathString pathMatch, System.Action<Owin.IAppBuilder> configuration) Unknown
Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, string pathMatch, System.Action<Owin.IAppBuilder> configuration)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, string path, Microsoft.AspNet.SignalR.HubConfiguration configuration)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, Microsoft.AspNet.SignalR.HubConfiguration configuration) Unknown
...

I don't know how to find out what it is trying to do with registry but it is always waiting there. I had suspected that it was related to permissions and tried with running VS as administrator; but it didn't solve the problem.

@smourier

This comment has been minimized.

Show comment
Hide comment
@smourier

smourier Sep 19, 2015

Like I said, it's a bug in this IsCustomCategory .NET's code here: http://referencesource.microsoft.com/#System/services/monitoring/system/diagnosticts/PerformanceCounterLib.cs,fb23e462a0395092,references when the current LCID/locale/language is not English, this code can go in a forever loop for some reason (not always). I you can force the culture to English before the code stack reaches this method, it should fix the problem.

smourier commented Sep 19, 2015

Like I said, it's a bug in this IsCustomCategory .NET's code here: http://referencesource.microsoft.com/#System/services/monitoring/system/diagnosticts/PerformanceCounterLib.cs,fb23e462a0395092,references when the current LCID/locale/language is not English, this code can go in a forever loop for some reason (not always). I you can force the culture to English before the code stack reaches this method, it should fix the problem.

@drphrozen

This comment has been minimized.

Show comment
Hide comment
@drphrozen

drphrozen Sep 22, 2015

I have the same issue.. Although only after having:

  1. attached a debugger
  2. stopped debugging
  3. build, rebuild or recycled
  4. verified that MapSignalR is the culprit (threads are looping forever)

Windows 10 (IIS 10), Visual Studio 2015 and SignalR 2.2.0

drphrozen commented Sep 22, 2015

I have the same issue.. Although only after having:

  1. attached a debugger
  2. stopped debugging
  3. build, rebuild or recycled
  4. verified that MapSignalR is the culprit (threads are looping forever)

Windows 10 (IIS 10), Visual Studio 2015 and SignalR 2.2.0

@alisabzevari

This comment has been minimized.

Show comment
Hide comment
@alisabzevari

alisabzevari Sep 22, 2015

@smourier Thanks your solution worked for me.

alisabzevari commented Sep 22, 2015

@smourier Thanks your solution worked for me.

@MojoDK

This comment has been minimized.

Show comment
Hide comment
@MojoDK

MojoDK Oct 24, 2015

No official fix for this yet? :(

MojoDK commented Oct 24, 2015

No official fix for this yet? :(

@zpqrtbnk

This comment has been minimized.

Show comment
Hide comment
@zpqrtbnk

zpqrtbnk Oct 31, 2015

See this blog post - we faced the same issue, in the OWIN Startup method, when creating a MemoryCache instance. The root cause is CultureInfo leaking from one AppDomain to another. A workaround has been identified, but the real issue lies within OWIN that should sanitize the culture before running the Startup method.

zpqrtbnk commented Oct 31, 2015

See this blog post - we faced the same issue, in the OWIN Startup method, when creating a MemoryCache instance. The root cause is CultureInfo leaking from one AppDomain to another. A workaround has been identified, but the real issue lies within OWIN that should sanitize the culture before running the Startup method.

@smourier

This comment has been minimized.

Show comment
Hide comment
@smourier

smourier Oct 31, 2015

Nice analysis. I would just add that for me the real issue is in the .NET framework (System.dll), not OWIN nor SignalR specifically. Everyone using PerformanceCounter directly or indirectly (in my case, I'm not using OWIN nor SignalR, just MemoryCache) has the problem. It's a quite big issue in fact. Don't know if Microsoft is aware of this BTW - I suppose their developers are mostly running with en English LCID so they don't see the problem often :-) ...

smourier commented Oct 31, 2015

Nice analysis. I would just add that for me the real issue is in the .NET framework (System.dll), not OWIN nor SignalR specifically. Everyone using PerformanceCounter directly or indirectly (in my case, I'm not using OWIN nor SignalR, just MemoryCache) has the problem. It's a quite big issue in fact. Don't know if Microsoft is aware of this BTW - I suppose their developers are mostly running with en English LCID so they don't see the problem often :-) ...

@zpqrtbnk

This comment has been minimized.

Show comment
Hide comment
@zpqrtbnk

zpqrtbnk Oct 31, 2015

One could definitively argue that the problem lies within the PerformanceCounterLib class that should not compare CultureInfo instances by reference.

zpqrtbnk commented Oct 31, 2015

One could definitively argue that the problem lies within the PerformanceCounterLib class that should not compare CultureInfo instances by reference.

@smourier

This comment has been minimized.

Show comment
Hide comment
@smourier

smourier Oct 31, 2015

Of course, yes but this class is internal. But as soon as you use public one PerformanceCounter (and work with non English LCID) you can face the issue.

smourier commented Oct 31, 2015

Of course, yes but this class is internal. But as soon as you use public one PerformanceCounter (and work with non English LCID) you can face the issue.

@harveytriana

This comment has been minimized.

Show comment
Hide comment
@harveytriana

harveytriana Nov 19, 2015

Thanks Jonas Syrstad. Really is the solution.

harveytriana commented Nov 19, 2015

Thanks Jonas Syrstad. Really is the solution.

moozzyk added a commit that referenced this issue Dec 5, 2015

HopeIsTheOnlyStrategy - fixing hangs when creating Perf Counters
The theory is that tha hangs are caused by a CultureInfo that leaked
from a different AppDomain (see this blog post for more details:
(http://www.zpqrtbnk.net/posts/appdomains-threads-cultureinfos-and-paracetamol).
Unfortunately, I was not however able to reproduce the issue (tried both English
and non-English Windows) but hope it will fix #3414 as indicated in the discussion
and the above blog post.
@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Dec 5, 2015

Contributor

@zpqrtbnk - this is a great write up. I prepared a fix (#3609) based on your post. Do you mind taking a look?

Contributor

moozzyk commented Dec 5, 2015

@zpqrtbnk - this is a great write up. I prepared a fix (#3609) based on your post. Do you mind taking a look?

@zpqrtbnk

This comment has been minimized.

Show comment
Hide comment
@zpqrtbnk

zpqrtbnk Dec 7, 2015

Looks good to me.

zpqrtbnk commented Dec 7, 2015

Looks good to me.

@moozzyk moozzyk added this to the 2.2.1 milestone Dec 7, 2015

@moozzyk moozzyk added the 5 - Done label Dec 7, 2015

@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Dec 7, 2015

Contributor

Fixed in 23e5b1b

Contributor

moozzyk commented Dec 7, 2015

Fixed in 23e5b1b

@moozzyk moozzyk closed this Dec 7, 2015

@Shazwazza

This comment has been minimized.

Show comment
Hide comment
@Shazwazza

Shazwazza Dec 14, 2015

Is it possible that this could this fix be applied at the Owin level (not sure where that repo lives)? This will fix the issue seen in SignalR but won't fix the issue for any other Owin based package

Shazwazza commented Dec 14, 2015

Is it possible that this could this fix be applied at the Owin level (not sure where that repo lives)? This will fix the issue seen in SignalR but won't fix the issue for any other Owin based package

@MojoDK

This comment has been minimized.

Show comment
Hide comment
@MojoDK

MojoDK Dec 15, 2015

When do you guys expect this update to be included in nuget? Last nuget version is 2.2.0 (jan 13th).

MojoDK commented Dec 15, 2015

When do you guys expect this update to be included in nuget? Last nuget version is 2.2.0 (jan 13th).

@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Dec 15, 2015

Contributor

@Shazwazza - we working on this as we speak

Contributor

moozzyk commented Dec 15, 2015

@Shazwazza - we working on this as we speak

@KimTranjan

This comment has been minimized.

Show comment
Hide comment
@KimTranjan

KimTranjan Dec 21, 2015

Same problem here. Glad you guys already fixed it.
Waiting for the next version.
Thanks.

KimTranjan commented Dec 21, 2015

Same problem here. Glad you guys already fixed it.
Waiting for the next version.
Thanks.

@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Dec 21, 2015

Contributor

@KimTranjan - the credits go to @smourier and @zpqrtbnk

Contributor

moozzyk commented Dec 21, 2015

@KimTranjan - the credits go to @smourier and @zpqrtbnk

@KimTranjan

This comment has been minimized.

Show comment
Hide comment
@KimTranjan

KimTranjan Dec 21, 2015

Yeah, I did read the thread and the blog post, btw.
This community is great.

:)

edit:

I'm using @zpqrtbnk's workaround and it's working perfectly.

KimTranjan commented Dec 21, 2015

Yeah, I did read the thread and the blog post, btw.
This community is great.

:)

edit:

I'm using @zpqrtbnk's workaround and it's working perfectly.

@MosheL

This comment has been minimized.

Show comment
Hide comment
@MosheL

MosheL Jan 31, 2016

I can see the problem also. when we can see the update in nuget ?

thanks.

MosheL commented Jan 31, 2016

I can see the problem also. when we can see the update in nuget ?

thanks.

@woutercx

This comment has been minimized.

Show comment
Hide comment
@woutercx

woutercx Feb 6, 2016

I have the problem too. I tried using the Unofficial nuget package, but this messes up my packages. I'd rather use the official release. When will this be made official?

woutercx commented Feb 6, 2016

I have the problem too. I tried using the Unofficial nuget package, but this messes up my packages. I'd rather use the official release. When will this be made official?

@flori27

This comment has been minimized.

Show comment
Hide comment
@flori27

flori27 Sep 14, 2016

I have SignalR version 2.2.1 from nuget and I'm using the solution from zpqrtbnk (https://www.zpqrtbnk.net/posts/appdomains-threads-cultureinfos-and-paracetamol) as well, but the problem still occurrs.
I'm using windows 10 with IIS 10

flori27 commented Sep 14, 2016

I have SignalR version 2.2.1 from nuget and I'm using the solution from zpqrtbnk (https://www.zpqrtbnk.net/posts/appdomains-threads-cultureinfos-and-paracetamol) as well, but the problem still occurrs.
I'm using windows 10 with IIS 10

@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Sep 14, 2016

Contributor

@flori27 - 2.2.1 should have this workaround (23e5b1b). The issue is that we have not been able to repro this issue ever so we took the fix hoping it will resolve the issue. If this is still occurring - how do I repro this? Is there anything special about your environment?

Contributor

moozzyk commented Sep 14, 2016

@flori27 - 2.2.1 should have this workaround (23e5b1b). The issue is that we have not been able to repro this issue ever so we took the fix hoping it will resolve the issue. If this is still occurring - how do I repro this? Is there anything special about your environment?

@MosheL

This comment has been minimized.

Show comment
Hide comment
@MosheL

MosheL Sep 14, 2016

Just to note - The problem is only on Windows 10, as I think. In Windows 7 or 2008R2 I do not see it.

MosheL commented Sep 14, 2016

Just to note - The problem is only on Windows 10, as I think. In Windows 7 or 2008R2 I do not see it.

@moozzyk

This comment has been minimized.

Show comment
Hide comment
@moozzyk

moozzyk Sep 14, 2016

Contributor

@MosheL - I did try Windows 10 (as well as Win 2008 R2) and could not repro this. Is there anything specific about your environment? One of the theories is that it happens if the OS/Thread is not using english culture. I did try non-english versions of both Win10 and Win2008R2 and still could not repro. Do you have any specific language settings on your Windows 10 machine where this happens?

Contributor

moozzyk commented Sep 14, 2016

@MosheL - I did try Windows 10 (as well as Win 2008 R2) and could not repro this. Is there anything specific about your environment? One of the theories is that it happens if the OS/Thread is not using english culture. I did try non-english versions of both Win10 and Win2008R2 and still could not repro. Do you have any specific language settings on your Windows 10 machine where this happens?

@MosheL

This comment has been minimized.

Show comment
Hide comment
@MosheL

MosheL Sep 14, 2016

I am using Hebrew-modified one, and also with web.config with culture (en-GB).

MosheL commented Sep 14, 2016

I am using Hebrew-modified one, and also with web.config with culture (en-GB).

@flori27

This comment has been minimized.

Show comment
Hide comment
@flori27

flori27 Sep 15, 2016

Hi,

In IIS, I have 2 bindings for my website
ts.localhost.comhttp://ts.localhost.com (configured in VS to start on this URL and points to my localhost)
pcwork.no-ip.bizhttp://pcwork.no-ip.biz (points to my internal IP address)

Other local computers are connected to pcwork.no-ip.bizhttp://pcwork.no-ip.biz.
When I use this url, IIS hangs.
I don't have the problem ts.localhost.comhttp://ts.localhost.com, but other devices don't know this url, so I need to use pcwork.no-ip.bizhttp://pcwork.no-ip.biz

Op 14 sep. 2016, om 18:04 heeft Pawel Kadluczka <notifications@github.commailto:notifications@github.com> het volgende geschreven:

@flori27https://github.com/flori27 - 2.2.1 should have this workaround (23e5b1bhttps://github.com/SignalR/SignalR/commit/23e5b1b5bcceec097cc4e4e0f32fd00303ad8746). The issue is that we have not been able to repro this issue ever so we took the fix hoping it will resolve the issue. If this is still occurring - how do I repro this? Is there anything special about your environment?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/SignalR/SignalR/issues/3414#issuecomment-247063874, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIXr4eIqMTUTW9K5Zri-OLbKuA5oOqOks5qqBr6gaJpZM4DkCkp.

flori27 commented Sep 15, 2016

Hi,

In IIS, I have 2 bindings for my website
ts.localhost.comhttp://ts.localhost.com (configured in VS to start on this URL and points to my localhost)
pcwork.no-ip.bizhttp://pcwork.no-ip.biz (points to my internal IP address)

Other local computers are connected to pcwork.no-ip.bizhttp://pcwork.no-ip.biz.
When I use this url, IIS hangs.
I don't have the problem ts.localhost.comhttp://ts.localhost.com, but other devices don't know this url, so I need to use pcwork.no-ip.bizhttp://pcwork.no-ip.biz

Op 14 sep. 2016, om 18:04 heeft Pawel Kadluczka <notifications@github.commailto:notifications@github.com> het volgende geschreven:

@flori27https://github.com/flori27 - 2.2.1 should have this workaround (23e5b1bhttps://github.com/SignalR/SignalR/commit/23e5b1b5bcceec097cc4e4e0f32fd00303ad8746). The issue is that we have not been able to repro this issue ever so we took the fix hoping it will resolve the issue. If this is still occurring - how do I repro this? Is there anything special about your environment?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/SignalR/SignalR/issues/3414#issuecomment-247063874, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIXr4eIqMTUTW9K5Zri-OLbKuA5oOqOks5qqBr6gaJpZM4DkCkp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment