SignalR.Owin works under Windows but returns 500 for Mono on Linux #1758

Closed
desunit opened this Issue Mar 27, 2013 · 20 comments

Projects

None yet

5 participants

desunit commented Mar 27, 2013

Just found strange behavior with the latest code of SignalR (self hosting, owin). It works perfectly in Windows box but returns error 500 without any explanation. Fiddler returns:

HTTP/1.1 500 Internal Server Error
Date: Wed, 27 Mar 2013 13:14:27 GMT
Server: nginx/1.1.19
Content-Length: 0
Via: 1.1 host.com
Connection: close
Content-Type: text/plain

I couldn't find how to get error details of that error, that's why modified PersistentConnectionHandler.Invoke with:

...
            try
            {
                return handler.Invoke(environment);
            }
            catch (Exception e)
            {
                Debug.WriteLine(e.Message, e.ToString());
                throw;
            }

and realized the cause of that error:

System.Security.Cryptography.CryptographicException: Data protection failed. ---> System.Security.Cryptography.CryptographicException: Improperly protected user's key pairs in '/home/user/.config/.mono/keypairs'.
  at Mono.Security.Cryptography.KeyPairPersistence.get_UserPath () [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.KeyPairPersistence.get_Filename () [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.KeyPairPersistence.Load () [0x00000] in <filename unknown>:0
  at System.Security.Cryptography.RSACryptoServiceProvider.Common (Int32 dwKeySize, System.Security.Cryptography.CspParameters p) [0x00000] in <filename unknown>:0
  at System.Security.Cryptography.RSACryptoServiceProvider..ctor (Int32 dwKeySize, System.Security.Cryptography.CspParameters parameters) [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.ManagedProtection.GetKey (DataProtectionScope scope) [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.ManagedProtection.Protect (System.Byte[] userData, System.Byte[] optionalEntropy, DataProtectionScope scope) [0x00000] in <filename unknown>:0
  at System.Security.Cryptography.ProtectedData.Protect (System.Byte[] userData, System.Byte[] optionalEntropy, DataProtectionScope scope) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at System.Security.Cryptography.ProtectedData.Protect (System.Byte[] userData, System.Byte[] optionalEntropy, DataProtectionScope scope) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Infrastructure.DefaultProtectedData.Protect (System.String data, System.String purpose) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0 : Data protection failed.

and another exception:

System.InvalidOperationException: Operation is not valid due to the current state of the object
  at System.Diagnostics.PerformanceCounter.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.Diagnostics.PerformanceCounter:set_RawValue (long)
  at Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterWrapper.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.TransportHeartbeat.AddConnection (ITrackingConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessConnectRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0 : Operation is not valid due to the current state of the object

I haven't seen that issue in v1.0. It's appeared only with trunk build. I'll try to provide pull request with couple other Mono related fixes but I really don't like the fact that I can't see error details without code modification. Have I overlooked anything?

desunit commented Mar 27, 2013

For the record - it appeared that .config/.mono/keypairs should have the following rights:

POSIX
- User
    - 0700 for directory
    - 0600 for key pairs (files)
- Machine:
    - 0755 for directory
    - 0644 for key pairs (files)

http://lists.ximian.com/pipermail/mono-devel-list/2004-April/005063.html

Though, the question how to identify the cause of the error without modifying code is still open...

Owner

What version of mono? It works fine when I last tried it.

desunit commented Mar 27, 2013
Mono JIT compiler version 2.11.4 (tarball Tue Oct 16 17:20:43 EEST 2012)
Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            Included Boehm (with typed GC and Parallel Mark)

What about tracing issues? What you can suggest for logging exceptions instead of modifying PersistentConnectionHandler.Invoke?

Owner

Can you get a managed call stack? What OS? It works on OSX for me

desunit commented Mar 27, 2013

What do you mean? I've included it to a task and explained that had to modify PersistentConnectionHandler.Invoke method to get it - without that modifications I only see HTTP error code "500" and no stack trace or error description...

OS: Linux 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Owner

By default the katana self host has no middleware to show exceptions and the only way to see them today is to use something like http://nuget.org/packages/Gate.Middleware/. You can add the UseShowExceptions middleware before SignalR and exceptions should start showing up.

Owner

@desunit before sending any mono fixes please read the contribution guidelines https://github.com/SignalR/SignalR/blob/master/CONTRIBUTING.md#contributing-code

I build signalr from source (under linux) and the problem disapear

desunit commented Apr 1, 2013

I've workaround issue with that exception:

System.InvalidOperationException: Operation is not valid due to the current state of the object
  at System.Diagnostics.PerformanceCounter.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.Diagnostics.PerformanceCounter:set_RawValue (long)
  at Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterWrapper.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.TransportHeartbeat.AddConnection (ITrackingConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessConnectRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0 : Operation is not valid due to the current state of the object

Modifying PerformanceCounterManager.cs so you can disable counters. I think the problem with mono that counters are not stable enough and works differently as under Windows. The best solution could be to provide ability in config to replace it with empty implementation... though that also means DefaultDependencyResolver.cs should be configurable because of:

            var perfCounterWriter = new Lazy<PerformanceCounterManager>(() => new PerformanceCounterManager(this));
            Register(typeof(IPerformanceCounterManager), () => perfCounterWriter.Value);

Hope that helps.

Owner

I can ifdef this out for mono builds or just try catch. Not going to add a configuration switch for this.

Owner

@desunit what branch were you using? dev or master?

desunit commented Apr 3, 2013

I was using master.

Owner

Can you try using the dev branch? We don't develop in master so this bug might not be an issue in dev.

desunit commented Apr 3, 2013

OK, I will try to setup test environment and get back to you with results.

mloenow commented May 21, 2013

@davidfowl I believe I'm having the same issue on Mono 3.0.10. I can reproduce this on master, but not on the current dev branch as it uses Microsoft.Owin.Security which uses System.Security.Claims.* and System.Security.Cryptography.DpapiDataProtector etc. — AFAIK those aren't available on Mono (yet?).

Here's the stack trace (master):

in /private/tmp/source/bockbuild-crypto-mono/profiles/mono-mac-xamarin/build-root/mono-3.0.10/mcs/class/System/System.Diagnostics/PerformanceCounter.cs:262 
  at (wrapper remoting-invoke-with-check) System.Diagnostics.PerformanceCounter:set_RawValue (long)
  at Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterWrapper.set_RawValue (Int64 value) [0x00001] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Infrastructure/PerformanceCounterWrapper.cs:28 
  at Microsoft.AspNet.SignalR.Transports.TransportHeartbeat.AddConnection (ITrackingConnection connection) [0x000ac] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Transports/TransportHeartBeat.cs:128 
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessReceiveRequest (ITransportConnection connection) [0x00010] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Transports/LongPollingTransport.cs:177 
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessRequest (ITransportConnection connection) [0x0004a] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Transports/LongPollingTransport.cs:141 
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00213] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/PersistentConnection.cs:227 
  at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00098] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Hubs/HubDispatcher.cs:243 
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00186] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Owin/Handlers/CallHandler.cs:74 
  at Microsoft.AspNet.SignalR.Owin.Handlers.HubDispatcherHandler.Invoke (IDictionary`2 environment) [0x00050] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Owin/Handlers/HubDispatcherHandler.cs:37 
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00025] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Owin/Handlers/PersistentConnectionHandler.cs:33 
  at Gate.Middleware.ShowExceptions+&lt;Middleware&gt;c__AnonStorey3C.&lt;&gt;m__53 (IDictionary`2 env) [0x00090] in /Users/*SNIPPED*/Test/gate/src/Main/Gate.Middleware/ShowExceptions.cs:69

I tried disabling counters like @desunit did, but that throws an ArgumentNullException in MessageBus.cs. Is there another way to disable counters or is another branch you'd like me to debug this on?

Owner

Fixed in dev 57e1c98

@davidfowl davidfowl closed this May 22, 2013
mloenow commented May 22, 2013

Awesome work, @davidfowl! Although, I hope to see a compilation symbol for this in the future, as it won't compile from source with Mono, perhaps defined in a new Microsoft.AspNet.SignalR.Core.Mono.csproj — or, if possible, move the DpapiDataProtector dependency to a seperate project e.g. Microsoft.AspNet.SignalR.Core.Security.

Owner

@mloenow not sure what you mean. It builds fine under mono and we won't be adding any projects specifically for mono.

The only version that works under Mono on Linux is 1.0.0. All later versions do not, as I have signaled a few times. Is there anyone that can give it a test and fix it ?

Owner

@ionelmoisuc I just fixed the build in the dev branch yesterday. It requires mono 3.0.10 to run and works fine now. If you have specific issues please file bugs but the bugs I closed are what I fixed. I'm running mono on osx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment