Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SignalR.Owin works under Windows but returns 500 for Mono on Linux #1758

Closed
desunit opened this issue Mar 27, 2013 · 20 comments

Comments

Projects
None yet
5 participants
@desunit
Copy link

commented Mar 27, 2013

Just found strange behavior with the latest code of SignalR (self hosting, owin). It works perfectly in Windows box but returns error 500 without any explanation. Fiddler returns:

HTTP/1.1 500 Internal Server Error
Date: Wed, 27 Mar 2013 13:14:27 GMT
Server: nginx/1.1.19
Content-Length: 0
Via: 1.1 host.com
Connection: close
Content-Type: text/plain

I couldn't find how to get error details of that error, that's why modified PersistentConnectionHandler.Invoke with:

...
            try
            {
                return handler.Invoke(environment);
            }
            catch (Exception e)
            {
                Debug.WriteLine(e.Message, e.ToString());
                throw;
            }

and realized the cause of that error:

System.Security.Cryptography.CryptographicException: Data protection failed. ---> System.Security.Cryptography.CryptographicException: Improperly protected user's key pairs in '/home/user/.config/.mono/keypairs'.
  at Mono.Security.Cryptography.KeyPairPersistence.get_UserPath () [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.KeyPairPersistence.get_Filename () [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.KeyPairPersistence.Load () [0x00000] in <filename unknown>:0
  at System.Security.Cryptography.RSACryptoServiceProvider.Common (Int32 dwKeySize, System.Security.Cryptography.CspParameters p) [0x00000] in <filename unknown>:0
  at System.Security.Cryptography.RSACryptoServiceProvider..ctor (Int32 dwKeySize, System.Security.Cryptography.CspParameters parameters) [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.ManagedProtection.GetKey (DataProtectionScope scope) [0x00000] in <filename unknown>:0
  at Mono.Security.Cryptography.ManagedProtection.Protect (System.Byte[] userData, System.Byte[] optionalEntropy, DataProtectionScope scope) [0x00000] in <filename unknown>:0
  at System.Security.Cryptography.ProtectedData.Protect (System.Byte[] userData, System.Byte[] optionalEntropy, DataProtectionScope scope) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at System.Security.Cryptography.ProtectedData.Protect (System.Byte[] userData, System.Byte[] optionalEntropy, DataProtectionScope scope) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Infrastructure.DefaultProtectedData.Protect (System.String data, System.String purpose) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0 : Data protection failed.

and another exception:

System.InvalidOperationException: Operation is not valid due to the current state of the object
  at System.Diagnostics.PerformanceCounter.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.Diagnostics.PerformanceCounter:set_RawValue (long)
  at Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterWrapper.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.TransportHeartbeat.AddConnection (ITrackingConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessConnectRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0 : Operation is not valid due to the current state of the object

I haven't seen that issue in v1.0. It's appeared only with trunk build. I'll try to provide pull request with couple other Mono related fixes but I really don't like the fact that I can't see error details without code modification. Have I overlooked anything?

@desunit

This comment has been minimized.

Copy link
Author

commented Mar 27, 2013

For the record - it appeared that .config/.mono/keypairs should have the following rights:

POSIX
- User
    - 0700 for directory
    - 0600 for key pairs (files)
- Machine:
    - 0755 for directory
    - 0644 for key pairs (files)

http://lists.ximian.com/pipermail/mono-devel-list/2004-April/005063.html

Though, the question how to identify the cause of the error without modifying code is still open...

@davidfowl

This comment has been minimized.

Copy link
Member

commented Mar 27, 2013

What version of mono? It works fine when I last tried it.

@desunit

This comment has been minimized.

Copy link
Author

commented Mar 27, 2013

Mono JIT compiler version 2.11.4 (tarball Tue Oct 16 17:20:43 EEST 2012)
Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            Included Boehm (with typed GC and Parallel Mark)

What about tracing issues? What you can suggest for logging exceptions instead of modifying PersistentConnectionHandler.Invoke?

@davidfowl

This comment has been minimized.

Copy link
Member

commented Mar 27, 2013

Can you get a managed call stack? What OS? It works on OSX for me

@desunit

This comment has been minimized.

Copy link
Author

commented Mar 27, 2013

What do you mean? I've included it to a task and explained that had to modify PersistentConnectionHandler.Invoke method to get it - without that modifications I only see HTTP error code "500" and no stack trace or error description...

OS: Linux 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

@davidfowl

This comment has been minimized.

Copy link
Member

commented Mar 28, 2013

By default the katana self host has no middleware to show exceptions and the only way to see them today is to use something like http://nuget.org/packages/Gate.Middleware/. You can add the UseShowExceptions middleware before SignalR and exceptions should start showing up.

@davidfowl

This comment has been minimized.

Copy link
Member

commented Mar 28, 2013

@desunit before sending any mono fixes please read the contribution guidelines https://github.com/SignalR/SignalR/blob/master/CONTRIBUTING.md#contributing-code

@paulofbr

This comment has been minimized.

Copy link

commented Mar 29, 2013

I build signalr from source (under linux) and the problem disapear

@desunit

This comment has been minimized.

Copy link
Author

commented Apr 1, 2013

I've workaround issue with that exception:

System.InvalidOperationException: Operation is not valid due to the current state of the object
  at System.Diagnostics.PerformanceCounter.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.Diagnostics.PerformanceCounter:set_RawValue (long)
  at Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterWrapper.set_RawValue (Int64 value) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.TransportHeartbeat.AddConnection (ITrackingConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessConnectRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessRequest (ITransportConnection connection) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00000] in <filename unknown>:0 : Operation is not valid due to the current state of the object

Modifying PerformanceCounterManager.cs so you can disable counters. I think the problem with mono that counters are not stable enough and works differently as under Windows. The best solution could be to provide ability in config to replace it with empty implementation... though that also means DefaultDependencyResolver.cs should be configurable because of:

            var perfCounterWriter = new Lazy<PerformanceCounterManager>(() => new PerformanceCounterManager(this));
            Register(typeof(IPerformanceCounterManager), () => perfCounterWriter.Value);

Hope that helps.

@davidfowl

This comment has been minimized.

Copy link
Member

commented Apr 1, 2013

I can ifdef this out for mono builds or just try catch. Not going to add a configuration switch for this.

@davidfowl

This comment has been minimized.

Copy link
Member

commented Apr 3, 2013

@desunit what branch were you using? dev or master?

@desunit

This comment has been minimized.

Copy link
Author

commented Apr 3, 2013

I was using master.

@davidfowl

This comment has been minimized.

Copy link
Member

commented Apr 3, 2013

Can you try using the dev branch? We don't develop in master so this bug might not be an issue in dev.

@desunit

This comment has been minimized.

Copy link
Author

commented Apr 3, 2013

OK, I will try to setup test environment and get back to you with results.

@mloenow

This comment has been minimized.

Copy link

commented May 21, 2013

@davidfowl I believe I'm having the same issue on Mono 3.0.10. I can reproduce this on master, but not on the current dev branch as it uses Microsoft.Owin.Security which uses System.Security.Claims.* and System.Security.Cryptography.DpapiDataProtector etc. — AFAIK those aren't available on Mono (yet?).

Here's the stack trace (master):

in /private/tmp/source/bockbuild-crypto-mono/profiles/mono-mac-xamarin/build-root/mono-3.0.10/mcs/class/System/System.Diagnostics/PerformanceCounter.cs:262 
  at (wrapper remoting-invoke-with-check) System.Diagnostics.PerformanceCounter:set_RawValue (long)
  at Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterWrapper.set_RawValue (Int64 value) [0x00001] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Infrastructure/PerformanceCounterWrapper.cs:28 
  at Microsoft.AspNet.SignalR.Transports.TransportHeartbeat.AddConnection (ITrackingConnection connection) [0x000ac] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Transports/TransportHeartBeat.cs:128 
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessReceiveRequest (ITransportConnection connection) [0x00010] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Transports/LongPollingTransport.cs:177 
  at Microsoft.AspNet.SignalR.Transports.LongPollingTransport.ProcessRequest (ITransportConnection connection) [0x0004a] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Transports/LongPollingTransport.cs:141 
  at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00213] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/PersistentConnection.cs:227 
  at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest (Microsoft.AspNet.SignalR.Hosting.HostContext context) [0x00098] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Core/Hubs/HubDispatcher.cs:243 
  at Microsoft.AspNet.SignalR.Owin.CallHandler.Invoke (IDictionary`2 environment) [0x00186] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Owin/Handlers/CallHandler.cs:74 
  at Microsoft.AspNet.SignalR.Owin.Handlers.HubDispatcherHandler.Invoke (IDictionary`2 environment) [0x00050] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Owin/Handlers/HubDispatcherHandler.cs:37 
  at Microsoft.AspNet.SignalR.Owin.Handlers.PersistentConnectionHandler.Invoke (IDictionary`2 environment) [0x00025] in /Users/*SNIPPED*/Test/SignalR/src/Microsoft.AspNet.SignalR.Owin/Handlers/PersistentConnectionHandler.cs:33 
  at Gate.Middleware.ShowExceptions+&lt;Middleware&gt;c__AnonStorey3C.&lt;&gt;m__53 (IDictionary`2 env) [0x00090] in /Users/*SNIPPED*/Test/gate/src/Main/Gate.Middleware/ShowExceptions.cs:69

I tried disabling counters like @desunit did, but that throws an ArgumentNullException in MessageBus.cs. Is there another way to disable counters or is another branch you'd like me to debug this on?

@davidfowl

This comment has been minimized.

Copy link
Member

commented May 22, 2013

Fixed in dev 57e1c98

@davidfowl davidfowl closed this May 22, 2013

@mloenow

This comment has been minimized.

Copy link

commented May 22, 2013

Awesome work, @davidfowl! Although, I hope to see a compilation symbol for this in the future, as it won't compile from source with Mono, perhaps defined in a new Microsoft.AspNet.SignalR.Core.Mono.csproj — or, if possible, move the DpapiDataProtector dependency to a seperate project e.g. Microsoft.AspNet.SignalR.Core.Security.

@davidfowl

This comment has been minimized.

Copy link
Member

commented May 22, 2013

@mloenow not sure what you mean. It builds fine under mono and we won't be adding any projects specifically for mono.

@ionelmoisuc

This comment has been minimized.

Copy link

commented May 22, 2013

The only version that works under Mono on Linux is 1.0.0. All later versions do not, as I have signaled a few times. Is there anyone that can give it a test and fix it ?

@davidfowl

This comment has been minimized.

Copy link
Member

commented May 22, 2013

@ionelmoisuc I just fixed the build in the dev branch yesterday. It requires mono 3.0.10 to run and works fine now. If you have specific issues please file bugs but the bugs I closed are what I fixed. I'm running mono on osx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.