Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

weird errors from MessageQueueException #566

Closed
stevengopmo opened this issue Nov 28, 2016 · 22 comments
Closed

weird errors from MessageQueueException #566

stevengopmo opened this issue Nov 28, 2016 · 22 comments

Comments

@stevengopmo
Copy link

Running in VS 2015, I continually see the following in my output window.

Event 10 was called with 5 argument(s) , but it is defined with 6 paramenter(s). Exception thrown: 'System.Messaging.MessageQueueException' in System.Messaging.dll

the event 10 part can be different (e.g. Event 7), but it always follows immediately with "Exception thrown". After a certain number of these, my debugging session shuts down. Any ideas?

IF you can point me to how I can get the specific error from the MessageQueueException I'll do it, but it seems I read somewhere this is what happens when there are no messages in the queue.

@mookid8000
Copy link
Member

That sounds weird.

It is normal that the debugger periodically detects MessageQueueExceptions in the background, but those are handled by Rebus and would be ignored completely if the debugger was no so eager.

I have never seen that particular exception message before though, so I am curious to know what caused it.

First off: Is there any more information in that exception message you're seeing in the output windows, or is it just that single line of information?

Second thing: Is your program a Console Application? If so, can you show me the structure of the Main method?

@stevengopmo
Copy link
Author

  1. the message is actually 2 lines
    Event 7 was called with 5 argument(s) , but it is defined with 6 paramenter(s).
    and also
    Exception thrown: 'System.Messaging.MessageQueueException' in System.Messaging.dll

Interestingly, I get the first message even when Rebus is all commented out (reference still exists), but not the Exception and it stops once the application has started up. If I leave my Rebus configuration in place, I continue to get both messages until the debugger shuts down (either because of my actions or the eventual shut down). So it seems they are at least partially related (MessageQueueException and Event message)

The application is a WebApi application and here is my configuration.

        Configure
            .With(adapter)
            .Options(o =>
            {
                o.SimpleRetryStrategy("Messages.Errors");
                o.LogPipeline(true);
            })
            .Serialization(s =>
            {
                var reSettings = new JsonSerializerSettings
                {
                    TypeNameHandling = TypeNameHandling.All,
                    NullValueHandling = NullValueHandling.Include,
                };
                reSettings.ConfigureForNodaTime(DateTimeZoneProviders.Tzdb);
                reSettings.Converters.Add(new Newtonsoft.Json.Converters.StringEnumConverter());
                s.UseNewtonsoftJson(reSettings);
            })
            .Logging(l => l.Serilog(Serilog.Log.Logger))
            .Transport(t => t.UseMsmq("Messages"))
            .Subscriptions(s => s.UseJsonFile(jsonSubcriptionStorage))
            .Start();

@mookid8000
Copy link
Member

Interestingly, I get the first message even when Rebus is all commented out (reference still exists), but not the Exception and it stops once the application has started up.

Sound to me like the Event 7 was called with 5 argument(s) , but it is defined with 6 paramenter(s). message is not related to Rebus at all....

I know the Exception thrown: 'System.Messaging.MessageQueueException' in System.Messaging.dll message, it is completely natural when you are running something MSMQ with the debugger attached.

But I have never seen that other message.

What other things are you using in your application?

How long does the application keep running with the debugger attached when you start it?

Does it still shut down if you comment out the Rebus bits?

@stevengopmo
Copy link
Author

But I have never seen that other message.
sorry but what message have you not seen? The two messages I receive are

Event 10 was called with 5 argument(s) , but it is defined with 6 paramenter(s).
and
Exception thrown: 'System.Messaging.MessageQueueException' in System.Messaging.dll

The Event 10 portion changes over time (event 7, or 6, etc.) Those messages continue to repeat as long as my app is in debug mode. I haven't measured the exact time it runs until shutdown, but I will attempt to capture today.

I'm using a few other 'things' like SimpleInjector, Serilog, etc. (there's quite a long list). If I comment out the Rebus bits the VS does NOT shut down, it only does this when I have Rebus configured as above.

@mookid8000
Copy link
Member

That's weird. Can you check which Rebus package versions you are using?

You can just post the full package.config here if you like 😄

@stevengopmo
Copy link
Author

<packages>
  <package id="BrockAllen.MembershipReboot" version="8.4.0" targetFramework="net46" />
  <package id="FluentValidation" version="6.2.1.0" targetFramework="net461" />
  <package id="goPMO.Common.Domain" version="2016.9.30.36" targetFramework="net461" />
  <package id="goPMO.Common.RavenDB" version="2016.11.20.121" targetFramework="net461" />
  <package id="goPMO.Common.Validation" version="2016.9.30.120" targetFramework="net461" />
  <package id="goPMO.Common.WebApi" version="2016.10.4.131" targetFramework="net461" />
  <package id="IdentityModel" version="1.9.2" targetFramework="net461" />
  <package id="IdentityServer3.AccessTokenValidation" version="2.11.0" targetFramework="net461" />
  <package id="JetBrains.Annotations" version="10.2.1" targetFramework="net461" />
  <package id="LibLog" version="4.2.6" targetFramework="net461" developmentDependency="true" />
  <package id="Microsoft.AspNet.Cors" version="5.2.3" targetFramework="net461" />
  <package id="Microsoft.AspNet.WebApi.Client" version="5.2.3" targetFramework="net46" />
  <package id="Microsoft.AspNet.WebApi.Core" version="5.2.3" targetFramework="net46" />
  <package id="Microsoft.AspNet.WebApi.Cors" version="5.2.3" targetFramework="net461" />
  <package id="Microsoft.AspNet.WebApi.Owin" version="5.2.3" targetFramework="net461" />
  <package id="Microsoft.IdentityModel.Protocol.Extensions" version="1.0.3.308261200" targetFramework="net461" />
  <package id="Microsoft.Owin" version="3.0.1" targetFramework="net461" />
  <package id="Microsoft.Owin.Cors" version="3.0.1" targetFramework="net461" />
  <package id="Microsoft.Owin.Security" version="3.0.1" targetFramework="net461" />
  <package id="Microsoft.Owin.Security.Jwt" version="3.0.1" targetFramework="net461" />
  <package id="Microsoft.Owin.Security.OAuth" version="3.0.1" targetFramework="net461" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net46" />
  <package id="NodaTime" version="1.3.2" targetFramework="net461" />
  <package id="NodaTime.Serialization.JsonNet" version="1.3.2" targetFramework="net461" />
  <package id="Owin" version="1.0" targetFramework="net461" />
  <package id="RavenDB.Client" version="3.0.30155" targetFramework="net461" />
  <package id="RavenDB.Client.NodaTime" version="3.0.0" targetFramework="net461" />
  <package id="RavenDB.Client.UniqueConstraints" version="3.0.30155" targetFramework="net461" />
  <package id="Rebus" version="2.1.3" targetFramework="net461" />
  <package id="Rebus.Msmq" version="2.0.1" targetFramework="net461" />
  <package id="Rebus.NewtonsoftJson" version="2.1.3" targetFramework="net461" />
  <package id="Rebus.Serilog" version="2.0.0" targetFramework="net461" />
  <package id="Rebus.SimpleInjector" version="2.0.0" targetFramework="net461" />
  <package id="Serilog" version="2.3.0" targetFramework="net461" />
  <package id="SimpleInjector" version="3.3.0" targetFramework="net461" />
  <package id="SimpleInjector.Extensions.ExecutionContextScoping" version="3.3.0" targetFramework="net461" />
  <package id="SimpleInjector.Integration.WebApi" version="3.3.0" targetFramework="net461" />
  <package id="System.IdentityModel.Tokens.Jwt" version="4.0.3.308261200" targetFramework="net461" />
</packages>

@stevengopmo
Copy link
Author

just as I was posting to you that it seemed to go away, VS shutdown again. I would guess it was between 10 and 12 minutes between the start and the shutdown.

There's nothing in the logs in terms of errors. I have the logs set to Warning and there's not even any warnings.

@stevengopmo
Copy link
Author

Investigations continue, I swapped to SQL Transport and I've been running 96+ minutes without any shutdown. The MSMQ transport is still shutting down after 10-12 minutes. I'm going to switch my apps to SQL, because of the other benefits discussed in a different thread but thought you should know about the MSMQ issue.

@SvenVandenbrande
Copy link
Contributor

Is it a small application? Or can you reproduce it in a smaller application?
That's also maybe easier to find the problem. So it's possible to run the same application as you to check the problem.

You shouldn't change the transport because it's not working @stevengopmo .
Although didn't check your other thread.

@stevengopmo
Copy link
Author

We are changing because the better timeout manager features, however I will try and take some time to replicate this issue in a smaller application when I have some time.

@mookid8000
Copy link
Member

If you can reproduce the issue in a piece of code that you can send to me, I will be happy to figure out what this is about.

@mookid8000
Copy link
Member

@stevengopmo any news about this issue?

@stevengopmo
Copy link
Author

stevengopmo commented Dec 15, 2016 via email

@mookid8000
Copy link
Member

I am on vacation this week and will work on a sample when I return.

Awesome, thanks 😄 have a nice vacation! 🌞

@stevengopmo
Copy link
Author

so of course my sample console app that I intend to replicate this issue with so far does not have the problem... SIGH! I will continue to add stuff from my real application until I find what is causing the problem, but it will take a while probably to find. If you want to close for now go ahead and I'll reply when I have more information.

@mookid8000
Copy link
Member

bummer dude 😄

I have never myself experienced the issue you are experiencing, and I have not had anyone report anything like it to me. At this point I will assume that it is somehow something else that is tricking you, so I really hope you can narrow it down and figure out what it is.

In case you figure out what it is, I am curious to know about it. For now, I will close this issue.

@danatcofo
Copy link

danatcofo commented Feb 15, 2017

This same thing is being reported as occurring with App Insights and Stackify Prefix where they can have this message start popping up and go away depending upon whether the profilers are active. Its not an exception in those cases, just a debug message being outputted.

My current running thought is that this is a setting that is set by the profilers to increase logging output or detail.

http://stackoverflow.com/questions/42123222/stackify-prefix-displays-event-argument-count-issue-after-upgrade

@stevengopmo
Copy link
Author

@danatcofo thanks for this I will check it out as I am using Prefix too. I don't think it will solve my msmq problem but it will be a start

@cldons
Copy link

cldons commented Nov 26, 2017

Encountering this problem too. However not with MSMQ or SQL transports but with the Rebus.AzureStorage transport and App Insight enabled.

My output log is being spammed with calls to my-sagas/messages as can be seen below. This is an App Insight event being logged against the Azure Storage Emulator.

{
  "name": "Microsoft.ApplicationInsights.Dev.XXX.RemoteDependency",
  "time": "2017-11-26T18:49:21.4853996Z",
  "iKey": "XXX",
  "tags": {
    "ai.internal.nodeName": "XXX",
    "ai.operation.id": "XXX",
    "ai.cloud.roleInstance": "XXX",
    "ai.internal.sdkVersion": "rdddsd:2.4.1-1362"
  },
  "data": {
    "baseType": "RemoteDependencyData",
    "baseData": {
      "ver": 2,
      "name": "GET /devstoreaccount1/my-sagas/messages",
      "id": "|XXX",
      "data": "http://127.0.0.1:10001/devstoreaccount1/my-sagas/messages?numofmessages=1&visibilitytimeout=300",
      "duration": "00:00:00.0600791",
      "resultCode": "200",
      "success": true,
      "type": "Http",
      "target": "127.0.0.1:10001",
      "properties": {
        "_MS.ProcessedByMetricExtractors": "(Name:'Dependencies', Ver:'1.0')",
        "DeveloperMode": "true"
      }
    }
  }
}

I think these calls are being caused by the code inside Windows.AzureStorage. What Rebus' AzureStorage does is waiting for the next message to arrive to handle it in it's pipeline. See https://github.com/rebus-org/Rebus.AzureStorage/blob/4.0.0/Rebus.AzureStorage/AzureStorage/Transport/AzureStorageQueuesTransport.cs line 101.

However what I don't understand (from an architectural point of view) is that the GetMessageAsync is being called even for a OneWayClient. Because I am using the Rebus instance only as OneWayClient, even then these messages are being spammed.

@mookid8000 is it true that the AzureStorage transport OneWayClient is listening for incoming messages too?

Edit: striking through my wrong claims

@mookid8000
Copy link
Member

is it true that the AzureStorage transport OneWayClient is listening for incoming messages too?

it definitely SHOULDN'T be trying to receive messages when it's a one-way client... could you maybe post your Rebus configuration?

@cldons
Copy link

cldons commented Nov 26, 2017

it definitely SHOULDN'T be trying to receive messages when it's a one-way client... could you maybe post your Rebus configuration?

I was totally wrong on my claims of the OneWayClient. Guess it has been a long weekend for me, but whilst testing I had my config wrong. I can verify now that this App Insight behavior occurs only on the two-way not the one-way transports.

@mookid8000
Copy link
Member

I am happy to hear that 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants