Azure Functions v2 .NET Core - ServiceBusTrigger Message not able to read BrokeredMessages

[DiaAWAY]

Hi!

We are attempting to upgrade our old ServiceBus solutions over to use the new .Net Core packages (both Azure Functions and Azure Webjobs).

When attempting to consume from our topics we encounter the following error:
System.Private.CoreLib: Exception while executing function: Function1. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'message'. System.Private.CoreLib: Unable to translate bytes [99] at index 60 from specified code page to Unicode.

This error occurs even if we attempt to use other types, such as string, byte[] and custom types (such as the standard Message class).

The issue seems to be the following configuration:
Body-Type: string
Content-Type: text/plain

Other combinations seem to work properly (where none is empty Content-Type);
stream+none, stream+text/plain and string+none.

We may be able to circumvent this (untested) in our webjobs, as we can access MessageReceiver or other ServiceBus configurations at runtime to handle this, but so far we have been unable to find any solution or method to handle this for Azure Functions as the framework throws an error before we are inside the function.

Is this behaviour a bug or by design?

Note: we do not control the creation of the servicebus messages, so we cannot simply change how we write the messages to the servicebus.

Edit: clarified that the exception occurs even with the standard Message class.

[DiaAWAY]

Others have similar issues combining the two different packages (BrokeredMessage->Message): brokeredmessage-send-and-consumer-in-azure-function-v2

Issue also posted on stackoverflow: https://stackoverflow.com/q/53524051/7797041

[ColbyTresness]

@jeffhollan

[giorgited]

Any update on this? I have a service hook setup in Azure Devops to send message to Azure Service Bus on release events. In addition, I have azure function setup to listen to new messages on this service bus queue but its been failing. It used to work for few weeks but then stopped working suddenly without any code changes on our end.

I create a stackoverflow question for this as well https://stackoverflow.com/questions/56590094/azure-functions-message-receive-issue

[ColbyTresness]

@jeffhollan thoughts on this?

[stackdev355]

having the same problem too. Whats the status?

[jeffhollan]

I suspect the issue may be that we try to default decode the message to a string using something like Encoding.UTF8 and maybe in this case the message bytes aren't actually in that format. I think the workaround here is to use the trigger to bind to type Message instead of string -- or even byte[] -- and then decode yourself based on the format of the message. Could use some confirmation that this is in fact the case. The error message could potentially be more clear if that is in fact the case

[JamiePed]

@jeffhollan I had a similar problem to this that is really easy to replicate - I posted a stack overflow question about it here: https://stackoverflow.com/questions/56481931/azure-function-servicebustrigger-exception-binding-message

In my instance I was able to uncheck this option from the service hook that was delivering the message, but not everyone will be able to do that. In my instance changing from Message/String/Byte[] made no difference either. It appears to be something in the functions that get triggered really early on

[giorgited]

I tried Message and string but made no difference. Going to try byte[] this evening.

[JamiePed]

From my experience changing the type of the input (into the message) doesn't make any difference because the exception is thrown as soon as the message is taken from the bus.

My experience was when I was using Service Hooks from Azure DevOps. In that world there is a toggle for "Send as non-serialized string". If that option wasn't ticked I would get the issue discussed in this thread - ticking it removed the problem for me.

If the help text with that check box is to be believed then that might shine a light on why others are getting this issue for messages from other sources. It would be good if this could be togglable within the functions. The help text is below:

If checked, messages will be sent to Service Bus as non-serialized strings instead of as .NET serialized strings. This setting should be checked when the receiver is a non-.NET client, such as a client utilizing the Azure Client Library for Node.

[giorgited]

@JamiePed Same here, once I checked that box in ADO service hook window, az function was able to start reading messages again.

Not sure on which end does the bug land but hopefully this is helpful.

[SamaraSoucy-MSFT]

@jeffhollan I think I've found the issue. When reading the message, the package is using the content type specified in the message instead of the binding. DevOps is sending the message as application/json so it is getting pushed into the UTF8 decoding regardless what type the function requests.

The issue is in Microsoft.Azure.WebJobs.ServiceBus.Triggers.UserTypeArgumentBindingProvider:

 private static TInput GetBody(Message message, ValueBindingContext context)
    {
        if (message.ContentType == ContentTypes.ApplicationJson) //This is looking at the message type
        {
            string contents;

            contents = StrictEncodings.Utf8.GetString(message.Body);

            try
            {
                return JsonConvert.DeserializeObject<TInput>(contents, Constants.JsonSerializerSettings);
            }
            catch (JsonException e)
            {
                // Easy to have the queue payload not deserialize properly. So give a useful error. 
                string msg = string.Format(
@"Binding parameters to complex objects (such as '{0}') uses Json.NET serialization. 
1. Bind the parameter type as 'string' instead of '{0}' to get the raw values and avoid JSON deserialization, or
2. Change the queue payload to be valid json. The JSON parser failed: {1}
", typeof(TInput).Name, e.Message);
                throw new InvalidOperationException(msg);
            }
        }
        else
        {
            return message.GetBody<TInput>();
        }
    }
}

[jeffhollan]

Feels like this has a workaround? Still trying to digest, but P1 was mostly for investigation and I think it may still be a vaild bug but workaround is to bind to Message and decode yourself

[SamaraSoucy-MSFT]

I have tried several different bindings: Message, string, byte[], and just a plain object. Because the decision to decode is being handled based on the incoming message type it doesn't matter what I set it to, I still get the error.

The workaround is to change the type of the incoming message to something other then application/json, and that is fairly simple in the case of DevOps by checking "Send as non-serialized string". In a case where the message type is out of the user's control, I have yet to find a viable workaround.

[adrobyazko-softheme]

Do we have any update for the issue?

[alrod]

Can you please try to use:
Microsoft.Azure.WebJobs.Extensions.ServiceBus 3.1.0-beta4-batch from: https://www.myget.org/feed/azure-appservice/package/nuget/Microsoft.Azure.WebJobs.Extensions.ServiceBus

It looks like the issue is fixed there.

[PlymouthRock]

Tried to install the beta package. Won't install because dependent package Microsoft.Azure.WebJobs.Sources is missing from NuGet.

[arunhallan]

I get the same issue too when trying to pull down 3.1.0-beta4

[alrod]

Sorry for that. I added https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Sources

[marioleed]

Still doesn't work for me in 3.1.0-beta4.

[remartens]

The Microsoft.Azure.WebJobs.Extensions.ServiceBus 3.1.0-beta4-batch from myget.org seems to do the trick. Any idea when this version will be released on NuGet?

[remartens]

Just after posting the last post, I noticed there is a new version on NuGet.... 3.1.0, but this version doesn't work. I guess it is based on the 3.10-beta4 from NuGet and not the 3.1.0-beta4-batch version on MyGet.

[alrod]

Just after posting the last post, I noticed there is a new version on NuGet.... 3.1.0, but this version doesn't work. I guess it is based on the 3.10-beta4 from NuGet and not the 3.1.0-beta4-batch version on MyGet.

@remartens, you are right. Please use Microsoft.Azure.WebJobs.Extensions.ServiceBus 3.1.0-beta4-batch from MyGet as workaround for now.

[marioleed]

When do you plan to release the 3.1.0-beta4-batch version on MyGet to NuGet?

[amandal1810]

version 3.1.1 is now on NuGet and upgrading to this resolved the issue 😄

[alrod]

@DiaAWAY, is 3.1.1 fixed the issue for you?

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[DiaAWAY]

@alrod I'll check as soon as I am able :)

Update:
Unfortunately new version didn't fix the issue as far as I can see, more info in the next comment.

[DiaAWAY]

@alrod
Upgrading Microsoft.Azure.WebJobs.Extensions.ServiceBus to 3.1.1 did not change the issue :(
Still unable to consume service bus messages between these two service bus libraries.

Sending:

var client = TopicClient.CreateFromConnectionString(conStr);
client.Send(new BrokeredMessage("test")
    {
        ContentType = "text/plain"
    }
);

Package used when sending:

• WIndowsAzure.ServiceBus 3.1.1 (same result with 6.0.0)

Receiving:

public static class Function1
{
    [FunctionName("Function1")]
    public static void Run(
        [ServiceBusTrigger("mytopic", "mysubscription", Connection = "SbConStr")] 
        Message mySbMsg, 
        ILogger log)
    {
        log.LogInformation($"C# ServiceBus topic trigger function processed message: {mySbMsg.ToString()}");
    }
}

Packages used when receiving:

• Microsoft.Azure.ServiceBus 3.4.0
• Microsoft.Azure.WebJobs 3.0.10
• Microsoft.Azure.WebJobs.Extensions.ServiceBus 3.1.1
• Microsoft.NET.Sdk.Functions 1.0.28

All of this was created using Visual Studio 2019 templates (.net 471 console for brokeredmessage, core 2.1 for functions).

Azure functions runtime:
Azure Functions Core Tools (2.7.1846 Commit hash: 458c671341fda1c52bd46e1aa8943cb26e467830)
Function Runtime Version: 2.0.12858.0

Note:
To run WindowsAzure.ServiceBus 3.1.1 you need to add the following to the App.config file otherwise it will crash on runtime.

<runtime>
  <AppContextSwitchOverrides value="Switch.System.IdentityModel.DisableMultipleDNSEntriesInSANCertificate=true" />
</runtime>

[remartens]

So when is this going to be fixed? Since November 19th there is version 4.0.0 of Microsoft.Azure.WebJobs.Extensions.ServiceBus on NuGet and it is still not working.

[alrod]

@DiaAWAY, the issue should be fixed in 4.1.0. Can you please check this?

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[lars-nystrom-mio]

@alrod With 4.1.0 it seems to be failing when ContentType is empty (null) with the following message when it tries to deserialize to Message:
System.Private.CoreLib: Exception while executing function: intimeresponse. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'mySbMsg'. Microsoft.Azure.WebJobs.ServiceBus: The Message with ContentType 'null' failed to deserialize to a string with the message: 'There was an error deserializing the object of type System.String. The input source is not correctly formatted.'. System.Private.DataContractSerialization: There was an error deserializing the object of type System.String. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted.

The Run function:

[FunctionName("myfunction")]
public static async Task Run(
    [ServiceBusTrigger("mytopic", "mysubscription", Connection = "sbconnection")]Message mySbMsg,
    ILogger log)       
{
    ...
}

The above still works in version 4.0.0.

[grahambunce]

@alrod To chip in on this

This is definitely not fixed and appears to be completely broken unless you use the latest .NET clients at both ends of the pipe.

My Java clients are sending messages to Azure Service Bus - they set a content-type of "application/x-gzip". We cannot change this. These messages are gzipped before sending to the bus to save space (they can be large and blow out the size limitations unless we zip them).

My .NET Framework 4.7.x clients could read these fine and then we ran it through an unzip routine to get the content. No issues. We didn't use the Web Job framework as such but we did use the Azure Service Bus SDK.

This custom subscriber framework didn't port to .NET Core so we decided to use Web Job Service Bus extensions as a stepping stone to Azure Functions.

The Web Jobs fail with the error "The Message with ContentType 'application/x-gzip' failed to deserialize to a string with the message: 'There was an error deserializing the object of type System.String. The input source is not correctly formatted.'"

No matter what the signature to receive the object is: message, string, byte[] it will fail with the same error - there is no way to get this content into the web job for us to handle it.

Where on earth do we go with this??? Can we provide a custom deserializer somehow? I think the option existed in the .NET Framework SDK e.g. config.MessagingProvider = new CustomMessagingProvider(config)

We're using the latest versions for everything:

"Microsoft.Azure.WebJobs" Version="3.0.14"
"Microsoft.Azure.WebJobs.Extensions" Version="3.0.6"
"Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="4.1.0"

If we moved to Azure Functions would we have better control of the message deserialization process?

[Jlopean]

so in the new world (.net 5 function Apps). we move away from [FunctionName("FunctionName")] to [Function("FunctionName")]

Additionally, you can no longer use 'Message' or Byte[]. it can only be a string!

public async Task Run([ServiceBusTrigger(topicName, SubscriberName, Connection= %conn% )] string messageText, string id, FunctionContext executionContext)

The magic now is in FunctionContext.

you can get the message properties from this e.g; executionContext.BindingContext.BindingData.Where(x => x.Key =="UserProperties")

[sonic1981]

the difference between [Function] and [FunctionName] is NOT the .net version. It's actually whether is it a In-proc process [FunctionName] or out-of-proc [Function]. So there are MANY changes to how function apps work in this