New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

queue message encoding issue #165

Closed
jamesdixon opened this Issue Jul 8, 2016 · 10 comments

Comments

Projects
None yet
2 participants
@jamesdixon
Contributor

jamesdixon commented Jul 8, 2016

Hi,

I started receiving the following error within my Azure Webjobs:

[07/08/2016 19:26:28 > cfb1f8: ERR ] Unhandled Exception: System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. 
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Convert.FromBase64String(String s)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage.get_AsString()
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.StorageQueueMessageExtensions.TryGetAsString(IStorageQueueMessage message)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.QueueCausalityManager.GetOwner(IStorageQueueMessage msg)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueTriggerExecutor.<ExecuteAsync>d__1.MoveNext()
[07/08/2016 19:26:28 > cfb1f8: ERR ] --- End of stack trace from previous location where exception was thrown ---
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ProcessMessageAsync>d__11.MoveNext()
[07/08/2016 19:26:28 > cfb1f8: ERR ] --- End of stack trace from previous location where exception was thrown ---
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[07/08/2016 19:26:28 > cfb1f8: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[07/08/2016 19:26:28 > cfb1f8: SYS ERR ] Job failed due to exit code -532462766

After doing some debugging, it appears that the message isn't being decoded properly because it's not base64. This didn't seem to be an issue before; has something changed?

@jamesdixon

This comment has been minimized.

Contributor

jamesdixon commented Jul 8, 2016

Note that prior to 1.1 (maybe 1.0), I don't recall having to manually encode or specify an encoder.

@jamesdixon

This comment has been minimized.

Contributor

jamesdixon commented Jul 9, 2016

I did some further investigation and can see that this is a change from 0.10 -> 1.x. Is there a reason for this change in behavior?

@hasonmsft

This comment has been minimized.

Member

hasonmsft commented Jul 11, 2016

@jamesdixon , refer to the breaking change log, originally the SDK only support a flag which indicates if the queue message should be encoded/decoded (base64 encoding). But we found that's not enough and not extensible. So we decided to introduce the encoding/decoding interface and provided some built-in encoders. For why we change the default encoding behavior from base64 encoding to XML encoding, it's because XML encoding will have smaller payload size for most scenarios. Please let me know if this makes sense.

Sorry again for the inconvenience.

@jamesdixon

This comment has been minimized.

Contributor

jamesdixon commented Jul 11, 2016

thanks for the reply, @hasonmsft.

Is encoding required with 1.x? Before 1.x, I seem to recall being able to pass a JSON string and see the contents in Azure Storage Explorer. Now it seems that encoding is required.

@hasonmsft

This comment has been minimized.

Member

hasonmsft commented Jul 11, 2016

If Azure Storage Explorer only support base64 encoding, the answer is yes, please set it to TextBase64QueueMessageEncoder - UTF8 text string to base64 encoding string.

@jamesdixon

This comment has been minimized.

Contributor

jamesdixon commented Jul 11, 2016

@hasonmsft thanks. I'm curious, why is it necessary to encode the message at all? Why can I not just pass a string into the queue?

@hasonmsft

This comment has been minimized.

Member

hasonmsft commented Jul 12, 2016

@jamesdixon , yes, you can. You can set the messageEncoder to null.

@jamesdixon

This comment has been minimized.

Contributor

jamesdixon commented Jul 12, 2016

Wow. Very simple! Thank you 👍

@jamesdixon jamesdixon closed this Jul 12, 2016

@jamesdixon

This comment has been minimized.

Contributor

jamesdixon commented Jul 19, 2016

@hasonmsft I tried setting the messageEncoder to null, but unfortunately when viewing the message in Azure Storage Explorer, everything appears garbled.

I also tried viewing it in the online version of Azure Storage Explorer and I get the following error:

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding character

@hasonmsft

This comment has been minimized.

Member

hasonmsft commented Jul 19, 2016

@jamesdixon , seems Azure Storage Explorer is using the base64 encoder. I don't know if Storage Explorer will support other decoders in the future. So to make your message can be displayed correctly in the Azure Storage Explorer, you need to use TextBase64QueueMessageEncoder to encode your message from plain text to base64 string.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment