Feature: Emit a log when Azure Service Bus de-duplicates a message for a queue/topic

[zeeshan-qadir]

Description

De-Duplication of messages in Azure Service Bus for a Queue or Topic is a very useful feature for different use cases including de-duplicating technical retries by the publishing application in case of intermittent fault etc.

However, when a message is de-duplicated by Azure Service Bus, there is no log which helps to identify that a message was de-duplicated by the Azure Service Bus. So, it results in longer triage times. Would be very beneficial if such log entry is emitted by Azure Service Bus when such de-duplication happens by Azure Service Bus.

Actual Behavior

Currently there is no way to identify through diagnostic logs, or metrics or any other mechanism when Service Bus De-duplicates a message.

Expected Behavior

Emit a log (which can be routed to Log Analytics Workspace via Diagnostic Settings) when Service Bus De-duplicates the message. Minimum information to be emitted in the log entry to include Timestamp in UTC, NameSpace, Queue/Topic Name, MessageID.

[EldertGrootenboer]

Thank you for suggesting this feature. We have opened an investigation task for this in our backlog, and will update this issue when we have more information. To help us get this prioritized, it would be helpful to see others vote and support this feature, as well as explain their scenarios.

[Stefanu83]

This would be a very helpful feature.

[serbrech]

+1 at least a metric. right now deduplication happens without any signal at all.
a metric would at least let us know how often that happens.
knowing which message id gets deduped would be even better.

[corner22]

+1 some metric/telemetry would be really useful to understanding the effectiveness of duplicate detection in a production environment.

[EldertGrootenboer]

Thank you for your feedback on this item, it helps us in our efforts to continuously prioritize the different requests we get from all our various feedback channels. We have added this feature in our backlog, however we currently don't have an ETA on when development might start on this. For now, to help us give this the right priority, it would be helpful to see others vote and support this feature, as well as explain their scenarios.

[ericleigh007]

Sorry for the bluntness here, but it is TOTALLY UNACCEPTABLE to delete data without at least a trace that it was done.
In the financial industry, we tend to send data on a queue to be archived for compliance reasons. If the queue itself does something to alter or remove data (whoever's fault it is), WITHOUT a peep, then we're in trouble.

What's the easiest thing to do? Have an endpoint that receives these kinds of logs? Have SB log to AI? What? Something must be done, or we just cannot use de-duplication.

Thanks for your kind and urgent attention to this matter.

[ericleigh007]

Stuck between a rock on a hard place on this one. The customer hates that the queue doesn't log these deletes and I don't blame them. However, we have to use de-duplication for just the simplicity of our application, and for replay/resiliency types of cases.

Can we talk about how hard this is have a log for that specifies the message id?
"Service bus received a duplicate Id in the time window [blah] for message id [msgid]"

Then, in our application, we can track down the msgid sent, and see the service bus-generated message, and realize what happened to the message.

That's it.

Thank you.

[EldertGrootenboer]

We have added this feature in our backlog, however we currently don't have an ETA on when development might start on this. For now, to help us give this the right priority, it would be helpful to see others vote and support this feature, as well as explain their scenarios.

[MichaelJones3BSL]

I was looking for this feature too.
Use case is similar to others, tracking of whether financial related data has been processed or not.
Our monitoring attempts to log reasons for failure or warning states.
A de-duplication of a service bus message would be considered a warning status, which would inform our users that the message did not process due to a warning message such as "duplicate message id 'abc123' detected on service bus, message has been de-duplicated".
At the moment we only have a success status from the Service Bus, which in this scenario adds no value to our monitoring.

[ruza-kavanka]

Hi everyone, I am missing this feature too. I am tracking the messages processing very carefully and not having any signs about message deduplication is causing me some troubles. Anyway, I need to stay with deduplication to keep my backend processing optimized. It would be very nice to have a log warning like "message id '123' skipped due to deduplication".