Blob triggers may not execute

[martinknafve]

Azure WebJobs relies on Azure Storage Analytics to determine which blobs have been written. Azure Storage Analytics logs are written on a (https://msdn.microsoft.com/en-us/library/azure/hh343262.aspx)[best-effort basis], so there's no guarantees that requests are logged.

As far as I understand it, the Azure Storage front-ends buffers logs for 5-10 minutes and if the machines crashes or restarts, the logs won't be written. We are processing millions of blobs every day, and several times a week we see requests not appearing in the logs (which is as expected).

When I'm reading about WebJobs I can't find any information that blob triggers may not execute, that execution is performed on a best-effort basis and that you probably should not design a system where you assume that they are executed. The Microsoft Azure Support have told me that I may want to look into using Azure WebJobs as a way to implement custom backup-procedures on top of Azure Storage, but considering that BlobTriggers in WebJobs are run on a best-effort basis, relying on them for backup purposes is probably a bad idea.

Is this behavior as-designed? If so, maybe it should be brought up in the documentation?

[mathewc]

In the doc (https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-storage-blobs-how-to/) we mention that we use analytic logs with inherent 5-10 minute delay and we recommend that people use a pattern of queue messages referencing the blob(s) to be processed. However, you are right that we don't point out that there is no absolute guarantee.

Note that logs are only used for blob detection when the webjob is up and running steady state. At startup, we do a container scan looking for unprocessed blobs (blobs with no receipts as detailed in the above link). So eventually all blobs should be processed. However, complicating matters is the fact that in the current release bits there was a bug in this scan logic (fixed here: 2b82daf) that might be why you're seeing some blobs not be processed.

@rustd Are you aware of any doc that mentions this? Can we update the existing doc?