Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When mixing Imperative bindings for blob and Declarative bindings for Queue messages, queue messages are written first even when Blob messages are explicitly flushed #1986

Open
hrkulkarMsft opened this Issue Oct 2, 2017 · 3 comments

Comments

Projects
None yet
5 participants
@hrkulkarMsft
Copy link
Member

hrkulkarMsft commented Oct 2, 2017

I am using a binder to construct the blob name at runtime in my Azure Function. Additionally, I have a queue binding that is declarative. I write to the blob and then flush and close the text writer. Then I add the queue message to the collection. However, the queue message is written before the blob, resulting in the next function in the chain seeing an error.

Investigative information

  • Timestamp: 2017-09-30T00:21:00Z
  • Invocation ID:
    bc72ea3d-0170-4e30-9a4e-6c78504b8498 (parent job, which publishes blob & queue).
    4b77107d-4a80-433a-8bf4-d50c2788b84f (sample queue message job)
  • Region: NCUS

Repro steps

  1. Create an Azure Function
  2. Set an Azure Queue as the output.
  3. Have the function output a blob using a binder.
  4. Write to the blob, and then the queue. Example:

` using (var writer = await binder.BindAsync(attributes))
{
writer.Write(output);
writer.Flush();
writer.Close();
}

await outputQueueItem.AddAsync(new SomePoco {Name =  blobName});

`
5. Create an Azure Function which takes a QueueMessage with SomePoco as a trigger, and takes a blob with the Name of SomePoco as input.

Expected behavior

I expect the write to the blob to be blocking, and for the queue message to be written after the blob has been written.

Actual behavior

The blob messages are not written until the parent function completely finishes execution. While the queue messages are written while the parent function is still running.

Known workarounds

Have a copy of the function which processes the SomePoco queue messages, except it accepts input from the poison queue message stream. Then use this to manually clear out poison queue messages.

Related information

Provide any related information

  • Programming language used: C#
@mathewc

This comment has been minimized.

Copy link
Contributor

mathewc commented Oct 2, 2017

I experimented, and it does appear that the binding returned from Binder (e.g. in this case the TextWriter) isn't finalized (the blob isn't written) when the TextWriter is disposed as I had initially thought. The dynamic bindings created by Binder are "finalized" after the function returns, as part of processing the rest of the output bindings. However, the queue message output via AddAsync will be output immediately.

The bindings created via Binder aren't fully flushed until the method returns, as part of processing/finalizing output bindings (see code SetValueAsync).

We need to decide whether this is by design (probably has been this way for years?) or whether we want to provide a Flush API on Binder/IBinder to allow users to force flush.

@paulbatum

This comment has been minimized.

Copy link
Member

paulbatum commented Jan 20, 2018

@MikeStall any thoughts on this? Also, is this bug in the right repo? Sounds like a sdk level issue to me..

@paulbatum paulbatum modified the milestones: Triaged, Unknown Mar 7, 2018

@DimaGershman

This comment has been minimized.

Copy link

DimaGershman commented Nov 21, 2018

Thanks! It took a while to figure out that something like that I going on there ;)

Regarding

Known workarounds

Have a copy of the function which processes the SomePoco queue messages, except it accepts input from the poison queue message stream. Then use this to manually clear out poison queue messages.

Another temporary workaround could be writing to blob storage explicitly, without the Binder, before creating queue message..

Something like:

var account = CloudStorageAccount.Parse(Environment.GetEnvironmentVariable("BlobStorageConnectionStringName", EnvironmentVariableTarget.Process));
var blobClient = account.CreateCloudBlobClient();
var blobContainer = blobClient.GetContainerReference("container name");
var cloudBlockBlob = blobContainer.GetBlockBlobReference("blobname");
cloudBlockBlob.UploadText("blob file contents");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.