New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function app: long-running queue trigger runs twice on AppService plan #1587

Closed
knikol opened this Issue Jun 9, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@knikol

knikol commented Jun 9, 2017

Repro steps
Step 1: Create function app which is hosted on AppService plan (Paid SKUs) - has no limit on execution time.
Step 2: Create C# queue trigger which runs 7 minutes
run.csx

using System;

public static async Task Run(string myQueueItem, TraceWriter log)
{
    log.Info($"Start queue trigger: {myQueueItem}");
    await Task.Delay(TimeSpan.FromMinutes(7));
    log.Info($"Completed queue trigger: {myQueueItem}");
}

function.json

{
  "disabled": false,
  "bindings": [
    {
      "name": "myQueueItem",
      "type": "queueTrigger",
      "direction": "in",
      "queueName": "test-trigger",
      "connection": "StorageConnectionString"
    }
  ]
}

host.json

{
  "id": "GUID",
  "functionTimeout": "01:00:00",
  "queues": {
    "maxPollingInterval": 5000
  }
}

Step 3: Add "test1" string to the queue which triggers the function

Expected behavior
Function executes once: succeeded, no message in the related poison queue.

Actual behavior
Function executes twice: succeeded, no message in the related poison queue.
Trace logs:

2017-06-09T07:36:25.346 Function started (Id=e7a607b2-c54c-4091-ab25-eb513a242f2d)
2017-06-09T07:36:25.346 Start queue trigger: test1
2017-06-09T07:43:25.358 Completed queue trigger: test1
2017-06-09T07:43:25.358 Function completed (Success, Id=e7a607b2-c54c-4091-ab25-eb513a242f2d, Duration=420012ms)
2017-06-09T07:51:27.048 Function started (Id=575f2f6a-5ed5-4148-9d0d-8b3bc3968d81)
2017-06-09T07:51:27.048 Start queue trigger: test1
2017-06-09T07:58:27.059 Completed queue trigger: test1
2017-06-09T07:58:27.059 Function completed (Success, Id=575f2f6a-5ed5-4148-9d0d-8b3bc3968d81, Duration=420005ms)

Related information
The issue appeared around 2-3 days ago. Before we did not observe such behavior.

@knikol

This comment has been minimized.

Show comment
Hide comment
@knikol

knikol Jun 9, 2017

BTW, the AlwaysOn has been disabled for the function app.
As recommended, enabled it to observe the behavior.

knikol commented Jun 9, 2017

BTW, the AlwaysOn has been disabled for the function app.
As recommended, enabled it to observe the behavior.

@knikol

This comment has been minimized.

Show comment
Hide comment
@knikol

knikol Jun 9, 2017

Did not help: triggered the second time after ~7 minutes or so.

knikol commented Jun 9, 2017

Did not help: triggered the second time after ~7 minutes or so.

@knikol

This comment has been minimized.

Show comment
Hide comment
@knikol

knikol Jun 9, 2017

According to the invocation log - trigger goes to infinite loop...
trigger

knikol commented Jun 9, 2017

According to the invocation log - trigger goes to infinite loop...
trigger

@christopheranderson

This comment has been minimized.

Show comment
Hide comment
@christopheranderson

christopheranderson Jun 12, 2017

Member

This is odd. Thanks for the investigation. We'll look into what might have caused this.

Member

christopheranderson commented Jun 12, 2017

This is odd. Thanks for the investigation. We'll look into what might have caused this.

@paulbatum paulbatum added this to the active questions milestone Jun 19, 2017

@brettsam

This comment has been minimized.

Show comment
Hide comment
@brettsam

brettsam Jun 19, 2017

Member

related to #1600

Member

brettsam commented Jun 19, 2017

related to #1600

@brettsam

This comment has been minimized.

Show comment
Hide comment
@brettsam
Member

brettsam commented Jun 19, 2017

@knikol

This comment has been minimized.

Show comment
Hide comment
@knikol

knikol Jun 20, 2017

@brettsam How can I know when the fix is deployed?

knikol commented Jun 20, 2017

@brettsam How can I know when the fix is deployed?

@brettsam

This comment has been minimized.

Show comment
Hide comment
@brettsam

brettsam Jun 20, 2017

Member

In general, watch here for the next release: https://github.com/Azure/azure-webjobs-sdk-script/releases.
But since I've seen multiple people hitting this issue, I'll swing back here and update the related issues when the release is ready.

Member

brettsam commented Jun 20, 2017

In general, watch here for the next release: https://github.com/Azure/azure-webjobs-sdk-script/releases.
But since I've seen multiple people hitting this issue, I'll swing back here and update the related issues when the release is ready.

@brettsam

This comment has been minimized.

Show comment
Hide comment
@brettsam

brettsam Jun 28, 2017

Member

Closing as it's fixed and we're waiting for it to appear in production.

Member

brettsam commented Jun 28, 2017

Closing as it's fixed and we're waiting for it to appear in production.

@brettsam brettsam closed this Jun 28, 2017

@brettsam

This comment has been minimized.

Show comment
Hide comment
@brettsam

brettsam Jul 3, 2017

Member

Update: the fix is now deployed in version 1.0.11015.0 (visible in the 'Settings' page of the Functions portal). If you are not running this version, manually restart your app and it will pick up the latest version.

Member

brettsam commented Jul 3, 2017

Update: the fix is now deployed in version 1.0.11015.0 (visible in the 'Settings' page of the Functions portal). If you are not running this version, manually restart your app and it will pick up the latest version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment