New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function not triggered by blob #691

Closed
JohanKroese opened this Issue Sep 15, 2016 · 97 comments

Comments

Projects
None yet
@JohanKroese

JohanKroese commented Sep 15, 2016

I have an Azure function with a single trigger of the type 'Azure Storage Blob'. While actively working on (developing/testing) the function it responds almost directly to blob activity. A few days later the function is not started on the trigger of a new blob. If this was a webjob if would judge, based on the symptoms, that it was not in 'always on' mode. The Azure Function however uses the dynamic service plan and should always run & react to triggers, right?
Once I go to the Azure portal and open the Azure Function, it immediately runs.

Function Name: vatreturnprep
The timeline:

  • Last function test (succes): 2016-09-11T21:45:38
  • New blob uploaded in the Azure blob container: 2016-09-14T14:48
  • Azure function opened & Azure Function triggered: 2016-09-15T20:11

Azure Function activity:
2016-09-11T21:45:38.591 Function started (Id=537bf088-0d23-4ea6-bc81-6a575e8c4096)
2016-09-11T21:45:51.595 Function completed (Success, Id=537bf088-0d23-4ea6-bc81-6a575e8c4096)
2016-09-15T20:11:06.456 Function started (Id=7b87f523-40a4-415f-80d5-5a46133d9591)
2016-09-15T20:11:20.285 Function completed (Success, Id=7b87f523-40a4-415f-80d5-5a46133d9591)

Azure Blob Container:
Name Modified Blob type Size
file1.csv ‎11‎/‎09‎/‎2016‎ ‎11‎:‎45‎:‎36‎ ‎PM Block blob 27.42 MB
file2.csv ‎14‎/‎09‎/‎2016‎ ‎02‎:‎48‎:‎16‎ ‎PM Block blob 27.42 MB

I fail to find a reason that the function is not triggered until the exact moment I monitor the function in the Azure Portal, any chance this is a bug?

@lindydonna

This comment has been minimized.

Show comment
Hide comment
@lindydonna

lindydonna Sep 19, 2016

Contributor

Can you confirm that you're on a Dynamic App Service Plan? Azure Functions can be used on both a classic app service plan (where you do need to manually turn on Always On, see Azure/azure-functions-ux#557), or a dynamic plan.

Contributor

lindydonna commented Sep 19, 2016

Can you confirm that you're on a Dynamic App Service Plan? Azure Functions can be used on both a classic app service plan (where you do need to manually turn on Always On, see Azure/azure-functions-ux#557), or a dynamic plan.

@JohanKroese

This comment has been minimized.

Show comment
Hide comment
@JohanKroese

JohanKroese Sep 19, 2016

Yes, it is using a Dynamic App Service Plan. I created the function from the 'Azure Functions page https://functions.azure.com, so the service plan was auto-created. Here's the service plan:

azurefunctions1

And the Azure Function:
azurefunctions2

JohanKroese commented Sep 19, 2016

Yes, it is using a Dynamic App Service Plan. I created the function from the 'Azure Functions page https://functions.azure.com, so the service plan was auto-created. Here's the service plan:

azurefunctions1

And the Azure Function:
azurefunctions2

@lindydonna

This comment has been minimized.

Show comment
Hide comment
@lindydonna

lindydonna Sep 19, 2016

Contributor

@tohling will investigate, it definitely sounds like a bug.

Contributor

lindydonna commented Sep 19, 2016

@tohling will investigate, it definitely sounds like a bug.

@bullsfan127

This comment has been minimized.

Show comment
Hide comment
@bullsfan127

bullsfan127 Sep 20, 2016

I am also experiencing the same behavior for my function app with blob triggers.

bullsfan127 commented Sep 20, 2016

I am also experiencing the same behavior for my function app with blob triggers.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Sep 20, 2016

Contributor

@JohanKroese & @bullsfan127 can you check whether your storage account has diagnostic logs enabled for blob? The notification system relies on this.

To check:

  • in Portal, find your storage account
  • go to Diagnostics
  • Check that Status is On and 'Blob logs' is checked.
Contributor

davidebbo commented Sep 20, 2016

@JohanKroese & @bullsfan127 can you check whether your storage account has diagnostic logs enabled for blob? The notification system relies on this.

To check:

  • in Portal, find your storage account
  • go to Diagnostics
  • Check that Status is On and 'Blob logs' is checked.
@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Sep 20, 2016

Contributor

One more thing: if blob logging is enabled, you look see a Blob container named $logs in the storage account, and it should contain entries that match the time where you updated blobs.

Contributor

davidebbo commented Sep 20, 2016

One more thing: if blob logging is enabled, you look see a Blob container named $logs in the storage account, and it should contain entries that match the time where you updated blobs.

@bullsfan127

This comment has been minimized.

Show comment
Hide comment
@bullsfan127

bullsfan127 Sep 21, 2016

I have 'Blob logs' checked and i had entries in $logs that matched the files that were placed

bullsfan127 commented Sep 21, 2016

I have 'Blob logs' checked and i had entries in $logs that matched the files that were placed

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Sep 21, 2016

Contributor

@bullsfan127 Can you share your function app name (that's the app name, not function name!), either directly or indirectly? This will help us investigate. Thanks!

Contributor

davidebbo commented Sep 21, 2016

@bullsfan127 Can you share your function app name (that's the app name, not function name!), either directly or indirectly? This will help us investigate. Thanks!

@JohanKroese

This comment has been minimized.

Show comment
Hide comment
@JohanKroese

JohanKroese Sep 21, 2016

Blob logging is enabled. The entry in $logs matches the time when the blob was inserted:
container: $logs\blob\2016\09\14\1200\000000.log - 14 sept - 12:56:33 GMT

In the storage account tied to the Azure Function I find the matching blob receipt:
container: azure-webjobs-hosts\blobreceipts...\myblobfile. Datetime: 15 sept - 20:11:05 GMT

Note that the storage account that is the trigger-source for the Azure Function is a different storage account than the one that the Azure Function uses internally (named azurefunctions6fde45d2).

JohanKroese commented Sep 21, 2016

Blob logging is enabled. The entry in $logs matches the time when the blob was inserted:
container: $logs\blob\2016\09\14\1200\000000.log - 14 sept - 12:56:33 GMT

In the storage account tied to the Azure Function I find the matching blob receipt:
container: azure-webjobs-hosts\blobreceipts...\myblobfile. Datetime: 15 sept - 20:11:05 GMT

Note that the storage account that is the trigger-source for the Azure Function is a different storage account than the one that the Azure Function uses internally (named azurefunctions6fde45d2).

@lindydonna lindydonna added this to the rc milestone Sep 26, 2016

@lindydonna

This comment has been minimized.

Show comment
Hide comment
@lindydonna

lindydonna Sep 26, 2016

Contributor

CC @mathewc Who was looking into it

Contributor

lindydonna commented Sep 26, 2016

CC @mathewc Who was looking into it

@tohling tohling assigned jocawtho and unassigned tohling Sep 27, 2016

@mathewc

This comment has been minimized.

Show comment
Hide comment
@mathewc

mathewc Sep 30, 2016

Contributor

Update on this: @jocawtho has identified the bug causing this. We have a fix, but it will take us a while to roll it out, since it is in core WebApps infrastructure, not the Functions runtime (i.e. not in this repo).

Contributor

mathewc commented Sep 30, 2016

Update on this: @jocawtho has identified the bug causing this. We have a fix, but it will take us a while to roll it out, since it is in core WebApps infrastructure, not the Functions runtime (i.e. not in this repo).

@martinabbott

This comment has been minimized.

Show comment
Hide comment
@martinabbott

martinabbott Oct 20, 2016

Any update on this, getting this in a customer deployment right now, although seems a bit schizophrenic as sometimes it is okay and other times I have to go in to kick it off?

martinabbott commented Oct 20, 2016

Any update on this, getting this in a customer deployment right now, although seems a bit schizophrenic as sometimes it is okay and other times I have to go in to kick it off?

@jocawtho

This comment has been minimized.

Show comment
Hide comment
@jocawtho

jocawtho Oct 20, 2016

Member

The fix is currently being rolled out into production. The upgrade should be completely finished by the end of next week.

Member

jocawtho commented Oct 20, 2016

The fix is currently being rolled out into production. The upgrade should be completely finished by the end of next week.

@lindydonna

This comment has been minimized.

Show comment
Hide comment
@lindydonna

lindydonna Nov 2, 2016

Contributor

@jocawtho Can you update this issue when the fix has been deployed?

Contributor

lindydonna commented Nov 2, 2016

@jocawtho Can you update this issue when the fix has been deployed?

@jocawtho

This comment has been minimized.

Show comment
Hide comment
@jocawtho

jocawtho Nov 2, 2016

Member

The fix to this issue has now been deployed fully to production.

Member

jocawtho commented Nov 2, 2016

The fix to this issue has now been deployed fully to production.

@lindydonna lindydonna closed this Nov 3, 2016

@lindydonna lindydonna removed the in progress label Nov 3, 2016

@ThorvaldBoe

This comment has been minimized.

Show comment
Hide comment
@ThorvaldBoe

ThorvaldBoe Dec 1, 2016

Sorry to bump this, but I am still experiencing this issue. Is the fix rolled out globally? My function is deployed in the west Europe region. BTW I don't have any dynamic plan available, instead I can choose between a consumption plan and an app service plan (mine is consumption). I suppose consumption and dynamic are the same..?

ThorvaldBoe commented Dec 1, 2016

Sorry to bump this, but I am still experiencing this issue. Is the fix rolled out globally? My function is deployed in the west Europe region. BTW I don't have any dynamic plan available, instead I can choose between a consumption plan and an app service plan (mine is consumption). I suppose consumption and dynamic are the same..?

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Dec 1, 2016

Contributor

Reactivating. @jocawtho can you investigate?

@ThorvaldBoe can you share your function app name (that's the app name, not function name!), either directly or indirectly?

Contributor

davidebbo commented Dec 1, 2016

Reactivating. @jocawtho can you investigate?

@ThorvaldBoe can you share your function app name (that's the app name, not function name!), either directly or indirectly?

@davidebbo davidebbo reopened this Dec 1, 2016

@davidebbo davidebbo removed this from the rc milestone Dec 1, 2016

@martinabbott

This comment has been minimized.

Show comment
Hide comment
@martinabbott

martinabbott Dec 1, 2016

martinabbott commented Dec 1, 2016

@Nosmadas

This comment has been minimized.

Show comment
Hide comment
@Nosmadas

Nosmadas Feb 3, 2017

I think we're experiencing a similar issue in which files are not being picked up by a blob trigger function.

Today we added 5 files to a container which weren't processed for just over 30 mins. Someone then loaded the function blade and they were picked up.

Is there anything we can do to diagnose this? From following this thread - it looks as if logs were piped into the $logs container shortly after the files were added. But in the function storage account blob receipts and host logs were only added after someone loaded the function app up via the portal.

Nosmadas commented Feb 3, 2017

I think we're experiencing a similar issue in which files are not being picked up by a blob trigger function.

Today we added 5 files to a container which weren't processed for just over 30 mins. Someone then loaded the function blade and they were picked up.

Is there anything we can do to diagnose this? From following this thread - it looks as if logs were piped into the $logs container shortly after the files were added. But in the function storage account blob receipts and host logs were only added after someone loaded the function app up via the portal.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 3, 2017

Contributor

@Nosmadas Can you share your function app name (that's the app name, not function name!), either directly or indirectly, as well as the UTC time of the incident? As far as we know, all cases are fixed now, so there are no know pending issues.

Contributor

davidebbo commented Feb 3, 2017

@Nosmadas Can you share your function app name (that's the app name, not function name!), either directly or indirectly, as well as the UTC time of the incident? As far as we know, all cases are fixed now, so there are no know pending issues.

@rmaziarka

This comment has been minimized.

Show comment
Hide comment
@rmaziarka

rmaziarka Feb 3, 2017

I am experiencing something similar with my bandata-complaint-photos-resize app. Logs was created in the log folder after uploading photos to blob, but no function running was occured. But then, after opening portal, photos were resized.

I went through this thread, by I haven't found anything that could help me run my function automatically. I would be grateful if you could help me in this matter.

I have Dynamic App Service Plan and Check that Status is On and 'Blob logs' is checked.

rmaziarka commented Feb 3, 2017

I am experiencing something similar with my bandata-complaint-photos-resize app. Logs was created in the log folder after uploading photos to blob, but no function running was occured. But then, after opening portal, photos were resized.

I went through this thread, by I haven't found anything that could help me run my function automatically. I would be grateful if you could help me in this matter.

I have Dynamic App Service Plan and Check that Status is On and 'Blob logs' is checked.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 4, 2017

Contributor

@rmaziarka as per above, to investigate, we need the approximate UTC time where you uploaded a blob and didn't see the function run. Also, please include the name of the function. Thanks!

Contributor

davidebbo commented Feb 4, 2017

@rmaziarka as per above, to investigate, we need the approximate UTC time where you uploaded a blob and didn't see the function run. Also, please include the name of the function. Thanks!

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 4, 2017

Contributor

Taking a look, I can see that your app is getting woken up on Feb 3 at 21:14, 21:20, 22:35, 22:44, 23:43. So it is correctly detecting new blobs and doing the right thing from what I can see. Do keep in mind that when the app is cold, blob detection can be delayed for several minutes.

Contributor

davidebbo commented Feb 4, 2017

Taking a look, I can see that your app is getting woken up on Feb 3 at 21:14, 21:20, 22:35, 22:44, 23:43. So it is correctly detecting new blobs and doing the right thing from what I can see. Do keep in mind that when the app is cold, blob detection can be delayed for several minutes.

@rmaziarka

This comment has been minimized.

Show comment
Hide comment
@rmaziarka

rmaziarka Feb 4, 2017

It was woken on Feb 3 because I opened portal. Photos were uploaded day before.

Today I uploaded 5 photos to my blob at 08:43 UTC (2017-02-04). Nothing happened. Then hour later I opened portal and automatically photos were resized (function has started), at 10:09 UTC (2017-02-04).

My app is bandata-complaint-photos-resize, function is complaint-photos-resize.

rmaziarka commented Feb 4, 2017

It was woken on Feb 3 because I opened portal. Photos were uploaded day before.

Today I uploaded 5 photos to my blob at 08:43 UTC (2017-02-04). Nothing happened. Then hour later I opened portal and automatically photos were resized (function has started), at 10:09 UTC (2017-02-04).

My app is bandata-complaint-photos-resize, function is complaint-photos-resize.

@Nosmadas

This comment has been minimized.

Show comment
Hide comment
@Nosmadas

Nosmadas Feb 4, 2017

@davidebbo The function app name is ... 5 files were added to the blob container used in the blob trigger at approx 10:10 UTC (2017-02-03), I loaded the function blade at around 10:43 and they were then picked up.

Thanks

Nosmadas commented Feb 4, 2017

@davidebbo The function app name is ... 5 files were added to the blob container used in the blob trigger at approx 10:10 UTC (2017-02-03), I loaded the function blade at around 10:43 and they were then picked up.

Thanks

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 4, 2017

Contributor

@rmaziarka so looking at your function, you are processing blobs in place, i.e. replacing the original image by the resized image.

Looking at our logs, it's a bit puzzling as I see the scale controller is detecting changes to the blob at 10:16, but not at 8:43. i.e. it appears to detect the change that is caused by the execution of your function, but not the change caused by you adding the new files in the first place.

Couple questions:

  • How exactly are you adding the new blobs? i.e. via code, some tool, ...?
  • I wonder if the fact that both your trigger and your output are the same blob is triggering some issue. If you don't mind, would you be able to try changing the output binding to go to a different container, to see if that makes a difference? Thanks!
Contributor

davidebbo commented Feb 4, 2017

@rmaziarka so looking at your function, you are processing blobs in place, i.e. replacing the original image by the resized image.

Looking at our logs, it's a bit puzzling as I see the scale controller is detecting changes to the blob at 10:16, but not at 8:43. i.e. it appears to detect the change that is caused by the execution of your function, but not the change caused by you adding the new files in the first place.

Couple questions:

  • How exactly are you adding the new blobs? i.e. via code, some tool, ...?
  • I wonder if the fact that both your trigger and your output are the same blob is triggering some issue. If you don't mind, would you be able to try changing the output binding to go to a different container, to see if that makes a difference? Thanks!
@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 4, 2017

Contributor

@Nosmadas your case is different, as the scale controller does not have any information about your triggers. There was an issue that was fixed yesterday that could cause that to happen when deploying via VS (or msdeploy). Can you try clicking the 'Refresh' button in the Functions UI (in the lower left, below Quickstart). Thanks!

Contributor

davidebbo commented Feb 4, 2017

@Nosmadas your case is different, as the scale controller does not have any information about your triggers. There was an issue that was fixed yesterday that could cause that to happen when deploying via VS (or msdeploy). Can you try clicking the 'Refresh' button in the Functions UI (in the lower left, below Quickstart). Thanks!

@Nosmadas

This comment has been minimized.

Show comment
Hide comment
@Nosmadas

Nosmadas Feb 4, 2017

@davidebbo Done! Thanks for looking into it, will monitor going forwards.

Nosmadas commented Feb 4, 2017

@davidebbo Done! Thanks for looking into it, will monitor going forwards.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 4, 2017

Contributor

@Nosmadas yep, I see the registration for both functions. Normally, processing will now happen with having to go to portal.

Contributor

davidebbo commented Feb 4, 2017

@Nosmadas yep, I see the registration for both functions. Normally, processing will now happen with having to go to portal.

@rmaziarka

This comment has been minimized.

Show comment
Hide comment
@rmaziarka

rmaziarka Feb 6, 2017

@davidebbo
I created new function complaint-photos-resize-2 (app name bandata-complaint-photos-resize) which adds temp at the end of the filename. And now it's working. Both from code (ASP.NET app) and Storage Exporer. I would not think about it :)

Would you be able to fix it somehow or should i need to create 2 functions:

  • first to resize photo and save with temp name
  • second to rename output file to previous name
    ?

rmaziarka commented Feb 6, 2017

@davidebbo
I created new function complaint-photos-resize-2 (app name bandata-complaint-photos-resize) which adds temp at the end of the filename. And now it's working. Both from code (ASP.NET app) and Storage Exporer. I would not think about it :)

Would you be able to fix it somehow or should i need to create 2 functions:

  • first to resize photo and save with temp name
  • second to rename output file to previous name
    ?
@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 7, 2017

Contributor

@rmaziarka I set up a similar scenario where the function updates the new blob inplace, and it's actually working fine for me.

My guess is that there is some alternate issue at play. Thinking about it, it's hard to see how the fact that you update inplace could break things, because the fact that you do an implace update doesn't come into play until your function gets called, and if it gets called, we don't have the issue in the first place. So the inplace part is very likely a red herring here.

Contributor

davidebbo commented Feb 7, 2017

@rmaziarka I set up a similar scenario where the function updates the new blob inplace, and it's actually working fine for me.

My guess is that there is some alternate issue at play. Thinking about it, it's hard to see how the fact that you update inplace could break things, because the fact that you do an implace update doesn't come into play until your function gets called, and if it gets called, we don't have the issue in the first place. So the inplace part is very likely a red herring here.

@edspivak

This comment has been minimized.

Show comment
Hide comment
@edspivak

edspivak Feb 27, 2017

Hi, Guys.

I am experiencing exactly the same issue but with function app triggered by the Queue. I am also on Consumption plan. App Name: cm-image-ops, Function Name: image-merge. Should I open a new issue or the same fix as for blob trigger can be applied in my case?

Thank you.

edspivak commented Feb 27, 2017

Hi, Guys.

I am experiencing exactly the same issue but with function app triggered by the Queue. I am also on Consumption plan. App Name: cm-image-ops, Function Name: image-merge. Should I open a new issue or the same fix as for blob trigger can be applied in my case?

Thank you.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Feb 27, 2017

Contributor

@edspivak Please do open a new issue, as this one is specific to blob triggers. Thanks!

Contributor

davidebbo commented Feb 27, 2017

@edspivak Please do open a new issue, as this one is specific to blob triggers. Thanks!

@ASeale

This comment has been minimized.

Show comment
Hide comment
@ASeale

ASeale Dec 6, 2017

@davidebbo
I'm seeing this issue.
At around 4:00pm UTC we uploaded a blob. It's now 4:19 UTC and it still hasn't processed. At 4:20 UTC I will open the app. The app name is cjep-Blob-Efiling-partners.

Opened the app and it immediately processed.

ASeale commented Dec 6, 2017

@davidebbo
I'm seeing this issue.
At around 4:00pm UTC we uploaded a blob. It's now 4:19 UTC and it still hasn't processed. At 4:20 UTC I will open the app. The app name is cjep-Blob-Efiling-partners.

Opened the app and it immediately processed.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Dec 6, 2017

Contributor

@ASeale it looks like there may be an issue with your connection string. I see an error on our backend: "The storage account connection string was null, empty, or not found." Can you check that the connection name is valid?

We're working on a way to better show these messages to the user, and this should be coming in the next month or two. /cc @tohling

Contributor

davidebbo commented Dec 6, 2017

@ASeale it looks like there may be an issue with your connection string. I see an error on our backend: "The storage account connection string was null, empty, or not found." Can you check that the connection name is valid?

We're working on a way to better show these messages to the user, and this should be coming in the next month or two. /cc @tohling

@ASeale

This comment has been minimized.

Show comment
Hide comment
@ASeale

ASeale Dec 6, 2017

@davidebbo ok so, the connection string names were accurate. As a test I moved them from the "Connection strings" section in the Application Settings to the "Application settings" section in the Application Settings. Everything seems to be working now. I will run some more delayed tests just to make sure.

ASeale commented Dec 6, 2017

@davidebbo ok so, the connection string names were accurate. As a test I moved them from the "Connection strings" section in the Application Settings to the "Application settings" section in the Application Settings. Everything seems to be working now. I will run some more delayed tests just to make sure.

@ASeale

This comment has been minimized.

Show comment
Hide comment
@ASeale

ASeale Dec 6, 2017

@davidebbo verified working. I waited 30 minutes before trying another test. It took it 8 minutes to finally process it, but it did end up getting it. Not sure why moving the connection strings from the connection string section to the application setting section worked, but it did.

ASeale commented Dec 6, 2017

@davidebbo verified working. I waited 30 minutes before trying another test. It took it 8 minutes to finally process it, but it did end up getting it. Not sure why moving the connection strings from the connection string section to the application setting section worked, but it did.

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo Dec 6, 2017

Contributor

@ASeale I would say it's a bug that it didn't work with Connection Strings. But for now, yes, please do use App Settings (there is no drawback).

Contributor

davidebbo commented Dec 6, 2017

@ASeale I would say it's a bug that it didn't work with Connection Strings. But for now, yes, please do use App Settings (there is no drawback).

@kkanna

This comment has been minimized.

Show comment
Hide comment
@kkanna

kkanna May 3, 2018

Facing the same issue now

kkanna commented May 3, 2018

Facing the same issue now

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo May 3, 2018

Contributor

@kkanna please share your function app name (that's the app name, not function name!), either directly or indirectly, as well as a sample UTC time where it didn't fire and you think it should have. Also, indicate how long you waited.

Contributor

davidebbo commented May 3, 2018

@kkanna please share your function app name (that's the app name, not function name!), either directly or indirectly, as well as a sample UTC time where it didn't fire and you think it should have. Also, indicate how long you waited.

@agostinocinelli

This comment has been minimized.

Show comment
Hide comment
@agostinocinelli

agostinocinelli May 18, 2018

Hello, I'm facing the same issue too
The connection string is in the app settings
'Blob logs' is checked
There are entries in $logs that matched the files
I have two function listening on the same container, but none of the two is triggering anymore.
The functions are in the same service plan where dummy6428 is
I been waiting for an hour now but nothing happened
I tried to run function manually with "Test" and the function worked
Can you @davidebbo help?
Thank you

agostinocinelli commented May 18, 2018

Hello, I'm facing the same issue too
The connection string is in the app settings
'Blob logs' is checked
There are entries in $logs that matched the files
I have two function listening on the same container, but none of the two is triggering anymore.
The functions are in the same service plan where dummy6428 is
I been waiting for an hour now but nothing happened
I tried to run function manually with "Test" and the function worked
Can you @davidebbo help?
Thank you

@davidebbo

This comment has been minimized.

Show comment
Hide comment
@davidebbo

davidebbo May 18, 2018

Contributor

@agostinocinelli I suspect there is some memory corruption that happened when the app last started around May 10th that put it in an unstable state.

Specifically, I see two kind of exceptions on the 10th and 11th:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.CreateVideoThumbCS ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer,Int32 offset,Int32 count)
   at System.IO.Stream.InternalCopyTo(Stream destination,Int32 bufferSize)
   at System.IO.Stream.CopyTo(Stream destination)
   at Submission#0.Run(Stream myBlob,Stream videoThumb,String blobname,String blobextension,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 107 
   End of inner exception
   at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.CreateVideoThumbCS ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.ComponentModel.Win32Exception : Only part of a ReadProcessMemory or WriteProcessMemory request was completed
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Submission#0.CreateThumbnail(String temp,String tempOut,Int32 duration,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 247
   at Submission#0.GetThumbnail(String temp,String tempOut,Int32 duration,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 209
   at Submission#0.Run(Stream myBlob,Stream videoThumb,String blobname,String blobextension,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 126 
   End of inner exception
   at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)

Restarting the app would put it back into a clean state, but it may happen again is there is some issue with what the function is doing. Also, you have 34 sites running on a Small Basic VM, which may contribute to the memory issues. Try scaling up to a medium, or use multiple App Service plans.

Contributor

davidebbo commented May 18, 2018

@agostinocinelli I suspect there is some memory corruption that happened when the app last started around May 10th that put it in an unstable state.

Specifically, I see two kind of exceptions on the 10th and 11th:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.CreateVideoThumbCS ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer,Int32 offset,Int32 count)
   at System.IO.Stream.InternalCopyTo(Stream destination,Int32 bufferSize)
   at System.IO.Stream.CopyTo(Stream destination)
   at Submission#0.Run(Stream myBlob,Stream videoThumb,String blobname,String blobextension,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 107 
   End of inner exception
   at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.CreateVideoThumbCS ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.ComponentModel.Win32Exception : Only part of a ReadProcessMemory or WriteProcessMemory request was completed
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Submission#0.CreateThumbnail(String temp,String tempOut,Int32 duration,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 247
   at Submission#0.GetThumbnail(String temp,String tempOut,Int32 duration,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 209
   at Submission#0.Run(Stream myBlob,Stream videoThumb,String blobname,String blobextension,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 126 
   End of inner exception
   at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)

Restarting the app would put it back into a clean state, but it may happen again is there is some issue with what the function is doing. Also, you have 34 sites running on a Small Basic VM, which may contribute to the memory issues. Try scaling up to a medium, or use multiple App Service plans.

@agostinocinelli

This comment has been minimized.

Show comment
Hide comment
@agostinocinelli

agostinocinelli May 21, 2018

Thank you @davidebbo for your help!
Most of the 34 sites are no longer or very rarely used, but we definitely need to organize them better :)
I will periodically check the log before moving the functions to production!
Thank you again

agostinocinelli commented May 21, 2018

Thank you @davidebbo for your help!
Most of the 34 sites are no longer or very rarely used, but we definitely need to organize them better :)
I will periodically check the log before moving the functions to production!
Thank you again

@mesani

This comment has been minimized.

Show comment
Hide comment
@mesani

mesani Aug 24, 2018

I'm encountering this same issue. If I load up the portal it seems to work, I can wait 2-3 mintues, upload again and it seems to work. If I wait around 10 minutes nothing triggers.

and I get the following :

2018-08-24T14:09:14 No new trace in the past 8 min(s).
2018-08-24T14:10:14 No new trace in the past 9 min(s).
2018-08-24T14:11:14 No new trace in the past 10 min(s).
2018-08-24T14:12:14 No new trace in the past 11 min(s).

mesani commented Aug 24, 2018

I'm encountering this same issue. If I load up the portal it seems to work, I can wait 2-3 mintues, upload again and it seems to work. If I wait around 10 minutes nothing triggers.

and I get the following :

2018-08-24T14:09:14 No new trace in the past 8 min(s).
2018-08-24T14:10:14 No new trace in the past 9 min(s).
2018-08-24T14:11:14 No new trace in the past 10 min(s).
2018-08-24T14:12:14 No new trace in the past 11 min(s).

@martinabbott

This comment has been minimized.

Show comment
Hide comment
@martinabbott

martinabbott Aug 24, 2018

martinabbott commented Aug 24, 2018

@mesani

This comment has been minimized.

Show comment
Hide comment
@mesani

mesani Aug 24, 2018

I've changed it from a consumption plan and will test it in a few mins to see if it runs ok . So is it because I'm using a consumption plan!!

I was using Runtime version
Runtime version: 1.0.11959.0 (~1)

THanks for your quick response.

mesani commented Aug 24, 2018

I've changed it from a consumption plan and will test it in a few mins to see if it runs ok . So is it because I'm using a consumption plan!!

I was using Runtime version
Runtime version: 1.0.11959.0 (~1)

THanks for your quick response.

@mesani

This comment has been minimized.

Show comment
Hide comment
@mesani

mesani Aug 24, 2018

ok this doesn't seem to have resolved it. I recreated the function but using the PremiumV2: 1 Small) plan and set:
always on -> On
HTTP version 2.0

I see in the analytics pane of the memory usage went right down about 10 mins ago which is I assume the app going dead.

I thought it might have been due to it being hosted on a consumption but it appears like it's not.

Is there something obvious I'm not doing?

mesani commented Aug 24, 2018

ok this doesn't seem to have resolved it. I recreated the function but using the PremiumV2: 1 Small) plan and set:
always on -> On
HTTP version 2.0

I see in the analytics pane of the memory usage went right down about 10 mins ago which is I assume the app going dead.

I thought it might have been due to it being hosted on a consumption but it appears like it's not.

Is there something obvious I'm not doing?

@martinabbott

This comment has been minimized.

Show comment
Hide comment
@martinabbott

martinabbott Aug 25, 2018

martinabbott commented Aug 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment