Timer Trigger No Longer Firing

[AngleOSaxon]

Investigative information

I have an Azure Function (identifying details below) that is configured to run on a timer, according to a cron set in the environment.

• Function App version: 2.0
• Invocation ID: 66e088dd-70b3-4e22-bc0d-96337968dd0e
• Execution Time: 2018-10-12 11:12:38.100
• Region: South Central US

The cron is 0 0 6 * * *.

Expected Behavior

The process will run at 0600 UTC every morning.

Actual Behavior

Up until recently, it was being run consistently. Starting on September 29th, however, the timer no longer triggers the function.

I can find no logs specifically pertaining to the timer attempting to fire and failing, or to the function running and failing. It just appears that nothing happens at the appointed hour.

Troubleshooting Steps Attempted

Manually running the process through the Portal is generally successful, although it did at one point produce a SocketException very similar to the ones mentioned below.

By rebooting the function and setting the schedule to run every few minutes, I was able to make it run again, but this was only temporary. After resetting the schedule to its normal 24-hour period, the timer again began failing to fire.

I also attempted the workaround mentioned in this issue by setting the WEBSITE_USE_PLACEHOLDER value to 0 and restarting the function. This appears to have had no effect.

Related information

We have a number of other Functions that run on a schedule within that app service plan, all of which have been functioning normally and firing regularly.

One of them (but only one) appears to record the same kind of SocketException errors mentioned below.

The same day that the function stopped running regularly, it began frequently logging SocketException errors at seemingly random times. Almost all of them occurred while the function was not executing.

From the appearance of the stack trace, they seem to have come from within the hosting environment:
System.Net.Sockets.SocketException at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw

[  
   {  
      "parsedStack":[  
         {  
            "method":"System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext",
            "level":0,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         },
         {  
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":1,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":2,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":3,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Threading.Tasks.ValueTask`1.get_Result",
            "level":4,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult",
            "level":5,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Net.Http.HttpConnectionPool+<CreateConnectionAsync>d__44.MoveNext",
            "level":6,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         },
         {  
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":7,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":8,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":9,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Threading.Tasks.ValueTask`1.get_Result",
            "level":10,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult",
            "level":11,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Net.Http.HttpConnectionPool+<WaitForCreatedConnectionAsync>d__49.MoveNext",
            "level":12,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         },
         {  
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":13,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":14,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":15,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Threading.Tasks.ValueTask`1.get_Result",
            "level":16,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult",
            "level":17,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__39.MoveNext",
            "level":18,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         },
         {  
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":19,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":20,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":21,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult",
            "level":22,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext",
            "level":23,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         },
         {  
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":24,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":25,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":26,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult",
            "level":27,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext",
            "level":28,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         }
      ],
      "outerId":"0",
      "message":"An attempt was made to access a socket in a way forbidden by its access permissions",
      "type":"System.Net.Http.HttpRequestException",
      "id":"17180427"
   },
   {  
      "parsedStack":[  
         {  
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":0,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":1,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":2,
            "line":0,
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"
         },
         {  
            "method":"System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext",
            "level":3,
            "line":0,
            "assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
         }
      ],
      "outerId":"17180427",
      "message":"An attempt was made to access a socket in a way forbidden by its access permissions",
      "type":"System.Net.Sockets.SocketException",
      "id":"34921712"
   }
]

[AngleOSaxon]

This issue is resolved. The cause was that this function was hosted on the same server instance as another function, which had begun attempting to open ~10,000 outgoing TCP connections every hour.

Fixing the other function resolved the issue.