On worker error, wait for worker initialization if one or more are initializing before sending request

[mhoeger]

We should not send requests while no worker channels are available AND we're currently trying to start one up. Kindof similar to #5624 , but in this case no worker channels are available and were NOT currently trying to start one (because things are shutting down).

In one case, I see a worker process executing find and then run into an uncaught exception (publish WorkerErrorEvent). We dispose the worker channel and go through "Initiating worker Process start up".

The Function Invocation Buffer and all is set up. However, it hasn't started yet and isn't found - likely due to this code:

azure-functions-host/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs

Line 269 in 3eb520c

IEnumerable<IRpcWorkerChannel> initializedWorkers = workerChannels.Where(ch => ch.State == RpcWorkerChannelState.Initialized);

[pragnagopa]

cc @yojagad
Ideally host should never send requests if there are no language workers. Following code:

azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs

Lines 91 to 107 in bd350f3

	private async Task DelayUntilFunctionDispatcherInitializedOrShutdown()
	{
	if (_functionDispatcher != null && _functionDispatcher.State == FunctionInvocationDispatcherState.Initializing)
	{
	_logger.LogDebug($"functionDispatcher state: {_functionDispatcher.State}");
	bool result = await Utility.DelayAsync((_functionDispatcher.ErrorEventsThreshold + 1) * WorkerConstants.ProcessStartTimeoutSeconds, WorkerConstants.WorkerReadyCheckPollingIntervalMilliseconds, () =>
	{
	return _functionDispatcher.State != FunctionInvocationDispatcherState.Initialized;
	});
	if (result)
	{
	_logger.LogError($"Final functionDispatcher state: {_functionDispatcher.State}. Initialization timed out and host is shutting down");
	_applicationLifetime.StopApplication();
	}
	}
	}

Waits till atlease one language worker starts. If starting worker continues to fail, functions host is shutdown.

Needs more investigation.

[mhoeger]

The code above only works if FunctionInvocationDispatcherState.Initializing. Maybe if we are down to 0 workers, we set dispatcher state back to "Initializing".

Each language worker channel has an action buffer that delays requests until they're ready too though:

azure-functions-host/src/WebJobs.Script/Workers/Rpc/RpcWorkerChannel.cs

Line 304 in bd350f3

var disposableLink = _functionInputBuffers[loadResponse.FunctionId].LinkTo(invokeBlock);

If we remove this check,

azure-functions-host/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs

Line 269 in 3eb520c

IEnumerable<IRpcWorkerChannel> initializedWorkers = workerChannels.Where(ch => ch.State == RpcWorkerChannelState.Initialized);

requests will go into that queue. One potential error case is if the initializing worker ends up failing to start and we don't modify code to keep those queued requests from being lost or ending in error.

I think either the approach would resolve these issues though

[pragnagopa]

Thanks for more details. Check to verify the status of worker channel:

azure-functions-host/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs

Line 269 in 3eb520c

IEnumerable<IRpcWorkerChannel> initializedWorkers = workerChannels.Where(ch => ch.State == RpcWorkerChannelState.Initialized);

is needed to avoid sending requests even before buffers are setup.

Resetting FunctionDispatcherState is the right approach!