Regression: functionInstanceInjector left null on concurrent cold-start load (2.19.2+) — reintroduces the NPE fixed by #684

[rgdick11]

Summary

Since worker 2.19.2, JavaFunctionBroker.initializeOneTimeLogics() sets oneTimeLogicInitialized = true before calling initializeFunctionInstanceInjector(). Because function-load requests are dispatched concurrently, a second load thread can observe the flag as already true, skip the double-checked-locked init block, and complete while functionInstanceInjector is still null. The field then stays null for the life of the worker, and every invocation fails in ExecutionContextDataSource.getFunctionInstance():

java.lang.NullPointerException: Cannot invoke
  "com.microsoft.azure.functions.spi.inject.FunctionInstanceInjector.getInstance(java.lang.Class)"
  because "this.functionInstanceInjector" is null
    at com.microsoft.azure.functions.worker.binding.ExecutionContextDataSource.getFunctionInstance(ExecutionContextDataSource.java:108)
    at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:20)
    ...

This is the same NPE that was fixed in #684 (which moved the flag assignment to after injector init). The 2.19.2 change reordered it back.

Affected versions

• Good: 2.19.1 and earlier (initializeFunctionInstanceInjector() runs, then oneTimeLogicInitialized = true).
• Regressed: 2.19.2, 2.19.3, 2.19.4 (flag set before injector init).

Confirmed by reading JavaFunctionBroker.java at each tag.

Root cause (current main / 2.19.4)

initializeOneTimeLogics():

if (!oneTimeLogicInitialized) {
    synchronized (oneTimeLogicInitializationLock) {
        if (!oneTimeLogicInitialized) {
            ...
            oneTimeLogicInitialized = true;          // <-- set before init
            initializeFunctionInstanceInjector();    // <-- slow ServiceLoader scan
        }
    }
}

JavaWorkerClient$StreamingMessagePeer.onNext submits every message (incl. per-function FunctionLoadRequests) to Executors.newCachedThreadPool(), so multiple loadMethod calls run concurrently at cold start. Thread A enters the block and sets the flag; thread B sees the flag true and returns from initializeOneTimeLogics() immediately; if B's invocation is dispatched before A finishes the ServiceLoader.load(FunctionInstanceInjector.class) scan, the injector is still null. The scan is slower for apps with a large lib/ classpath and a custom injector provider (e.g. Spring Cloud Function's spring-cloud-function-adapter-azure AzureFunctionInstanceInjector), widening the window.

Impact

An affected instance fails ~100% of invocations until recycled; healthy instances coexist, so it looks intermittent in aggregate. Especially damaging for non-HTTP triggers that fire immediately at cold start (e.g. SQL trigger with a pending change), and for always-on / elastic plans where cold starts happen on deploy, restart, and scale/patch events.

Suggested fix

Restore the #684 ordering: assign oneTimeLogicInitialized = true after initializeFunctionInstanceInjector() completes (and after any other one-time init), so a concurrent thread cannot observe the flag as set while the injector is still null. Consider also initializing the injector eagerly/synchronously before any load request is acked.

Environment

• Java worker 2.19.4 (host 4.1051.300), Linux, Java 21.
• Spring Cloud Function Azure adapter (custom FunctionInstanceInjector provider).

Workaround

Pin the Functions host to a worker <= 2.19.1 via linuxFxVersion (mcr.microsoft.com/azure-functions/java:4.1040.300-0-java21[-appservice]).

[ahmedmuhsin]

Thanks for the detailed report, this is exactly right. The injector init got moved after the flag assignment in 2.19.2, which reopened the race that #684 had closed. A second load thread sees the flag set, skips init, and then hits the null injector on every call.

The fix is up in #880. It puts injector init back before the flag flips and adds a regression test. This will ship in our next release.