Waiting for WorkerThreadPool task completion occasionally hangs despite is_task_completed

[j-zeppenfeld]

Godot version

v4.2.beta6.official [6415006]

System information

Godot v4.2.beta6 - Windows 10.0.19045 - Vulkan (Mobile) - dedicated NVIDIA RTX A2000 Laptop GPU (NVIDIA; 31.0.15.2737) - 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz (16 Threads)

Issue description

I have been looking into the occasional crash I mentioned in #84888, and while I haven't been able to reliably replicate the crash, I have found a more reproducible issue that seems to be a prerequisite for the crash.

When running the code below, i.e., keeping a largish number of tasks running in the WorkerThreadPool, occasionally waiting for completion will hang the main thread even if is_task_completed has previously indicated that the task is completed. I would expect the wait_for_task_completion to always return instantly (well below 1ms) since is_task_completed indicates that the task is already done.

The crash mentioned in #84888 is correlated to when it takes over 1 second waiting for the task to complete, and very rarely happens here as well.

Steps to reproduce

extends Node2D

var _tasks: Array[int] = []

func work(data: Array):
	for i in randi_range(50000, 2000000):
		data[i % 1000] += data[(i + 11) % 1000]

func _process(_delta: float):
	if _tasks.size() < 100:
		var data = []
		data.resize(1000)
		data.fill(0)
		var task: int = WorkerThreadPool.add_task(work.bind(data))
		_tasks.push_back(task)
	
	var t: int = 0
	while t < _tasks.size():
		if WorkerThreadPool.is_task_completed(_tasks[t]):
			var us: int = Time.get_ticks_usec()
			WorkerThreadPool.wait_for_task_completion(_tasks[t])
			us = Time.get_ticks_usec() - us
			if us > 1000:
				print("Took ", us, "us waiting for task to complete.")
			_tasks.pop_at(t)
		else:
			t += 1

func _exit_tree():
	for task in _tasks:
		WorkerThreadPool.wait_for_task_completion(task)

On my machine, this results in output similar to the following (over the course of ~1 minute):

Took 1009us waiting for task to complete.
Took 1793612us waiting for task to complete.
Took 1798884us waiting for task to complete.
Took 2160us waiting for task to complete.
Took 1390us waiting for task to complete.

In Godot v4.1.3.stable.official [f06b6836a], I do not get any extreme hangs of over a second. However, I regularly get hangs for several milliseconds (much more often than in v4.2, several times a second).

Took 1123us waiting for task to complete.
Took 3836us waiting for task to complete.
Took 2146us waiting for task to complete.
Took 6938us waiting for task to complete.
Took 3780us waiting for task to complete.
...

Minimal reproduction project

A single node with the script above is sufficient to reproduce this issue. If a project is needed I can happily supply one upon request.

[j-zeppenfeld]

This appears to have been fixed in v4.2.rc2.official [1ba920f]