Jolt physics using Godot/M4 on Switch2

[TimLutherRS]

I realise this is quite a specific ask but I'm having some headaches with this and could use some advice.

As the title indicates, I'm trying to get Jolt to work on the Switch inside the Godot engine. I'm using the M4 console friendly branch of Godot which disables Jolt support.. probably because of the problems I'm seeing.
I've reenabled it and I'm having what I think are probably race conditions?

It hits here:

JPH::JobHandle JoltJobSystem::CreateJob(const char *p_name, JPH::ColorArg p_color, const JPH::JobSystem::JobFunction &p_job_function, JPH::uint32 p_dependency_count) {
	Job *job = nullptr;
	while (true) {
		JPH::uint32 job_index = jobs.ConstructObject(p_name, p_color, this, p_job_function, p_dependency_count);

		if (job_index != JPH::FixedSizeFreeList<Job>::cInvalidObjectIndex) {
			job = &jobs.Get(job_index);
			break;
		}

		WARN_PRINT_ONCE("Jolt Physics job system exceeded the maximum number of jobs. This should not happen. Please report this. Waiting for jobs to become available...");

		OS::get_singleton()->delay_usec(100);

		_reclaim_jobs();
	}

	// This will increment the job's reference count, so must happen before we queue the job
	JPH::JobHandle job_handle(job);

	if (p_dependency_count == 0) {
		QueueJob(job);
	}

	return job_handle;
}

And then, after a while and depending upon the complexity of the scene, gets stuck in an infinite loop at that WARN_PRINT ONCE - the job_index is 0xFFFFFFFF forever.
I've tried toying with 'cMaxPhysicsJobs' (upping it from 2048) but that just causes memory access crashes, so I'm thinking I need to change other things as well to get that idea to work.
Within 'FixedSizeFreeList.inl' there seems to be a lot of interesting things going on with atomics and I'm wondering if some of the logic here about what should be locked and unlocked when might not be playing nice on the switch but I'm sure people here understand the intent of this code better than I, hence my plea for help.

Does anybody know what might be going on here, is any of this familiar sounding?

[jrouwe]

This may be related to godotengine/godot#110724 with the difference that I've never heard of anyone actually livelocking on this. If you have a lot more threads than you have CPUs then you might run into this message, although it should never livelock.

Looking at the code I think there could be an ABA problem in JoltJobSystem::Job::pop_completed vs JoltJobSystem::Job::push_completed that could cause completed jobs to become orphaned and thus slowly leak. But without more information it's hard to tell.

I'd appreciate if you moved this ticket to the godot repository or alternatively added information to godotengine/godot#110724 as this may not be a Jolt issue at all.

What would help is if you added the callstacks of all other threads when the livelock occurs (so we can see if there are any Jolt jobs still executing) and if you could look into the state of JoltJobSystem::jobs to see if the jobs in that list still have to be executed by Jolt (!IsDone()) and if completed_head == nullptr while there are some jobs have a completed_next != nulptr (indicating the ABA problem happened). If the debugger makes it hard to see the jobs, you could add a bit of code that prints out the state of all jobs if that while loop executes more than e.g. 1000 times.