Fiber based Job System

Fiber based job system, based on:
Christian Gyrling's GDC talk: Parallelizing the Naughty Dog Engine Using Fibers
Game Engine Architecture 3rd Edition

Implementation is for Windows only.
Besides std::optional from C++17, it should compile with C++11.

API:

class JobSystem {
public:
	// entry point for each job
	using EntryPoint = void(void* param);

	// counter used for synchronizng jobs
	using Counter = std::atomic<I32>;

	// jobs' priority
	enum class Priority {
		LOW = 0, NORMAL = 1, HIGH = 2
	};

	// declaration of each job
	struct Declaration {
		EntryPoint* m_pEntryPoint  = nullptr;
		void*		m_param	   = nullptr;
		Priority	m_priority = Priority::LOW;
		Counter*	m_pCounter = nullptr;
	};

	// kick jobs
	void KickJob(const Declaration& decl);
	void KickJobs(int count, const Declaration aDecl[]);
	
	// wait for counter to become 0
	void WaitForCounter(Counter* pCounter);

	// kick jobs and wait for completion
	void KickJobAndWait(Declaration& decl);
	void KickJobsAndWait(int count, Declaration aDecl[]);
};

Example usage

#include <iostream>		// std::cout
#include <string>		// std::string

#include "JobSystem.h"		// JobSystem

JobSystem g_jobSystem;

void TheMostCreativeWayToCalculateFibonacci(void* pNumberVoid) {
	int* pNumber = reinterpret_cast<int*>(pNumberVoid);
	int n = *pNumber;
	if (n > 1) {
		int fibNMinus1 = n-1;
		int fibNMinus2 = n-2;
		JobSystem::Declaration adecl[2];
		adecl[0].m_pEntryPoint = TheMostCreativeWayToCalculateFibonacci;
		adecl[0].m_param = &fibNMinus1;
		adecl[1].m_pEntryPoint = TheMostCreativeWayToCalculateFibonacci;
		adecl[1].m_param = &fibNMinus2;

		g_jobSystem.KickJobsAndWait(2, adecl);
		n = fibNMinus1 + fibNMinus2;
		*pNumber = n;
	}
	std::cout << n << "\n";
}

int main() {
	int numberOfThreads = std::thread::hardware_concurrency();
	g_jobSystem.Initialize(numberOfThreads);
	int n = 11;
	JobSystem::Declaration decl;
	decl.m_pEntryPoint = TheMostCreativeWayToCalculateFibonacci;
	decl.m_param = &n;
	g_jobSystem.KickJob(decl);

	g_jobSystem.Join();
	g_jobSystem.Terminate();
}

Last printed number is our Fibonacci number.
Higher numbers with kicking two jobs at the time will make you run out of fibers, so increase sizes of queues, fiber pool and wait list appropriately.

Differences between GDC talk

All fibers have the same stack space, but different sizes can be easily implemented.
No adaptive mutexes were implemented, only basic spin lock.
No visualization method was implemented.

How to make it multiplatform

Provide equivalents for things listed below and you are good to go.

Fibers related:

::CreateFiber, ::SwitchToFiber, ::ConvertThreadToFiber

Setting affinity for worker threads:

SetThreadAffinityMask

TLS access

/GT compiler flag, or use workaround mentioned in GDC talk

Sidenote:

G++ emits NOP instruction with _mm_pause() instead of desired pause instruction, so you probably would want to use inline assembly instead when working with that compiler.