[O2-4578] DPL: attempt at removing a race condition in the rate limiting

[ktf]

[O2-4578] DPL: attempt at removing a race condition in the rate limiting

[davidrohr]

ok, but please merge only once the retry handling is implemted.

[ktf]

This should be what we need, but it needs testing.

[alibuild]

Error while checking build/O2/fullCI for b9937bc at 2024-03-04 19:53:

## sw/BUILD/O2-latest/log
c++: error: unrecognized command-line option '--rtlib=compiler-rt'
c++: error: unrecognized command-line option '--rtlib=compiler-rt'


## sw/BUILD/O2-full-system-test-latest/log
Detected critical problem in logfile digi.log
digi.log:[30484:internal-dpl-injected-dummy-sink]: [18:52:58][ERROR] Exception caught: Unable to find requested service N2o29framework15ServiceRegistryE with hash 1165381451 using salt 0 for service kind 3 
[30484:internal-dpl-injected-dummy-sink]: [18:52:58][ERROR] Exception caught: Unable to find requested service N2o29framework15ServiceRegistryE with hash 1165381451 using salt 0 for service kind 3 
[ERROR] Workflow crashed - PID 30484 (internal-dpl-injected-dummy-sink) did not exit correctly however it's not clear why. Exit code forced to 128.
[ERROR]  - Device internal-dpl-injected-dummy-sink: pid 30484 (exit 128)
[ERROR] SEVERE: Device internal-dpl-injected-dummy-sink (30484) returned with 128


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/613ad6de4306a6b9657fe4da6d81e026faa81b58/slc8_x86-64/o2checkcode/1.0-local2059/etc/modulefiles
++ cat
--

Full log here.

[davidrohr]

@ktf : What is the status of this? All CIs are red :(

[ktf]

I am on it right now.

[ktf]

This seems to now work now. Tests are passing and AFAICT async reco works. How can we test the FIT issues?