No more use of unordered reactions in federated programs

[arengarajan99]

The current runtime support for federated execution relies on a set of blocking unordered "control reactions" for each input port that to block the handling of reactions until the required port statuses are known. The drawbacks of this design are discussed in Rengarajan 2023, along with a solution that no longer relies on the use of blocking unordered reactions. This PR implements that solution.

This work was initiated by @arengarajan99 and completed by @petervdonovan.

[lhstrh]

This is looking very good! Only some comments missing, but other than that, this looks ready to merge...

[petervdonovan]

I think the timeout in ParallelSourcesMultiport is likely to be a (legitimate) deadlock, although it is hard to be sure since the output was truncated. I do not remember seeing that test deadlock before, so this must be something new. I am trying to reproduce now.

Edit: Failed to reproduce locally on Ubuntu after running 348 times with log level LOG.

Edit: Also failed to reproduce locally on Ubuntu after running 348 times with log level DEBUG.

Edit: Also failed to reproduce on a macOS GH actions runner after running 265 times with log level DEBUG.

[petervdonovan]

Could we maybe pretend that everything is fine? I have run ParallelSourcesMultiport hundreds of times locally and in a macOS GH actions runner without being able to reproduce the latest failure; additionally, I have never seen ParallelSourcesMultiport fail when running it on Wessel with random delays inserted.

For the record, this is all the information we got from CI about the apparent deadlock:

Execution output:
    Federate ParallelSourcesMultiport in Federation ID '3da11054f538fe46ef345bbdfe648c7c7e52d705cef6ded8'
    #### Launching the federate federate__s1.
    #### Launching the federate federate__s2.
    #### Launching the federate federate__d1.
    #### Launching the federate federate__t4.
    #### Bringing the RTI back to foreground so it can receive Control-C.
    RTI -i ${FEDERATION_ID} -n 4 -c init exchanges-per-interval 10

It looks like the program never completed startup. It is possible to imagine that it is somehow not a true deadlock (some resource not available on GH's server, etc.) but I cannot say with confidence that it is not a true deadlock.

[lhstrh]

Could we maybe pretend that everything is fine? I have run ParallelSourcesMultiport hundreds of times locally and a macOS GH actions runner without being able to reproduce the latest failure; additionally, I have never seen ParallelSourcesMultiport fail when running it on Wessel with random delays inserted.

For the record, this is all the information we got from CI about the apparent deadlock:

Execution output:

    Federate ParallelSourcesMultiport in Federation ID '3da11054f538fe46ef345bbdfe648c7c7e52d705cef6ded8'

    #### Launching the federate federate__s1.

    #### Launching the federate federate__s2.

    #### Launching the federate federate__d1.

    #### Launching the federate federate__t4.

    #### Bringing the RTI back to foreground so it can receive Control-C.

    RTI -i ${FEDERATION_ID} -n 4 -c init exchanges-per-interval 10

It looks like the program never completed startup. It is possible to imagine that it is somehow not a true deadlock (some resource not available on GH's server, etc.) but I cannot say with confidence that it is not a true deadlock.

Sounds fair to me. Should this reappear we'll deal with it then.