Put query updates on a diet

[gbiggs]

Closes open-rmf/rmf#75.

New feature implementation

Implemented feature

Remove the query and query ID from the MirrorUpdate message.

Implementation description

The query ID was previously used to know if a query update should be ignored or applied to the mirror's internal copy of the database. This is no longer necessary because query updates are separated by topic, so a mirror will only receive the updates for its registered query. Thus the query ID is removed from the MirrorUpdate message along with checks on it in the MirrorManager class.

The query was added to the query update message as part of the initial redundancy work to allow a MirrorManager to verify that it is receiving updates for the correct query (it hasn't subscribed to the wrong topic, or the schedule node hasn't got its queries mixed up, etc.). This is inefficient, as the query may be large and query updates are frequent. Therefore the query is removed from the query update message. To retain the functionality that it allowed, the MirrorManager now subscribes to the latched "registered queries" broadcast topic. Each time it receives the list of registered queries from the Schedule node, it verifies both that its own query is present and that the query ID assigned by the Schedule node matches its own opinion of the query ID.

When the query is missing, or the ID is incorrect, the MirrorManager will re-register its query and request a new update.

To handle possible corner cases when the Schedule node fails and the redundant backup takes over (such as a query being registered just before failure and the redundant backup not being aware of it), the MirrorManager will force query validation mode on when it receives the fail-over announcement. However there is a possibility of a query update from the new schedule node arriving before the fail-over announcement, which may not be for the correct query anymore. To identify this situation, the Schedule node is given an "edition" number. Each time fail-over occurs, the new schedule node gets a new edition number. The edition number is sent with each query update. The MirrorManager checks the received edition number, and if it is greater than the expected edition number, it also enters query validation mode. While in query validation mode, it stashes all updates. If the query is validated, then it processes the stashed updates. If the query is found to be missing or the ID is found to be incorrect, the stashed updates are dumped and a fresh update is requested.

[mxgrey]

I haven't reviewed everything yet, but I've looked over the MirrorManager class pretty closely and left some feedback. This design and implementation is looking fantastic; I just have some recommendations, most of which are nitpicks, but some may be actual soundness concerns.

[mxgrey]

This is relevant to #71

[gbiggs]

That just shows how easy a mistake it is to make...

[mxgrey]

I've finished reviewing the rest of the files! Just adding a few more comments. I'll start doing some manual tests of the fail over now.

[codecov]

Codecov Report

Merging #86 (22ab9e0) into main (3d8874d) will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #86      +/-   ##
==========================================
- Coverage   22.11%   21.95%   -0.16%     
==========================================
  Files         410      828     +418     
  Lines       32898    66160   +33262     
  Branches    16050    32044   +15994     
==========================================
+ Hits         7274    14528    +7254     
- Misses      17795    36043   +18248     
- Partials     7829    15589    +7760     

Flag	Coverage Δ
tests	21.95% <ø> (-0.16%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...2/rmf_fleet_adapter/src/rmf_fleet_adapter/Task.hpp
...rmf_fleet_adapter/test/phases/test_RequestLift.cpp
.../src/rmf_task_ros2/bidding/internal_Auctioneer.hpp
...xcpp/RxCpp-4.1.0/Rx/v2/src/rxcpp/rx-observable.hpp
...et_adapter/src/rmf_fleet_adapter/jobs/Planning.cpp
...os2/rmf_traffic_ros2/examples/participant_node.cpp
...lude/rmf_fleet_adapter_python/graph/PyExecutor.hpp
...c_ros2/src/rmf_traffic_ros2/convert_Trajectory.cpp
...src/rmf_fleet_adapter/jobs/detail/impl_Rollout.hpp
.../src/rmf_traffic_ros2/schedule/NegotiationRoom.cpp
... and 1228 more