tie_binding_to_dest_with_keep_while_cond fails with keep_while_conditions_not_met on headers exchange with inbound-only E2E after 4.2.x → 4.3.x upgrade

[akshayagarwal-hub]

Describe the bug

Upgrading from 4.2.1 to 4.3.2 fails to enable the feature flag tie_binding_to_dest_with_keep_while_cond. I have a setup which has E2E with alternate exchange where the destination exchange does not have any outbound bindings but only has an alternate exchange. Enabling the flag looks for a binding for the destination exchange but fails to find one as alternate exchanges are not added as bindings. In my case i want to add the binding to the destination exchange dynamically when a particular job is triggered so exchanges without outbound bindings are expected.

Reproduction steps

• Create a 4.2.1 cluster on Khepri.
• Declare a topology matching a multi-tiered exchange-to-exchange cascade: Topic Exchange -> Headers Exchange (with alternate-exchange). Leave the intermediate headers exchange with no direct outbound queue bindings.
• Perform an inline upgrade of the data volume to RabbitMQ 4.3.2.
• Run rabbitmqctl enable_feature_flag tie_binding_to_dest_with_keep_while_cond.

Try running the below script in a local kubernetes setup

repro2.sh

Error:

{:exception, {:badmatch, {:error, {:khepri, :keep_while_conditions_not_met, %{node_name: "turbine.qos.container_jobs", node_path: [:rabbitmq, :vhosts, "/", :exchanges, "turbine.qos.container_jobs"], keep_while_reason: {:pattern_matches_no_nodes, [:rabbitmq, :vhosts, "/", :exchanges, "turbine.qos.container_jobs", :bindings, {:if_name_matches, :any, :undefined}, {:if_name_matches, :any, :undefined}, {:if_name_matches, :any, :undefined}]}}}}}, [{:rabbit_db_queue, :"-tie_binding_to_dest_with_keep_while_cond_enable/1-fun-0-", 2, [file: ~c"rabbit_db_queue.erl", line: 1032]}, {:khepri_tx, :"-foreach/3-fun-0-", 4, [file: ~c"src/khepri_tx.erl", line: 580]}, {:khepri_tree, :find_matching_nodes_cb, 5, [file: ~c"src/khepri_tree.erl", line: 614]}, {:khepri_tree, :walk_down_the_tree1, 1, [file: ~c"src/khepri_tree.erl", line: 1250]}, {:khepri_tree, :handle_branch, 3, [file: ~c"src/khepri_tree.erl", line: 1316]}, {:maps, :fold_1, 4, [file: ~c"maps.erl", line: 860]}, {:khepri_tree, :walk_down_the_tree1, 1, [file: ~c"src/khepri_tree.erl", line: 1196]}, {:khepri_tree, :handle_branch, 3, [file: ~c"src/khepri_tree.erl", line: 1316]}]}

Expected behavior

Enabling the flag should work in this case and handle the scenario.

Additional context

No response

[michaelklishin]

There's prior art around this feature flag.

Our best understanding of the conditions that can cause this feature flag's transition function to fail are covered in detail in #16587 and the known scenario was addressed in #16590.

[michaelklishin]

@akshayagarwal-hub thanks for the script and the initial analysis.

Compared to #16587, this is a similar scenario but the culprit metadata store object is an auto-delete exchange that has no outbound bindings.

Workarounds

Clusters without such auto-delete exchanges would not run into this exception, so
one workaround is

1. Exporting definitions
2. Deleting the auto-delete exchanges
3. Upgrading
4. Enabling all stable feature flags, then re-importing importing the definitions file

Or

1. Adding an outbound binding to the auto-delete exchange(s) (that is, where the exchange(s) is the source, not the destination: add an unused durable queue with a unique name and use it the binding)
2. Upgrading
3. Enabling all stable feature flags

Non-auto-delete exchanges should not exhibit this behavior.

Fixes

#16824 addresses this exception and will ship in 4.3.3.