Upgrade to 4.3.0 can fail when the tie_binding_to_dest_with_keep_while_cond feature flag is enabled

[olakunlecyferd]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify)

Erlang version used

27.3.x

Operating system (distribution) used

Ubuntu 24.04.4 LTS

How is RabbitMQ deployed?

Kubernetes Operator(s) from Team RabbitMQ

rabbitmq-diagnostics status output

rabbitmqctl enable_feature_flag tie_binding_to_dest_with_keep_while_cond

Enabling feature flag "tie_binding_to_dest_with_keep_while_cond" ...
Error:
{:exception, :function_clause,
 [{:rabbit_db_queue,
   :"-tie_binding_to_dest_with_keep_while_cond_enable/1-fun-0-",
   [[:rabbitmq, :vhosts, "dev.domain", :exchanges, ".*"],
    %{payload_version: 1}],
   [file: ~c"rabbit_db_queue.erl", line: 1024]},
  {:khepri_tx, :"-foreach/3-fun-0-", 4,
   [file: ~c"src/khepri_tx.erl", line: 580]},
  {:khepri_tree, :find_matching_nodes_cb, 5,
   [file: ~c"src/khepri_tree.erl", line: 614]},
  {:khepri_tree, :walk_down_the_tree1, 1,
   [file: ~c"src/khepri_tree.erl", line: 1250]},
  {:khepri_tree, :handle_branch, 3,
   [file: ~c"src/khepri_tree.erl", line: 1316]},
  {:maps, :fold_1, 4,
   [file: ~c"maps.erl", line: 860]},
  {:khepri_tree, :walk_down_the_tree1, 1,
   [file: ~c"src/khepri_tree.erl", line: 1196]},
  {:khepri_tree, :handle_branch, 3,
   [file: ~c"src/khepri_tree.erl", line: 1316]}]}

Logs from node 1 (with sensitive values edited out)

2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags: error while running:
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:   rabbit_db_queue:tie_binding_to_dest_with_keep_while_cond_enable...
...
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:   exception error: {exception,function_clause,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                        [{rabbit_db_queue,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             '-tie_binding_to_dest_with_keep_while_cond_enable/1-fun-0-',
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             [[rabbitmq,vhosts,<<"dev.cyferd">>,exchanges,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                               <<".*">>],
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                              #{payload_version => 1}],
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             [{file,"rabbit_db_queue.erl"},{line,1024}]},

Logs from node 2 (if applicable, with sensitive values edited out)

2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags: error while running:
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:   rabbit_db_queue:tie_binding_to_dest_with_keep_while_cond_enable...
...
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:   exception error: {exception,function_clause,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                        [{rabbit_db_queue,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             '-tie_binding_to_dest_with_keep_while_cond_enable/1-fun-0-',
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             [[rabbitmq,vhosts,<<"dev.cyferd">>,exchanges,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                               <<".*">>],
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                              #{payload_version => 1}],
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             [{file,"rabbit_db_queue.erl"},{line,1024}]},

Logs from node 3 (if applicable, with sensitive values edited out)

2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags: error while running:
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:   rabbit_db_queue:tie_binding_to_dest_with_keep_while_cond_enable...
...
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:   exception error: {exception,function_clause,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                        [{rabbit_db_queue,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             '-tie_binding_to_dest_with_keep_while_cond_enable/1-fun-0-',
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             [[rabbitmq,vhosts,<<"dev.cyferd">>,exchanges,
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                               <<".*">>],
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                              #{payload_version => 1}],
2026-05-28 16:56:50.767623+00:00 [error] <0.13671.0> Feature flags:                             [{file,"rabbit_db_queue.erl"},{line,1024}]},

rabbitmq.conf

queue_master_locator                       = min-masters
disk_free_limit.absolute                   = 2GB
cluster_partition_handling                 = pause_minority
cluster_formation.peer_discovery_backend   = rabbit_peer_discovery_k8s
cluster_formation.k8s.host                 = kubernetes.default
cluster_formation.k8s.address_type         = hostname
cluster_formation.target_cluster_size_hint = 3
cluster_name                               = rabbitmq-dev
auth_mechanisms.1                          = PLAIN
auth_mechanisms.2                          = AMQPLAIN

RabbitMQ is deployed using the RabbitMQ Kubernetes Cluster Operator on Kubernetes. No custom Khepri, feature flag, exchange, queue, or binding configuration has been applied beyond the configuration shown above.

Steps to deploy RabbitMQ cluster

Deploy RabbitMQ using the RabbitMQ Kubernetes Cluster Operator.
Create a 3-node RabbitMQ cluster using a RabbitmqCluster custom resource.

Steps to reproduce the behavior in question

Steps to reproduce the behavior

• Deploy a 3-node RabbitMQ cluster on Kubernetes using the RabbitMQ Cluster Operator.
• Install RabbitMQ version 4.2.6.
• Create and use a vhost (in my case dev.domain) with exchanges, queues, and bindings.
• Upgrade the cluster from 4.2.6 to 4.3.0.
• Verify that all RabbitMQ nodes are healthy and the cluster is operational.
• Check feature flags and observe that tie_binding_to_dest_with_keep_while_cond remains disabled.
• Attempt to enable the feature flag:

  rabbitmqctl enable_feature_flag tie_binding_to_dest_with_keep_while_cond
  Observe the following error:
  {:exception, :function_clause,
   [{:rabbit_db_queue,
     :"-tie_binding_to_dest_with_keep_while_cond_enable/1-fun-0-",
     [[:rabbitmq, :vhosts, "dev.domain", :exchanges, ".*"],
      %{payload_version: 1}],
     ...}]}
  

Additional observations
The cluster remains healthy and fully functional after the upgrade.
Khepri metadata store reports healthy status on all nodes.
No deprecated features are reported by rabbitmq-diagnostics check_if_any_deprecated_features_are_used.
No exchange named .* exists when listing exchanges in the affected vhost.
The issue is reproducible every time the feature flag enablement is attempted.

What problem are you trying to solve?

After upgrading RabbitMQ from 4.2.6 to 4.3.0, I want to complete the upgrade by enabling all required feature flags and removing the warning shown in the Management UI.

The cluster is healthy and fully operational, but the feature flag tie_binding_to_dest_with_keep_while_cond cannot be enabled because it consistently fails with a function_clause exception.

I am trying to understand whether this is a known issue with the upgrade path from 4.2.x to 4.3.x, whether there is a supported workaround, or whether the feature flag migration is encountering an unexpected metadata state that requires remediation.

[lukebakken]

It would be extremely helpful if you detailed the exact upgrade steps you took.

https://www.rabbitmq.com/docs/upgrade

For instance, did you ensure that all feature flags were enabled prior to the upgrade? https://www.rabbitmq.com/docs/feature-flags#how-to-enable-feature-flags

[michaelklishin]

A proposed fix #16590.

[dumbbell]

I can’t reproduce with the instructions you shared. Could you please give more details if you can reproduce? Like what exchanges/queues/bindings you create exactly?

I need to understand why you end up with an exchange tree node in the metadata store without an exchange stored underneath.

[michaelklishin]

And what exchanges, bindings are deleted since our current understanding is that deletions play a role here.

[michaelklishin]

@dumbbell I have some additional details.

Khepri uses khepri_payload:none/0 as a placeholder (e.g. in khepri_tree:insert_or_update_node_cb/5) and only prunes a parent through a keep_while watcher, there is no automatic GC of empty branches. Once the orphan is left behind, it persists.

Three code paths can leave a node at [..., exchanges, X, …] without a corresponding #exchange{} payload one level up.

Topic Permissions

The exchange field of #topic_permission{} is taken as an opaque binary and not validated
for existgence. rabbit_db_user:set_topic_permissions_in_khepri_tx/3 writes straight to [..., exchanges, Exchange, user_permissions, Username].

The .* value in the report is likely a "match anything" pattern passed to rabbitmqctl set_topic_permissions.

Definition import and rabbit_db_user_m2k_converter:copy_to_khepri/3 run against the rabbit_topic_permission table have the same effect because they end up calling rabbit_db_user:set_topic_permissions_in_khepri_tx/3.

Binding Creation

Binding creation seems to have a genuine race with tthe deletion of the source exchange. rabbit_db_binding:create/2 does the source/destination resource lookup outside the Khepri transaction. If the source is deleted between the lookup and the khepri_tx:put/2, the put recreates [..., exchanges, X] as a data-less node.

Before tie_binding_to_dest_with_keep_while_cond the leaf carries no keep_while,
so it is never cleaned up.

Exchange Serial Bump

rabbit_db_exchange:next_serial_in_khepri_tx/1 puts to [..., exchanges, X, serial]
and can resurrect the parent immediately after X's deletion.

khepri_tx:foreach/2 then hands each match to the callback as a node_props_map.
For a data-less node khepri_tree:gather_node_props/2 returns #{payload_version => 1} (note the missing data key), which the #{data := Exchange} head rejects, failing with a function_clause.

The #if_has_data{} in #16590 takes the same approach as rabbit_db_exchange:recover/1
and a few listing functions already use.

Possible Reproduction Example

Consider a cluster on 4.2.6 with a virtual host named dev.cyferd.

Operator wanted user app to publish on any topic exchange in dev.cyferd and ran

rabbitmqctl set_topic_permissions -p dev.cyferd app ".*" ".*" ".*"

The exchange-name argument (the first ".*") is taken as an opaque binary by rabbit_auth_backend_internal:set_topic_permissions/6. Only the read and write patterns are regex-validated, the exchange field isn't checked against anything.

The record lands at [..., exchanges, <<".*">>, user_permissions, <<"app">>] in the metadata store, the store then creates the intermediate exchanges/<<".*">> as khepri_payload:none/0,
and the orphan sits there.

Definitions import that carries the same topic permissions can have the same effect and is
very common to use during deployments via a definitions file referenced from rabbitmq.conf.

Things work well until rabbitmqctl enable_feature_flag tie_binding_to_dest_with_keep_while_cond walks [rabbitmq, vhosts, *, exchanges, *] and the callback's #{data := Exchange} head meets a node with no data key. Boom.

set_topic_permissions accidentally invoked as if it was set_permissions

I have a feeling the difference between set_permissions and set_topic_permissions is what triggers the incorrect invocation here:

• set_permissions's arguments: a username + three patterns (configure, write, read)
• set_topic_permissions's arguments: a username + an exchange name + two patterns (write, read)

[dumbbell]

• Topic permissions: indeed, nothing enforces that the exchange exists before setting a topic permission on it. I checked the Mnesia-based code and that’s the same. So we always allowed to set a topic permission on a non-existing exchange.

• Binding creation: this is a known issue, we should put a FIXME there until we fix it.

• Exchange serial bump: like topic permissions, the code with Mnesia didn’t check the existence of the exchange first. I wonder if we should change the put to ensure the exchange exists. Unfortunately, the next_serial/1 function always returns a serial: it has no room for error handling.

With this in mind, I think your proposed patch is fine. I will review it in #16590.

[michaelklishin]

The simplest explanation I can think of:

1. A user runs rabbitmqctl set_topic_permissions and screws up the exchange name argument (passes it as .*) due to the set_permissions muscle memory
2. We do not validate exchange existence and store the permission in Khepri
3. The Khepri tree node becomes a data-less orphan
4. We do not filter it out the same way we do elsewhere in a specific recent feature flag path
5. Boom, a function_clase, the upgrade fails, and there is no easy workaround on top (you cannot really delete a bogus permission)

Filtering out data-less nodes should be enough.