Shovel duplicates exchange messages for each node in RabbitMQ Cluster since 3.8.16

[J12934]

Hi you'all 👋

I'm running into Problems with the Shovel plugin since upgrading RabbitMQ from 3.8.15 to 3.8.16.
We are using the Shovel plugins to replicate all messages from our production RabbitMQ cluster to our dev cluster.
Before v3.8.16 the shovel job was running on a single node and worked brilliantly. After the upgrade the Shovel is executed on every node of the cluster. As we're running with 3 nodes per cluster this causes all messages to be multiplicated 3 times. 😕

I found a similar report (#3065), but for 3.8.14 where it was definitely working fine on our clusters so I'm opening this as a new discussion.

I can't anything in the changelog which sounds like it could cause this problem, anybody have any ideas?

Shovel Status

Shovel status in 3.8.15

Shovel status in 3.8.16

Message Rate

The Message rate of the exchange should be around 110 - 130 messages/s

Message Rate in 3.8.15

Message Rate in 3.8.16

Configuration

We deploy the Shovel via a definition file:

{
    "users": [
        {
            "name": "shovel",
            "password_hash": "",
            "hashing_algorithm": "rabbit_password_hashing_sha256",
            "tags": "",
            "limits": {}
        }
    ],
    "vhosts": [
        {
            "name": "/"
        }
    ],
    "permissions": [
        {
            "user": "shovel",
            "vhost": "/",
            "configure": ".*",
            "write": ".*",
            "read": ".*"
        }
    ],
    "topic_permissions": [],
    "parameters": [
        {
            "value": {
                "ack-mode": "no-ack",
                "dest-add-forward-headers": false,
                "dest-exchange": "vessel_positions",
                "dest-protocol": "amqp091",
                "dest-uri": "amqps://broker-playground.example.com?cacertfile=/etc/destination-ca.cert.pem&certfile=/etc/destination.cert.pem&keyfile=/etc/destination.key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external",
                "src-delete-after": "never",
                "src-exchange": "vessel_positions",
                "src-exchange-key": "#",
                "src-prefetch-count": 100,
                "src-protocol": "amqp091",
                "src-uri": "amqps://broker.example.com?cacertfile=/etc/source-ca.cert.pem&certfile=/etc/source.cert.pem&keyfile=/etc/source.key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external"
            },
            "vhost": "/",
            "component": "shovel",
            "name": "shovel"
        }
    ],
    "exchanges": [
        {
            "name": "vessel_positions",
            "vhost": "/",
            "type": "topic",
            "durable": true,
            "auto_delete": false,
            "internal": false,
            "arguments": {}
        }
    ]
}

[michaelklishin]

Moved to #3154 and hopefully answered there.

[michaelklishin]

Follow-up: #3260, #3263