Repository navigation
A replicated __compaction subscription stops geo-replication (IllegalArgumentException in PulsarCompactorSubscription.acknowledgeMessage)
#27030
SummaryWhen the system subscription We saw this in production on Pulsar 4.0.14. One topic built a replication backlog of more than 95 million messages. The script below reproduces it on 4.0.13 and 4.0.14 with two standalone clusters. The broker log below is from 4.0.14, and its line numbers are the same as in our production log. Pulsar does not stop a client from setting the replicated status on Broker logRoot cause
Code references (v4.0.14):
On How to reproduceThe script needs Docker. It starts two standalone clusters (
Only to make the stall quick, the script sets repro.sh#!/usr/bin/env bash
# Reproduce: a replicated __compaction subscription stalls the geo-replicator (Pulsar 4.0.13 and 4.0.14).
# Needs Docker. Runs two standalone clusters (cluster-a, cluster-b) with geo-replication.
# Cleanup: ./repro.sh --cleanup
set -euo pipefail
IMAGE=${IMAGE:-apachepulsar/pulsar:4.0.14}
NET=pulsar-geo-repro
TOPIC=persistent://geo/ns/compacted
CONTROL=persistent://geo/ns/control
cleanup() { docker rm -f pulsar-a pulsar-b >/dev/null 2>&1 || true; docker network rm "$NET" >/dev/null 2>&1 || true; }
if [[ "${1:-}" == "--cleanup" ]]; then cleanup; exit 0; fi
cleanup
docker network create "$NET" >/dev/null
start() { # container cluster
docker run -d --name "$1" --hostname "$1" --network "$NET" \
-e clusterName="$2" -e advertisedAddress="$1" \
-e brokerDeleteInactiveTopicsEnabled=false \
-e PULSAR_MEM="-Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g" \
"$IMAGE" bash -c "bin/apply-config-from-env.py conf/standalone.conf && exec bin/pulsar standalone -nfw -nss" >/dev/null
}
admin() { docker exec "$1" bin/pulsar-admin "${@:2}"; }
wait_up() { until docker exec "$1" curl -sf http://localhost:8080/admin/v2/clusters >/dev/null 2>&1; do sleep 2; done; }
compact() { # container topic: run one compaction and wait until the __compaction cursor moves
local before; before=$(md "$1" "$2")
admin "$1" topics compact "$2" >/dev/null
until [[ "$(md "$1" "$2")" != "$before" ]]; do sleep 1; done
}
md() { admin "$1" topics stats-internal "$2" | python3 -c 'import json,sys; c=json.load(sys.stdin)["cursors"]; print(c.get("__compaction",{}).get("markDeletePosition",""))'; }
repl() { # container topic remote
admin "$1" topics stats "$2" | python3 -c '
import json,sys; d=json.load(sys.stdin); r=d["replication"][sys.argv[1]]
print("msgRateIn=%.0f/s replication msgRateOut=%.0f/s replicationBacklog=%d connected=%s" % (d["msgRateIn"], r["msgRateOut"], r["replicationBacklog"], r["connected"]))' "$3"
}
start pulsar-a cluster-a
start pulsar-b cluster-b
wait_up pulsar-a; wait_up pulsar-b
echo "== 0. two standalone clusters are up; set up geo-replication for geo/ns"
for n in a b; do
for o in a b; do
[[ $n == "$o" ]] && continue
admin pulsar-$n clusters create cluster-$o --url http://pulsar-$o:8080 --broker-url pulsar://pulsar-$o:6650
done
admin pulsar-$n tenants create geo --allowed-clusters cluster-a,cluster-b
admin pulsar-$n namespaces create geo/ns --clusters cluster-a,cluster-b
done
# Optional, only to make the stall quick. With the default (1000), the stall needs more failures
# with large read batches (for example, while the replicator catches up on a backlog).
admin pulsar-a brokers update-dynamic-config --config replicationProducerQueueSize --value 50
for t in "$TOPIC" "$CONTROL"; do
admin pulsar-a topics create "$t"
until admin pulsar-b topics list geo/ns 2>/dev/null | grep -q "$t"; do sleep 1; done
done
echo "== 1. continuous traffic on both clusters, on both topics"
for t in "$TOPIC" "$CONTROL"; do
docker exec -d pulsar-a bash -c "bin/pulsar-perf produce -r 3000 -s 200 $t > /dev/null 2>&1"
docker exec -d pulsar-b bash -c "bin/pulsar-perf produce -r 20 -s 200 $t > /dev/null 2>&1"
done
sleep 15
echo "== 2. compact $TOPIC on both clusters (creates __compaction on A and B)"
compact pulsar-a "$TOPIC"; compact pulsar-b "$TOPIC"
echo "== 3. mark __compaction as replicated on both clusters"
admin pulsar-a topics set-replicated-subscription-status "$TOPIC" -s __compaction --enable
admin pulsar-b topics set-replicated-subscription-status "$TOPIC" -s __compaction --enable
sleep 10
repl pulsar-a "$TOPIC" cluster-b
echo "== 4. compact on B a few times; each run makes B send a subscription-update marker for __compaction to A"
for i in 1 2 3 4 5 6; do compact pulsar-b "$TOPIC"; sleep 10; done
repl pulsar-a "$TOPIC" cluster-b
echo "== 5. unload the topic on A (same as a broker restart or a bundle move); the replicator re-reads the old markers"
admin pulsar-a topics unload "$TOPIC"
for i in 1 2 3; do compact pulsar-b "$TOPIC"; sleep 10; done
echo "-- replicator cluster-a --> cluster-b of $TOPIC:"
for i in 1 2 3; do repl pulsar-a "$TOPIC" cluster-b; sleep 15; done
echo "-- control topic (no replicated __compaction), same brokers and same traffic:"
repl pulsar-a "$CONTROL" cluster-b
echo "== 6. broker log of cluster-a"
echo "failures: $(docker logs pulsar-a 2>&1 | grep -c 'Unexpected exception in replication task')"
docker logs pulsar-a 2>&1 | grep -m1 -A8 "Unexpected exception in replication task" | sed -E 's/^[0-9T:,+.-]+ //'Output on 4.0.14: After the unload, the replicator of Workaround (tested on 4.0.13 with the setup of the script)
Possible fixes
We can open an issue and a pull request for one or more of these fixes. Which fix do the maintainers prefer? |
Replies: 3 comments 1 reply
|
I opened a bug issue for this: #27031 It also covers the current state of |
|
Update: the fix is ready for review.
The PR ignores Feedback from the maintainers is welcome, also about a backport to |
|
Update: #27032 (ignore replicated subscription updates for |
Update: the fix is ready for review.
The PR ignores
REPLICATED_SUBSCRIPTION_UPDATEmarkers for__compactioninReplicatedSubscriptionsController.receiveSubscriptionUpdated. The compactor owns this cursor, so a remote update can never be applied to it. The check is on the receiving side, so it also clears markers that are already in the backlog of a stuck cluster. It also stops a remote update from creating__compactionas an ordinary replicated subscription on a cluster that has no compaction subscription yet.Feedback from the maintainers is welcome, also about a backport to
branch-4.0.