LDAPReplicationDomain.recoverFromReplayFailure() logs WARN_REPLAY_RETRYING_CHANGE once for
every delivery of a change whose replay failed
(opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:3066),
and nothing bounds how many times that happens.
The session is left down for REPLAY_RETRY_DELAY_IN_MS * restarts (:337), capped at
MAX_REPLAY_RETRY_DELAY_IN_MS = 10 s (:343), so a change which keeps failing has its warning
logged every ten seconds for as long as it is retried - roughly 8600 lines a day, per stuck change,
all of them saying the same thing.
Until #901 that flood was bounded by the give-up budget being a constant: five minutes of retrying
is a few dozen lines, then the change is skipped and the warning stops. With replay-give-up-delay
configurable, the bound is whatever the administrator set - and unlimited, a value the property
documents for an operator who would rather have a domain stop than have it diverge, removes it
altogether.
The alert next to it is already throttled
/**
* How long the alert telling that this replica diverges is not sent again. A single
* cause - a schema which does not match, a backend which is gone - makes every change
* in flight unreplayable, and one alert per change would be a storm.
*/
private static final long UNREPLAYED_CHANGE_ALERT_INTERVAL_IN_MS = 60000;
(:349)
The log line has no equivalent guard, for the same storm and the same reasons.
What it would look like
The shape of the alert throttle: the first failure of a change logged as it is now, the repeats
folded - one line per change and per interval, or a line which says how many deliveries have been
retried since the previous one. It has to be per change rather than per domain, unlike the alert
throttle, because the message identifies the change by its CSN; the failures are already kept on the
PendingChange itself, so there is somewhere to hang the last-logged timestamp.
Raised from the review of the #901 branch. Nothing here is a regression of it: the flood is reachable
with any deliberately raised finite budget too, unlimited only takes the ceiling off.
LDAPReplicationDomain.recoverFromReplayFailure()logsWARN_REPLAY_RETRYING_CHANGEonce forevery delivery of a change whose replay failed
(
opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java:3066),and nothing bounds how many times that happens.
The session is left down for
REPLAY_RETRY_DELAY_IN_MS * restarts(:337), capped atMAX_REPLAY_RETRY_DELAY_IN_MS= 10 s (:343), so a change which keeps failing has its warninglogged every ten seconds for as long as it is retried - roughly 8600 lines a day, per stuck change,
all of them saying the same thing.
Until #901 that flood was bounded by the give-up budget being a constant: five minutes of retrying
is a few dozen lines, then the change is skipped and the warning stops. With
replay-give-up-delayconfigurable, the bound is whatever the administrator set - and
unlimited, a value the propertydocuments for an operator who would rather have a domain stop than have it diverge, removes it
altogether.
The alert next to it is already throttled
(
:349)The log line has no equivalent guard, for the same storm and the same reasons.
What it would look like
The shape of the alert throttle: the first failure of a change logged as it is now, the repeats
folded - one line per change and per interval, or a line which says how many deliveries have been
retried since the previous one. It has to be per change rather than per domain, unlike the alert
throttle, because the message identifies the change by its CSN; the failures are already kept on the
PendingChangeitself, so there is somewhere to hang the last-logged timestamp.Raised from the review of the #901 branch. Nothing here is a regression of it: the flood is reachable
with any deliberately raised finite budget too,
unlimitedonly takes the ceiling off.