Skip to content

Stop PersistentActor by default when RecoveryCompleted handler fails#3359

Merged
He-Pin merged 1 commit into
apache:mainfrom
He-Pin:fix-3234-recovery-completed-failure
Jul 17, 2026
Merged

Stop PersistentActor by default when RecoveryCompleted handler fails#3359
He-Pin merged 1 commit into
apache:mainfrom
He-Pin:fix-3234-recovery-completed-failure

Conversation

@He-Pin

@He-Pin He-Pin commented Jul 17, 2026

Copy link
Copy Markdown
Member

Motivation

A classic PersistentActor can enter an unbounded restart loop when its RecoveryCompleted handler throws because the Classic default decider restarts ordinary exceptions. Issues #3234 and #3249 describe the same trigger and root cause.

RecoveryCompleted runs after event replay has succeeded, so this failure should remain visible to the parent supervisor instead of being translated into an onRecoveryFailure replay failure.

Modification

  • Wrap NonFatal exceptions from the RecoveryCompleted handler in ActorInitializationException and pass them to Classic supervision.
  • Preserve the existing processing-state transition and outer recovery-permit cleanup path.
  • Add directional regression coverage for both the default Stop decision and a custom parent Restart decision.
  • Document the resulting supervision contract.

Result

The default supervision strategy stops the actor instead of repeatedly restarting it. A custom parent strategy can still Resume or Restart the same ActorRef, and the original exception is retained as the cause.

Tests

  • sbt "persistence / Test / testOnly org.apache.pekko.persistence.RecoveryPermitterSpec" — restart-direction test failed before the production change; 7 passed after it
  • sbt "persistence / Test / test" — 62 passed
  • sbt docs/paradox — passed with existing duplicate-anchor warnings
  • sbt headerCreateAll — passed
  • sbt +headerCheckAll — passed for Scala 2.13.18 and 3.3.8
  • sbt checkCodeStyle — passed
  • sbt validatePullRequest — ran for 24m44s; affected persistence, typed, query, cluster-sharding, multi-JVM, and docs tests completed. Overall exit 1 is limited to unchanged baseline/environment failures: LeveldbJournalNativeSpec cannot load the x86 leveldbjni-all 1.8 binary on macOS ARM64, and IntegrationDocTest references the existing invalid class pekko.dispatch.UnboundedMailbox
  • sbt sortImports — not completed because the repository Scalafix rule fails against the current scala.meta with NoSuchMethodError; no unrelated changes were retained
  • scalafmt --list --mode diff-ref=origin/main — passed
  • git diff --check — passed
  • Qoder stdout review — No must-fix findings

References

Fixes #3234
Fixes #3249

@He-Pin He-Pin added this to the 2.0.0-M4 milestone Jul 17, 2026
Motivation:
A classic PersistentActor can restart indefinitely under the default supervision strategy when its RecoveryCompleted handler throws.

Modification:
Wrap exceptions from the RecoveryCompleted handler in ActorInitializationException before passing them to the parent supervisor. Keep recovery permit cleanup in the existing outer catch, add directional tests for the default Stop decision and a custom parent Restart decision, and document the supervision contract.

Result:
The default strategy stops the actor instead of looping, while a custom parent supervisor can still resume or restart it. The original failure remains available as the exception cause.

Tests:
- sbt "persistence / Test / testOnly org.apache.pekko.persistence.RecoveryPermitterSpec" (7 passed; restart test failed before the fix)
- sbt "persistence / Test / test" (62 passed)
- sbt docs/paradox
- sbt headerCreateAll
- sbt +headerCheckAll
- sbt checkCodeStyle
- sbt validatePullRequest (24m44s; failed only on unchanged baseline/environment issues: x86 LevelDB JNI on macOS ARM64 and IntegrationDocTest using pekko.dispatch.UnboundedMailbox)
- sbt sortImports (not completed: repository Scalafix rule failed with scala.meta NoSuchMethodError)
- scalafmt --list --mode diff-ref=origin/main
- git diff --check
- Qoder stdout review: No must-fix findings

References:
Fixes apache#3234, Fixes apache#3249
@He-Pin
He-Pin force-pushed the fix-3234-recovery-completed-failure branch from 4818e0e to 074671f Compare July 17, 2026 06:25
@He-Pin He-Pin changed the title Stop PersistentActor when RecoveryCompleted handler fails Stop PersistentActor by default when RecoveryCompleted handler fails Jul 17, 2026

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@He-Pin
He-Pin merged commit 1fa250d into apache:main Jul 17, 2026
10 checks passed
@He-Pin
He-Pin deleted the fix-3234-recovery-completed-failure branch July 17, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants