Skip to content

fix: bound nesting depth when deserializing enclosed payloads - #3500

Merged
pjfanning merged 1 commit into
apache:mainfrom
pjfanning:nested-payload-depth
Sep 1, 2026
Merged

fix: bound nesting depth when deserializing enclosed payloads#3500
pjfanning merged 1 commit into
apache:mainfrom
pjfanning:nested-payload-depth

Conversation

@pjfanning

@pjfanning pjfanning commented Sep 1, 2026

Copy link
Copy Markdown
Member

Motivation

Several of the remoting wire formats carry a payload whose enclosed message is itself a
serialized payload — Some, Optional, Status.Failure, StatusReply, a Throwable
cause, or an ActorSelectionMessage. Each level is parsed on its own, so neither the
protobuf parser's own nesting limit nor the size of the message bounds how deep the chain
can go: the recursion tracks the nesting rather than the number of bytes. A sufficiently
deep chain fails with a StackOverflowError rather than a serialization error, which is
not how the rest of the deserialization path reports a message it cannot read.

Modification

Add a per-thread nesting-depth counter (NestedDeserialization, @InternalApi). A level is
counted where a serializer is actually invoked: Serialization.deserializeByteArray counts
one, and in WrappedPayloadSupport.deserializePayload the two branches that call a serializer
directly count one each, while the branch that delegates back to Serialization leaves the
counting to it. Nesting deeper than pekko.actor.serialization-max-nesting-depth (default 32)
is rejected with a NotSerializableException. Ordinary nesting is unaffected; the default sits
well above anything the wrapper types produce in practice.

Result

An over-nested payload is reported as an ordinary serialization failure instead of a
StackOverflowError. No behaviour change for messages within the depth limit.

Tests

  • sbt "remote/testOnly org.apache.pekko.remote.serialization.NestedPayloadDepthSpec" - 5 passed
  • sbt "actor-tests/testOnly org.apache.pekko.serialization.SerializeSpec org.apache.pekko.serialization.WireManifestClassLoadingSpec" - passed unchanged
  • sbt "actor/mimaReportBinaryIssues" "remote/mimaReportBinaryIssues" - no issues
  • scalafmt on the changed Scala sources

References

None - robustness of nested payload deserialization

Motivation:
Several of the remoting wire formats enclose a serialized payload whose enclosed
message is itself a serialized payload (for example Some, Optional, Status.Failure,
StatusReply, a Throwable cause, or an ActorSelectionMessage). Each level is parsed
separately, so neither a protobuf parser's own nesting limit nor the size of the
message bounds how deep the chain can go relative to the stack: the recursion tracks
the nesting rather than the number of bytes. A sufficiently deeply nested message
fails with a StackOverflowError rather than a serialization error.

Modification:
Add a per-thread nesting-depth counter (NestedDeserialization). A level is counted
where a serializer is actually invoked: Serialization.deserializeByteArray counts
one, and in WrappedPayloadSupport.deserializePayload the two branches that call a
serializer directly count one each, while the branch that delegates back to
Serialization leaves the counting to it. Nesting deeper than
pekko.actor.serialization-max-nesting-depth (default 32) is rejected with a
NotSerializableException. Ordinary nesting depths are unaffected.

Result:
An over-nested payload is rejected as an ordinary serialization error instead of
exhausting the stack.

Tests:
- sbt "remote/testOnly org.apache.pekko.remote.serialization.NestedPayloadDepthSpec" - 5 passed
- sbt "actor-tests/testOnly org.apache.pekko.serialization.SerializeSpec org.apache.pekko.serialization.WireManifestClassLoadingSpec" - 21 passed
- sbt "actor/mimaReportBinaryIssues" "remote/mimaReportBinaryIssues" - no issues
- scalafmt on the changed Scala sources

References:
None - robustness of nested payload deserialization
@pjfanning
pjfanning merged commit 3def397 into apache:main Sep 1, 2026
10 checks passed
@pjfanning
pjfanning deleted the nested-payload-depth branch September 1, 2026 11:34
@pjfanning pjfanning added this to the 2.0.0-M5 milestone Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants