reflect: adopt message values from another descriptor pool on set#308
Conversation
A generated type reflects against its *defining* crate's DescriptorPool: a
Duration view always reports buffa-types' pool, never the consumer pool that
references it. So a nested cross-crate message value can never be
pointer-identical to its parent's pool, and validating message values by
Arc::ptr_eq rejected every one of them — with a self-contradictory error:
field "timeout" (#2) on verify.Job
expects message google.protobuf.Duration,
got message google.protobuf.Duration
That made the vtable rebuild walk (for_each_set + set(fd, vr.to_owned()))
panic for any message with a well-known-type field. It is not WKT-specific:
any extern_path-mapped proto hits it, and so do two packages generated in one
codegen run, which get distinct pool Arcs. Views, owned types, and the bridge
path are all affected; only lazy views escape, having no reflect impls.
try_set now adopts rather than rejects. A message of the same full name from a
foreign pool is re-homed into the target pool by a wire round-trip, which
re-homes the whole subtree in one decode, recursively covering singular, list
and map values. Descriptors stay strict — you still pass field descriptors
resolved from the message's own pool — but values are adopted, because the
codegen makes pool-identical values impossible for a cross-crate type. Full
name is the adoption key, the same criterion Any's type URL uses, so a
genuinely wrong type is still WrongValueKind whatever pool it came from.
The strict check doubles as the "already homed and well-shaped" predicate, so a
value that needs nothing is returned untouched and the common path — every
value the decoder and the JSON parser produce — allocates nothing, not even for
repeated and map fields. Re-homing goes through try_encode_to_vec: a foreign
message over the 2 GiB limit is rejected, never a panic inside set.
CI ran only three of the seven conformance modes, which is why this shipped
green; via-lazy, view-json, via-reflect and via-vtable are now wired in too.
The via-vtable run goes from 20 panicking failures to 1246 successes, 0
failures, and the suite is 14/14.
Adds MapValue::into_entries, the by-value counterpart to from_entries, so a map
can be rebuilt without cloning its values.
|
All contributors have signed the CLA ✍️ ✅ |
…l cause Adoption takes equal full names to mean equal schemas, and the doc says what follows when two pools disagree — but nothing tested it. Cover the three outcomes: a wire-compatible type is reinterpreted, a wire-incompatible one lands in unknown fields and reads unset, and a field the target cannot name round-trips intact. Nothing is lost in any of them. The round-trip decode applies the standard recursion limit, so a message nested past it was reported as coming "from an incompatible pool" — naming the wrong cause for pools that agree perfectly. Report what the decoder said instead, and scope the encode comment's claim to size, which is what it actually guards. Correct the try_set performance note: the vtable rebuild it cites as the motivating caller pays two round-trips for a cross-pool field, not one, since to_owned has already materialized the subtree in its defining pool.
|
[claude code] Merged current main in (this branch predated #312/#313/#299/#306) and finished the review pass. Ready for review. The soundness question worth your attention: adoption is keyed on the message's full name. Two pools can hold structurally different messages under the same FQN, so I probed what actually happens rather than reasoning about it. Building two pools that disagree about
Nothing is lost in any case. Every outcome is what protobuf's wire contract already does when those bytes arrive from a peer on a different schema vintage, and protobuf's own identity model is name-based — it is what Verified against the full conformance suite: 14/14 suites pass, zero failures — std 5549, no_std 5529, via-view 2797, via-lazy 2797, view-json 1246, via-reflect 2783, via-vtable 1246, plus 883 text × 2. That includes via-vtable, the mode whose absence from CI is how #297 shipped green in the first place. All four newly-wired modes have their known-failure lists present. Two Low findings from review, both applied:
One hypothesis I chased and can now rule out: the rebuild walk is not quadratic on nesting. Also confirmed the two unreleased changelog fragments are coherent: this branch already reconciles the #272 fragment ("nested message descriptor identity" → "nested messages of the wrong type"), since #297 supersedes that identity claim. |
Fixes #297.
The bug
A generated type reflects against its defining crate's
DescriptorPool. ADurationview always reports buffa-types' pool — never the consumer pool that references it. So a nested cross-crate message value can never be pointer-identical to its parent's pool, and #272'sArc::ptr_eqcheck on message values rejected every one of them, with an error that says the quiet part out loud:That makes the vtable rebuild walk —
for_each_set+set(fd, vr.to_owned())— panic for any message with a well-known-type field.It is broader than the issue title suggests. It is not WKT-specific (any
extern_path-mapped proto hits it, and so do two packages generated in a single codegen run, which get distinct poolArcs), and it is not view-specific (owned types and the bridge path are affected identically; only lazy views escape, having no reflect impls at all).The fix
try_setadopts rather than rejects. A message of the same full name from a foreign pool is re-homed into the target pool by a wire round-trip — which re-homes the whole subtree in one decode — recursively covering singular, list, and map values.Descriptors stay strict: you still pass field descriptors resolved from the message's own pool. Values get adopted, because the codegen makes pool-identical values impossible for a cross-crate type. Full name is the adoption key — the same criterion protobuf itself uses across compilations, and what
Any's type URL carries — so a genuinely wrong type is stillWrongValueKindwhatever pool it came from.Two details worth a reviewer's eye:
try_encode_to_vec, notencode_to_vec: after encode: enforce the protobuf 2 GiB message-size limit #271 the latter panics above 2 GiB, and a value handed tosetmust never take the process down. An over-limit foreign message is rejected like any other value the pool cannot hold.Why CI didn't catch it
The conformance job ran only three of the seven modes.
via-lazy,view-json,via-reflectandvia-vtableexisted solely in the local Docker script, and via-vtable is the one that walks this exact rebuild path. All four are now wired into CI. The via-vtable run goes from 20 panicking failures to 1246 successes, 0 failures; the suite is 14/14.Behavior change
A cross-pool value that previously returned
Err(WrongValueKind)now succeeds. Callers that leaned on that rejection as a provenance check must compareReflectMessage::poolthemselves — called out in the changelog fragment. #272's own fragment is still unreleased and claimed "nested message descriptor identity included" among the rejections, so it would have shipped contradicting this one in the same release; it is amended here to say what it actually delivers (nested messages of the wrong type).Also adds
MapValue::into_entries, the by-value counterpart tofrom_entries, so a map can be rebuilt without cloning its values.Verification
Beyond the unit tests (which reproduce the cross-pool condition with two pools decoded from identical FDS bytes — no buffa-types needed), I drove the real cross-crate scenario from a scratch consumer crate: a
Jobmessage withDuration/Timestamp/repeated Durationfields, generated with vtable reflection, walked throughfor_each_set+set. Onmainit panics with the error above. With this change the walk completes, the rebuiltDynamicMessageencodes byte-identically to the generated message (49 bytes), the adopted sub-messages report the consumer's pool, the reflective JSON is correct ("timeout":"90.000500s"), and aTimestampset into aDurationfield is still rejected — now with an error that actually names two different types.Full workspace tests, clippy
-D warnings, and conformance 14/14.