Fix comm_cfg anchor-holder key leaking into config dicts - #773
Conversation
pre_process_yaml's whole-file {include file} splices the entire included
file, including any key whose sole purpose is holding a YAML anchor for
<<: *anchor use elsewhere (e.g. comm.shared.yaml's comm_cfg: &comm). That
leaked key then reached Object.__init__'s **kwargs and was silently
dropped there -- masking real config typos, since the include leak looked
identical to a typo.
Drop anchor-holder keys from whole-file splices using the (keyword, anchor)
pairs reload_anchors() already extracts; keyed includes of the same key are
left untouched. Handle the resulting all-keys-stripped case (comm.shared.yaml's
only top-level key is the anchor holder) by dropping the splice placeholder
entirely instead of emitting an empty "{}" mapping, which broke YAML parsing
when followed by block-style content.
Verified against a real pyobs-monet config: comm_cfg no longer leaks, and the
comm: <<: *comm alias still resolves correctly.
See specs/plans/2026-08-09-object-kwarg-validation.md for the investigation
and remaining open items (environment/database wrapper keys, Object.__init__
enforcement level).
|
Approach is sound and it fixes the 1. Empty-dict splice isn't scoped to whole-file includes ( The # inc.yaml
somekey: {}
# main.yaml
outer:
{include inc.yaml somekey}Before: 2. Anchor-holder detection isn't top-level-scoped (
# inc.yaml
type: Foo
camera:
type: &cam
model: XAfter a whole-file include, top-level Both are silent-data-loss shapes, i.e. exactly the class of bug this PR is meant to kill. They're unlikely in today's fleet configs (only Otherwise good: |
Two silent-data-loss bugs found in review of the comm_cfg anchor-leak fix:
1. The "drop an empty splice placeholder" special case wasn't scoped to
whole-file includes, so a keyed include that legitimately selects an
empty mapping (`{include file key}` where key's value is `{}`) silently
became `null` instead of `{}`.
2. Anchor-holder detection reused reload_anchors(), which matches
`keyword: &anchor` at any nesting depth via a plain (not line-anchored)
regex. A top-level key could be incorrectly dropped from a whole-file
include just because some unrelated *nested* key elsewhere in the file
happened to share its name and carry an anchor.
Fixed by gating the empty-splice case on the same whole-file condition as
the anchor-drop itself, and by adding top_level_anchor_keywords() -- a
line-anchored regex restricted to unindented keys -- used only for the
drop decision. reload_anchors() itself is unchanged, since replace_aliases()
still needs to resolve anchors at any nesting depth.
Added regression tests for both. Full suite: 1490 passed, 25 skipped.
Re-verified against all 803 yaml files in pyobs-monet, pyobs-iagvt, and
pyobs-iag50: same result as before (2 pre-existing, unrelated errors; no
comm_cfg leaks in any consuming config).
|
Both fixed in 8124467. 1. Empty-splice drop now scoped to whole-file includes only. Gated that branch on the same 2. Anchor-holder detection now top-level-scoped. Added Reproduced both bugs from your examples before fixing, to confirm they were real. Added a regression test for each ( |
comm_cfg fix (#773) and this pass together close out every confirmed dead/misplaced/typo'd kwarg found by re-running the investigation as a static check across pyobs-monet, pyobs-iagvt, pyobs-iag50, and pyobs-polaris (815 real config files). environment/database, the last open blocker on the Object.__init__ warn/raise decision, is confirmed gone -- nothing found is blocking that decision anymore.
* Fix comm_cfg anchor-holder key leaking into config dicts
pre_process_yaml's whole-file {include file} splices the entire included
file, including any key whose sole purpose is holding a YAML anchor for
<<: *anchor use elsewhere (e.g. comm.shared.yaml's comm_cfg: &comm). That
leaked key then reached Object.__init__'s **kwargs and was silently
dropped there -- masking real config typos, since the include leak looked
identical to a typo.
Drop anchor-holder keys from whole-file splices using the (keyword, anchor)
pairs reload_anchors() already extracts; keyed includes of the same key are
left untouched. Handle the resulting all-keys-stripped case (comm.shared.yaml's
only top-level key is the anchor holder) by dropping the splice placeholder
entirely instead of emitting an empty "{}" mapping, which broke YAML parsing
when followed by block-style content.
Verified against a real pyobs-monet config: comm_cfg no longer leaks, and the
comm: <<: *comm alias still resolves correctly.
See specs/plans/2026-08-09-object-kwarg-validation.md for the investigation
and remaining open items (environment/database wrapper keys, Object.__init__
enforcement level).
* Address review: scope empty-splice drop and anchor detection correctly
Two silent-data-loss bugs found in review of the comm_cfg anchor-leak fix:
1. The "drop an empty splice placeholder" special case wasn't scoped to
whole-file includes, so a keyed include that legitimately selects an
empty mapping (`{include file key}` where key's value is `{}`) silently
became `null` instead of `{}`.
2. Anchor-holder detection reused reload_anchors(), which matches
`keyword: &anchor` at any nesting depth via a plain (not line-anchored)
regex. A top-level key could be incorrectly dropped from a whole-file
include just because some unrelated *nested* key elsewhere in the file
happened to share its name and carry an anchor.
Fixed by gating the empty-splice case on the same whole-file condition as
the anchor-drop itself, and by adding top_level_anchor_keywords() -- a
line-anchored regex restricted to unindented keys -- used only for the
drop decision. reload_anchors() itself is unchanged, since replace_aliases()
still needs to resolve anchors at any nesting depth.
Added regression tests for both. Full suite: 1490 passed, 25 skipped.
Re-verified against all 803 yaml files in pyobs-monet, pyobs-iagvt, and
pyobs-iag50: same result as before (2 pre-existing, unrelated errors; no
comm_cfg leaks in any consuming config).
comm_cfg fix (#773) and this pass together close out every confirmed dead/misplaced/typo'd kwarg found by re-running the investigation as a static check across pyobs-monet, pyobs-iagvt, pyobs-iag50, and pyobs-polaris (815 real config files). environment/database, the last open blocker on the Object.__init__ warn/raise decision, is confirmed gone -- nothing found is blocking that decision anymore.
Summary
pre_process_yaml's whole-file{include file}spliced the entire included file in, including keys whose sole purpose is holding a YAML anchor for<<: *anchoruse elsewhere -- e.g.comm.shared.yaml'scomm_cfg: &comm, used across every monti/monet/iagvt/polaris config. That leakedcomm_cfgreachedObject.__init__'s**kwargsand was silently dropped, indistinguishable from a real config typo.reload_anchors()identifies as an anchor holder for that file before splicing; keyed includes of the same key ({include file key}) are untouched.comm.shared.yaml's only top-level key), by dropping the splice placeholder instead of emitting an empty{}mapping, which broke YAML parsing when followed by block-style content.pyobs-monetconfig (config/central/imagedb.yaml):comm_cfgno longer leaks,comm: <<: *commstill resolves correctly.Implements the
comm_cfgportion ofspecs/plans/2026-08-09-object-kwarg-validation.md. Theenvironment/databasewrapper-key question andObject.__init__'s warn/raise enforcement are intentionally left open -- see that plan's Decision section for why.Test plan
tests/utils/test_config.py: 3 new tests -- alias resolution (previously uncovered), keyed-include-of-anchor-holder is kept, whole-file-include no longer leaks the anchor holderpytest -m "not integration and not xmpp"-- 1488 passed, 25 skipped, 0 failedruff check/black --check/pyrefly checkclean on changed filespyobs-monet/config/central/imagedb.yaml