Five defects of the same shape have been found independently over the last few weeks — #276/#478, #467, #468, #477, #425. Every one of them degrades to something that succeeds: GAMG converges, richardson converges, the solve returns a correct answer. So none presented as a failure. They presented months later as "this is a bit slow", or as a parameter sweep returning identical rows, which reads as "this variable does not matter" rather than "this variable was never set". That cost a twelve-cell sweep on #477 and a wrong premise in #471.
This issue is about the property that makes them all invisible, rather than any one of them.
Inventory of the preconditioner-configuration fallbacks
| site |
substitutes |
observable? |
_apply_preconditioner_options single-field gate |
geometric MG → GAMG |
warns only on explicit fmg |
_apply_preconditioner_options n_levels <= 1 |
geometric MG → GAMG |
warns only on explicit fmg |
_apply_preconditioner_options override latch |
user's bundle → managed bundle |
nothing (fixed in #471) |
auto_inject_custom_mg barycentric→RBF retry |
sparse transfer → dense |
warns |
auto_inject_custom_mg build failure |
geometric MG → default PC |
warns |
auto_inject_custom_mg dimensional guard |
skips the auto-pickup |
warns |
_assert_finest_spans_operator swallow |
skips a correctness guard |
nothing |
CustomMGHierarchy.build setUp swallow |
proceeds on a possibly-stale DM section |
nothing |
_install_velocity_block_transfers PETSc-error path |
alternative assembly route |
nothing (same outcome) |
_enforce_galerkin_for_geometric_mg |
forces pc_mg_galerkin=both |
warns |
rotated_bc no hierarchy |
geometric MG → GAMG |
velocity_pc |
rotated_bc no pressure mass |
1/μ-mass Schur → selfp + jacobi |
schur_pre |
Only the last two leave readable state. Both were added in #465 — and they are the only reason #467 was findable at all.
Three things the current state cannot do
1. A warning is not observability. Warnings are filtered, deduplicated by Python's warning registry (so a fallback that fires every timestep reports once), and invisible in a long log. Readable state can be asserted on in a test. There is currently no public way to ask a solver which preconditioner did I actually get — _pc_managed_value and _pc_user_override are private.
2. "Couldn't" and "chose not to" are indistinguishable. GAMG-because-no-hierarchy and GAMG-because-we-do-not-trust-DMCreateInjection (#478) read identically, and only one of them is a bug. A user cannot tell whether to go and build a hierarchy or to file an issue.
3. Two silent swallows disable correctness guards. _assert_finest_spans_operator returns early when it cannot read the operator — its own docstring calls it a guarantee. CustomMGHierarchy.build swallows a failed snes.setUp() and proceeds with whatever DM section is current, which is exactly the case (a stale section on an adapt() child) the guard exists for. Both are defensible as written; both chose "do not crash the solve" over "say the check did not run".
Suggested direction
Generalise the velocity_pc / schur_pre pattern: a small reported record of what the managed block actually resolved to and why — the preconditioner that was installed, whether it was the requested one, and if not, which of unavailable / declined / build failed / user override applies. Readable on any solver, not just the rotated path, and assertable in a test.
Related, and cheap alongside it: the barycentric→RBF retry silently switches to a dense transfer whose Galerkin coarse operators are dense too. The code comment is explicit that this "rescues correctness but does not scale" and should be treated as a performance cliff — but a long run cannot tell you it happened, and no test can assert which builder ran.
This is a design change touching every managed solver, so it wants its own PR rather than riding on a fix. Filing it because the individual instances keep being found one at a time, and the reason they are hard to find is this.
Found by the solver configuration audit, docs/reviews/2026-07/solver-configuration-reachability-audit.md (branch docs/solver-config-audit), findings F-6, F-7 and F-8.
Underworld development team with AI support from Claude Code
Five defects of the same shape have been found independently over the last few weeks — #276/#478, #467, #468, #477, #425. Every one of them degrades to something that succeeds: GAMG converges, richardson converges, the solve returns a correct answer. So none presented as a failure. They presented months later as "this is a bit slow", or as a parameter sweep returning identical rows, which reads as "this variable does not matter" rather than "this variable was never set". That cost a twelve-cell sweep on #477 and a wrong premise in #471.
This issue is about the property that makes them all invisible, rather than any one of them.
Inventory of the preconditioner-configuration fallbacks
_apply_preconditioner_optionssingle-field gatefmg_apply_preconditioner_optionsn_levels <= 1fmg_apply_preconditioner_optionsoverride latchauto_inject_custom_mgbarycentric→RBF retryauto_inject_custom_mgbuild failureauto_inject_custom_mgdimensional guard_assert_finest_spans_operatorswallowCustomMGHierarchy.buildsetUpswallow_install_velocity_block_transfersPETSc-error path_enforce_galerkin_for_geometric_mgpc_mg_galerkin=bothrotated_bcno hierarchyvelocity_pcrotated_bcno pressure massselfp+ jacobischur_preOnly the last two leave readable state. Both were added in #465 — and they are the only reason #467 was findable at all.
Three things the current state cannot do
1. A warning is not observability. Warnings are filtered, deduplicated by Python's warning registry (so a fallback that fires every timestep reports once), and invisible in a long log. Readable state can be asserted on in a test. There is currently no public way to ask a solver which preconditioner did I actually get —
_pc_managed_valueand_pc_user_overrideare private.2. "Couldn't" and "chose not to" are indistinguishable. GAMG-because-no-hierarchy and GAMG-because-we-do-not-trust-
DMCreateInjection(#478) read identically, and only one of them is a bug. A user cannot tell whether to go and build a hierarchy or to file an issue.3. Two silent swallows disable correctness guards.
_assert_finest_spans_operatorreturns early when it cannot read the operator — its own docstring calls it a guarantee.CustomMGHierarchy.buildswallows a failedsnes.setUp()and proceeds with whatever DM section is current, which is exactly the case (a stale section on anadapt()child) the guard exists for. Both are defensible as written; both chose "do not crash the solve" over "say the check did not run".Suggested direction
Generalise the
velocity_pc/schur_prepattern: a small reported record of what the managed block actually resolved to and why — the preconditioner that was installed, whether it was the requested one, and if not, which of unavailable / declined / build failed / user override applies. Readable on any solver, not just the rotated path, and assertable in a test.Related, and cheap alongside it: the barycentric→RBF retry silently switches to a dense transfer whose Galerkin coarse operators are dense too. The code comment is explicit that this "rescues correctness but does not scale" and should be treated as a performance cliff — but a long run cannot tell you it happened, and no test can assert which builder ran.
This is a design change touching every managed solver, so it wants its own PR rather than riding on a fix. Filing it because the individual instances keep being found one at a time, and the reason they are hard to find is this.
Found by the solver configuration audit,
docs/reviews/2026-07/solver-configuration-reachability-audit.md(branchdocs/solver-config-audit), findings F-6, F-7 and F-8.Underworld development team with AI support from Claude Code