Placement and split gather per region; the geometric tail solves on a distributed band (#670, #671) by lmoresi · Pull Request #672 · underworldcode/underworld3

lmoresi · 2026-09-02T20:05:05Z

Stacked on #669 (the base branch here); three commits.

The gather-first surgery moves base cells onto one rank so that nothing it deletes or creates is shared. Three things were wrong with how much and where (#670): the mark reached two cell widths beyond the carve before the star and layer were grown from it; the whole network was one region with one target rank; and a region already interior to a rank was moved anyway.

1. The mark covers only what the carve drops (victims plus the crossed cells' vertices, one cell diameter out), the same rule at every placement path. _gather_region reports the moved count, and ptest_0855 bounds it by the cells within three median diameters of the zone.

2. The gather takes regions. _gather_regions marks a chart of region ids, claims each region's star and layer, merges regions whose shells touch (collective union-find), and sends each to the rank already holding most of it — or leaves it where it is when it is interior to one rank. One shell partition moves them all. ptest_0857 covers the three behaviours at np=2, 3, 4.

3. The thin volume and the split use the regions. One region per connected component of the assembly; the owning ranks carve and fill their own components concurrently on their compacted share of the assembly, and the collective rebuild sews them at once. The split's redistribution takes groups of faults and delegates to the same gather; the network passes it the regions the placement reported, so what the placement kept apart is not gathered together afterwards. Outcrop and ladder paths keep one region.

Measured (base 27,544 cells, two crossing patches; or two zones a domain apart):

case	before	after
crossing patches, cells gathered	9831	5046
two zones apart, np=2, cells moved	8451, one rank	304, two owners
two faults in one network, split, np=2	one rank	nothing moved: 14,339 / 14,332 cells
same, np=4	one rank	each region to its own rank, 2507 moved

Tests: ptest_0855 (+2 cases), ptest_0857 (new), ptest_0863 (+1 case) at np=2 and 4 (0855 and 0857 also at 3); lines, sheet, remove-embedded, fault-split, 3-D split, network and place-split-contact parallel suites at np=2; the serial thin-volume and network suites.

Not fixed here. The contact solve on the balanced two-fault layout fails in the co-located multigrid tail's setup (#671) even at np=2, where nothing moved: the band is then genuinely distributed, and the tail was built for a band on one rank. The placement and the split complete. Also pre-existing: the gap fill's placed-vertex count varies with np by one or two (611 vs 612 on the same input under the previous build), a gmsh ordering effect.

Underworld development team with AI support from Claude Code

lmoresi · 2026-09-02T20:05:08Z

Adversarial review (we read the diff against what it does not measure).

The bound is calibrated, not derived. Three median diameters was chosen because the new mask sits at 5046 against 7269 and the old at 9831 on this box. A mesh with a wider spread of cell sizes than a refined gmsh box could move the shell past the median-based bound without any regression in the rule. The rule itself, victims plus star plus layer, is not what the test checks.
The mark is still a distance, and the rule is topological. "Crossed cells' vertices within one cell diameter" is an estimate of which cells the carve will drop; the carve decides by geometry afterwards. The surgery raises if the gathered region turns out to touch a shared point, so an under-reach is loud, and no path hit it at np=2, 3 or 4. A mark taken from the carve's own victim set would remove the estimate.
The interior short-cut was not exercised by any test. It fires when the star counts are non-zero on exactly one rank; on every fixture here the region straddles a seam at every rank count, so the branch has been measured only in that it did not fire. A test needs a partition it controls.
n_gathered is reported by two of the six paths. Lines, the 2-D thin volume and the ribbon carry the same mask change without the count.
The one-target rule stands. Two surfaces interior to different ranks are still gathered to one of them; the mask change halves what moves, it does not change where.

None of these change the measured numbers. Item 2 is the one that would make the rule exact rather than estimated.

lmoresi · 2026-09-02T20:44:57Z

Review addendum for the per-region commits (we read the diff against what it does not cover).

A merged region's owner is the majority rank of the union, chosen after the merge. Two zones whose shells touch become one region and move together; that is correct for the seam rule, and it means "close" is measured in shells (three base cells each), not in zone widths. Two faults 0.8 apart on 0.126 cells merged in our measurement; 1.4 apart did not. The docs say so; a user cannot see the merge except through n_regions.
The per-rank surgery compacts its share of the assembly, so the assembly node numbering the surgery sees is per rank. Everything downstream that keyed on global assembly indices was checked (made-cell codes, placed rows, skin rows, labels); embedded_nodes are coordinates and unaffected. The outcrop path was deliberately left on one region because its cap and bowl machinery broadcasts from a single root.
The split's groups come from the placement's report, not from the mesh. info["embedded_regions"] identifies each mid-surface's region by matching its first node's coordinates against the assembly at 1e-12; a mid-surface that failed to match falls into region 1. A split called without the network (no groups) is one region, as before.
The interior short-cut is now exercised (ptest_0857 and the two-fault network at np=2), which the first review asked for.
The solve is the open end. Every measurement here stops at the sewn, split mesh; on the balanced layout the multigrid tail refuses (The co-located multigrid tail fails at np=3 on the 3-D width network: a Galerkin coarse operator with a missing diagonal, and a 10% slip error #671). The claim of this PR is placement and split parallelism, not the solve.

lmoresi · 2026-09-02T21:24:08Z

Fourth commit: the tail on a distributed band (#671). Per-region placement produced the first layouts where the band spans ranks, and the co-located multigrid tail failed on them. Two defects, both measured:

The rotated prolongation zeroed its constrained rows by LOCAL block index; zeroRows takes global ones, so every rank but the first zeroed the wrong rows. That is the "New nonzero caused a malloc" in PCSetUp_MG: PETSc repairs a zero row of the Galerkin coarse operator with an identity diagonal, and the diagonal was never allocated.
The "auto" transfer mode replaced a co-partitioned transfer with 3 orphan columns by a cross-partition one with 16,791 (np=2) or 43,047 (np=4) of 96,009; the repair then injected all of them and the velocity KSP failed at its first iteration. The rebuild is now kept only where it does better, with a warning otherwise.

With both, the two-fault network solves on custom-FMG at np=2 and np=4 in one Newton step, slips within 0.5% of serial. The np=3 answer discrepancy on the crossing fixture (10% on the smallest piece) is unchanged and stays open in #671. The parallel custom-MG (1017) and rotated free-slip (1064, 1066, 1068) suites pass at np=2.

Review note on ourselves: the cross-partition builder losing 17 to 45 percent of the coarse columns on a co-partitioned pair is a defect in its own right, not fixed here; we only stop it from being chosen when it is worse. It is measured, warned about, and left for #671.

… region is already interior (#670) The gather-first surgery moves base cells onto one rank so that no point it deletes or creates is shared. That needs three layers: the cells the carve drops, their vertex star (the ring the fill attaches to), and one more layer so the ring's points are unshared. _gather_region grows the star and the layer from the mark, but the mark itself reached two cell widths beyond the carve, so the margin was paid twice: on the crossing-patches fixture the gather moved 5087 of 5592 base cells for a cavity of 334, and every rank count ended with the surgery rank holding 95% of the mesh. The mark now covers what the carve drops: the victims within the clearance plus the crossed cells' vertices, one cell diameter out. Every placement path carries the same rule (lines, sheet, the 2-D and 3-D thin volumes, the ribbon). On a box large enough to hold a shell (eight times the fixture, 27,544 base cells, two crossing patches) the gather moves 5046 cells instead of 9831, identical at np=2 and np=4; the remaining extra load on the surgery rank is the shell plus the band and fill it creates, which is the design's accepted trade. A region whose star and layer already sit on one rank is placed there with no redistribution at all. The decision is collective: the per-rank star counts are gathered before it is taken. _gather_region returns the moved cell count, and the sheet and thin volume report it as info["n_gathered"]. ptest_0855 bounds it by the cells within three median cell diameters of the zone (7269 on that box): the old mask fails the test, the new one passes it at np=2, 3 and 4. The other placement paths' parallel tests and the serial thin-volume suite pass unchanged. Not changed here, recorded in #670: the network is still one region with one target rank, so two surfaces interior to different ranks are gathered together; and the moved cells stay where the surgery put them. The np=3 failures of the network solve predate this change and are #671. Underworld development team with AI support from Claude Code

_gather_regions generalises the one-region gather: a chart of region ids marks the vertices, each region's star and layer is claimed, regions that touch (a shared cell or vertex, on any rank) are merged by a collective union-find, and each merged region goes to the rank that already holds most of it — or stays where it is when its star and layer are already interior to one rank. One shell partition moves them all. The count returned is the regions' size (the seam rule's footprint, a function of the mesh alone) alongside the cells that actually changed rank. _gather_region is now the one-region form of it, unchanged for its callers. ptest_0857 covers the three behaviours at np=2, 3 and 4: two regions a domain apart keep their own owners, two that touch are merged, and one vertex marked deep inside every rank moves nothing. Underworld development team with AI support from Claude Code

…not at all (#670) The thin volume marks one region per connected component of the assembly (zones fused through shared faces are one component; zones a domain apart are separate) and gathers them with _gather_regions: regions whose shells touch are merged, each goes to the rank already holding most of it, and one whose shell is interior to a rank is not moved. The owning ranks then carve and fill their own components concurrently — each on its compacted share of the assembly and skin — and the collective rebuild sews them at once. The outcrop and ladder paths keep one region; their bowl, cap and extrusion are single-rank. The split follows the same regions. _redistribute_fault_interior takes groups of faults and delegates to _gather_regions; split_faults passes them through, and the network derives them from the placement's report of which region each embedded mid-surface's zone became (info["embedded_regions"]), so what the placement kept apart is not gathered together afterwards. The network's info now carries n_regions, n_gathered and n_moved. Measured. Two zones a domain apart on the 8x box, np=2: 304 cells moved to two different owners where one region for the pair moved 8451 to one rank; the sewn mesh has the same zone, skin and removed counts. Two faults in one network, split realisation, np=2: both regions already interior to their ranks, nothing moved, 14,339 and 14,332 cells per rank; np=4: each region to a different rank, 2507 moved. Two faults 0.8 apart on 0.126 cells merge into one region (their shells touch); 1.4 apart they do not. Tests: ptest_0855 gains the two-zone case (np=2, 4); ptest_0863 the two-fault network (np=2, 4). The placement, split and network parallel suites and the serial thin-volume and network suites pass. Not fixed here: the contact solve on the balanced two-fault layout fails in the co-located multigrid tail's setup (#671) even at np=2, where nothing moved — the band is then genuinely distributed, and the tail was built for a band on one rank. The placement and the split complete; the tail is the next item. Underworld development team with AI support from Claude Code

…y their global index, keep the better transfer (#671) Two defects in the co-located multigrid tail surfaced once a band could sit on more than one rank (the per-region placement of #670). The rotated prolongation zeroes the constrained rows — the rotated normals and the fault-pair contact rows — so they take no coarse correction. It enumerated this rank's entries of the velocity IS and passed those LOCAL block rows to zeroRows, which takes global ones: on every rank but the first the wrong rows were zeroed. Where the rows wrongly zeroed were a coarse DOF's only fine images, the Galerkin coarse operator got a zero row, and PETSc's own remedy (MatGalerkin writes an identity diagonal into zero rows) failed on a row whose diagonal was never allocated — the "New nonzero caused a malloc" in PCSetUp_MG. The rows are now the rank's ownership offset plus the local index. The "auto" transfer mode rebuilt any transfer with a zero column as a cross-partition one. On the balanced two-fault layout the co-partitioned build left 3 orphan columns (coarse DOFs under the band with no fine image) and the cross-partition build 16,791 of 96,009 at np=2 and 43,047 at np=4; the repair then injected a nearest fine DOF into every one, and the coarse operator was nonsense (the velocity KSP failed with reason -11 at its first iteration). The rebuild is now kept only where it leaves fewer orphans than the co-partitioned build, with a warning when it would have made things worse; the few genuine orphans are the repair's job, as before. Measured. Two faults a domain apart in one network, split, on a [-0.5, 1.5]^3 base: the contact solve now converges in one Newton step on custom-FMG at np=2 and np=4 with slips within 0.5% of the serial answer (the gap fill's node count varies by one or two with the partition, so the meshes are not identical). The crossing-patches fixture passes at np=2 and np=4; at np=3 it no longer fails in PCSetUp_MG but still reports the smallest piece's slip 10% low, which remains open in #671. The parallel custom-MG and rotated free-slip suites pass at np=2. ptest_0863's two-fault case now runs the solve and pins both. Underworld development team with AI support from Claude Code

…st, and the gather's growth as a parameter (#670, #671) docs/developer/design/fault-parallel-placement-2026-09.md is the governing note for parallel placement: the adapt-on-top frame, the three parallel strategies in the stack, what the gather moves and why, the measurements that showed it moving far too much, what PR #672 changed, the decision on long faults (faults as distributable objects, cut in the CAD along strike when one is too long for a rank, the strip along a cut plane gathered before the split), the rulings not to rebalance the child and to pre-refine the base, and what remains open. It is linked from the developer index (toctree and authority map), the conforming-surfaces subsystem note and the fault-networks page. The throughput test beside it holds everything fixed but where the faults sit relative to the seams: a 6 x 1 x 1 box at np=3, three faults one per slab, then one moved onto a seam, then all forced into one region, plus serial and GAMG baselines. Measured: per-region placement balances the mesh to 6% and gives a warm solve 1.8x serial where the old single gather gave 1.0x; one straddling fault costs 15%; the answer is layout-independent to four digits; and on this contact fixture the pressure Schur solve caps at 200 iterations in every configuration, so wall time is the pressure block's and GAMG beats the tail in wall time despite 26 velocity iterations against 4. _gather_regions takes the growth beyond the star as ``layers``. The placement was tried at ``layers=0`` (the star alone) and its own gate refused at np=2, 3 and 4: the carve drops cells beyond the marked vertices' star. Both the placement and the split keep one layer, and the note records the measurement. Underworld development team with AI support from Claude Code

lmoresi · 2026-09-03T05:37:39Z

Fifth commit: the design note (docs/developer/design/fault-parallel-placement-2026-09.md, linked from the developer index's toctree and authority map) and the layout throughput test beside it. The table from the test, np=3 on a 6 x 1 x 1 box with three faults:

layout	cells moved	cells per rank (max/mean)	warm solve	velocity its	pressure its
serial	0	20778	81.0 s	3	200
local (one fault per slab)	122	1.06	45.8 s	4	200
one fault straddling a seam	834	1.22	52.9 s	4	200
all gathered to one rank (pre-#672)	5440	1.91	78.3 s	4	200
local, GAMG	122	1.06	17.6 s	26	200

The answer is layout-independent to four digits. Per-region placement turns np=3 from no gain (78 s against 81 s serial) into 1.8x; a straddling fault costs 15%. And on this fixture the pressure Schur solve is at its 200-iteration cap in every configuration, so wall time is the pressure block's, and GAMG beats the tail in wall time despite six times the velocity iterations. That is the #625 cap seen on a contact fixture, not a placement matter, and the note says so.

Also in this commit: the gather's growth beyond the star is a parameter, and the placement was measured at zero extra layers — its own gate refuses, so the third layer stays.

Copilot

🔵 Needs a closer look

It changes core parallel placement/split behavior and multigrid transfer logic across several subsystems, which warrants final human review despite targeted tests and documentation.

Pull request overview

This PR refines Underworld3’s parallel fault-network “gather-first” workflow so placement and split operate per connected region (instead of gathering an entire network onto one rank), and fixes geometric multigrid tail behavior for distributed bands.

Changes:

Implement per-region gathering (_gather_regions) with merge-on-touch logic and “don’t move if already rank-interior”, then use regions to drive concurrent per-component placement/surgery.
Propagate region information through fault-network build and split redistribution (split_faults(..., groups=...)) to prevent re-gathering independent regions together.
Fix multigrid tail issues on distributed bands (rotated prolongation constrained-row indexing, and safer “auto” cross-partition transfer selection) and add/extend parallel tests + developer docs.

File summaries

File	Description
tests/parallel/ptest_0863_fault_network_3d_width_parallel.py	Adds a two-faults-apart placement/split/solve regression case and asserts new gather-region metrics.
tests/parallel/ptest_0857_gather_regions_parallel.py	New test coverage for `_gather_regions` (separate regions, merge-on-touch, and no-move interior regions).
tests/parallel/ptest_0855_place_thin_volume_parallel.py	Adds assertions bounding gather footprint and verifying multi-region placement behavior.
src/underworld3/utilities/rotated_bc.py	Fixes constrained-row zeroing in rotated prolongation for distributed matrices (global row indexing).
src/underworld3/utilities/place_surface.py	Introduces `_assembly_components`, implements `_gather_regions`, tightens gather masks, and runs per-region thin-volume surgery.
src/underworld3/utilities/fault_split.py	Adds region-aware redistribution via `groups` and routes through `_gather_regions`.
src/underworld3/utilities/custom_mg.py	Improves “auto” transfer selection to avoid choosing worse cross-partition transfers; keeps orphan repair.
src/underworld3/meshing/fault_network.py	Passes placement regions into split grouping; records `n_regions/n_gathered/n_moved` into `net.info`.
docs/developer/subsystems/conforming-surfaces-and-fault-zones.md	Updates subsystem narrative for per-region gathering and reported metrics.
docs/developer/index.md	Adds the new governing design note to the developer index.
docs/developer/design/fault-parallel-placement-2026-09.md	New governing design/measurement note documenting the decision and throughput results.
docs/developer/design/fault_parallel_layouts.py	New script to reproduce throughput measurements across layouts and tail vs GAMG.
docs/advanced/fault-networks.md	Updates user-facing advanced docs to reflect per-region gather behavior and reference the design note.

Review details

Suppressed comments (1)

src/underworld3/utilities/place_surface.py:1762

Same issue as above: this error message is hard-coded to place_sheet even though _gather_regions() is a general utility and can be invoked from other operations (e.g. place_thin_volume, fault_split).

    counts = np.asarray(comm.allgather(local))          # (size, n_regions)
    if counts.sum() == 0:
        raise ValueError("place_sheet: the sheet meets no cell on any rank")
    owner = {}

Files reviewed: 13/13 changed files
Comments generated: 5
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+                    n_zero = _count_zero_columns_parallel(P, comm)
+                    if self.cross_partition == "auto" and n_zero > 0:
+                        P_x = _build_crosspart_transfer(*args)
+                        n_x = _count_zero_columns_parallel(P_x, comm)
+                        if n_x < n_zero:
+                            P.destroy()
+                            P = P_x
+                        else:
+                            P_x.destroy()
+                            if n_x > n_zero:
+                                import warnings
+                                warnings.warn(
+                                    f"custom_mg: the cross-partition transfer "
+                                    f"{l - 1}->{l} left {n_x} coarse DOF(s) "
+                                    f"without a fine image where the "
+                                    f"co-partitioned one left {n_zero}; kept "
+                                    f"the co-partitioned transfer (#671).")
                # the same orphan repair the serial path has: a coarse DOF no
                # fine node reaches gets its nearest fine DOF as an injection
                if _count_zero_columns_parallel(P, comm) > 0:


+    The one-region form of :func:`_gather_regions` (a 0/1 mask is one
+    region). Returns ``(new_dm, n_moved)``: the global count of cells
+    gathered, ``0`` when nothing moved (serial, or a region already
+    interior to one rank), so callers can both branch on it and report
+    it. The input is untouched.
+    """


+layer from it. The result is a shell about three cells thick around the
+surface; a region whose shell is already interior to one rank is placed
+there with nothing moved. The moved count is reported as
+`info["n_gathered"]`, and `ptest_0855` bounds it by the cells within three
+median cell diameters of the zone.


+    n_ids = int(comm.allreduce(int(ids_in.max()) if ids_in.size else 0,
+                               op=MPI.MAX))
+    if n_ids == 0:
+        raise ValueError("place_sheet: the sheet meets no cell on any rank")
    if comm.size == 1:


        uw.pprint(f"[place_sheet {label!r}] placed {info['n_placed']} "
                  f"vertices, removed {info['n_removed']}; "
                  f"{info['n_surface_facets']} sheet faces")
+    info["n_gathered"] = int(moved)   # cells the gather moved (#670)


… tolerance as a measurement knob (#625, #670) The first pass of the layout throughput test ran every configuration's pressure Schur solve to its 200-iteration cap. The monitor shows why: the pressure residual falls by 5e-3 in twenty iterations, then creeps to a floor of about 7e-6 relative — the inexact velocity solve inside the Schur application (3.3e-7 of its own residual) amplified by the Schur complement — while the rotated path asks the pressure solve for 1e-6. The margin rule is inverted between the two inner solves. Each wasted pressure iteration is a velocity-preconditioner apply, which is what made GAMG look 2.6 times faster than the tail. solver._rotated_pres_rtol is the measurement knob (TODO(MEASURE)); the default stays. At 1e-4 the pressure solve converges in 26 iterations and the table of record reads: serial 14.0 s warm, np=3 local 6.8 s (2.1x), one straddling fault 8.7 s (+28%), GAMG 3.3 s. The answer is unchanged to four digits. GAMG remains twice as fast as the tail on this linear, uniform-viscosity fixture; the note says so and why that is not the placement's question. Underworld development team with AI support from Claude Code

lmoresi · 2026-09-03T15:43:16Z

Sixth commit: the layout table re-measured with the pressure solve converging. The first pass had every configuration's pressure Schur solve at its 200-iteration cap; the monitor showed a floor set by the inexact velocity solve inside the Schur application (mechanism on #625). With the pressure tolerance at 1e-4 (knob solver._rotated_pres_rtol, default unchanged):

layout	cells per rank (max/mean)	warm solve	velocity its	pressure its
serial	1.00	14.0 s	3	26
np=3 local	1.06	6.8 s	4	26
np=3 one fault straddling a seam	1.22	8.7 s	4	26
np=3 local, GAMG	1.06	3.3 s	26	26

The answer is unchanged to four digits. np=3 per region is 2.1x serial; one non-local fault costs 28%. GAMG is still twice as fast as the tail in wall time on this linear, uniform-viscosity fixture, which the note records as a red flag for a contrast-fixture measurement rather than a placement result.

…y and the traction correction (#625, #670) At a solver tolerance of 1e-3 the default margins converge the pressure sub-solve in 26 iterations with no knob: the fixture's 1e-5 was one or two orders stricter than a mesh of 0.08 to 0.2 cells deserves, which is what put the pressure solve at its cap. The table carries that block too: serial 10.1 s warm, np=3 local 5.4 s, straddle 5.3 s, GAMG 2.2 s; the slips move by 0.2 to 0.7% between tolerances. The cost of one non-local fault is stated as a bound (at most a quarter of the solve, possibly nothing measurable) rather than a point value, since the three passes read 28%, 15% and none on single runs. The penalty is recorded as a recovery matter, not a discretisation one: the traction is the multiplier plus the augmentation, and the Coulomb fault law's reaction-fed normal stress needs the same correction before a penalty is used with friction (Louis's note on non-planar boundaries). Penalty 1 at 1e-5 does not lift the cap and shifts the slips by 0.6 to 0.9% at this resolution. Underworld development team with AI support from Claude Code

lmoresi · 2026-09-03T16:02:08Z

Seventh commit: the table at tolerance 1e-3, where the default margins converge the pressure solve in 26 iterations without any knob — the 1e-5 of the first pass was stricter than the resolution deserves. Warm: serial 10.1 s, np=3 local 5.4 s, one fault straddling a seam 5.3 s, GAMG 2.2 s. The non-local fault's cost is now stated as a bound (none to 28% across the three passes, single runs) rather than a number. The penalty is recorded as a traction-recovery matter per Louis's note on non-planar boundaries: the multiplier plus the augmentation is the traction, and the Coulomb law's reaction-fed normal stress needs that correction before a penalty is used with friction.

…d level; the preconditioner question deferred to 2-D (#670) The renders (script beside the note) show the tail as the gmsh box at 0.24, its single bisection, and the placed mesh differing in 1,314 cells in three pockets — the finest transfer is the identity on 94% of its rows, and the band sits two cells across inside fill four to eight times larger with no grading, since the band builder hard-codes one refinement. Louis's ruling: multigrid is unsurprisingly not useful on this isoviscous two-grid fixture; the preconditioner study moves to 2-D. Recorded so the GAMG row is not read as a verdict. Underworld development team with AI support from Claude Code

…embed once, TI immediately, split locally with seam ligaments (#670) Louis, 3 September: the embedding is the same for both realisations and only the split needs pairs on one rank; the weak and TI band need mesh continuity across a seam and nothing else. So: embed the band with its mid-surface reserved as conforming faces, cut the split only where the facet star is interior, and leave a user-set ligament at each seam crossing for the band's weak rheology to bridge — the hybrid recipe at seams, which the stepover and junction studies measured. Partition independence becomes a bound to measure. Build order: the seam-conforming fill in 2-D, the bridging split mode, the 2-D study. Underworld development team with AI support from Claude Code

…#670) The mechanism in four steps (interface planes chosen collectively, a strip alignment of cavity-adjacent cells to the plane's sides, an interface surface meshed once and broadcast, two fills and the existing rebuild extended with star-forest entries for the shared interface vertices), the signatures (place_thin_volume seams=, split_faults at_seams= and ligament=, FaultNetwork.build pass-through), the collective discipline, the tests in order (2-D serial, 2-D np=2 and 3, then 3-D), and the traps known in advance. For implementation in a fresh session. Underworld development team with AI support from Claude Code

…athering, and split it rank-local (#670) Partition-crossing structures stay transversely isotropic (the ruling of 3 September): the mesh only has to conform on each side of the seam, and the band's weak plane is the glue across it. place_thin_volume gains seams="ligament" (2-D): every rank carves its own cavities, each stops one cell short of the seam, the assembly is clipped to what the cavity holds and made manifold, and the base cells the clipped band covered are left as the ligament, labelled zone and <label>_ligament. Nothing moves. In serial it reduces to the gather path exactly. The split never sees a seam: add_fault(cut=False) labels the embedded spine edges outside the ligament, split_fault splits a fault that crosses a rank in and out as sub-chains (one collective pass per piece), and FaultNetwork.apply paints the weak plane on the ligament cells for the split realisation too, so it takes eta_1 in both. Measured on a 35-cell fault (h = 0.02): the weak plane loses 1.5% of peak slip at np=2 and 3.6% at np=3 with the cells balanced (1216/1050 against the gather's 1629/635); the split loses 13% per crossing, insensitive to eta_1, because a sub-chain's tip is pinned. On the unit box the np=2 seam runs along y = 0.5, so a horizontal network is all ligament: the design note records both findings and what follows (a free tip at a ligament end; a segmented gather to steer the seam). Also: Mesh.cells_labelled aborted on an absent label (a NULL label wrapper, not None); it now returns the all-False mask it promised. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

The seam ligament's first build protected every cell touching a shared vertex, so the band stopped a base cell short of the seam and the two sub-chain tips sat six band cells apart in base mesh, each touching the weak plane on one side only. The rebuild's contract is narrower: no shared vertex deleted, no placed vertex shared. So the placement now protects the shared vertices only, carves the seam cells like any other, runs the cavity ring along the seam's own edges (a seam vertex whose local fan is wholly dropped is kept as a ring vertex, not orphaned), and caps the band against them; the ligament is one strip of fill on the seam. add_fault(cut=False, blind=1) leaves one band cell uncut at each clipped chain end, and FaultNetwork.bridge_cells paints the weak plane on the ligament and on the band around each blind tip, so the tip is enclosed as it is under a free surface. Measured on the long-fault fixture: the TI realisation is now within the fill's noise (0.5684 at np=2 and 0.5643 at np=3 against 0.5688); the split's weld per crossing halves (0.4703 against 0.5094 at np=2), the far-field profile recovers to 0.95-0.98 of the gathered slip, and what remains is the three band cells the crack does not cross. The design note carries the numbers and the before/after diagrams. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

…ict on the ligament (#670) A rank's cleared cells may fall into more than one hole — a network whose zones sit apart, a graded base at two refinement levels — and each hole is now carved and filled on its own in every mode, which lifts the "one simple hole" refusal the fine S-fault rig hit. Each cavity takes the assembly cells whose vertices lie inside it, except a single seamless cavity, which keeps the whole assembly: an outcrop vertex snapped to a curved wall lies just outside the straight base cells. The design note records the S-fault rig in parallel with TI: within 3% at the coarse width, but 8-10% down on the crossing strand at the fine width with the partition shifting onto the Branch, halving with each refinement of the seam's cells. That is the ligament standing in for the band at base resolution, and it will not do for a sensible-width fault. The decision is to mesh the band through the seam — the conforming interface — as the next build; the note lays out its four steps. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

… ribbon (#670) The seam-conforming placement, built and measured. Nothing moves. A seam vertex is deleted only where the band itself reaches it, and the decision is reconciled over the star forest so both sides agree. Every band cell, band vertex and skin edge has one owner, the rank whose cavity holds it (three global reductions), so the seam inside the band runs along band edges with no construction of its own. Each rank's fill boundary is a graph — its ring edges minus the crossed seam spans, its skin edges, and a connector from each span end to the nearest free end of its skin runs, which is the same edge on both sides — and the fill loops are read off it, junctions and skirting seams included. The rebuild's vertex star forest gains the band vertices both sides place, keyed by the band's own numbering, and a shared vertex deleted on both sides drops out. The fill is graded from the band arcs in the ring as from holes. Measured: the long fault gives the gathered answer at np=2 and np=3 with the cells balanced; the S-fault rig at the fine width (w = 0.01), where the ligament had lost 8-10% of the Main's slip and shifted the partition onto the Branch, is within 2% along the Main and 0.1% on the other three strands at np=2 and np=4, with the repeat solve 5.0 s against 7.4 s serial. In serial the mode reduces to the gather path exactly. A refused fill saves its inputs beside the script. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

…p its islands (#670) The fill's cell size is now a power of the relative distance across the cavity (grading, FaultNetwork.build(fill_grading=)), default 0.35: on the fine S-fault rig the fill cells within three band widths drop from 3924 to 1931 and the mesh from 8170 to 5816 cells, the worst angle from 17.7 to 17.2 degrees, the weak plane's answer unchanged to 0.5%. The ladder mesher keeps the linear fill (its sequential cavities need the room between close strands). Two repairs: kept cells enclosed by a cavity are dropped instead of refusing the ring, and with several cavities on a rank a band cell goes to the one holding its centroid. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

…670) The patch smoother's blocks were built only in the serial branch of the transfer builder, so a parallel run silently fell back to whole-level smoothing at the finest level. The parallel branch now builds the same blocks in global rows: the fault-zone blocks through the level layout's ghost-resolved numbering (a masked cell's off-rank nodes join this rank's subdomain, the seam halo), and the structural block from this rank's owned rows of the distributed transfer, with the cover gate decided collectively. The installation takes them unchanged. Measured on the fine S-fault rig it engages as in serial and is a net loss in both (44 iterations without the patch against 60 with it in serial, 57 against 66 at np=2): the fault blocks alone stall the solve, the structural block is mandatory and is 88% of the level. The design note records the matrix; UW_FAC_STRUCTURAL=0 is the measurement knob. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

… and its parallel decision made collective (#670) With a full hierarchy and a plain mesh the velocity solve should be easy, and it is: 4 iterations with no viscosity contrast on the fine S-fault rig, 9 at a tenfold weak plane, 44 at the thousandfold one, unchanged by another base level. The cost is the weak plane in a band thinner than the coarse cells, a smoother problem. The patch as built replaced the level's smoother and so had to cover everything the coarse level could not represent, the whole placed region. The composite form keeps the whole-level smoother and applies the fault blocks on top, with an exact LU sub-solve (nonzero shift) and one overlap layer: 10 velocity iterations in serial, 11 at np=2, 12 at np=4, against 44, 57 and 58. It is the default when a zone is keyed; the structural split is off then (the measurement knobs remain). The parallel structural split's cover gate is a collective; a rank with no band cells took it alone and hung np=4, and the decision now comes from what any rank holds. The composed-ribbon test resolves the composite's ASM member and expects the zone block alone in the default form. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

…aults from the size sweep (#670) A mask block is cut into pieces of at most fac_block_rows rows (default 2000, before the overlap layer) along its leading principal direction, whole nodes kept together, so the exact sub-solve stays bounded however long the band grows. Measured on the fine S-fault rig (TI 1e-3): one block of 15736 rows takes 10 velocity iterations, blocks of ~2000 rows 12 in serial, at np=2 and at np=4, blocks of ~600 rows 15 in serial and 30 at np=2; ILU instead of LU 27; a second overlap layer 9 for 2700 more rows. The defaults are the composite form, LU with a nonzero shift, one overlap layer, 2000 rows per block, no structural block. The design note carries the sweep and the 3-D arithmetic it settles. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

…ent-local-gather

The band meshed through the seam (seams="conform") left the split gathering. The 2-D cut now runs through the seam: the chain is assembled globally from the star-forest identities of its vertices, a vertex on the seam is duplicated on every rank holding it with the replica owned where the original is, the seam edges re-homed onto a replica keep their star-forest entries, and the pair's Plus->Minus normal at a seam vertex is recorded by the split so the contact solve needs no exchange. Reached through split_fault(across_seams=True), add_fault(cut=False, across_seams=True) and FaultNetwork.build(seams="conform", realisation="split"). Nothing is redistributed. The long vertical fault gives the serial peak (0.5094, 69 owned pairs) at np=2 and np=3 with balanced cells; the three-piece network gives its serial peaks at both rank counts. ptest_0865 pins both. Two things the measurement turned up, fixed here: the conform placement now keeps both cells of a spine edge with one owner (the np=3 partition had put a spine facet on the seam, which the cut cannot represent), and the blind rule no longer drops an unclipped one-edge piece of a spine between two seam vertices (it broke the global chain into two pinned pieces). A fill ring that crosses itself is refused before gmsh, which used to spin on it; the two conform fill limits that produce such rings (a tip on the seam beside a junction, a junction within a band width or two of the seam) are recorded in the design note as open. Underworld development team with AI support from Claude Code Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzK7JXSxsc2GoG7TGnQgUS

lmoresi · 2026-09-04T19:20:56Z

2926248 adds the split through the partition seam in 2-D, the cut the conform placement was missing. The chain is assembled globally from the star-forest identities of its vertices, a vertex on the seam is duplicated on every rank holding it (replica owned with the original, so a coincident pair never straddles ranks), re-homed seam edges keep their star-forest entries, and the pair's normal at a seam vertex is recorded by the split so the contact solve needs no exchange. Nothing is redistributed.

Measured against the gathered answer: the long vertical fault gives 0.5094 with 69 owned pairs at np=1, 2 and 3 (cells 1212/1042 and 785/890/571); the three-piece network gives its serial peaks at np=2 and 3. ptest_0865 pins both.

Two defects the measurement exposed are fixed in the same commit: the conform placement could put the two cells of a spine facet on different ranks (np=3), and the blind rule dropped an unclipped one-edge piece of a spine between two seam vertices. A fill ring that crosses itself is now refused before gmsh rather than spinning in it; the two conform fill configurations that produce one (a tip on the seam beside a junction; a junction within a band width or two of the seam) are recorded in the design note as open.

lmoresi changed the title ~~Placement gathers only the shell the seam rule needs, and nothing when the region is already interior (#670)~~ Placement gathers per region, only the shell each region needs, and nothing when a region is already interior (#670) Sep 2, 2026

lmoresi mentioned this pull request Sep 2, 2026

The co-located multigrid tail fails at np=3 on the 3-D width network: a Galerkin coarse operator with a missing diagonal, and a 10% slip error #671

Open

lmoresi changed the title ~~Placement gathers per region, only the shell each region needs, and nothing when a region is already interior (#670)~~ Placement and split gather per region; the geometric tail solves on a distributed band (#670, #671) Sep 2, 2026

lmoresi added 4 commits September 2, 2026 14:24

lmoresi force-pushed the feature/placement-local-gather branch from 9eaf2c9 to 0d5b8bb Compare September 2, 2026 21:24

lmoresi changed the base branch from bugfix/fault-network-mg-tail-and-gauge to development September 2, 2026 21:24

Copilot AI lite review requested due to automatic review settings September 3, 2026 05:37

Copilot started reviewing on behalf of lmoresi September 3, 2026 05:38 View session

lmoresi mentioned this pull request Sep 3, 2026

Stokes pressure sub-solve runs at its 200-iteration cap — the inner Schur solve is silently unconverged and dominates solve time #625

Open

Copilot AI reviewed Sep 3, 2026

View reviewed changes

lmoresi and others added 8 commits September 3, 2026 09:15

lmoresi and others added 5 commits September 3, 2026 17:59

Merge remote-tracking branch 'origin/development' into feature/placem…

8690d64

…ent-local-gather

Navigation Menu

Uh oh!

Placement and split gather per region; the geometric tail solves on a distributed band (#670, #671) - #672

Placement and split gather per region; the geometric tail solves on a distributed band (#670, #671)#672
lmoresi wants to merge 20 commits into
developmentfrom
feature/placement-local-gather

lmoresi commented Sep 2, 2026 •

edited

Loading

Uh oh!

lmoresi commented Sep 2, 2026

Uh oh!

lmoresi commented Sep 2, 2026

Uh oh!

lmoresi commented Sep 2, 2026

Uh oh!

lmoresi commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

lmoresi commented Sep 3, 2026

Uh oh!

lmoresi commented Sep 3, 2026

Uh oh!

lmoresi commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lmoresi commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lmoresi commented Sep 2, 2026

Uh oh!

lmoresi commented Sep 2, 2026

Uh oh!

lmoresi commented Sep 2, 2026

Uh oh!

lmoresi commented Sep 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (1)

Uh oh!

lmoresi commented Sep 3, 2026

Uh oh!

lmoresi commented Sep 3, 2026

Uh oh!

lmoresi commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lmoresi commented Sep 2, 2026 •

edited

Loading