Add node-based catamorphism POC

[Adam-Vandervorst]

I purpose we use this paradigm under the cata interface.

[Adam-Vandervorst]

@luketpeterson any problem merging this?

[luketpeterson]

@luketpeterson any problem merging this?

In concept no. There is a (small) bit of work to make the API is consistent with the cata that's there already.

[adamv-symbolica]

Giving Fable some time with it:

Blocking

B1 — goat_val_count double-counts the root value

trie_map.rs:509–521. PathMap::recursive_cata already ends with
collapse_f(self.root_val(), Some(w), &[]), so the closure counts the root value; the caller
then adds root_val again.

let mut map = PathMap::new();
map.insert(b"", ());
map.insert(b"a", ());
assert_eq!(map.val_count(), 2);      // ok
assert_eq!(map.goat_val_count(), 2); // FAILS: returns 3

Fix: drop the + root_val (and the now-pointless match self.root()), or make the collapse
closure ignore the final root-val invocation.

B2 — LineListNode "Case 10" (Val, Child with different first bytes) violates the branch contract

line_list_node.rs:2952–2977. The PR's own tests establish the contract:
at a branch, branch_f is called once per branch in ascending mask-bit order with the branch
mask, so the algebra can attribute each W to its byte (mask.indexed_bit(acc.idx)). Case 10
breaks it twice:

• the first branch_f call passes &ByteMask::new() (empty) instead of &mask
  (line 2960 — contrast with Cases 4/5/8 which pass &mask on every call);
• it processes the child slot first, but slots are stored sorted by first byte, so for
  (Val@a, Child@b) the calls arrive in descending byte order.

// keys {"a", "b1", "b2"} -> root pair node (Val@'a', Child@'b')
// Reconstructing paths via the documented mask contract yields:
//   [[98], [238,49], [238,50]]   (0xEE = sentinel for "mask had no bit at idx")
// instead of [[97], [98,49], [98,50]]

The value gets attributed to byte b, and the child subtrie to an empty mask. Any
mask-sensitive algebra (including the PR's own recursive_cata_jumping_total_len bench closure,
which would panic on .unwrap()) is wrong or crashes on tries that contain this node shape.
Fix: pass &mask on both calls and emit the value (lower byte) before the child.

B3 — Zipper recursive_cata silently returns "empty" for a mid-node focus

morphisms.rs:513–521. The blanket impl uses get_focus().0.borrow(), which
returns Some only for the BorrowedRc/OwnedRc variants of AbstractNodeRef. A zipper
focused part-way into a node (BorrowedTiny/BorrowedDyn — exactly what TinyRefNode exists
for) gets None and is treated as an empty trie:

let mut map = PathMap::new();
map.insert(b"abc1", ());
map.insert(b"abc2", ());
let mut rz = map.read_zipper();
rz.descend_to(b"ab");                       // path exists
rz.recursive_cata::<..>(count_vals ...)     // returns 0, expected 2

Fix: handle the remaining variants — e.g. go through as_tagged()/try_as_tagged() and add a
TaggedNodeRef-level entry point, or fall back to into_option() (accepting the clone), or
panic loudly rather than returning a wrong answer.

B4 — Branch-byte convention differs between node types, so results depend on physical layout

DenseByteNode includes the branch byte in the collapse_f prefix
(dense_byte_node.rs:415, core::slice::from_ref(&key_byte)) and
represents it in the mask given to branch_f. LineListNode strips the byte from the prefix
(&key0[1..]) and represents it only in the mask. An algebra therefore cannot know whether
prefix[0] is the branch byte or a distinct following byte:

// Path-reconstruction algebra (byte taken from mask, prefix appended):
//   correct on LineListNode pair shapes,
//   duplicates the first byte on DenseByteNode: [1,7,13] -> [1,1,7,13]

The same logical trie yields different W depending on which physical nodes back it. The PR's
own sum-digits tests only pass because the algebra threads a bool flag ("value was at empty
prefix") through W to compensate — a workaround that no external user will discover from the
docs. This is the "API consistency with the existing cata" work the reviewer already flagged on
the PR. Fix: pick one convention (LLN's byte-in-mask-only matches the existing
into_cata_jumping_* sub_path semantics best) and align ByteNode::node_recursive_cata;
then document it on Summarization.

B5 — Unbounded recursion: stack overflow on deep tries; PR's own test aborts the all_dense_nodes suite

recursive_cata_cached recurses once per physical node. The PR's
recursive_cata_stack_overflow_smoke (morphisms.rs:2232) documents overflow
between 8–10 KB of path depth on default features — and under --features all_dense_nodes
(1 byte per node) the very same test overflows and SIGABRTs the whole test binary at
PATH_LEN = 8_000. As a public API this is a panic-free-abort footgun on adversarial/deep data,
and as merged it leaves a red test config. Minimum: gate or shrink the smoke test per feature
and document the depth limit prominently; proper fix: explicit work-stack or segmented stacks
(e.g. stacker::maybe_grow) in recursive_cata_cached.

Non-blocking cleanups

• Dead code: node_goat_val_count (trait method + 6 impls) lost its only consumer when
  traverse_physical was removed; either delete the chain or keep goat_val_count on it for
  the comparison's sake — not both.
• Commented-out blocks left in line_list_node.rs (old generic implementation, old
  node_goat_val_count) and dense_byte_node.rs.
• unreachable_unchecked on header patterns (line_list_node.rs:2980):
  header values 1–7 (slot1 used, slot0 free) are assumed impossible; if that invariant is ever
  violated this is UB rather than a panic. A debug_assert!/unreachable! in debug builds
  would be cheap insurance.
• Option<Acc> dance in ByteNode::node_recursive_cata (Some(Acc::default()) +
  unwrap_unchecked) — ws is always Some; a plain Acc binding works.
• pub(crate) values on ByteNode is only applied to the non-nightly field variant and no
  code outside the module reads it — revert the visibility change.
• Docs: Summarization docs still carry GOAT/dev-branch placeholders;
  recursive_cata_stepping links to [Catamorphism::recursive_cata] (wrong trait); the
  COMPUTE_PATH=false caveat ("no reliable child_masks") deserves a loud, user-facing warning
  since it silently changes what the closures receive.
• Bench honesty: recursive_cata_jumping_total_len asserts only the count, not the length —
  the two implementations being compared do not agree on total length today (a consequence of B4).
• slim_dispatch's TaggedNodeRef::node_val_count now takes the gxhash map while the node impls
  take std — moot while the feature is bitrotted on master, but worth aligning if it's revived;
  same for the missing BridgeNode arm in recursive_cata_dispatch under bridge_nodes.

[adamv-symbolica]

Path byte should not be represented in prefix.

[adamv-symbolica]

Fable says:

Still broken, plus two regressions — all in the recursive engine only

The iterative engine passed every probe, including all shapes below. The cross-engine
equivalence suite doesn't yet include these pair-node shapes, which is why they slipped through.

• R1 (= B2, order half — still broken). LLN Case 10 folds the child slot before the
  lower-byte value slot: keys {a, b1, b2} reconstruct as {a1, a2, b} — byte attribution
  swapped. Fix: emit in ascending byte order.
• R2 (new regression). LLN Case 7 (Val, Val, shared first byte) places the slot-0 value one
  byte too deep when the slot-1 key is ≥3 bytes: keys {a, abc} reconstruct as {ab, abc}.
  The old code summarized the slot-0 value with prefix key1[0..1]; the refactored chain of
  summarize_run calls uses key1[1..], and summarize_run positions the value before the
  last prefix byte, not at the run root. ({a, ab} passes — the bug needs a ≥2-byte tail.)
• R3 (new). summarize_run's (Some(val), Some(w), []) arm folds the downstream W under
  an empty mask (reached via LLN Case 6 with a passed-in value): keys
  {x, xy, xyz, xa1, xa2, q} reconstruct with a corrupted entry for xyz — the z branch is
  folded with no mask bit to attribute it to. The child byte is known at the call site but
  discarded.
• B3 — unfixed. A zipper focused mid-node still computes over an empty trie: count = 0
  instead of 2 both for a plain mid-key focus (descend_to(b"ab") over {abc1, abc2}) and for
  a valued focus with continuation ({ab, abcd} — the focus value is dropped too, a step
  worse than the audited version). The focus.0.borrow() → None fallback in the
  CatamorphismCached zipper impl is unchanged; the branch's new focus test only covers a
  node-aligned focus. Note the iterative trait handles all of these correctly — routing
  non-node foci to it (or via TaggedNodeRef) is a ready-made fix.

[adamv-symbolica]

The probes

    /// The branch bytes a node owes its `fold_child` calls.  The contract: folds arrive once
    /// per child, in mask-bit order, so the k-th fold binds the k-th set bit of the mask.
    struct BranchBytes(ByteMaskIter);

    impl BranchBytes {
        fn of(child_mask: &ByteMask) -> Self {
            Self(child_mask.iter())
        }
        fn take(&mut self) -> u8 {
            self.0.next().expect("contract violation: more fold_child calls than bits in the child mask")
        }
        fn finish(mut self) {
            assert!(self.0.next().is_none(), "contract violation: fewer fold_child calls than bits in the child mask");
        }
    }

    /// One write zipper per node, created with the accumulator and reused for every fold,
    /// the prefix insertion, and the value placement.  Leaves never construct a zipper.
    struct ReconAcc {
        branch_bytes: BranchBytes,
        wz: WriteZipperOwned<()>,
    }

    fn recon_start(child_mask: &ByteMask) -> Result<ReconAcc, Infallible> {
        Ok(ReconAcc {
            branch_bytes: BranchBytes::of(child_mask),
            wz: PathMap::new().into_write_zipper(b""),
        })
    }

    /// A child's `W` is the sub-map hanging just below its branch byte: graft it there,
    /// reusing the node's zipper (one byte down, graft, one byte up).
    fn recon_fold(_mask: &ByteMask, child: PathMap<()>, acc: &mut ReconAcc) -> Result<(), Infallible> {
        let branch_byte = acc.branch_bytes.take();
        acc.wz.descend_to_byte(branch_byte);
        acc.wz.graft_map(child);
        acc.wz.ascend_byte();
        Ok(())
    }

    /// Contract: the returned `W` summarizes the subtrie from the start of `prefix`; the value
    /// (if any) sits at the end of `prefix`, and the children hang below the end of it.
    fn recon_summarize(_mask: &ByteMask, value: Option<&()>, children: Option<ReconAcc>, prefix: &[u8]) -> Result<PathMap<()>, Infallible> {
        match children {
            Some(ReconAcc { branch_bytes, mut wz }) => {
                branch_bytes.finish();
                if !prefix.is_empty() {
                    wz.insert_prefix(prefix); // pushes the grafted children down under `prefix`
                }
                if value.is_some() {
                    wz.descend_to(prefix); // insert_prefix leaves the focus value in place, so
                    wz.set_val(());        // the value is set at the prefix end afterwards
                }
                Ok(wz.into_map())
            },
            None => {
                let mut map = PathMap::new();
                if value.is_some() {
                    map.set_val_at(prefix, ());
                }
                Ok(map)
            },
        }
    }

    /// Rebuilds `$subject`'s trie through the named engine ($Engine is only a name; both
    /// cached-cata traits expose the identical method).
    macro_rules! reconstruct_trie {
        ($Engine:ident, $subject:expr) => {
            <_ as $Engine<_, GlobalAlloc>>::factored_cata_jumping::<_, _, _, _, _, _, true>(
                $subject, recon_start, recon_fold, recon_summarize,
            ).unwrap()
        };
    }

    /// Streaming equality on value paths — no materialized path list, so it is as cheap as
    /// one iteration of each map.  On divergence the assert prints the first differing path.
    #[track_caller]
    fn assert_same_paths(got: &PathMap<()>, expected: &PathMap<()>, who: &str) {
        let mut got = got.iter();
        let mut expected = expected.iter();
        loop {
            match (got.next(), expected.next()) {
                (None, None) => break,
                (g, e) => assert_eq!(g.map(|(p, _)| p), e.map(|(p, _)| p), "{who} diverged"),
            }
        }
    }

    #[track_caller]
    fn assert_roundtrips(map: &PathMap<()>) {
        assert_same_paths(&reconstruct_trie!(CatamorphismCachedIterative, map), map, "oracle");
        assert_same_paths(&reconstruct_trie!(CatamorphismCached, map), map, "recursive engine");
    }

    #[track_caller]
    fn assert_keys_roundtrip(keys: &[&[u8]]) {
        let mut map = PathMap::<()>::new();
        for k in keys { map.set_val_at(k, ()); }
        assert_roundtrips(&map);
    }

    /// Validates the probe algebra itself: on these shapes both engines already agree with
    /// the input (pair nodes in every currently-correct arrangement, plus dense nodes), so a
    /// failure in the `audit_*` tests below isolates an engine bug, not a probe bug.
    #[test]
    fn audit_probe_algebra_sanity() {
        assert_keys_roundtrip(&[b"a1", b"a2", b"b"]);         // (Child, Val) pair
        assert_keys_roundtrip(&[b"b", b"a1", b"a2"]);         // same, reversed insert order
        assert_keys_roundtrip(&[b"a1", b"a2", b"b1", b"b2"]); // (Child, Child) pair
        assert_keys_roundtrip(&[b"a", b"ab"]);                // (Val, Val) shared byte, short
        assert_keys_roundtrip(&[b"a", b"a1", b"a2"]);         // value + child at same byte
        assert_keys_roundtrip(&[b"abc1", b"abc2"]);           // key run into a branch
        assert_keys_roundtrip(&[b"", b"q1", b"q2"]);          // root value
        let dense: Vec<Vec<u8>> = (0u8..200)
            .map(|b| vec![b, b.wrapping_mul(7), b.wrapping_mul(13)])
            .collect();
        let dense: Vec<&[u8]> = dense.iter().map(|k| k.as_slice()).collect();
        assert_keys_roundtrip(&dense);                        // DenseByteNode layouts
    }

    /// B2: pair node (Val@'a', Child@'b') — the child is folded before the lower-byte value,
    /// so byte attribution comes out swapped: {a, b1, b2} rebuilds as {a1, a2, b}
    #[test]
    fn audit_b2_pair_val_child_fold_order() {
        assert_keys_roundtrip(&[b"a", b"b1", b"b2"]);
    }

    /// R2: (Val, Val) sharing a first byte with a >=2-byte tail — the short value lands one
    /// byte too deep: {a, abc} rebuilds as {ab, abc}
    #[test]
    fn audit_r2_val_val_shared_byte_value_position() {
        assert_keys_roundtrip(&[b"a", b"abc"]);
    }

    /// R3: a value passed into a single-value node folds its downstream under an EMPTY mask;
    /// the probe panics with "more fold_child calls than bits in the child mask"
    #[test]
    fn audit_r3_passed_val_empty_mask_fold() {
        assert_keys_roundtrip(&[b"x", b"xy", b"xyz", b"xa1", b"xa2", b"q"]);
    }

    /// Large-instance round-trip: grafting + O(1) `W` clones keep the probe at
    /// Θ(trie bytes), so scale is limited by the map itself, not the algebra.
    #[test]
    fn audit_roundtrip_large_random_trie() {
        use rand::prelude::*;
        let mut rng = StdRng::from_seed([17; 32]);
        let mut map = PathMap::<()>::new();
        for _ in 0..50_000 {
            let len = rng.random_range(0..=12usize);
            let key: Vec<u8> = (0..len).map(|_| b'a' + rng.random_range(0..4u8)).collect();
            map.set_val_at(&key, ());
        }
        assert_roundtrips(&map);
    }

    /// B3: a zipper focused inside a node's key run (with or without a value at the focus)
    /// must summarize the subtrie below the focus, not an empty trie
    #[test]
    fn audit_b3_mid_node_zipper_focus() {
        macro_rules! count_vals {
            ($Engine:ident, $z:expr) => {
                <_ as $Engine<_, GlobalAlloc>>::factored_cata_jumping::<usize, usize, Infallible, _, _, _, false>(
                    $z,
                    |_| Ok(0),
                    |_mask, child_count, total| { *total += child_count; Ok(()) },
                    |_mask, value, children, _prefix| Ok(value.is_some() as usize + children.unwrap_or(0)),
                ).unwrap()
            };
        }
        let mut map = PathMap::<()>::new();
        map.set_val_at(b"abc1", ());
        map.set_val_at(b"abc2", ());
        let mut rz = map.read_zipper();
        rz.descend_to(b"ab");
        assert_eq!(count_vals!(CatamorphismCachedIterative, &rz), 2, "oracle diverged");
        assert_eq!(count_vals!(CatamorphismCached, &rz), 2, "recursive engine ignored the mid-node focus");

        let mut map = PathMap::<()>::new();
        map.set_val_at(b"ab", ());
        map.set_val_at(b"abcd", ());
        let mut rz = map.read_zipper();
        rz.descend_to(b"ab");
        assert_eq!(count_vals!(CatamorphismCachedIterative, &rz), 2, "oracle diverged");
        assert_eq!(count_vals!(CatamorphismCached, &rz), 2, "recursive engine dropped the focus value + subtrie");
    }

    /// hash() must be a function of the logical trie alone, so the two engines must agree on
    /// any focus.  Random maps over a small alphabet hit the pair-node shapes (B2/R2/R3), and
    /// random foci sampled with `random::FairTriePath` hit mid-node positions (B3).
    /// Run with `--features random`.
    #[cfg(feature = "random")]
    #[test]
    fn audit_hash_engines_agree_on_random_subtries() {
        use rand::prelude::*;
        use rand::distr::Distribution;
        use crate::random::FairTriePath;

        let mut rng = StdRng::from_seed([31; 32]);
        for round in 0..64 {
            let mut map = PathMap::<u64>::new();
            for i in 0..48u64 {
                let len = rng.random_range(0..=6usize);
                let key: Vec<u8> = (0..len).map(|_| b'a' + rng.random_range(0..3u8)).collect();
                map.set_val_at(&key, i);
            }

            let root_oracle = <_ as CatamorphismCachedIterative<_, GlobalAlloc>>::hash(&map);
            assert_eq!(<_ as CatamorphismCached<_, GlobalAlloc>>::hash(&map), root_oracle, "root hash diverged (round {round})");

            let sampler = FairTriePath { source: map.clone() };
            for _ in 0..8 {
                let (path, _val) = sampler.sample(&mut rng);
                let mut rz = map.read_zipper();
                rz.descend_to(&path);
                let oracle = <_ as CatamorphismCachedIterative<_, GlobalAlloc>>::hash(&rz);
                let got = <_ as CatamorphismCached<_, GlobalAlloc>>::hash(&rz);
                assert_eq!(got, oracle, "subtrie hash diverged at {path:?} (round {round})");
            }
        }
    }

[luketpeterson]

Wow that took a lot more work than expected to get the API into shape. But it should be ready to merge now.

[adamv-symbolica]

Fable 5.1

1. Recursive engine misplaces the value in LineListNode Case 9 with a ≥3-byte child key

Blocker. src/line_list_node.rs ~line 2915 (summarize!(Some(val), Some(child_w), &key1[1..])).

map = {a, abcd, abce}          // plain inserts
recursive rebuild -> {ab, abcd, abce}   // value "a" lands at "ab"

summarize_run positions a value before the last byte of the prefix it is given, so a value at the
start of a run followed by a ≥2-byte valueless tail is pushed one level too deep. Case 7 had the same
construction error and was fixed (lines 2876–2884); Case 9 was not. Count-only algebras are unaffected,
but hash() disagrees between the engines on this map and any path-sensitive algebra is wrong. The
iterative engine is correct. The randomized tests miss the shape because it needs exactly one value
above a ≥3-byte valueless run.

Fix: mirror the Case 7 fix — summarize the child under key1[2..], fold it under mask{key1[1]},
finalize with the value, then wrap with key0. Add {a, abcd, abce} to the equivalence suite.

2. Iterative engine aborts (debug) / commits UB (release) on dangling branches

Blocker for relying on the iterative engine. src/zipper.rs ~line 2497, ZipperConcrete::is_shared.

map = {aa, ab, ba, bb, b}
map.remove_val_at(b"ba", false);   // prune = false
map.remove_val_at(b"bb", false);
CatamorphismCached::val_count(&map)          // 3 — recursive engine guards the empty node
CatamorphismCachedIterative::val_count(&map) // SIGABRT: non-unwinding UB-check panic in refcount()

A no-prune removal leaves a valueless branch that is logical structure (path_exists() is true, it is
in child_mask). The iterative cache-key lookup calls is_shared(), which does
node_get_child(..).unwrap() and then refcount() on the child with no empty-sentinel check, reading
through the sentinel pointer. Inherited from master (into_cata_cached aborts identically at
d967f0c), but the PR routes CatamorphismCachedIterative, ACT val_count, and CatamorphismDebug
through it and presents it as the stack-safe engine.

Fix: guard with !node.is_empty() && node.refcount() > 1, as recursive_cata_cached already does.
Add the dangling-branch map to the equivalence suite.

3. PrefixZipper under the blanket recursive impl drops the prefix

src/morphisms.rs ~line 647 (blanket CatamorphismCached for Z: ZipperInfallibleSubtries),
src/prefix_zipper.rs ~line 702.

pz = PrefixZipper::new(b"xy", {ab, ac}.read_zipper())
pz.make_map()        -> {xyab, xyac}
recursive rebuild    -> {ab, ac}        // iterative rebuild is correct

PrefixZipper::get_focus() delegates to the source while its make_map() includes the prefix, so the
blanket impl (which rebuilds from get_focus()) disagrees with the zipper's own subtrie semantics.
Any wrapper zipper whose logical position differs from its focus node has the same problem.

Fix (decision needed): exclude wrappers from the blanket impl (marker trait or explicit impls), or
give PrefixZipper a CatamorphismCached impl that delegates to the iterative body.

[adamv-symbolica]

One more

Iterative engine hangs on a LineListNode shape left behind by join_k_path_into (found by a randomized edit-program run against 3839f31)

join_k_path_into shortens two value keys in place: (Val@"aaa", Val@"bab") → (Val@"aa", Val@"ab"). factor_prefix accepts that (rule A: "slot 0 contains a value"), but insertion never builds it — a fresh {aa, ab} is Child@"a" -> {a, b} — and nothing that indexes children can walk it: descend_indexed_byte fails (master's side-effecting cata trips its debug_assert at morphisms.rs:~844), the recursive engine trips Case 7's debug_assert_eq!(key0.len(), 1), and CatamorphismCachedIterative loops forever (summarize/fold on byte a, never advancing to b). Plain iter() works, which is why it went unnoticed. The shape is pre-existing master behavior; the hang is new with the iterative engine.

Standalone test (drop into tests/); the assertions are ordered so it fails before it can hang:

use pathmap::PathMap;
use pathmap::zipper::*;
use pathmap::morphisms::{CatamorphismCached, CatamorphismCachedIterative};

#[test]
fn drop_head_shared_first_byte_shape() {
    let mut m = PathMap::<()>::new();
    m.set_val_at(b"aaa", ());
    m.set_val_at(b"bab", ());
    assert!(m.write_zipper().join_k_path_into(1, false));
    assert_eq!(m.iter().map(|(k, _)| k).collect::<Vec<_>>(), vec![b"aa".to_vec(), b"ab".to_vec()]);

    let mut rz = m.read_zipper();
    rz.descend_to_byte(b'a');
    assert_eq!(rz.child_count(), 2);
    assert!(rz.descend_indexed_byte(1).is_some(), "indexed descent cannot reach the second child"); // fails here today

    assert_eq!(CatamorphismCached::val_count(&m), 2);           // debug build: Case 7 assert
    assert_eq!(CatamorphismCachedIterative::val_count(&m), 2);  // never returns
}

Fix that makes this pass (and keeps both full suites green): in LineListNode::factor_prefix, rule A should also require a 1-byte slot-0 key —

let legal_overlap = overlap == 1 && (
    (!self.is_child_ptr::<0>() && key0.len() == 1) ||
    (!self.is_child_ptr::<1>() && key0.len()==1 && key1.len()==1 ));

— so the shape gets factored into the canonical Child@"a" -> {a, b}. Alternatively the engines could learn to traverse it, but since indexed descent itself can't, canonicalizing at the producer seems right. Worth adding to the recursive-vs-iterative equivalence suite either way.