Skip to content

RecodeHierarchy derives a secondary's state space from observed token *strings*, so an ambiguity token becomes a novel third state and XFORM over-counts. #11

Description

@ms609

Severity: P1 · Area: 10 (HSJ/XFORM kernels)

Location: R/recode_hierarchy.R:103-106 (secLevels <- sort(setdiff(unique(origMat[, d]), c("-", "?")))), :184 (match(secVals[s], secLevels[[s]]))

An ambiguity token such as "{01}" is neither "-" nor "?", so it survives the setdiff and is admitted as its own level. It then sits at Hamming distance 1 from both "0" and "1" in the generated cost matrix instead of matching either. This is ordinary polymorphic morphological data, not malformed input: MatrixToPhyDat puts "{01}" in allLevels with contrast row (0, 1, 1, 0), and ValidateHierarchy guards only the primary (:216-224), so nothing rejects it and nothing warns. Verified REAL and demonstrated (opus verifier, build at bbab0f7c): with a "{01}" secondary at one tip, TreeLength(..., inapplicable = "xform") = 6, while every concrete resolution control gives 5 ({01}->"0" = 5, {01}->"?" = 5) and HSJ on the same data gives 5. Since {01} is a subset of {0,1}, the correct cost is the minimum over resolutions and is therefore <= 5; 6 is strictly an over-count. The finder's n_states 7-not-5 arithmetic reproduced exactly in its 2-secondary layout (verifier's indices differ -- cm[2,4] = cm[4,2] = cm[3,4] = cm[4,3] = 1 -- because the finder's [2,3]/[3,2] were for a 1-secondary layout; same substance). Second consequence: multiplicative state-space inflation ((k+1)^m rather than k^m present states), which trips the > 32 states guard on data that does not warrant it -- four binary secondaries each with a single polymorphic cell yields an 82 states (> 32) warning where the truth is 17, i.e. an O(states^2)-per-node Sankoff cost as well as a wrong score. Root cause shared with T-394 (both are R/recode_hierarchy.R:103-105 deriving the state space from token strings rather than from the contrast/levels state space) but neither subsumes the other: patching ambiguity handling leaves T-394's zero-level Inf, and guarding secNStates == 0 leaves this over-charge. Fix should derive secondary levels from the phyDat contrast matrix, resolving ambiguity tokens to their state sets exactly as HSJ already does via token_states.


Migrated 2026-08-04 from dev/red-team/findings.md; pre-tracker ID T-393. References to T-393 in source comments, dev/red-team/log.md, commit messages and PR bodies mean this issue — see dev/red-team/migration-map.tsv. Cross-repo references use the fully-qualified form agent-issues/TreeSearch#<n>; a bare #n means this repo.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:10Red-team focus area 10in-progressBeing fixed; claiming comment names the branchred-teamFiled by the /red-team rotationsev:highP1: wrong user-visible result / crash

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions