HSJ scoring reads past the end of a `std::vector<int>`, caught by `-D_GLIBCXX_ASSERTIONS`

[ms609]

Found while validating feature/xform-statespace; pre-existing on cpp-search, not introduced by that branch.

Reproduce

Build cpp-search (verified at 99b79d8f) with libstdc++ assertions on — the flag has to go in PKG_CPPFLAGS, since ~/.R/Makevars.win zeroes PKG_CXXFLAGS:

PKG_CPPFLAGS="-D_GLIBCXX_ASSERTIONS" R CMD INSTALL --library=<lib> --preclean TreeSearch_*.tar.gz

Confirm the flag took (grep -c _GLIBCXX_ASSERTIONS the build log; expect ~34), then:

NOT_CRAN=true Rscript -e "library(TreeSearch, lib.loc=<lib>); testthat::test_file('tests/testthat/test-tree_length.R', reporter='location')"

Result

Aborts on entry to test_that("TreeLength HSJ returns valid score") (tests/testthat/test-tree_length.R:318) — a 4-taxon dataset, CharacterHierarchy("1" = 2:3), BalancedTree, inapplicable = "hsj":

bits/stl_vector.h:1149: std::vector<_Tp, _Alloc>::const_reference
std::vector<_Tp, _Alloc>::operator[](size_type) const [with _Tp = int; ...]:
Assertion '__n < this->size()' failed.

_Tp = int and a const operator[], so this is an out-of-bounds read. The likeliest candidate is the tip_labels[t * n_orig_chars + char_idx] indexing in src/ts_hsj.cpp (e.g. :166, :390, :418) against the flattened vector built in ts_hsj_score (src/ts_rcpp.cpp:3189-3197), but I did not narrow it further.

Attribution

Both cpp-search @ 99b79d8f and the feature branch abort at the identical test under the identical flag, so the defect is on trunk. It is silent in ordinary builds — the whole suite passes without assertions, and ubuntu-arm64 is green on cpp-search — which is why it has gone unnoticed.

Why it matters

An OOB read in a shipped scoring path. It is reading uninitialised or foreign memory, so an HSJ score could in principle depend on heap contents; and it is exactly the class of defect that becomes a hard fault under a different allocator or on a different architecture.

Suggested next step

Re-run the above and bisect within score_hierarchy_block() / fitch_label_char(), checking n_orig_chars against the actual tip_labels_r column count and each block.primary_char / secondary_chars index against it. Note that -D_GLIBCXX_ASSERTIONS is the cheap local reproducer here; plain builds will not show it.

[ms609]

Second, cheaper reproducer for the same class of fault — and one that does not involve inapplicable = "hsj", so the defect may be broader than the HSJ path this issue was filed against.

With the same -D_GLIBCXX_ASSERTIONS build:

NOT_CRAN=true Rscript -e "library(TreeSearch, lib.loc=<lib>); testthat::test_file('tests/testthat/test-ts-xform.R', reporter='summary')"

aborts after just 5 assertions with the identical signature:

ts-xform: .....bits/stl_vector.h:1149: ... [with _Tp = int; ...]:
Assertion `__n < this->size()` failed.

Confirmed pre-existing: cpp-search @ 99b79d8f and feature/xform-statespace abort at the same point after the same five assertions. test-recode-hierarchy.R, test-ts-t330-collapse-hsj-xform.R and test-ts-xform-statespace.R are all clean under the same build, so it is not the recoding or the Sankoff tip-cost construction.

This is a faster entry point than the test-tree_length.R one above — five assertions rather than fifty — so it is probably the better one to bisect from. Whether the two are the same site is unverified; both are const std::vector<int>::operator[] reads.