Propose absence semantics: one diagnosis, three different remedies (#294, #304, #307)

proposals/absence_semantics.md

@@ -0,0 +1,228 @@
+# Absence semantics in CommunityMech (#294, #304, #307)
+
+Three open issues turn out to be one diagnosis. This proposal states it, quantifies each
+instance, and recommends a remedy per instance — which is **not** the same remedy three
+times.
+
+**Decision requested:** approve or reject the two remaining remedies in §4 — the enum in
+§4.1 and the counter-selection block in §4.3. They are independent of each other. The third
+(§4.2) has since been implemented and is kept here as the worked example.
+
+---
+
+## 1. The shared diagnosis
+
+The schema can say *"this slot has value X"* and *"this slot is empty"*, and nothing else.
+Empty is therefore overloaded. In the records it currently means at least four different
+things:
+
+| what the curator meant | example |
+|---|---|
+| **not done yet** | a taxon nobody has tried to ground |
+| **impossible** | a virus, which GTDB will never classify |
+| **undecidable from the source** | GTDB splits the NCBI taxon and the paper does not say which |
+| **deliberately excluded** | a strain screened out of a consortium for antagonising its mutualist |
+
+A consumer reading a record cannot tell these apart, and neither can a gate. Worse, in one
+case (§3.2) the audit actively *rewards* filling a slot with an unsourced guess, because a
+filled slot is treated as more complete than an honest blank.
+
+The three issues are the three places this has already caused a concrete problem.
+
+---
+
+## 2. Why one mechanism will not fix all three
+
+It is tempting to propose a single general "absence annotation" reused everywhere. That
+would be wrong here, because the three instances differ in kind:
+
+- **#294** is a *missing vocabulary* problem. The information mostly exists —
+  `gtdb_ground.py` already computes three of the states — it simply is not persisted.
+- **#307** is a *missing slot* problem. The information exists in the source and in the
+  curator's head, and there is nowhere to put it.
+- **#304** is **not a schema problem at all**. It is auditor logic. The schema is fine; the
+  rule that reads it is wrong.
+
+Bundling them into one schema change would drag a code fix into a data migration for no
+benefit. They are proposed separately below and can be done in any order.
+
+---
+
+## 3. The three instances, with numbers
+
+### 3.1 #294 — GTDB grounding: 378 blanks encoding four different situations
+
+Re-measured over **311 records / 1024 taxonomy entries** (figures refreshed after PRs
+#308 and #311 landed):
+
+| | count | share of ungrounded |
+|---|---:|---:|
+| grounded | 646 of 1024 (63%) | — |
+| **no GTDB equivalent** — eukaryote, virus, environmental pseudo-taxon, absent from mapping | 288 | 76% |
+| **ambiguous** — GTDB splits the NCBI taxon with no majority | 87 | 23% |
+| **groundable by the tool** | 6 occurrences | 2% |
+
+Checking what those last six are is what makes the case, and it took running the grounding
+tool across the whole KB and cross-referencing to find out. Five are entries **deliberately
+withheld** under #292 — *Bacteroides ovatus* on `NCBITaxon:821` and `Nitrospiraceae
+bacterium` on `NCBITaxon:1236`, whose ids name a different organism, pinned by
+`tests/test_gtdb_withheld_groundings.py`. The sixth is a **genuine gap** (#314): a taxon
+whose `term.id` was corrected *after* grounding ran, leaving the derived block absent.
+
+So one blank column encodes four different situations — permanently impossible,
+undecidable, deliberately withheld, and accidentally stale — and distinguishing them
+required re-deriving the whole computation. That is the argument in miniature.
+
+It also means #294 has a **fourth** state to represent, distinct from the other three:
+It also means #294 has a **fourth** state to represent, distinct from the other three:
+grounding withheld pending an upstream correction.
+
+### 3.2 #304 — DISCONNECTED fired on the wrong criterion *(now fixed — PR #311)*
+
+Two rules interact in `network/auditor.py`:
+
+1. `connected_taxa` is built only from `source_taxon`/`target_taxon` (lines ~189, ~229). A
+   `COMMUNITY_LEVEL` interaction has neither by design, so it contributes **no**
+   connections. **107 of 302 records with interactions (35%) are entirely
+   `COMMUNITY_LEVEL`** — structurally, a third of the KB has zero connected taxa.
+2. A taxon carrying `abundance_level` **or** `functional_role` was exempt.
+   **931 of 1007 taxa (92%) were exempt on this basis** at the time of measurement.
+
+The exemption is doing essentially all the work. `DISCONNECTED` does not mean "this taxon
+has no curated interaction"; it means "this taxon has no *pairwise* interaction **and** no
+membership metadata" — a compound of connectivity and slot-completeness that the name does
+not convey.
+
+**The consequence is a perverse incentive.** In PR #298 I had invented `abundance_level`
+values (`AbundanceEnum` is quantitative — `DOMINANT` is ">1% relative abundance" — and the
+paper reported no abundances). Review removed them, which is unambiguously correct, and the
+immediate result was five new `DISCONNECTED` findings. A curator optimising against the
+audit would put the guesses back.
+
+### 3.3 #307 — counter-selection has nowhere to live
+
+`SynCom_ARC` (CommunityMech:000314) is *defined* by an exclusion: candidate *Bacillus*
+isolates that inhibited *Bradyrhizobium* were screened out, so the shipped community is the
+subset of effective antifungal strains that spares the nitrogen-fixing mutualist.
+
+There were two places to put that, and both are bad:
+
+- **As an `ecological_interaction`** — machine-readable but false. Excluded and retained
+  isolates are indistinguishable at the genus-level grounding the source supports, so the
+  typed edge asserts that an ARC *member* antagonises the mutualist ARC exists to spare.
+- **As `engineering_design.notes` prose** — accurate but invisible to any query.
+
+PR #305 chose prose. That was right for the record and means the fact is no longer
+queryable. Screening is how most SynComs are built, so this recurs: **197 records** carry
+`engineering_design` and only **40** carry free-text `notes` — the rest have no natural
+place for a negative result at all.
+
+---
+
+## 4. Proposed remedies
+
+### 4.1 #294 — add a grounding-status enum *(schema, small)*
+
+Add to `TaxonDescriptor`, alongside `gtdb_classification`:
+
+```yaml
+gtdb_grounding_status:
+  range: GtdbGroundingStatusEnum
+  # GROUNDED | NO_GTDB_EQUIVALENT | AMBIGUOUS | WITHHELD | NOT_ATTEMPTED
+  required: false
+```
+
+`WITHHELD` is needed because §3.1 found two real instances of it: grounding is possible and
+deliberately not applied, pending an upstream fix (#292). Without it those two collapse into
+`NOT_ATTEMPTED`, which is what the pin in `tests/test_gtdb_withheld_groundings.py` currently
+exists to prevent — a test compensating for vocabulary the schema lacks.
+
+`AMBIGUOUS` should carry the candidate GTDB taxa, since that is precisely what a curator
+needs in order to resolve it. Either a companion multivalued `gtdb_ambiguous_candidates`
+slot, or fold the status into the existing `GtdbClassification` class and allow it without
+a `gtdb_id`.
+
+`gtdb_ground.py` already distinguishes GROUNDED, AMBIGUOUS and NO_GTDB_EQUIVALENT
+internally, so populating those three is a mechanical pass rather than a curation effort;
+only WITHHELD needs to be asserted by hand, and there are two of them. Add a gate afterwards asserting the status matches
+what the tool computes, so it cannot drift — the shape used by `tests/test_enum_groundings.py`.
+
+**Cost:** one schema change, one datamodel regeneration, one backfill run, one test.
+**Benefit:** ~63% coverage stops looking like ~63% completion.
+
+### 4.2 #304 — fix the auditor, change no data — **IMPLEMENTED (PR #311)**
+
+Shipped as proposed: `COMMUNITY_LEVEL` interactions now credit their members, and the
+`abundance_level`/`functional_role` exemption is removed. Both halves together, as the
+simulation required.
+
+| configuration | `DISCONNECTED` reported |
+|---|---:|
+| before | 32 |
+| drop the exemption only | 390 |
+| credit `COMMUNITY_LEVEL` only | 1 |
+| **both — shipped** | **19** |
+
+The prediction held exactly: network issues went 49 → 23, of which `DISCONNECTED` 32 → 19,
+and the surviving findings are genuine (spot-checked a *"Desulfovibrio piger comparator"*
+and *"Sporomusa spp. in KB-1"*, both in `taxonomy` while taking part in no interaction of
+any scope).
+
+Two things surfaced during that work and are tracked separately: the credit is
+all-or-nothing because the schema cannot say *which* members a community-level interaction
+concerns (#312, which overlaps §4.3's design question), and dangling-edge detection turns
+out to live only in an orphaned script that nothing runs (#313).
+
+**This retired the blocker on #273**, which could not decide whether to restore the network
+gate while `DISCONNECTED` measured something other than its name.
+
+### 4.3 #307 — add a counter-selection block *(schema, medium)*
+
+Add to `CommunityEngineeringDesign`:
+
+```yaml
+counter_selection:
+  multivalued: true
+  range: CounterSelection      # excluded_taxon (TaxonDescriptor, optional), criterion, evidence
+```
+
+Deliberately **not** in `ecological_interactions`: excluded candidates are not members, and
+these are not interactions within the community. Making them edges is what produced the
+#300 defect.
+
+`excluded_taxon` must be optional, because ARC is exactly the case where the excluded
+strains cannot be distinguished from the retained ones at the available resolution. A
+counter-selection with a criterion and evidence but no resolved taxon is still far more
+useful than prose.
+
+**Open question for the curator:** should excluded candidates ever appear in `taxonomy`?
+Recommendation: **no** — they are not members, and adding them would re-create #304-style
+connectivity noise.
+
+---
+
+## 5. Recommended sequence
+
+1. ~~**#304, both changes together**~~ — **done, PR #311.** Landed at 19 findings and
+   retired the blocker on #273.
+2. **#294** — status enum plus mechanical backfill. Now the next step. Self-contained, and
+   it retires the recurring misreading of GTDB coverage as GTDB backlog. §3.1 shows the
+   distinction currently costs a full re-derivation to recover.
+3. **#307** — the largest, and the one most worth designing carefully rather than quickly,
+   since it introduces a concept the schema does not yet have. Worth designing together with
+   **#312**, which asks the same question from the other direction: how to say *which*
+   members a community-level statement concerns.
+
+Independent of all three: **#314** (a taxon whose id changed after grounding, leaving a
+stale blank) is a one-command data fix, and the `ncbi_source_id` mismatch it exposes is
+detectable with a cheap test today — no schema change required.
+
+## 6. What is deliberately not proposed
+
+- **A general absence-annotation framework.** Three instances is not enough evidence to
+  design one, and §2 argues they are not the same kind of problem.
+- **A `NOT_APPLICABLE` value anywhere.** It reintroduces the ambiguity this proposal is
+  trying to remove; `NO_GTDB_EQUIVALENT` says why, and "not applicable" does not.
+- **Backfilling `abundance_level`.** The perverse incentive in §3.2 should be removed by
+  fixing the audit, never by filling the slot. `AbundanceEnum` is quantitative and most
+  sources do not report abundances.