Add the confident contact count (CCC) - #24
Merged
Conversation
CCC counts inter-chain residue pairs that are both in physical contact and confidently placed relative to one another, judged by the predicted aligned error. Lambourne et al. introduced it as a screening statistic and report it beating the usual confidence scores at low false-positive rates. Our own benchmark reproduces that on AlphaFold2, where CCC leads contact probability on early retrieval (normalised partial AUC over the first 1% of controls, 0.482 against 0.430) while losing clearly on global AUROC (0.822 against 0.862); on AlphaFold3 contact probability wins both. It is a specialist score for screens whose priority is the first few candidates, not a new default. Adds src/alphajudge/confident_contacts.py, Interface.confident_contacts(...) and Interface.ccc, with three tests covering the counting, the Interactome3D geometry and the boundary convention. Both conventions the count depends on are explicit parameters rather than hard-coded. Contact geometry defaults to INTERACTOME3D, following Mosca et al. as cited by Lambourne et al.; that definition lists four rules but its hydrogen-bond rule (N-O <= 3.5 A) is strictly subsumed by its salt-bridge rule (N-O <= 5.5 A), so three tests decide membership. REPRESENTATIVE_ATOM reuses AlphaJudge's own CB/CA contact definition instead, so CCC, contact_pairs and cLIS can share one pair set. The PAE boundary needed the published record to settle: the paper's main text twice says PAE <= 4 A while its Methods heading says PAE < 4 A. The authors' released analysis code decides it - the column carrying the count is named n_contacts_PAE_lt_4A - so strict is the default here, with inclusive=True available. The PAE direction is the one convention the published material does not fix, since the paper refers to "the" PAE of a contact without saying which entry of the asymmetric matrix; ab is our default and the other four are available to audit the asymmetry. Verified against this project's own prior implementation on 60 real benchmark predictions across both backends, both classes and all four organisms: identical contact-pair sets and identical counts on all five direction variants, 60/60. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The slider panel listed six AlphaFold-derived rows, five of which summarised the same predicted- aligned-error matrix: ipSAE, LIS, pDockQ2, average interface PAE and ipTM. ipSAE and LIS correlate at rho ~ 0.9 on the benchmark, so a reader seeing six sliders reasonably inferred six pieces of evidence when the panel carried roughly two. Drop LIS and pDockQ2 from the panel and add CCC, so the rows span the distinct constructions instead: the distogram (contact probability), PAE gated by contact geometry (CCC), interface-restricted PAE (ipSAE), AlphaFold's own global number (ipTM), and the raw error the rest are built from (average interface PAE). Both dropped scores are still computed and still written to the score table; only the report layout changed. Emit interface_ccc in the per-interface score table. A slider row whose feature has no frozen benchmark ladder is now omitted rather than raising KeyError. CCC has no ladder yet: its deciles must be frozen on the full benchmark, and the stratified sample available today caps each organism at 250 rows, which would over-weight the small organisms about sixteen-fold against the benchmark's real composition. Drawing a bar off that would be worse than drawing none. The CCC row appears automatically once the deciles are frozen. interface_meta_score keeps its eleven-feature basis, having measured the alternative rather than assumed it. A reduced basis with one representative per construction scores 0.8332/0.8287 (AF2/AF3) against the current 0.8312/0.8320 - indistinguishable, and changing it would invalidate every previously reported meta-score value for no gain. The more useful finding is that neither aggregate beats the best single score: contact probability alone reaches 0.8333/0.8428. The meta-score is therefore a convenience summary of the panel, not the recommended ranking column, and the CHANGELOG now says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
|
Full-benchmark follow-up is now pushed as dafbbb8. The report's CCC row is calibrated on positive interface-present rows with parsed CCC values (12,160 pooled; 6,034 AF2; 6,126 AF3), with backend-specific ladders selected from model provenance. The manual freezer now reproduces the pooled/AF2/AF3 ladders via --ccc-csv and --backend-filter. Focused verification: 10 report/meta-score tests passed; 3 CCC geometry/boundary tests passed; all touched modules compile. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the confident contact count — inter-chain residue pairs that are both in physical contact and confidently placed relative to one another, judged by the predicted aligned error. Introduced by Lambourne et al., Nat. Commun. 17:4894.
Why
The benchmark paper's recommendations already point readers at CCC for screens whose priority is the first few follow-up candidates, and until now AlphaJudge could not compute the score it recommends.
It is not a new default. On our benchmark sample it loses clearly on global AUROC (0.822 AF2 / 0.797 AF3 against contact probability's 0.862 / 0.881), but leads on early retrieval on AlphaFold2 — normalised partial AUC over the first 1% of controls, 0.482 against 0.430. Lambourne et al. report the same ordering independently on yeast. On AlphaFold3 contact probability wins both.
API
Conventions, and how they were settled
Both choices the count depends on are explicit parameters, not hard-coded.
Contact geometry defaults to
INTERACTOME3D, following Mosca et al. as cited by Lambourne et al. That definition lists four rules, but its hydrogen-bond rule (N–O ≤ 3.5 Å) is strictly subsumed by its salt-bridge rule (N–O ≤ 5.5 Å), so three tests decide membership: Cys S–S ≤ 2.56 Å, N–O ≤ 5.5 Å, C–C ≤ 5.0 Å.REPRESENTATIVE_ATOMreuses AlphaJudge's own CB/CA definition so CCC,contact_pairsand cLIS can share one pair set.The PAE boundary needed the published record to settle. The paper contradicts itself — main text twice says
PAE ≤ 4 Å, Methods heading saysPAE < 4 Å. The authors' released analysis code decides it: the column carrying the count is namedn_contacts_PAE_lt_4A. Strict is the default;inclusive=Trueis available.The PAE direction is the one convention the published material does not fix. The paper refers to "the" PAE of a contact without saying which entry of the asymmetric matrix.
ab(chain1 → chain2) is our default;ba/min/mean/maxexist to audit the asymmetry.Verification
Against the benchmark project's prior independent implementation, on 60 real predictions spanning both backends, both classes and all four organisms: identical contact-pair sets and identical counts on all five direction variants, 60/60.
Three new tests cover the counting, the Interactome3D geometry and the boundary convention.