Scientific Methodology and Limitations

Scientific Methodology and Limitations

Author: Ciprian Ștefan Pleșca

OpenLongevity's documentation is unusually direct about the boundary between what its software demonstrably does and what a reader might be tempted to infer from it. This page consolidates that boundary into a single reference, organized as a set of claims paired with their explicit scope limits, so that anyone building on top of the project — or citing it — can state precisely what a given result establishes.

The core epistemic distinction

flowchart TD
    A["A component executes correctly"] -.does NOT imply.-> B["A scientific claim is validated"]
    C["A source adapter retrieves metadata"] -.does NOT imply.-> D["A trustworthy finding was extracted"]
    E["A database preserves revisions"] -.does NOT imply.-> F["A tamper-proof review history exists"]
    G["A graph endpoint returns an edge"] -.does NOT imply.-> H["Biological causation is supported"]
    I["Type-checking passes on the frontend"] -.does NOT imply.-> J["A production website has been verified"]

Loading

These four pairs are drawn directly from the project's own README and represent the architectural philosophy underlying every module documented elsewhere in this wiki: correctness of execution and validity of scientific content are independent axes, and the system is built to keep them visibly separate rather than let one imply the other.

Claim-by-claim scope table

Component	What a successful call demonstrates	What it does not demonstrate
/api/v1/publications/* (API-Reference)	A publication record was retrieved from, or persisted to, PostgreSQL with intact provenance	That the underlying publication's findings are accurate, unretracted in practice, or clinically meaningful
/api/v1/evidence*	Software correctly filtered and graded a synthetic fixture	Anything about real-world evidence for the queried topic — the fixture set has exactly one record
/api/v1/research-gaps (Research-Gap-Detection)	The six gap heuristics ran correctly against the fixture set	That these are the actual research gaps in the live literature for that topic
/api/v1/graph	The graph response shape renders correctly	Any biological relationship — the nodes and edge are explicitly illustrative strings
EvidenceEngine.score() (Evidence-Grading-Engine)	A transparent, reproducible arithmetic combination of study design, confidence, replication, sample size, and recency	A probability of truth, a treatment effect size, or a validated evidence-certainty scale
BiologicalAgeModel (Analysis-Toolkit)	An ordinary-least-squares fit on the features and targets supplied by the caller	Clinical validity of "biological age" as a construct, or of any specific feature set used to estimate it
kaplan_meier()	Correct point estimates of a survival curve given event/censoring data	Confidence intervals, group comparisons, or handling of competing risks
enrich_gene_set()	A hypergeometric p-value and a BH-style adjustment over the pathways that passed the overlap filter	A complete multiple-testing correction over the originally supplied pathway set (see Analysis-Toolkit for the exact limitation)
apps/web TypeScript shell	The frontend type-checks and its build/lint/test scripts execute	A deployed, accessibility-reviewed, browser-tested production application

The A–G hierarchy is a navigation convention, not a ranking of truth

As detailed in Evidence-Grading-Engine, the letter grade assigned to a record reflects study design category — it is "not a claim that every review is stronger than every experiment or a substitute for evaluating bias and relevance." The implementation itself acknowledges a specific sharp edge here: retracted records are mapped to grade G alongside genuinely computational work, so any client displaying grades must also surface retraction_status separately, or risk implying that a retracted RCT and an exploratory simulation are the same kind of "low-grade" evidence.

A four-stage workflow with explicit stopping points

The project's README lays out a disciplined research workflow, which this wiki reproduces here because it directly governs how the software's outputs should be used:

flowchart TD
    S1["1. Formulate a question with clear\npopulation, measurement, and inference target"]
    S1 --> S2["2. Retrieve and log:\nprovider, query, retrieval time, IDs, pagination.\nReport incomplete retrieval as incomplete — never as exhaustive."]
    S2 --> S3["3. Inspect normalized metadata against\nthe original provider record.\nA missing abstract is missing information,\nnot a negative finding."]
    S3 --> S4["4. Only then consider a structured\nEvidenceRecord — with its own provenance\nand review decision, distinct from the\npublication's bibliographic envelope."]

Loading

Each stage has a corresponding module in this codebase — provider adapters for stage 2 (Literature-Provider-Adapters), the persistence layer's provenance requirements for stage 3 (Persistence-and-Data-Governance), and the EvidenceRecord/ReviewStatus types for stage 4 (Domain-Model) — but the workflow discipline itself is a human responsibility that no amount of correct software enforces automatically.

Practical guidance for reporting results from this system

1. Always state the mode. Every relevant API response already carries mode: "persisted" or mode: "fixture-only" — preserve that distinction in any downstream report, chart, or publication.
2. Never present a navigation score as a probability. State it as "OpenLongevity navigation score: 0.88," not "88% likely to be true."
3. Report retraction status alongside grade, not instead of it. A grade of G alone loses the distinction between "retracted" and "computational."
4. Treat analysis-toolkit outputs as inputs to further scrutiny, not conclusions. A Kaplan–Meier curve, an OLS fit, or an enrichment p-value from this toolkit is a calculation on the data supplied — its scientific weight depends entirely on the quality and provenance of that data, which the toolkit itself does not assess.
5. Distinguish software capability claims from research claims when describing the project. "OpenLongevity can retrieve and normalize PubMed records with provenance" is a verifiable software claim; "OpenLongevity has identified evidence for a given longevity intervention" is not currently a claim the implementation supports.

Next

Return to Home for the full page index, or see Contribution-and-Governance-Model for how these standards are enforced in review.