2 Core Data Pipeline

Core Data Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• docs/ARCHITECTURE.md
• src/core/pipeline.js

The purpose of this page is to provide a high-level architectural overview of the src/core data pipeline within VariantScope. The core data layer encapsulates all business logic, data fetching, parsing, joining, and statistical computations with zero DOM or UI dependencies, allowing it to be tested entirely in isolation docs/ARCHITECTURE.md:41-41.

For detailed technical breakdowns, see the child pages:

• Data Source Adapters — Endpoint configurations, query construction, and parsing.
• Joining and Statistics — Coordinate/protein-change matching, numbering validation, and rarity buckets.
• Networking and Caching — HTTP abstraction, caching layers, and offline behaviors.
• Saved Library and Folders — Persistence schemas, hierarchical folders, and storage sizing.

---

Pipeline Execution Flow

When a user searches for a gene symbol (e.g., "TP53"), the core pipeline coordinates asynchronous network fetches across multiple biological data sources, handles partial failures gracefully, validates reference numbering, and computes summary statistics src/core/pipeline.js:29-82.

graph TD
  Start["UserSearchInput"] -->|Calls| LoadGene["loadGene(symbol, options)<br/>src/core/pipeline.js:29"]
  LoadGene -->|1. Required| Uniprot["fetchUniprot()<br/>src/core/uniprot.js"]
  Uniprot -->|Accession & Sequence| ParallelTasks["Parallel Settle Phase<br/>src/core/pipeline.js:36"]
  
  ParallelTasks --> AlphaFold["fetchAlphaFold()<br/>src/core/alphafold.js"]
  ParallelTasks --> ClinVar["fetchClinvar()<br/>src/core/clinvar.js"]
  ParallelTasks --> GnomADVar["fetchGnomadVariants()<br/>src/core/gnomad.js"]
  ParallelTasks --> GnomADCon["fetchGnomadConstraint()<br/>src/core/gnomad.js"]

  AlphaFold --> ParseStage["parse.js Mapping"]
  ClinVar --> ParseStage
  GnomADVar --> ParseStage
  GnomADCon --> ParseStage

  ParseStage --> JoinStage["joinVariants()<br/>src/core/join.js:59"]
  JoinStage --> StatsStage["summarize()<br/>src/core/stats.js"]
  StatsStage --> ResultObj["PipelineResultPayload<br/>src/core/pipeline.js:70"]

Loading

Sources: docs/ARCHITECTURE.md:43-56, src/core/pipeline.js:29-82

---

Natural Language to Code Entity Space

To bridge biological domain concepts with the codebase implementation, the following diagram maps high-level data sources and operations directly to their corresponding JavaScript modules and functions in src/core.

graph LR
  subgraph "Biological Data Sources"
    BioUni["UniProt Reference Sequence"]
    BioAF["AlphaFold 3D Structure"]
    BioCV["ClinVar Clinical Variants"]
    BioGN["gnomAD Frequencies & Constraint"]
  end

  subgraph "Code Entity Space"
    CodeUni["fetchUniprot()<br/>src/core/uniprot.js"]
    CodeAF["fetchAlphaFold()<br/>src/core/alphafold.js"]
    CodeCV["fetchClinvar()<br/>src/core/clinvar.js"]
    CodeGN["fetchGnomadVariants()<br/>src/core/gnomad.js"]
    CodeParse["parseResponse()<br/>src/core/parse.js"]
    CodeJoin["joinVariants()<br/>src/core/join.js"]
  end

  BioUni --> CodeUni
  BioAF --> CodeAF
  BioCV --> CodeCV
  BioGN --> CodeGN

  CodeUni --> CodeParse
  CodeAF --> CodeParse
  CodeCV --> CodeParse
  CodeGN --> CodeParse

  CodeParse --> CodeJoin

Loading

Sources: docs/ARCHITECTURE.md:43-56, src/core/pipeline.js:3-10

---

Sub-Topics and Child Page Overview

2.1 Data Source Adapters

The adapters in src/core handle endpoint construction, payload transformation, and robust parsing. UniProt acts as the fundamental gatekeeper: its lookup is mandatory because it provides the reference protein accession and amino acid sequence against which all subsequent variants are checked docs/ARCHITECTURE.md:60-60. AlphaFold, ClinVar, and gnomAD execute concurrently via settle() wrappers, allowing individual sources to fail without crashing the entire search docs/ARCHITECTURE.md:48-61.

For detailed endpoint construction, queries, and parsing rules, see Data Source Adapters.

Sources: src/core/pipeline.js:12-47, docs/ARCHITECTURE.md:46-61

2.2 Joining and Statistics

Once raw variant records are fetched and parsed, src/core/join.js matches ClinVar clinical assertions to gnomAD population frequencies using genomic coordinates primarily, falling back to protein-change matching when necessary docs/ARCHITECTURE.md:62-63. It performs rigorous numbering validation against the reference sequence docs/ARCHITECTURE.md:63-63. Subsequently, src/core/stats.js calculates rarity buckets, density distributions, and summary metrics.

For implementation details on coordinate matching, validation thresholds, table models, and CSV formatting, see Joining and Statistics.

Sources: src/core/pipeline.js:59-66, docs/ARCHITECTURE.md:62-64

2.3 Networking and Caching

Network abstraction is handled uniformly through getJson(url, options) in src/core/http.js docs/ARCHITECTURE.md:15-15. Depending on the deployment target, this function delegates to the Electron main process via IPC or executes a standard browser fetch combined with localStorage caching docs/ARCHITECTURE.md:67-71. Cached data never expires automatically but tracks freshness (triggering an "Out of date" warning after 7 days) docs/ARCHITECTURE.md:74-78.

For deeper coverage of IPC fetchers, TTL freshness rules, and disk caching mechanics, see Networking and Caching.

Sources: src/core/http.js:1-60, docs/ARCHITECTURE.md:65-90

2.4 Saved Library and Folders

Saved genes, folder hierarchies, and user collections are managed through library.js, folders.js, and folderStore.js docs/ARCHITECTURE.md:93-95. Folders act purely as organizational metadata labels without duplicating underlying gene payloads docs/ARCHITECTURE.md:101-101. Folder nesting is restricted to 6 levels with cycle-prevention sanitization docs/ARCHITECTURE.md:95-95, docs/ARCHITECTURE.md:103-103.

For schemas, persistence strategies, and folder mutation rules, see Saved Library and Folders.

Sources: src/core/library.js:1-40, docs/ARCHITECTURE.md:91-106

---