4.3 Fetcher and Disk Cache

Fetcher and Disk Cache

Relevant source files

The following files were used as context for generating this wiki page:

• electron/fetcher.js
• tests/unit/cache.test.js
• tests/unit/disk-cache.test.js
• tests/unit/http-electron.test.js

The purpose of this page is to document the main-process networking and persistent caching infrastructure implemented in electron/fetcher.js and electron/disk-cache.js. This subsystem handles secure HTTPS requests, per-host rate limiting, streaming size caps, request retries, and file-system-backed caching grouped by gene symbols for VariantScope's desktop target.

Sources: electron/fetcher.js:1-127, electron/disk-cache.js:1-93

---

1. Main Process Fetcher Architecture

The fetcher runs entirely in the Electron main process, exposing a secure IPC handler created by createFetcher electron/fetcher.js:81-127 that processes requests initiated from the preload bridge and renderer.

Core Safeguards and Constants

• Timeouts: Every request is bounded by a strict timeout (TIMEOUT_MS = 20_000 ms) via AbortSignal.timeout electron/fetcher.js:6, electron/fetcher.js:60.
• Size Caps: Responses are read iteratively through readCapped electron/fetcher.js:30-46. If a response exceeds MAX_BYTES (25 MB, well above the largest expected 1.5 MB biological data payload), the reader is canceled and an error is thrown electron/fetcher.js:7, electron/fetcher.js:39-42.
• Redirect Blocking: net.fetch is explicitly configured with redirect: 'error' to prevent external servers from redirecting requests to hosts outside the security allowlist electron/fetcher.js:56-57.
• No-Store Cache Directive: Chromium's built-in HTTP cache is bypassed (cache: 'no-store') so that data freshness and disk-cache rules remain fully controlled by VariantScope electron/fetcher.js:58-59.

Rate Limiting and Queues

To respect third-party API usage policies (such as NCBI E-utilities restrictions), electron/fetcher.js enforces per-host throttling using asynchronous request queues electron/fetcher.js:9-27:

• MIN_INTERVAL_MS specifies custom gaps per host (e.g., 340 ms for eutils.ncbi.nlm.nih.gov, translating to roughly 3 requests per second) and defaults to 100 ms electron/fetcher.js:10-11.
• acquireSlot(host) maintains a promise chain per hostname to ensure requests are appropriately spaced electron/fetcher.js:18-27.

Retries and Error Handling

requestWithRetry wraps HTTP executions to handle transient failures electron/fetcher.js:66-78:

• Status codes 429 (Too Many Requests) and 5xx (Server Errors) are classified as retryable (retryable check at electron/fetcher.js:71).
• A single retry is executed after an exponential/fixed delay (RETRY_DELAY_MS * 2) electron/fetcher.js:76.

Sources: electron/fetcher.js:1-127

---

2. Disk Cache Layout and Persistence

The disk cache (electron/disk-cache.js) provides persistent storage for API responses in a dedicated user-data directory. Unlike transient memory caches, files are retained indefinitely for offline use, but are flagged as stale after a fixed window.

Staleness and Expiration

• OUT_OF_DATE_MS defines the freshness TTL (7 days) tests/unit/disk-cache.test.js:6, tests/unit/disk-cache.test.js:31.
• cache.get(key, group) inspects the stored file metadata; if the file age exceeds 7 days, it is returned with stale: true rather than being deleted tests/unit/disk-cache.test.js:31-41.

Gene Grouping and File Naming

Cached files are associated with genomic entities using a group tag (e.g., a gene symbol like TP53) electron/fetcher.js:90-91:

• isValidGroup(group) validates that group tags conform to strict alphanumeric/hyphen constraints (maximum 20 characters) and rejects path traversal attempts like ../ or absolute paths tests/unit/disk-cache.test.js:52-61.
• Files are written to disk using the format group~hash.json, where un-grouped or invalidly grouped requests default to a fallback prefix (_~) tests/unit/disk-cache.test.js:43-51.

Management Operations

• listGroups() scans the cache directory, aggregating entry counts and total byte sizes per gene group tests/unit/disk-cache.test.js:63-73.
• deleteGroups(groups) selectively removes files associated with specified gene arrays or clears the entire cache when passed 'all' tests/unit/disk-cache.test.js:75-95.

Sources: electron/disk-cache.js:1-93, tests/unit/disk-cache.test.js:1-96

---

3. Request Flow and Code Architecture

The following diagram bridges natural language descriptions of the request lifecycle to the actual functions and modules inside the codebase.

graph TD
  subgraph "Renderer Process"
    A["getJson src/core/http.js"] -->|IPC Invoke| B["window.api.fetchJson electron/preload.cjs"]
  end

  subgraph "Main Process"
    B -->|Calls| C["createFetcher electron/fetcher.js"]
    C -->|Validates URL| D["checkApiUrl electron/allowlist.js"]
    C -->|Checks Cache| E["createDiskCache electron/disk-cache.js"]
    
    E -->|Hit & Fresh| F["Return Cached Data"]
    E -->|Miss or Stale/Refresh| G["acquireSlot electron/fetcher.js"]
    
    G -->|Rate Limited| H["requestWithRetry electron/fetcher.js"]
    H -->|Fetch with Net| I["net.fetch electron"]
    I -->|Read Body| J["readCapped electron/fetcher.js"]
    J -->|Success| K["cache.set electron/disk-cache.js"]
    K -->|Store File| L["FileSystem Disk Storage"]
  end

Loading

Sources: electron/fetcher.js:81-127, electron/disk-cache.js:1-93, electron/allowlist.js:1-1

Sources: electron/fetcher.js:81-127, electron/disk-cache.js:1-93, electron/allowlist.js:1-1

---

4. Disk Cache File Storage Architecture

The structural relationship between group validation, disk storage naming, and grouped metrics is mapped out below.

graph LR
  subgraph "Input Layer"
  M["fetchJson options.group"] --> N["isValidGroup electron/disk-cache.js"]
  end

  subgraph "Validation & Sanitization"
  N -->|Valid Group| O["Prefix: TP53~"]
  N -->|Invalid/Missing| P["Prefix: _~"]
  end

  subgraph "Disk Storage Layout"
  O --> Q["file: TP53~hash1.json"]
  O --> R["file: TP53~hash2.json"]
  P --> S["file: _~hash3.json"]
  end

  subgraph "Management APIs"
  Q & R --> T["listGroups electron/disk-cache.js"]
  S --> T
  T --> U["Delete Groups / Calculate Bytes"]
  end

Loading

Sources: electron/disk-cache.js:1-93, tests/unit/disk-cache.test.js:43-74

Sources: electron/disk-cache.js:1-93, tests/unit/disk-cache.test.js:43-74

---

5. Error Classification and Network Failures

When network execution fails or access is refused, createFetcher normalizes errors before returning them to the UI layer electron/fetcher.js:121-125:

• Allowlist Blocks: If checkApiUrl rejects a URL, the fetcher immediately returns an error object without initiating network activity electron/fetcher.js:83-84.
• Offline Detection: Connection-level errors (such as TimeoutError or error messages matching net::ERR_) set the offline flag to true, allowing the UI to gracefully fall back to cached data or display offline banners electron/fetcher.js:122-124.
• Invalid JSON: If a server responds with status 2xx but returns malformed body text when JSON parsing is requested, a descriptive error is returned ('The server did not return valid JSON.') electron/fetcher.js:110-117.

Sources: electron/fetcher.js:81-127, tests/unit/http-electron.test.js:62-75

---