Skip to content

feat(pipeline): add retry logic, perClass conversion, and tuneable concurrency - #235

Merged
ddeboer merged 2 commits into
mainfrom
feat/pipeline-resilience
Mar 13, 2026
Merged

feat(pipeline): add retry logic, perClass conversion, and tuneable concurrency#235
ddeboer merged 2 commits into
mainfrom
feat/pipeline-resilience

Conversation

@ddeboer

@ddeboer ddeboer commented Mar 13, 2026

Copy link
Copy Markdown
Member

Summary

Improves pipeline resilience against cold-cache 504 timeouts on external SPARQL endpoints.

  • Retry transient HTTP errors: SparqlConstructExecutor now retries on 502/503/504 using p-retry (configurable, default 3 retries)
  • Convert classPropertySubjects/classPropertyObjects to perClass: these stages previously ran monolithic queries across all classes, making them vulnerable to timeouts. Now they use per-class execution with inner subqueries compatible with injectValues()
  • Tuneable concurrency: new VoidStageOptions interface exposes batchSize and maxConcurrency on all VoID stage factory functions
  • voidStages() convenience function: returns all stages in recommended order, optimised for cache warming — classPartitions() runs before per-class stages so the ?s a ?class pattern is cached. Replaces 16-item Promise.all() boilerplate in consumers

Changes

@lde/pipeline

  • Add p-retry dependency
  • Add retries option to SparqlConstructExecutorOptions (default 3)
  • Wrap fetchTriples() in pRetry(), only retrying on transient HTTP errors (502/503/504)
  • Add isTransientHttpError() helper parsing the fetch-sparql-endpoint error format
  • Three new tests: retry succeeds, non-transient not retried, retries exhausted

@lde/pipeline-void

  • Add VoidStageOptions and VoidStagesOptions interfaces
  • All stage factories accept optional VoidStageOptions
  • classPropertySubjects() and classPropertyObjects() now use selection: 'perClass'
  • SPARQL queries rewritten: ?type?class, added inner subqueries for injectValues() compatibility
  • New voidStages() function with cache-warming stage ordering
  • README updated: document voidStages() with options table and usage example, move classPropertySubjects/classPropertyObjects to per-class table, fix query file link paths

ddeboer added 2 commits March 13, 2026 15:50
…ncurrency

- Add p-retry for transient HTTP errors (502/503/504) in SparqlConstructExecutor
- Convert classPropertySubjects/classPropertyObjects to perClass execution,
  wrapping queries in inner subqueries for injectValues() compatibility
- Add VoidStageOptions interface for configurable batchSize/maxConcurrency
- Add voidStages() convenience function with cache-warming stage ordering
- Update coverage thresholds for new code paths
- Add voidStages() section with cache-warming rationale and options table
- Move classPropertySubjects/classPropertyObjects to per-class table
- Fix query file link paths (src/queries → queries)
- Update uriSpaces parameter name to uriSpaceMap
@ddeboer
ddeboer merged commit 3a450d8 into main Mar 13, 2026
2 checks passed
@ddeboer
ddeboer deleted the feat/pipeline-resilience branch March 13, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant