Spool is a modular, event-driven data pipeline framework for Java. It handles the full lifecycle from raw source extraction to a queryable data mart — with built-in observability, pluggable adapters, and no vendor lock-in.

Spool-FRAMEWORK-Labs is the examples & use cases organization for the Spool Framework — a collection of real-world, production-style pipelines built with Spool, covering data sources like public APIs, government datasets, healthcare records, and financial filings.

External Sources  (HTTP · DB · File · Queue)
        │
        ▼
     Crawler    fetch → normalize → split → persist
        │
      Inbox     (S3 · FileSystem · custom)
        │
     Janitor    housekeeping: republish stuck · quarantine failed · remove expired
        │
    Event Bus   (Kafka · InMemory)
        │
     Ingester   validate → batch → flush
        │
    Data Lake   (S3 · FileSystem)
        │
     Mounter    aggregate → promote
        │
    Data Mart   (silver · gold)


     Watchdog   heartbeat monitoring  ←── Crawler

Each module communicates through an event bus. Every envelope state transition (CAPTURED → INGESTED → PERSISTED) is an event — fully auditable and reproducible.

Create src/main/resources/Spool.yaml:

infrastructure:
  eventBus:
    type: "inMemory"
  inbox:
    filesystem:
      path: "/var/spool/inbox"
  dataLake:
    filesystem:
      path: "/var/spool/datalake"

modules:

  - crawler:
      id: "my-crawler"
      source:
        id: "my-source"
        format: JSON_ARRAY
        poll:
          http:
            url: "https://api.example.com/data"
      schedule:
        everyMilliseconds: 60000

  - janitor:
      id: "my-janitor"
      everyMilliseconds: 5000

  - ingester:
      id: "my-ingester"
      flush:
        size: 100
        every: 30
        unit: SECONDS

Boot the pipeline:

SpoolRuntime.builder()
        .OpenTelemetryConfiguration(otelConfig)
        .withNodeFromDSL("/Spool.yaml")
        .build()
        .start();

Crawler crawler = CrawlerBuilderFactory.poll(new HTTPPollSource("https://api.example.com/data", "my-api"))
        .source()
            .ports(CrawlerPorts.builder().inbox(inboxWriter).bus(eventBus).build())
            .schedule(PollingConfiguration.every(Duration.ofMinutes(1)))
            .and()
        .createWith(StandardNormalizer.JSON_ARRAY);

Janitor janitor = JanitorBuilderFactory.polling()
        .from(inboxQuery)
        .with(inboxUpdater)
        .removeWith(envelopeRemover)
        .on(eventPublisher)
        .subscribeWith(eventSubscriber)
        .every(Duration.ofSeconds(5))
        .create();

Ingester ingester = IngesterBuilderFactory.reactive()
        .readWith(envelopeResolver)
        .flushPolicy(FlushPolicy.whenReaches(100).orEvery(Duration.ofSeconds(30)))
        .writingTo(dataLakeWriter)
        .subscribingWith(eventSubscriber)
        .create();

SpoolNode.create()
        .register(crawler)
        .register(janitor)
        .register(ingester)
        .start();

• One interface to extend anything — implement PollSource to crawl any system; @SpoolPlugin to add any adapter
• Event-sourced audit trail — every state change is an immutable event on the bus
• Observability out of the box — OpenTelemetry traces, Prometheus metrics, Loki logs, and Grafana dashboards included
• Technology-agnostic — swap Kafka for any bus, S3 for any store, without touching pipeline logic
• Declare or code — full YAML DSL for operations, fluent builder API for fine-grained control

Spool was developed as a final-year thesis project (Trabajo de Fin de Título) at the Universidad de Las Palmas de Gran Canaria, within the Bachelor's degree in Computer Science Engineering.

Built with Java 21 · Apache License 2.0