Architecture Overview

Architecture Overview

This page is for anyone reading the codebase for the first time — to get oriented before diving into source, or to understand why a change lands where it does. It covers the three ways S9TerpSync is invoked, the modules those three paths share, and why state lives in files instead of a database. It is not a how-to-contribute guide; see the closing note below.

Three entry paths, one set of modules

S9TerpSync is a modular monolith — one Node.js codebase, no microservices — invoked in three different ways:

1. The long-running service starts the scheduler and exposes read-only operational endpoints (health, readiness, metrics) for systemd and monitoring to poll.
2. The manual CLI invokes the same synchronization and administrative logic directly, on demand, from a terminal.
3. The installer/configurator creates or validates the configuration that both the service and the CLI consume.

None of these three paths has its own copy of the sync logic. All of them compose the same modules underneath, and all of them go through the same locking, validation, audit, idempotency, and dry-run behavior — a manual run --dry-run and a scheduled run take an identical path through the code once they're past the entry point.

File-based state, on purpose

S9TerpSync keeps its durable state — run history, idempotency records, locks, retention metadata — as files on disk, not in a database. There's no ORM, no database server to provision or back up, and no Docker to run. This is a deliberate fit for the target deployment: one RHEL/systemd box per institution, managed by a single administrator who shouldn't need to also operate a database. Atomic writes, an exclusive run lock, and append-only audit logs stand in for what a database would otherwise give you for free.

Module boundaries

Module	Purpose
ConfigurationModule	Loads and validates the YAML configuration, resolves secret references, and produces one immutable, effective configuration for everything else to use.
SchedulerModule	Creates the configured cron schedules, in the institution's own time zone, and triggers runs through the same shared run logic every other entry path uses.
IngestionModule	Discovers or fetches source records — from SFTP or the Slate API — and maps every supported format into one canonical applicant model.
ProcessingModule	Orchestrates a run and its records: matching, dependency ordering between operations, and computing each record's terminal outcome.
EthosModule	Provides typed adapters over Ethos resources, on top of one shared authenticated transport.
ReferenceDataModule	Loads only the reference domains a run actually needs and freezes them into a run-scoped snapshot.
CrosswalkModule	Validates and resolves source-to-target value mappings against that reference snapshot.
InstitutionRulesModule	Evaluates institution-specific rules that live in configuration, not hardcoded in the engine.
FileStateModule	Owns the durable state itself: atomic writes, locking, indexes, recovery, and retention metadata.
ReportingModule	Produces file- and record-level reports with sensitive values redacted.
ArchiveModule	Copies and checksum-verifies processed source files, then deletes them only once verification passes.
OperationsModule	Exposes local health, readiness, and metrics endpoints — read-only, no mutation operations reachable through it.

How it fits together

flowchart TB
  service["Long-running service<br/>starts scheduler, exposes operations endpoints"]
  cli["Manual CLI<br/>invokes sync and admin logic directly"]
  installer["Installer / configurator<br/>creates or validates configuration"]

  config["ConfigurationModule"]
  scheduler["SchedulerModule"]
  ingestion["IngestionModule"]
  processing["ProcessingModule"]
  refdata["ReferenceDataModule"]
  crosswalk["CrosswalkModule"]
  rules["InstitutionRulesModule"]
  ethos["EthosModule"]
  filestate["FileStateModule"]
  reporting["ReportingModule"]
  archive["ArchiveModule"]
  operations["OperationsModule"]

  target["Ethos APIs / Banner"]

  installer --> config
  service --> config
  cli --> config
  service --> scheduler
  service --> operations
  config --> scheduler
  config --> processing
  scheduler --> processing
  ingestion --> processing
  processing --> refdata --> crosswalk --> processing
  processing --> rules
  processing --> filestate
  processing --> ethos --> target
  processing --> reporting
  processing --> archive

  classDef pipeline fill:#EEEDFE,stroke:#534AB7,color:#26215C
  classDef mutation fill:#FAECE7,stroke:#993C1D,color:#4A1B0C
  classDef failure fill:#FAEEDA,stroke:#854F0B,color:#412402
  classDef readonly fill:#E1F5EE,stroke:#0F6E56,color:#04342C
  classDef neutral fill:#F1EFE8,stroke:#5F5E5A,color:#2C2C2A
  class config,scheduler,ingestion,processing,refdata,crosswalk,rules,ethos,filestate,reporting,archive,operations pipeline
  class service,cli,installer neutral
  class target mutation

Loading

Where to go next

This page is a starting point for understanding how the pieces fit together, not a contribution guide — accepting outside contributions is a separate workstream that isn't open yet. For everything else about running the software day to day, start from the Home page and its links to installation, configuration, running a sync, and diagnostics.