Skip to content

refactor: consolidate terminal ETL scheduler into single sequential handler #221

@GitAddRemote

Description

@GitAddRemote

User Story

As a backend engineer, I want the terminal and terminal-distances ETL scheduler to run as a single sequential handler, so that the ordering dependency between the two steps is guaranteed by code rather than a 5-minute cron offset.

Definition of Done

  • Replace the two independent @Cron methods in CatalogEtlScheduler with a single @Cron('0 * * * *') handler that calls runStep('terminals-sync') then awaits completion before calling runStep('terminal-distances-sync')
  • If `terminals-sync` fails, `terminal-distances-sync` is skipped for that cycle and the failure is logged
  • Remove the offset cron expression `'5 * * * *'` (no longer needed)
  • Unit tests updated to cover the sequential-execution and early-abort behaviors

Acceptance Criteria

  • Terminal distances step never starts in the same scheduler cycle until terminals step has completed
  • A failure in terminals step prevents the distances step from running that cycle
  • Both steps still appear in `station_etl_run` with separate run rows

Technical Elaboration

Current implementation uses two independent @Cron decorators at '0 * * * *' and '5 * * * *'. The 5-minute gap is a pragmatic workaround but does not guarantee ordering — if terminals sync is slow or the server is under load, the gap may not be sufficient on a fresh database.

The fix: collapse into one handler that awaits runStep('terminals-sync') and only proceeds to runStep('terminal-distances-sync') on success.

@Cron('0 * * * *', { name: 'terminal-etl' })
async scheduledTerminalEtl(): Promise<void> {
  try {
    await this.catalogEtlService.runStep('terminals-sync');
  } catch (err) {
    this.logger.error({ err }, 'terminals-sync failed; skipping terminal-distances-sync');
    return;
  }
  try {
    await this.catalogEtlService.runStep('terminal-distances-sync');
  } catch (err) {
    this.logger.error({ err }, 'terminal-distances-sync failed');
  }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendBackend services and logicenhancementNew feature or requesttech-storyTechnical implementation story

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions