Skip to content

[FAQ] Why are SQL row counts not part of the orchestration layer? #138

@AsherJD-io

Description

@AsherJD-io

Course

data-engineering-zoomcamp

Question

Why are dataset row counts computed in PostgreSQL instead of within the Kestra workflow?

Answer

Kestra is designed to orchestrate tasks rather than perform analytical computations. Row counts represent dataset-level analytics that are best computed in the database layer after ingestion, ensuring correctness and separation of responsibilities.

Checklist

  • I have searched existing FAQs and this question is not already answered
  • The answer provides accurate, helpful information
  • I have included any relevant code examples or links

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions