Terminology: pipeline

[treysp]

It's critical for the documentation to be internally consistent with respect to terminology. We're not trying to figure out the "true" definition of terms in the sense of how the words are used in the wild, we just need to have a specific and stable definition for use in the docs.

The term pipeline is ubiquitous in both data engineering lingo and our docs. We don't need to come up with a perfect definition, but we want to make sure it's being used at the correct level of generality in the docs.

Specifically, I'd like to clarify the relationship between pipelines and other terms we're using, and I'd like to understand enumeration of pipelines.

Here is the definition in the SQLMesh glossary:
"The set of tools and processes for moving data from one system to another. Datasets are then organized, transformed, and inserted into some type of database, tool, or app, where data scientists, engineers, and analysts can access the data for analysis, insights, and reporting."

Here is Iaroslav's comment on a concepts/overview.md pull request:
I don't think we have a concept of a "pipeline" internally but even if we did a it would be something like a series of execution steps applied to a single model: eg. create a table, evaluate model's logic, write results, run audits, etc.

Questions to answer in this issue:

1. What is the right level of generality for the docs?
   • We could keep it very general (more like glossary def), precluding statements like "there are X models in this pipeline." This would allow us to use it frequently but not in specific technical contexts.
   • We could make it defined at the model level (each model constitutes one pipeline). This would make it very clear how to use it in technical contexts, but would restrict how we used it in introductory paragraphs and other non-technical contexts.
2. How many pipelines are in each of A-D? Is that even a sensible question?

[izeigerman]

In my mind anything that involves more than one model is not a pipeline, it's a dependency DAG or lineage. So none of the cases depicted on the diagram are pipelines. Pipeline is what happens within the individual model, like: read data -> filter data -> transform data -> write data -> audit data.

IMHO our definition in the glossary is wrong.

Here are some reasonable definitions from the internet:

A data pipeline is a method in which raw data is ingested from various data sources and then ported to data store, like a data lake or data warehouse, for analysis. Before data flows into a data repository, it usually undergoes some data processing. This is inclusive of data transformations, such as filtering, masking, and aggregations, which ensure appropriate data integration and standardization.

Source: https://www.ibm.com/topics/data-pipeline#:~:text=A%20data%20pipeline%20is%20a,usually%20undergoes%20some%20data%20processing.

A data pipeline is a set of actions that ingest raw data from disparate sources and move the data to a destination for storage and analysis. A pipeline also may include filtering and features that provide resiliency against failure.

Source: https://www.stitchdata.com/resources/what-is-data-pipeline/

IMHO All these definitions indicate that each model is an individual pipeline. While dependencies between models / pipelines is a DAG.

[izeigerman]

My suggestion would be to just stick to models and not mention pipelines anywhere for now. I think one value of the "model" abstraction is that it shields users from having to deal with data pipelines.

[eakmanrq]

Yeah I see why there is confusion here. Personally I generally have seen the "pipeline" term used to describe a series of transformations (DAG) that complete a given task. "Listening Metrics Pipeline" for a Spotify like company could be a series of tables that in conjunction take in source data and transform to an end result of listening metrics. This grouping is used because team X owns it and it has it's own cadence/SLA. Ex: "Listening Metrics Pipeline" runs hourly and has SEV1 alerting on it.

In my mind then with SQLMesh users don't have to worry about constructing dedicated "pipelines". They just make tables and it takes care of the "pipelining". As a result I agree with @izeigerman that within the context of SQLMesh a pipeline is the loading of an individual model. This though can be confusing as you are pointing out @treysp.

I agree with trying to avoid pipeline entirely. How do we want to word something like this though:
"By default, SQLMesh will halt the when an audit fails in...."

"By default, SQLMesh will halt the loading of a model when an audit fails in..."?

[izeigerman]

"By default, SQLMesh will halt the loading of a model when an audit fails in..."?

yeah, "loading" / "evaluation" / "processing".