Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create column lineage endpoint proposal #2077

Merged
merged 4 commits into from
Sep 12, 2022
Merged

Create column lineage endpoint proposal #2077

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bd9cea9
Create column lineage endpoint proposal
julienledem Aug 19, 2022
258f38c
add more details
julienledem Aug 19, 2022
c4fdaea
extend existing proposal with comment suggestion
pawel-big-lebowski Aug 30, 2022
071fdc1
Merge branch 'main' into column-lineage-proposal
pawel-big-lebowski Sep 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions proposals/2045-column-lineage-endpoint.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Proposal: Column lineage endpoint proposal

Author(s): @julienledem

Created: 20022-08-18

Dicussion: [column lineage endpoint issue #2045](https://github.com/MarquezProject/marquez/issues/2045)

## Overview

### Use cases
- Find the current upstream dependencies of a column. A column in a dataset is derived from columns in upstream datasets.
- See column-level lineage in the dataset level lineage when available.
- Retrieve point-in-time upstream lineage for a dataset or a column. What did the lineage look like yesterday compared to today?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What did the lineage look like yesterday compared to today?

The compare use case needs a proposal on it's own 😉

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair enough, I'm hoping someone else can take over that part and go in the details

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking to start with just point-in-time upstream lineage. And have compare later

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this proposal should be limited to point-in-time within column-level lineage. We should leave compare feature and also point-in-time for lineage endpoint which has nothing to do with column level.


### Existing elements

- OpenLineage defines a [column-level lineage facet]- (https://github.com/OpenLineage/OpenLineage/blob/ff0d87d30ed6c9fe39472788948266a6d3190585/spec/facets/ColumnLineageDatasetFacet.md).
pawel-big-lebowski marked this conversation as resolved.
Show resolved Hide resolved
- Marquez has a lineage endpoint `GET /api/v1/lineage` that returns the current lineage graph connected to a job or a dataset

### New Elements
We propose to add the following:
- Add column lineage to the lineage endpoint
- A new column-lineage endpoint leveraging the column lineage facet to retrieve lineage for a given column.
- Point-in-time upstream (dataset or column level) lineage given a version of a dataset.

## Proposal

### add column lineage to existing endpoint
In the GET /lineage api, add column lineage to DATASET nodes' data
```diff
{
"id": "dataset:food_delivery:public.categories",
"type": "DATASET",
"data": {
"type": "DATASET",
"id": {
"namespace": "food_delivery",
"name": "public.categories"
},
"type": "DB_TABLE",
...
"fields": [{
...
}],
> columnLineage: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calls to GET /lineage returns an array of nodes. Meaning, we'll want to add columnLineage to DatasetData. We can use the generated classes in the openlineage-java lib. for column level lineage (vs maintaining our own). Which, I think we still need to generate @julienledem? I don't see them in the javadocs for OpenLineage.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think yes, we would reuse the columnLineage facet object. The OL javadoc needs to be updated. It is not an automated process at the moment

> "a": {
> inputFields: [
> {namespace: "ns", name: "name", "field": "a"},
> ... other inputs
> ],
> transformationDescription: "identical",
> transformationType: "IDENTITY"
> },
> "b": ... other output fields
> }
},
"inEdges": [{
"origin": "job:food_delivery:etl_orders.etl_categories",
"destination": "dataset:food_delivery:public.categories"
}],
"outEdges": [{
"origin": "dataset:food_delivery:public.categories",
"destination": "job:food_delivery:etl_orders.etl_orders_7_days"
}]
}
```

### add a column-level-lineage endpoint:

```
GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days&column=a
Copy link
Member

@wslulciuc wslulciuc Aug 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we have the following endpoint to query lineage:

GET /lineage?nodeId=<node-id>

I'm not sure there's much advantage to defining a separate endpoint for column-level lineage. Although a new endpoint would contextualize the API call; with proper documentation, we can extend out current lineage endpoint to support columns:

GET /lineage?nodeId=<node-id>,column=<column>

If the query param column is present, the backend will assume the nodeId to be a dataset node ID and return an upstream lineage graph with only dataset-to-dataset relationships. That said, column-level lineage is an upstream query from the origin <node-id> (as defined in this proposal). The /lineage call assumes both upstream and downstream lineage. To further contextualize the call, we should (and would prefer) an upstream specific lineage endpoint:

GET /lineage/upstream?nodeId=<node-id>
GET /lineage/downstream?nodeId=<node-id> # Add for completeness

On the backend, these calls would be handled differently. When querying for upstream lineage, the graph returned would consists of only nodes upstream of <node-id>; similarly for upstream lineage, only nodes downstream.

LineageService.upstreamOf(NodeID)
LineageService.downstreamOf(NodeID)

You can then recursively follow the in edges to traverse the upstream graph consisting of job-to-dataset relationships:

{
  .
  .
  "inEdges": [{
    "origin": "job:{namespace}:{job}",
    "destination": "dataset:{namespace}:{dataset}"
  }],
  "outEdges": [{
    "origin": "job:{namespace}:{job}",
    "destination": "dataset:{namespace}:{dataset}"
  }]
}

For column-level lineage, the in / out node edges in the upstream lineage graph would contain both job and dataset node IDs, though only dataset nodes would be present. This means, the in / out edges would still be a job-to-dataset relationships, but now you wouldn't be able to recursively follow the in edges as before given that the job nodes in the returned upstream graph aren't present; and though the dataset node contains column-level lineage metadata via columnLineage, the in/out edges of the node doesn't feel consistent.

By consistent, I mean that backend can assist in better representing the dataset-to-dataset relationship (or rather dataset-column-to-dataset-column relationship) on a given dataset for a particular column by defining the following node ID:

dataset:{namespace}:{dataset}#{field}

Note: As an alternative, we can use datasetField:{namespace}:{dataset}:{field}.

For example, with the node ID defined, an upstream lineage call would now be:

GET /lineage/upstream?nodeId=dataset:my-namespace:my-dataset#my-field

{
  .
  .
  "inEdges": [{
    "origin": "dataset:my-namespace:my-dataset#my-field",
    "destination": "dataset:my-namespace:my-other-dataset#my-other-field"
  }],
  "outEdges": [{
    "origin": "dataset:my-namespace:my-other-dataset#my-other-field",
    "destination": "dataset:my-namespace:some-other-dataset#some-other-field"
  }]
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm proposing a different endpoint fot /column-lineage because the payload would be different, containing only datasets. I was considering that the columnLineage facet was already providing edges and that the inEdges and outEdges fields of the lineage graph became unnecessary.

To me /upstream or /downstream is not an endpoint as they are more of a filter on the lineage than a different result.

Copy link
Member

@wslulciuc wslulciuc Aug 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was considering that the columnLineage facet was already providing edges and that the inEdges and outEdges fields of the lineage graph became unnecessary

I would then change the payload from a graph consisting of nodes (with in/out edges that aren't really relevant), to more an array of datasets objects that don't have in / out edges as much of the metadata that is relevant for lineage, wouldn't apply here.

My thinking is this: the lineage call returns a set of nodes, but doesn't specify if they all have to be datasets, or all have to be jobs. It's generic in that way. What matters are the nodeIDs and that the origin and destination point to a node in the return node set. Adding a new node type datasetField would fulfill the API contract. But, like you said, whether a query is upstream or downstream can be a backend implementation that can be based on the node type:

GET /lineage?nodeId=datasetField:{namespace}:{dataset}:{field}

Basically, I think column-level lineage should still be represented via a graph data structure. If we will only be using columnLineage to establish relationships between datasets, then it's less of a graph and more of a list of objects that are assumed to be connected.

Copy link
Collaborator

@pawel-big-lebowski pawel-big-lebowski Aug 30, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohh man, it's great discussion although it took me 10 times reading to get to know what are you talking about.

I tried to include the initial idea of Julien mixed with the feedback of Willy.

Some clue design decisions:

  • existing lineage endpoint will be enriched with column lineage as-is (column lineage facet included within dataset)
  • new column-lineage will return column lineage graph with edges between dataset fields. It will reuse existing Graph data structure with new new dataset_field node type.
  • Jobs won't be included in the graph, as a single job may have tons of edges to the fields. Edges will connect dataset_field nodes directly.

Other:

  • graph depth can be controlled by url parameter,
  • downstream lineage will be turned off by default and can be turned on when requested,
  • depth of a returned graph can be controlled by URL parameter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @pawel-big-lebowski This looks good to me. I left a minor comment bellow

```
`column` is a ne parameter that must be a column in the schema of the provided dataset `nodeId`.

The logic is layered on the existing lineage endpoint, filtering down to the datasets that contribute to that column.
It only returns dataset nodes.

```diff
{
graph: [
{
"id": "dataset:db1:table2",
"type": "DATASET",
data: {
namespace: "DB1",
name: "table2",
> columnLineage: {
> "a": {
> inputFields: [
> {namespace: "DB1", name: "table1, "field": "a"}
> ],
> transformationDescription: "identical",
> transformationType: "IDENTITY"
> },
> "b": ... other output fields
> }
},
...
}
]
}
```

### Point in time upstream lineage
return historical upstream lineage from a given Dataset version.
This adds the version element to the nodeId in both the existing `/api/v1/lineage` and newly proposed `/api/v1/column-lineage` endpoint
```
GET /lineage?nodeId=dataset:food_delivery:public.delivery_7_days:{version}
GET /column-lineage?nodeId=dataset:food_delivery:public.delivery_7_days:{version}&column=a
pawel-big-lebowski marked this conversation as resolved.
Show resolved Hide resolved
```
This returns only upstream lineage in this current proposal. This is because upstream lineage is well defined to a specific version while downstream lineage is not. The data payload would add a version field.
```diff
{
graph: [
{
< "id": "dataset:db1:table2",
> "id": "dataset:db1:table2#{VERSION UUID}",
"type": "DATASET",
data: {
namespace: "DB1",
name: "table2",
> version: "{VERSION UUID}"
...
}
}
]
}
```

## Implementation

### columne lineage facet in lineage
Adding the columnLineage facet requires a formatting of existing facet data.
### column lineage endpoint
The `/column-lineage` endpoint leverages the `/lineage` endpoint and then filters down the payload to return the expected result.
### point-in-time upstream lineage
The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version.
Dataset version -> run that produced it -> consumed Dataset Versions.

## Next Steps

Review of this proposal and production of detailed design for the implementation, in particular for the point in time lineage which might affect the dabtabase schema.