Runless events - consume job event #2661

pawel-big-lebowski · 2023-10-26T13:42:11Z

Problem

As a followup of #2641, this PR introduces support for JobEvent. PR shall be merged after #2641.

Solution

Schema changes: job_version_uuid column is added to job_facets table. This requires db migration to backfill existing job_facets entries.
JobDao as findJobByName method should work without join to runs table.
Add spec_event_type to lineage_events table to indicate which type of event is stored.

Limitations:

listLineage endpoints filters RunEvent only and does not support introduced event types. This can be implemented within other PR and issue.

Note: All database schema changes require discussion. Please link the issue for context.

One-line summary:

Checklist

You've signed-off your work
Your changes are accompanied by tests (if relevant)
Your change contains a small diff and is self-contained
You've updated any relevant documentation (if relevant)
You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
You've included a header in any source code files (if relevant)

codecov · 2023-10-31T13:00:55Z

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (3a26e50) 83.76% compared to head (0a3f98a) 84.05%.

Files	Patch %	Lines
api/src/main/java/marquez/db/OpenLineageDao.java	92.10%	4 Missing and 2 partials ⚠️
.../migrations/V66_3_JobFacetsBackfillJobVersion.java	81.81%	2 Missing ⚠️
...src/main/java/marquez/api/OpenLineageResource.java	75.00%	0 Missing and 1 partial ⚠️
api/src/main/java/marquez/db/models/ModelDaos.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2661      +/-   ##
============================================
+ Coverage     83.76%   84.05%   +0.28%     
- Complexity     1338     1379      +41     
============================================
  Files           247      248       +1     
  Lines          6112     6297     +185     
  Branches        281      286       +5     
============================================
+ Hits           5120     5293     +173     
- Misses          843      851       +8     
- Partials        149      153       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

CHANGELOG.md

wslulciuc · 2023-11-06T07:37:53Z

api/src/main/java/marquez/db/JobDao.java

-      ) e
-      GROUP BY e.run_uuid
-    ) f ON f.run_uuid=jv.latest_run_uuid
+    LEFT OUTER JOIN job_versions_facets f ON j.current_version_uuid = f.job_version_uuid


wslulciuc · 2023-11-06T07:48:51Z

api/src/main/java/marquez/db/JobVersionDao.java

+   * @param jobRow The job.
+   * @return A {@link BagOfJobVersionInfo} object.
+   */
+  default BagOfJobVersionInfo upsertRunlessJobVersion(


Minor: We can consider factoring out common code in JobVersionDao.upsertJobVersionOnRunTransition() and upsertRunlessJobVersion() to avoid duplication. But, not a major concern as we are aware we'll need to revisit some of this code later.

Yes, we can do later further refactor with this section.

api/src/main/resources/marquez/db/migration/V65__dataset_facets_lineage_event_type_nullable.sql

api/src/main/resources/marquez/db/migration/V66.1__job_facets_changes.sql

wslulciuc · 2023-11-06T08:08:31Z

api/src/main/resources/marquez/db/migration/V66.1__job_facets_changes.sql

+CREATE INDEX job_facets_job_version_uuid ON job_facets (job_version_uuid);
+
+ALTER TABLE lineage_events ADD COLUMN spec_event_type VARCHAR(64);
+UPDATE lineage_events SET spec_event_type = 'RunEvent';


do we want to set all events to RunEvent? Also, I'd define the _event_type enum as RUN_EVENT, DATASET_EVENT , or JOB_EVENT.

I switched to enum and set a default value instead of running update table.

netlify · 2023-11-06T09:52:14Z

✅ Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
🔨 Latest commit	`0a3f98a`
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/65549cb91e31ca0008c63c38

wslulciuc · 2023-11-15T08:22:26Z

api/src/main/resources/marquez/db/migration/V66.2__alter_lineage_events_add_event_type.sql

@@ -0,0 +1,4 @@
+CREATE TYPE EVENT_TYPE AS ENUM ('RUN_EVENT', 'DATASET_EVENT', 'JOB_EVENT');


Minor: We standardized on not defining enums types in the DB layer but rather enforce them in the application layer. This way, it avoids a DB migration every time we (might) add a new event type.

tl;dr, I'd just define _event_type as a string

added extra commit for that: 0a3f98a

wslulciuc · 2023-11-15T08:25:46Z

api/src/main/java/marquez/db/OpenLineageDao.java

+    DATASET_EVENT,
+    JOB_EVENT;
+  }
+
  @SqlUpdate(
      "INSERT INTO lineage_events ("
          + "event_type, "


Minor: We can consider defining a _run_state column and eventually dropping the event_type. That is, we can consider columns prefixed with _ to be "remappings" of OL properties to Marquez.

wslulciuc

Minor comments / suggests, otherwise 💯 💯 🥇

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>

boring-cyborg bot added the api API layer changes label Oct 26, 2023

pawel-big-lebowski force-pushed the static/job-event branch 3 times, most recently from 5158fc2 to bdcfec7 Compare October 31, 2023 12:48

pawel-big-lebowski force-pushed the static/job-event branch 4 times, most recently from 32deb1a to 151ed6b Compare November 1, 2023 11:09

boring-cyborg bot added the docs label Nov 1, 2023

pawel-big-lebowski marked this pull request as ready for review November 1, 2023 11:32

pawel-big-lebowski requested a review from wslulciuc November 1, 2023 11:32

pawel-big-lebowski marked this pull request as draft November 1, 2023 11:54

pawel-big-lebowski marked this pull request as ready for review November 1, 2023 12:30

Base automatically changed from static/dataset-event to main November 6, 2023 07:16

wslulciuc reviewed Nov 6, 2023

View reviewed changes

pawel-big-lebowski force-pushed the static/job-event branch from 84f4670 to de98803 Compare November 6, 2023 09:52

pawel-big-lebowski requested a review from wslulciuc November 6, 2023 13:16

wslulciuc reviewed Nov 15, 2023

View reviewed changes

wslulciuc approved these changes Nov 15, 2023

View reviewed changes

pawel-big-lebowski added 5 commits November 15, 2023 11:16

Runless events - consume job event

04829f1

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>

Runless events - fix listLineage API

2fef646

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>

fix rebase

edce54b

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>

modify event type column

80e8bf7

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>

make _event_type varchar(64) instead of enum

0a3f98a

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>

pawel-big-lebowski force-pushed the static/job-event branch from b7e40f8 to 0a3f98a Compare November 15, 2023 10:25

pawel-big-lebowski merged commit 60d7d90 into main Nov 16, 2023
16 checks passed

pawel-big-lebowski deleted the static/job-event branch November 16, 2023 06:57

wslulciuc mentioned this pull request Nov 28, 2023

Update OpenLineage implementation to support metadata outside of runs #1930

Closed

wslulciuc added this to the 0.43.0 milestone Dec 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runless events - consume job event #2661

Runless events - consume job event #2661

pawel-big-lebowski commented Oct 26, 2023 •

edited

codecov bot commented Oct 31, 2023 •

edited

wslulciuc Nov 6, 2023

wslulciuc Nov 6, 2023

pawel-big-lebowski Nov 6, 2023

wslulciuc Nov 6, 2023

pawel-big-lebowski Nov 6, 2023

netlify bot commented Nov 6, 2023 •

edited

wslulciuc Nov 15, 2023 •

edited

pawel-big-lebowski Nov 15, 2023

wslulciuc Nov 15, 2023 •

edited

wslulciuc left a comment

		@@ -0,0 +1,4 @@
		CREATE TYPE EVENT_TYPE AS ENUM ('RUN_EVENT', 'DATASET_EVENT', 'JOB_EVENT');

Runless events - consume job event #2661

Runless events - consume job event #2661

Conversation

pawel-big-lebowski commented Oct 26, 2023 • edited

Problem

Solution

Checklist

codecov bot commented Oct 31, 2023 • edited

Codecov Report

wslulciuc Nov 6, 2023

Choose a reason for hiding this comment

wslulciuc Nov 6, 2023

Choose a reason for hiding this comment

pawel-big-lebowski Nov 6, 2023

Choose a reason for hiding this comment

wslulciuc Nov 6, 2023

Choose a reason for hiding this comment

pawel-big-lebowski Nov 6, 2023

Choose a reason for hiding this comment

netlify bot commented Nov 6, 2023 • edited

✅ Deploy Preview for peppy-sprite-186812 canceled.

wslulciuc Nov 15, 2023 • edited

Choose a reason for hiding this comment

pawel-big-lebowski Nov 15, 2023

Choose a reason for hiding this comment

wslulciuc Nov 15, 2023 • edited

Choose a reason for hiding this comment

wslulciuc left a comment

Choose a reason for hiding this comment

pawel-big-lebowski commented Oct 26, 2023 •

edited

codecov bot commented Oct 31, 2023 •

edited

netlify bot commented Nov 6, 2023 •

edited

wslulciuc Nov 15, 2023 •

edited

wslulciuc Nov 15, 2023 •

edited