Skip to content

SAMZA-2470: Add Samza job lineage interfaces#1294

Closed
alnzng wants to merge 3 commits into
apache:masterfrom
alnzng:SAMZA-2470
Closed

SAMZA-2470: Add Samza job lineage interfaces#1294
alnzng wants to merge 3 commits into
apache:masterfrom
alnzng:SAMZA-2470

Conversation

@alnzng

@alnzng alnzng commented Feb 26, 2020

Copy link
Copy Markdown
Contributor

Background

Add interfaces to provide the ability to allow the users to define their own job lineage data model and store in the specified outside systems.

Changes

  • Add LineageFactory class to allow users to build their own job lineage data model
  • Add LineageReporter and LineageReporterFactory classes to allow users to have their own way to store lineage information in outside systems.
  • Emit job lineage data in JobModelManager.readJobModel() function, because this function provides the ability to dynamically load Kafka topics matched specified regex pattern.

Test

  • All unit tests and integration tests are passed

Upgrade Instructions

None

Usage Instructions

The users need to config lineage.factory and lineage.reporter.factory to enable the lineage function.

Signed-off-by: Alan Zhang <shuai.xyz@gmail.com>
Signed-off-by: Alan Zhang <shuai.xyz@gmail.com>
Signed-off-by: Alan Zhang <shuai.xyz@gmail.com>
@alnzng

alnzng commented Feb 26, 2020

Copy link
Copy Markdown
Contributor Author

@rmatharu can you please help review this PR?

@alnzng

alnzng commented Feb 28, 2020

Copy link
Copy Markdown
Contributor Author

talked with @rmatharu, closed it for now.

@alnzng alnzng closed this Feb 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant