Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task] Default transform metrics like throughput, latency & watermark progress for Beam Samza Runner #26456

Open
1 of 15 tasks
Sanil15 opened this issue Apr 27, 2023 · 0 comments
Open
1 of 15 tasks

[Task] Default transform metrics like throughput, latency & watermark progress for Beam Samza Runner #26456

Sanil15 opened this issue Apr 27, 2023 · 0 comments
Labels
awaiting triage P3 samza task

Comments

@Sanil15
Copy link
Contributor

Sanil15 commented Apr 27, 2023

What needs to happen?

Summary

  • Added Support for generating throughput, watermark progress & latency metric emission
  • A MetricOp is attached to each input & output
  • Throughput Metrics are emitted for input and output throughput
  • Watermark Progress Metrics are emitting at the during-process watermark of output
  • Latency Metrics are emitted by collecting avg arrival time at input & output per watermark for each transform using formula Latency = Avg
  • Arrival Time of Elements at Output - Avg Arrival Time of Elements at Input
  • For GroupByKey translator based operators we aggregate avg arrivial time per window id and then only emit the latency using bove formula for elements under the same window

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@Sanil15 Sanil15 changed the title Default transform metrics like throughput, latency & watermark progress for Beam Samza Runner [Task] Default transform metrics like throughput, latency & watermark progress for Beam Samza Runner Apr 27, 2023
@Sanil15 Sanil15 mentioned this issue Apr 27, 2023
Merged
3 tasks
xinyuiscool pushed a commit that referenced this issue May 9, 2023
@Sanil15
#26456 : Basic Transform metric like throughput, watermark progress &…
0ac9fc1 
… latency support for non data shuffle transforms for Samza Runner (#26437)
@Sanil15 Sanil15 mentioned this issue May 10, 2023
Merged
3 tasks
Sanil15 added a commit to Sanil15/beam that referenced this issue May 22, 2023
@Sanil15
apache#26456 : Basic Transform metric like throughput, watermark prog…
272cb62 
…ress & latency support for non data shuffle transforms for Samza Runner (apache#26437)
Sanil15 added a commit to linkedin/beam that referenced this issue May 22, 2023
@Sanil15
apache#26456 : Basic Transform metric like throughput, watermark prog…
325358c 
…ress & latency support for non data shuffle transforms for Samza Runner (apache#26437)
Dippatel98 pushed a commit to Dippatel98/beam that referenced this issue Jun 5, 2023
@Sanil15 @Dippatel98
apache#26456 : Basic Transform metric like throughput, watermark prog…
0717595 
…ress & latency support for non data shuffle transforms for Samza Runner (apache#26437)
bullet03 pushed a commit to akvelon/beam that referenced this issue Aug 11, 2023
@Sanil15 @bullet03
apache#26456 : Basic Transform metric like throughput, watermark prog…
6a7786c 
…ress & latency support for non data shuffle transforms for Samza Runner (apache#26437)
cushon pushed a commit to cushon/beam that referenced this issue May 24, 2024
@Sanil15 @cushon
apache#26456 : Basic Transform metric like throughput, watermark prog…
5e8d03f 
…ress & latency support for non data shuffle transforms for Samza Runner (apache#26437)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting triage P3 samza task
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Sanil15