Skip to content

SAMZA-2706: Clean up specific handling of diagnostics-specific metrics reporter#1550

Merged
cameronlee314 merged 2 commits intoapache:masterfrom
cameronlee314:diagrefactor
Nov 8, 2021
Merged

SAMZA-2706: Clean up specific handling of diagnostics-specific metrics reporter#1550
cameronlee314 merged 2 commits intoapache:masterfrom
cameronlee314:diagrefactor

Conversation

@cameronlee314
Copy link
Contributor

Issues: Specific checks for the diagnosticsreporter metrics reporter were making the code for setting up diagnostics and the diagnosticsreporter metrics reporter a bit harder to work with. For the new StaticResourceJobCoordinator, it made object wiring harder for diagnostics-related components. #1021 initially had a reason to have special logic for creating the diagnosticsreporter metrics reporter, but #1369 made that special logic unnecessary.

Changes:

  1. Changed DiagnosticsUtil.buildDiagnosticsManager to only return a DiagnosticsManager instead of Pair<DiagnosticsManager, MetricsSnapshotReporter>.
  2. MetricsReporterLoader.getMetricsReporters no longer excludes the diagnosticsreporter metrics reporter. It just creates all configured reporters.
  3. Since MetricsReporterLoader.getMetricsReporters returns all reporters, any classes that use that method no longer need to inject the diagnosticsreporter explicitly.

Tests:
./gradlew build

API changes:

  1. In the YARN application master,diagnosticsreporter metrics reporter now is given a processorId of "ApplicationMaster" instead of "samza-container-ApplicationMaster".
  2. When using MetricsReporterLoader.getMetricsReporters, if diagnosticsreporter is in metrics.reporters, then diagnosticsreporter metrics reporter is always created.
  3. The diagnosticsreporter metrics reporter may still be created even if job.diagnostics.enabled is set to false. Note that Samza will only automatically create the stream for the diagnosticsreporter metrics reporter if diagnostics is enabled (since the DiagnosticsManager also uses that same stream), so if there is a case in which diagnosticsreporter metrics reporter is needed while diagnostics is disabled, then the stream for the reporter needs to be created through some other means.

@rmatharu-zz
Copy link
Contributor

LGTM,

I'll create a jira for writing an integ test that tests out the behavior of data being emitted into diag-stream.
Currently we rely on rel-test for this.

@cameronlee314 cameronlee314 merged commit 3054a86 into apache:master Nov 8, 2021
ehoner pushed a commit to ehoner/samza that referenced this pull request Apr 11, 2023
…s reporter (apache#1550)

API changes:
1. In the YARN application master,diagnosticsreporter metrics reporter now is given a processorId of "ApplicationMaster" instead of "samza-container-ApplicationMaster".
2. When using MetricsReporterLoader.getMetricsReporters, if diagnosticsreporter is in metrics.reporters, then diagnosticsreporter metrics reporter is always created.
3. The diagnosticsreporter metrics reporter may still be created even if job.diagnostics.enabled is set to false. Note that Samza will only automatically create the stream for the diagnosticsreporter metrics reporter if diagnostics is enabled (since the DiagnosticsManager also uses that same stream), so if there is a case in which diagnosticsreporter metrics reporter is needed while diagnostics is disabled, then the stream for the reporter needs to be created through some other means.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants