Skip to content

NIFI-16297 Prevent stale instance ID metrics after cluster join - #11625

Merged
pvillard31 merged 1 commit into
apache:mainfrom
abij:NIFI-16297
Sep 5, 2026
Merged

NIFI-16297 Prevent stale instance ID metrics after cluster join#11625
pvillard31 merged 1 commit into
apache:mainfrom
abij:NIFI-16297

Conversation

@abij

@abij abij commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

NIFI-16297 - Prometheus jvm/cluster/connection metrics permanently retain a stale pre-cluster-join instance ID series

A clustered NiFi node's Prometheus instance label starts out as a temporary, per-process value and switches to the node's persisted cluster UUID once it finishes its initial cluster-join handshake. Because jvmMetricsRegistry, connectionAnalyticsMetricsRegistry, and clusterMetricsRegistry were long-lived singleton fields on StandardNiFiServiceFacade that were never cleared or recreated, any scrape of /nifi-api/flow/metrics/prometheus landing in the (typically brief, but non-zero) window before that transition would permanently register nifi_jvm_*/connection-analytics/cluster_* metrics under the temporary instance ID — a frozen, stale series returned on every subsequent scrape for the life of the process, alongside the correct, continuously-updated series under the real node UUID.

This directly re-opens the symptom described by NIFI-14014 ("Metrics endpoints may return data for 'dead' instance IDs"). That fix only changed which identifier is preferred once available (node.getId() over controllerFacade.getInstanceId()); it didn't stop a scrape landing before the transition from permanently registering data under the temporary identifier, because it never addressed the actual root cause: these registries' never-cleared, singleton lifecycle. This is the same root cause behind NIFI-8272, NIFI-11899, and NIFI-16296.

What this PR does

Extends the pattern already established by NIFI-11899 (which converted NiFiMetricsRegistry and BulletinMetricsRegistry from singleton fields to fresh, locally-scoped instances created on every invocation of populateFlowMetrics()) to the three registries that were left out of that fix: JvmMetricsRegistry, ConnectionAnalyticsMetricsRegistry, and ClusterMetricsRegistry. All Prometheus flow metrics registries are now fully refreshed on every scrape, so no registry in this method can retain a stale label combination across process restarts (or any other future source of label churn).

  • Removed the three permanent singleton fields from StandardNiFiServiceFacade
  • Added them as local variables inside populateFlowMetrics(), alongside the other "fully refreshed upon each invocation" registries

No behavior changes for a node that has already completed its cluster-join handshake — the fix only prevents a stale series from surviving past the moment the node's identity transitions.

Verification

Build

  • Build completed using ./mvnw clean install -P contrib-check (scoped to nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api and its dependencies via -pl ... -am; full reactor build not run locally, deferred to CI) — 0 Checkstyle violations, RAT license check passed, 642/642 tests passed, BUILD SUCCESS
  • JDK 21
  • JDK 25 — not available (not offered by internal software catalog); project's maven.compiler.release targets 21

Licensing

  • No new dependencies added
  • N/A — no LICENSE/NOTICE changes required

Documentation

  • N/A — no documentation changes in this PR

…istries on every scrape to prevent stale instance ID series
@abij abij changed the title NIFI-16297 Refresh JVM, connection analytics, and cluster metrics registries on every scrape NIFI-16297 Prevent stale instance ID metrics after cluster join Sep 4, 2026
@abij

abij commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

I think the Mac Java25 test: FlowSynchronizationIT>NiFiSystemIT.teardown:187->NiFiSystemIT.destroyFlow:277 » IO Timed out waiting for queue <UUID> to empty is unrelated, should I just re-ran the testsuite?

@pvillard31 pvillard31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change makes sense and follows the NIFI-11899 pattern. +1, merging, thanks!

@pvillard31
pvillard31 merged commit 23c3894 into apache:main Sep 5, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants