NIFI-16297 Prevent stale instance ID metrics after cluster join - #11625
Merged
Conversation
…istries on every scrape to prevent stale instance ID series
Contributor
Author
|
I think the Mac Java25 test: |
pvillard31
approved these changes
Sep 5, 2026
pvillard31
left a comment
Contributor
There was a problem hiding this comment.
Change makes sense and follows the NIFI-11899 pattern. +1, merging, thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NIFI-16297 - Prometheus jvm/cluster/connection metrics permanently retain a stale pre-cluster-join instance ID series
A clustered NiFi node's Prometheus
instancelabel starts out as a temporary, per-process value and switches to the node's persisted cluster UUID once it finishes its initial cluster-join handshake. BecausejvmMetricsRegistry,connectionAnalyticsMetricsRegistry, andclusterMetricsRegistrywere long-lived singleton fields onStandardNiFiServiceFacadethat were never cleared or recreated, any scrape of/nifi-api/flow/metrics/prometheuslanding in the (typically brief, but non-zero) window before that transition would permanently registernifi_jvm_*/connection-analytics/cluster_*metrics under the temporary instance ID — a frozen, stale series returned on every subsequent scrape for the life of the process, alongside the correct, continuously-updated series under the real node UUID.This directly re-opens the symptom described by NIFI-14014 ("Metrics endpoints may return data for 'dead' instance IDs"). That fix only changed which identifier is preferred once available (
node.getId()overcontrollerFacade.getInstanceId()); it didn't stop a scrape landing before the transition from permanently registering data under the temporary identifier, because it never addressed the actual root cause: these registries' never-cleared, singleton lifecycle. This is the same root cause behind NIFI-8272, NIFI-11899, and NIFI-16296.What this PR does
Extends the pattern already established by NIFI-11899 (which converted
NiFiMetricsRegistryandBulletinMetricsRegistryfrom singleton fields to fresh, locally-scoped instances created on every invocation ofpopulateFlowMetrics()) to the three registries that were left out of that fix:JvmMetricsRegistry,ConnectionAnalyticsMetricsRegistry, andClusterMetricsRegistry. All Prometheus flow metrics registries are now fully refreshed on every scrape, so no registry in this method can retain a stale label combination across process restarts (or any other future source of label churn).StandardNiFiServiceFacadepopulateFlowMetrics(), alongside the other "fully refreshed upon each invocation" registriesNo behavior changes for a node that has already completed its cluster-join handshake — the fix only prevents a stale series from surviving past the moment the node's identity transitions.
Verification
Build
./mvnw clean install -P contrib-check(scoped tonifi-framework-bundle/nifi-framework/nifi-web/nifi-web-apiand its dependencies via-pl ... -am; full reactor build not run locally, deferred to CI) — 0 Checkstyle violations, RAT license check passed, 642/642 tests passed, BUILD SUCCESSmaven.compiler.releasetargets 21Licensing
Documentation