NIFI-16130 - Fix ReplayLastEventEndpointMerger to count per-node replays correctly#11450
Merged
Conversation
…ays correctly Provenance event IDs are local counters per node and reset to zero on each new NiFi installation. In a multi-node cluster, two different nodes can legitimately assign the same numeric event ID to different physical provenance events. ReplayLastEventEndpointMerger used a HashSet<Long> to accumulate event IDs from all nodes. When two nodes each replay their last event and happen to have the same local event ID (which occurs whenever both nodes have processed an equal number of events), the set deduplicates them and reports eventsReplayed.size() as 1 instead of 2. This caused ClusteredReplayProvenanceIT to fail with "expected: <2> but was: <1>". Change the accumulator to ArrayList<Long> so each node's replay is counted independently, regardless of colliding local event IDs. Add ReplayLastEventEndpointMergerTest to cover both the colliding-ID case (regression test for this fix) and the distinct-ID and partial-failure cases.
exceptionfactory
requested changes
Jul 21, 2026
exceptionfactory
left a comment
Contributor
There was a problem hiding this comment.
Thanks @pvillard31, the functional change to List<Long> for Provenance IDs looks good. For the new test, the public modifiers can be removed, and it looks like static values, as well as the NodeIdentifier objects themselves, could be declared once as static final members.
exceptionfactory
approved these changes
Jul 22, 2026
exceptionfactory
left a comment
Contributor
There was a problem hiding this comment.
Thanks @pvillard31, the latest version looks good.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NIFI-16130 - Fix ReplayLastEventEndpointMerger to count per-node replays correctly
Provenance event IDs are local counters per node, starting at 0 and incrementing independently. In a multi-node cluster two different nodes can legitimately have the same numeric event ID for entirely different physical provenance events.
ReplayLastEventEndpointMergerwas using aHashSet<Long>to accumulate event IDs returned by each node after a "replay last event" request. Because both nodes in a 2-node cluster where each has processed the same number of events will return the same local event ID, theHashSetsilently deduplicated them and reportedeventsReplayed.size() == 1instead of2. This causedClusteredReplayProvenanceIT.testReplayLastEvent(withReplayEventNodes.ALL) to fail withexpected: <2> but was: <1>.Change: Replace
HashSet<Long>withArrayList<Long>for the event ID accumulator so each node's replay is counted independently regardless of matching local event IDs.Also adds:
ReplayLastEventEndpointMergerTest— there was no unit test for this merger class. The test covers the colliding-ID case (regression test for this fix), the distinct-ID case, and the partial-failure case.Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation