NIFI-15938 Improve TailFile startup and state recovery#11245
NIFI-15938 Improve TailFile startup and state recovery#11245ing-mattioni wants to merge 1 commit into
Conversation
exceptionfactory
left a comment
exceptionfactory
left a comment
There was a problem hiding this comment.
Thanks for proposing these improvements @ing-mattioni.
On initial review, I raised some concerns about the introduction of several try-catch blocks. These seem unnecessary without further explanation.
Some of the other changes also raise implementation questions, so overall, some clarification and refinement is needed before this can go forward.
|
|
||
| private void initStates(final List<String> filesToTail, final Map<String, String> statesMap, final boolean isCleared) { | ||
| int fileIndex = 0; | ||
| final Set<String> filesToTailSet = new HashSet<>(filesToTail); |
There was a problem hiding this comment.
| final Set<String> filesToTailSet = new HashSet<>(filesToTail); | |
| final Set<String> uniqueFilesToTail = new HashSet<>(filesToTail); |
| } catch (final RuntimeException e) { | ||
| getLogger().warn("Ignoring malformed TailFile state key {}", key, e); | ||
| continue; | ||
| } |
There was a problem hiding this comment.
Is there a particular reason for introducing this try-catch? The Processor should control the state content, avoiding potential incorrect formatting.
| states.put(value, new TailFileObject(index, statesMap, preAllocatedBufferSize)); | ||
| } catch (final RuntimeException e) { | ||
| getLogger().warn("Ignoring malformed TailFile state for {}", value, e); | ||
| } |
There was a problem hiding this comment.
This catch also seems too broad and not necessary.
|
|
||
| private void initStates(final List<String> filesToTail, final Map<String, String> statesMap, final boolean isCleared) { | ||
| int fileIndex = 0; | ||
| final Set<String> filesToTailSet = new HashSet<>(filesToTail); |
There was a problem hiding this comment.
What is the reason for creating this wrapping Set?
2 participants
Summary
NIFI-15938
This pull request updates
TailFileto improve multi-file startup behavior and make persisted state recovery more robust.The changes focus on startup performance, rollover state restoration, malformed persisted state handling, read-loop buffer behavior, and buffer-size validation.
Changes
HashSetmembershiptailingPostRolloverstate correctlyKey fixes, more technical description:
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkModule-level verification performed:
.\mvnw.cmd --% -pl :nifi-standard-processors -am -Dtest=TestTailFile,TestTailFileSimpleScenarios,TestTailFileGeneratedScenarios -Dsurefire.failIfNoSpecifiedTests=false testAdditional verification performed:
.\mvnw.cmd --% -pl :nifi-standard-processors -Pcontrib-check validateLicensing
LICENSEandNOTICEfilesNo new dependencies were added.
Documentation
No user-facing documentation files were changed.