chore(utilities): add SQS backlog / in-flight visibility to S3 events source - #19333
Merged
voonhous merged 2 commits intoJul 22, 2026
Merged
Conversation
Adds a lifecycle log line to S3EventsSource.onCommit recording how many SQS messages are being deleted at commit and the checkpoint, improving operational visibility into the SQS-backed S3 events ingestion path. Placeholder / first increment of a broader observability change tracked for an Onehouse internal hotfix; the full receive-loop / in-flight logging will be upstreamed here subsequently.
hudi-agent
reviewed
Jul 21, 2026
hudi-agent
left a comment
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
Thanks for working on this! This PR adds a single informational log line to S3EventsSource.onCommit recording the number of processed SQS messages being deleted and the checkpoint. It's a logging-only change with no behavioral impact. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. One small nit on the log message; otherwise the change is clean.
cc @yihua
voonhous
changed the title
SLF4J emits the logger name already (every log4j2 pattern in the repo uses %c), so the "S3EventsSource.onCommit:" prefix duplicated it. Also names the queue as SQS in the message.
voonhous
enabled auto-merge (squash)
July 22, 2026 08:15
wombatu-kun
approved these changes
Jul 22, 2026
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
The SQS-backed S3 events ingestion path gives no signal about message cleanup. When
S3EventsSource.onCommitruns, it deletes the processed messages from the queue and clears its tracker, but nothing is logged, so there is no way to tell from the logs how many messages were reclaimed at a given checkpoint or whether cleanup ran at all.Summary and Changelog
Operators get a per-commit log line showing how many SQS messages were deleted and at which checkpoint, making the ingestion path's queue cleanup observable. This is the first increment of a broader observability change (receive-loop plan/summary and in-flight queue counters) that will follow.
S3EventsSource: added an SLF4J logger.S3EventsSource.onCommit: log the processed-message count and the checkpoint before deleting the messages from the queue.No existing code paths were modified and no code was copied.
Impact
Logging only. No public API, config, or behavior change. One additional INFO line per commit on the S3 events source path.
Risk Level
none
Documentation Update
none
Contributor's checklist