chore: add timing logs for file index partition and file listing#18417
chore: add timing logs for file index partition and file listing#18417suryaprasanna wants to merge 1 commit intoapache:masterfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18417 +/- ##
=========================================
Coverage 68.21% 68.22%
+ Complexity 27709 27698 -11
=========================================
Files 2440 2440
Lines 134249 134265 +16
Branches 16179 16181 +2
=========================================
+ Hits 91578 91599 +21
+ Misses 35565 35560 -5
Partials 7106 7106
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
voonhous
left a comment
There was a problem hiding this comment.
Main concern here is that the logs are too noisy.
- Are these logs only used for debugging?
- Should we use
log.debug(...)for routine timing logs instead oflog.info(...)? - We can consider logging at info level when something is slow, i.e.
if (elapsed > threshold) log.info(...)or unexpected (e.g., large cache miss ratio)
| .collect(Collectors.toList()) | ||
| )); | ||
| } finally { | ||
| log.info("On {} with query instant as {}, it took {}ms to filter {} files into file slices across {} partitions", |
There was a problem hiding this comment.
My main concern here is that the logs generated here will be too noisy.
This fires on every single query, even if it took 1ms. For a busy Spark application hitting multiple Hudi tables, this alone generates one info line per query.
| log.info("On {}, out of {} partition paths, {} are missing from cache. Loading them.", | ||
| metaClient.getTableConfig().getTableName(), partitionPaths.size(), missingPartitionPaths.size()); |
There was a problem hiding this comment.
uard the "missing from cache" log with if (missingPartitionPaths.size() > 0) there is no point logging when everything is cached.
Describe the issue this Pull Request addresses
File index operations currently do not provide enough visibility into how much time is spent listing partitions, fetching files from metadata, and filtering files into file slices. This makes it harder to debug slow queries and identify where file-index time is being spent.
Summary and Changelog
Adds timing and context logs in
BaseHoodieTableFileIndexfor key file-index stages.getAllFilesInPartitionsfilterFileswhile building file slicesImpact
No public API or user-facing behavior change. This improves observability for file-index execution and helps diagnose performance issues in partition and file listing paths.
Risk Level
low
This change only adds logging and slightly updates an internal exception message with table context. No functional behavior is intended to change.
Documentation Update
none
Contributor's checklist