[AMORO-1503][mixed-hive] Add INFO logs for partition operations#4200
Open
lintingbin wants to merge 1 commit intoapache:masterfrom
Open
[AMORO-1503][mixed-hive] Add INFO logs for partition operations#4200lintingbin wants to merge 1 commit intoapache:masterfrom
lintingbin wants to merge 1 commit intoapache:masterfrom
Conversation
Add INFO/WARN logging on the Mixed Hive Table partition operation paths (create / drop / alter location) so that issues like missing or mis-located partitions can be diagnosed from the logs. Logging added at two layers: * HivePartitionUtil: per-partition LOG.info on create, drop, and update-location paths, plus matching LOG.warn on failures. The alterPartition log now reports both old and new locations. * UpdateHiveFiles / ReplaceHivePartitions: per-batch LOG.info for the drop / create / alter sets emitted by commitPartitionedTable, with a count and a sampled list of partitions to avoid log explosion on large commits, plus matching LOG.warn on commit failures. Each log includes the table identifier, partition values, location (old -> new for alter), and transaction id where available.
Contributor
Author
|
Hi @zhoujinsong, could you help take a look at this PR when you have time? Thanks! Quick note on CI: the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1503.
Adds INFO/WARN logs on the Mixed Hive Table partition operation paths
(create / drop / alter location). These operations are infrequent, so
logging them at INFO level makes after-the-fact diagnosis of missing
or mis-located partitions much easier with negligible runtime cost.
Logs are added at two layers:
HivePartitionUtil— per-partitionLOG.infooncreatePartitionIfAbsent,dropPartition, andupdatePartitionLocation, plus matchingLOG.warnonthe failure paths. The existing
alterPartitionlog now reports both oldand new locations.
UpdateHiveFiles/ReplaceHivePartitions— per-batchLOG.infofor thedrop / create / alter sets emitted by
commitPartitionedTable, with acount and a sampled list of partitions (capped at 5, with
...for therest) to avoid log explosion on large commits, plus matching
LOG.warnon commit failures.
Each log line carries the table identifier, partition values, location
(old → new on alter), and transaction id where available, so a single
log record is enough to identify the partition and trace it back to a
commit.
Sample logs
Per-partition (single-partition path,
HivePartitionUtil):Per-batch (transaction commit path,
UpdateHiveFiles):Failure path:
Tests
No new unit tests; existing tests in the module continue to pass:
mvn spotless:checkandmvn checkstyle:checkare clean for thetouched module.