[HUDI-7011] a metric to indicate whether rollback has occurred in final compaction state#9956
Conversation
|
|
||
| public enum CompactionState{ | ||
| SUCCESS(0),FAILED(1); | ||
| int state; |
There was a problem hiding this comment.
The metric only has 0 and 1 as values, does it make sense to represent it as a metric?
There was a problem hiding this comment.
en, this metric works like a state signal used to alert people to know the final compaction state this time. when the mor table happens many rollback, may user can manual operate the hudi mor table or flink job. the pending count | delay cost may cant't represent the final result.
|
|
||
| /** | ||
| * flag indicates whether rollback has occurred in final compaction state | ||
| */ |
There was a problem hiding this comment.
flag indicates whether rollback has occurred in final compaction state ->
Flag saying whether the compaction is completed or been rolled back.
| * flag indicates whether rollback has occurred in final compaction state | ||
| */ | ||
| private long compactionFailedState; | ||
|
|
| } | ||
|
|
||
| public void setCompactionFailedState(CompactionState compactionState){ | ||
| this.compactionFailedState = compactionState.state; |
There was a problem hiding this comment.
Mayne we have 2 methods here:
markCompactionCompleted and markCompactionRolledBack.
|
@danny0405 hello, i have fixed the change, would you please review one more |
| this.compactionFailedState = compactionState.state; | ||
| public void markCompactionCompleted(CompactionState compactionState) { | ||
| this.compactionStateSignal = compactionState.state; | ||
| } |
There was a problem hiding this comment.
There is no need to pass around CompactionState, just set up the constant 1 or 0 internally, the CompactionState can be eliminated.
|
@danny0405 OK, it get you |
|
@hudi-bot run azure |
LXin96
force-pushed
the
hudi_compaction_metric_failed_count
branch
from
ac92a41 to
bd6cc40
Compare
…al compaction state
LXin96
force-pushed
the
hudi_compaction_metric_failed_count
branch
from
bd6cc40 to
184321e
Compare
…al compaction state
|
@danny0405 I have modified, please check |
3 participants
currently, when flink job start async compaction on a mor table, the metrics in org.apache.hudi.metrics.FlinkCompactionMetrics
will update including pendingCompactionCount,compactionDelay,compactionCost. However, when a compaction failed need a metric to
help user to know specific instant whether the final compaction has occured rollback.
so attemp to add a metric named compactionFailedState in org.apache.hudi.sink.compact.CompactionCommitSink to record the instance
happend rollback, which also means the current compaction failed in current time
Change Logs
add a metric named compactionFailedState in org.apache.hudi.sink.compact.CompactionCommitSink to record the instance
happend rollback, which also means the current compaction failed in current time
Impact
no impact
Risk level (write none, low medium or high below)
low medium
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist