New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-34777][UI] Fix StagePage input size/records not show when records greater than zero #35498
Conversation
…ecords greater than zero
Can one of the admins verify this patch? |
dataToShow.showInputData = responseBody.inputBytes > 0 || responseBody.inputRecords > 0 ; | ||
dataToShow.showOutputData = responseBody.outputBytes > 0 || responseBody.outputRecords > 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How to reproduce this issue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be easily reproduced by any spark structured streaming job reading from Kafka source.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wangyum Any more comments for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding responseBody.inputRecords > 0
seems correct. But why do we need to add responseBody.outputRecords > 0
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if sending message into Kafka, output will have similar bug.
cc @HeartSaVioR |
@wangyum @HeartSaVioR Could you help take a look? |
Merged to master. |
What changes were proposed in this pull request?
Determine whether show input/output size and records based on either has value, rather than only size.
Before:
After:
Why are the changes needed?
Stage page UI not show input/output size and records even when records greater than zero. This is common when data source only have records metrics updated.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manually