-
Notifications
You must be signed in to change notification settings - Fork 134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Fix incorrect spark metrics #324
Conversation
Codecov Report
@@ Coverage Diff @@
## master #324 +/- ##
============================================
+ Coverage 61.21% 61.32% +0.11%
- Complexity 1506 1526 +20
============================================
Files 185 186 +1
Lines 9360 9441 +81
Branches 908 924 +16
============================================
+ Hits 5730 5790 +60
- Misses 3325 3341 +16
- Partials 305 310 +5
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
...gration-test/spark-common/src/test/java/org/apache/uniffle/test/WriteAndReadMetricsTest.java
Outdated
Show resolved
Hide resolved
I'm confused that the |
Could you debug in your local machine? |
Fixed @jerqi PTAL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @zuston
Fix incorrect spark metrics 1. The corresponding shuffle-read records number and shuffle-write records number is not consistent in our internal cluster 2. Log wont show the correct fetch bytes, always return 0 like `22/11/15 13:54:53 INFO RssShuffleDataIterator: Fetch 0 bytes cost 30791 ms and 53 ms to serialize, 347 ms to decompress with unCompressionLength[274815736] ` No 1. UTs 2. Online spark3 jobs test Co-authored-by: Kaijie Chen <ckj@apache.org>
Fix incorrect spark metrics 1. The corresponding shuffle-read records number and shuffle-write records number is not consistent in our internal cluster 2. Log wont show the correct fetch bytes, always return 0 like `22/11/15 13:54:53 INFO RssShuffleDataIterator: Fetch 0 bytes cost 30791 ms and 53 ms to serialize, 347 ms to decompress with unCompressionLength[274815736] ` No 1. UTs 2. Online spark3 jobs test
import org.junit.jupiter.api.Test; | ||
import scala.collection.Seq; | ||
|
||
public class WriteAndReadMetricsTest extends SimpleTestBase { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test case seems flaky
https://github.com/apache/incubator-uniffle/actions/runs/3511234782/jobs/5881817090
What changes were proposed in this pull request?
Fix incorrect spark metrics
Why are the changes needed?
22/11/15 13:54:53 INFO RssShuffleDataIterator: Fetch 0 bytes cost 30791 ms and 53 ms to serialize, 347 ms to decompress with unCompressionLength[274815736]
Does this PR introduce any user-facing change?
No
How was this patch tested?