Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[jvm-packages] fix spark tests on machines with many cores #4634

Merged
merged 1 commit into from Jul 7, 2019

Conversation

Projects
None yet
4 participants
@rongou
Copy link
Contributor

commented Jul 3, 2019

This does seem to work better on my desktop which has 12 cores. Should fix #3955.

@hcho3 @CodingCat

@hcho3 hcho3 changed the title fix spark tests on machines with many cores [jvm-packages] fix spark tests on machines with many cores Jul 3, 2019

@hcho3 hcho3 requested a review from CodingCat Jul 3, 2019

@hcho3

hcho3 approved these changes Jul 3, 2019

@thesuperzapper

This comment has been minimized.

Copy link
Contributor

commented Jul 4, 2019

This finally allowed me to build master locally! @CodingCat can we be 100% sure that we aren't hiding some bigger issue here?

@CodingCat
Copy link
Member

left a comment

I am ok to merge this for now, but the root cause of the issue is actually on the repartitioning algorithm in ranking part...that means, users are possible to experience “empty partition” issue in the real scenario...further, limiting core number to 4 is dangerous for new algorithm testing, for example, when I work on distributed hist, the test can pass the accuracy threshold with 4 or less cores even the syncing part had some issues

@CodingCat CodingCat merged commit 30204b5 into dmlc:master Jul 7, 2019

10 checks passed

Jenkins Linux: Build Stage built successfully
Details
Jenkins Linux: Formatting Check Stage built successfully
Details
Jenkins Linux: Get sources Stage built successfully
Details
Jenkins Linux: Test Stage built successfully
Details
Jenkins Win64: Build Stage built successfully
Details
Jenkins Win64: Get sources Stage built successfully
Details
Jenkins Win64: Test Stage built successfully
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

thesuperzapper added a commit to thesuperzapper/xgboost that referenced this pull request Jul 8, 2019

sperlingxx added a commit to alipay/ant-xgboost that referenced this pull request Jul 17, 2019

sperlingxx added a commit to alipay/ant-xgboost that referenced this pull request Jul 17, 2019

@rongou rongou deleted the rongou:fix-spark-tests branch Jul 18, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.