cover change with gtest unittests, clean up cmake script and code includes #106

chenqin · 2019-09-17T04:42:24Z

This PR do two things

clear unused environment variables, accept those parameters as arguments
add gtest unittests, cleanup includes and cmake scripts to work with xgboost project

@trivialfis @hcho3

src/allreduce_base.cc

trivialfis

Please use deterministic waiting and point me to some unittests.

hcho3 · 2019-09-18T02:28:24Z

@chenqin Can we use exponential backoff to ease congestion? Unlike random waiting, it will have deterministic behavior.

chenqin · 2019-09-18T05:39:25Z

@chenqin Can we use exponential backoff to ease congestion? Unlike random waiting, it will have

removed random with expoential back off plus rank %10 to avoid ddos tracker

chenqin · 2019-09-18T06:19:22Z

Please use deterministic waiting and point me to some unittests.

working on googletest to support unittests

trivialfis · 2019-09-19T02:51:59Z

@chenqin You can consider using gtest main: https://github.com/dmlc/xgboost/blob/master/tests/cpp/test_main.cc

chenqin · 2019-09-19T14:51:52Z

@chenqin You can consider using gtest main: https://github.com/dmlc/xgboost/blob/master/tests/cpp/test_main.cc

done

chenqin · 2019-09-19T17:03:29Z

@hcho3 @trivialfis

trivialfis

Will give a proper review tomorrow, one thing though, try find_package first then download gtest ...

chenqin · 2019-09-20T20:43:30Z

Will give a proper review tomorrow, one thing though, try find_package first then download gtest ...

I tried to use find_package along with exteranl project which doesn't ends well.
gtest download is shelled with RABIT_BUILD_TESTS

trivialfis · 2019-09-21T09:38:13Z

@chenqin Let me try.

trivialfis · 2019-09-21T09:44:23Z

Also, why is it necessary to download gtest? Can we assume that the developer can install gtest test somewhere? CMake accepts a parameter called GTEST_ROOT, which can be used to find gtest: https://cmake.org/cmake/help/v3.0/module/FindGTest.html

Clean up CMake scripts. will work on a bit cleanup

chenqin · 2019-09-22T17:24:40Z

@trivialfis let's get this in to unblock xgb CMake house cleaning

chenqin · 2019-09-23T22:34:38Z

done with cleanup, xgboost now build with rabit

CodingCat

we should avoid mixing things together, especially "cleanup/refactoring" and "new feature adding"

and the other thing I don't understand is. the description says "clear unused environment variables, accept those parameters as arguments" why it contains so many CMake file changes

src/allreduce_base.cc

chenqin · 2019-09-23T23:20:14Z

we should avoid mixing things together, especially "cleanup/refactoring" and "new feature adding"

and the other thing I don't understand is. the description says "clear unused environment variables, accept those parameters as arguments" why it contains so many CMake file changes

removed from this pr, let's do back off in separate pr

chenqin · 2019-09-23T23:24:54Z

we should avoid mixing things together, especially "cleanup/refactoring" and "new feature adding"

and the other thing I don't understand is. the description says "clear unused environment variables, accept those parameters as arguments" why it contains so many CMake file changes

fixed, remvoed code change touch sleep, this pr focus on make it buildable with xgboost project and adding googletest backed unit tests

chenqin · 2019-09-24T02:10:40Z

@trivialfis

trivialfis · 2019-09-24T09:39:15Z

cmake/googletest-download.cmake

@@ -0,0 +1,20 @@
+# code copied from https://crascit.com/2015/07/25/cmake-gtest/


Doesn't seem necessary anymore. I'm OK for keeping it here for the convenience of others.

@trivialfis ~~I added this because this was the easiest way to install Google Test on Windows platform.~~ Sorry, I forgot that this PR is in Rabit. I added something like this in dmlc-core, and it's quite convenient. I suggest we keep it.

test/cpp/allreduce_base_test.cpp

trivialfis · 2019-09-24T09:56:33Z

CMakeLists.txt

+
+option(DMLC_ROOT "Specify root of external dmlc core.")
+# by default point to xgboost/dmlc-core
+set(DMLC_ROOT ${CMAKE_CURRENT_LIST_DIR}/../dmlc-core)


You spent so much effort on rabit I assume you don't want to make it XGBoost exclusive right? Since last time I check rabit is only using some headers from dmlc-core, there's no linkage involved. So technically rabit can use dmlc-core in system path, say /usr/ or /home/chenqin/.local/ or any other directories that follow the bin/include/lib/lib64 pattern without any problem, and you can set above path as DMLC_ROOT for your convenience. With this, you can simply throw an error if DMLC_ROOT is not specified.

In the future (not for this PR), I may add this to rabit as dmlc-core has very good support for CMake target.

if (NOT DMLC_ROOT) find_package(dmlc) target_link_libraries(rabit dmlc::dmlc) # This line also handles include directory. endif (NOT DMLC_ROOT)

trivialfis · 2019-09-24T09:59:18Z

CMakeLists.txt

 ENDIF(R_LIB OR MINGW OR WIN32)

 if(RABIT_BUILD_MPI)
  find_package(MPI REQUIRED)
+  if (NOT MPI_CXX_FOUND)
+    message(FATAL_ERROR "CXX Interface for MPI is required for building MPI backend.")


I wrote this line, but do we actually want to support MPI backend? I thought it's a proof of concept for the paper. And now that the MPI CXX interface is deprecated, there will be some more works for bring it up. @chenqin What do you think?

We might discuss this in another thread how to support MPI runtime. The rationale is there are lots of similar frameworks highly optimized such as facebook/gloo that just do MPI optimization. None of which offers failed single worker recover. But I don't have clear picture who is using rabit other than distributed XGBoost.

trivialfis · 2019-09-24T10:00:35Z

Other than the line making rabit XGBoost exclusive, LGTM. Since this PR is a whole with dmlc/xgboost#4876 , will merge them together.

trivialfis · 2019-09-26T06:29:28Z

@chenqin Merged. You can focus on the jvm-package PR. ;-)

trivialfis · 2019-09-26T06:29:48Z

And, a big thanks!

* Clean up CMake scripts and related include paths. * Add unittests.

chenqin and others added 2 commits September 17, 2019 04:22

remove no need env

36e4c2f

use rand back off to avoid swap tracker

77eea8d

chenqin mentioned this pull request Sep 17, 2019

[jvm-packages] expose rabit configuration to user dmlc/xgboost#4861

Closed

chenqin changed the title ~~cleanup unused environment variables rand to smooth traffic to tracker~~ cleanup unused environment variables rand to smooth traffic send to tracker Sep 17, 2019

trivialfis self-requested a review September 18, 2019 01:52

trivialfis approved these changes Sep 18, 2019

View reviewed changes

src/allreduce_base.cc Outdated Show resolved Hide resolved

src/allreduce_base.cc Show resolved Hide resolved

trivialfis requested changes Sep 18, 2019

View reviewed changes

use determinstic backoff retry considering smooth traffic

4ff7cee

Chen Qin added 2 commits September 18, 2019 16:24

adding googletests support

415f4c6

add unittests on allreduce parameters init

6aefaa0

chenqin added 5 commits September 18, 2019 20:53

Merge branch 'master' of github.com:chenqin/rabit

8224bda

per feedback

566e7bc

revert main

7a44877

fix failure

bd8227f

use test_main

700c86c

chenqin changed the title ~~cleanup unused environment variables rand to smooth traffic send to tracker~~ cover change with gtest unittests, smooth tracker reconnect Sep 19, 2019

chenqin mentioned this pull request Sep 19, 2019

[EPIC] Allow failed worker retry in distributed training dmlc/xgboost#4753

Closed

10 tasks

trivialfis reviewed Sep 19, 2019

View reviewed changes

fix rabit_reduce_ring_mincount parser

58344af

chenqin mentioned this pull request Sep 20, 2019

[jvm-packages] update rabit, surface new changes to spark, add parity and failure tests dmlc/xgboost#4876

Merged

add README

3346a46

trivialfis and others added 8 commits September 21, 2019 10:31

Install tree.

72a5f33

Use before_install.

5650d99

Specify GTEST_ROOT.

64963dc

Force c++11 in test.

0f6bc00

Merge pull request #5 from trivialfis/clean-cmake

3fa3803

Clean up CMake scripts. will work on a bit cleanup

add back cpplint

8188f0a

move pip3 install before setup script

6e13671

try fix cmake test issue on osx

16d5f0d

chenqin and others added 5 commits September 22, 2019 21:10

spring clean includes

0e90e01

downgrade cmake

492024a

more cleanup on DMLC_ROOT

65eb5c0

build engempty for win32 asis

4114087

fix relateive path of install prefix

5bb3045

hcho3 requested a review from CodingCat September 23, 2019 22:48

CodingCat reviewed Sep 23, 2019

View reviewed changes

src/allreduce_base.cc Outdated Show resolved Hide resolved

revert backoff change for another pr

f0cfd4c

chenqin changed the title ~~cover change with gtest unittests, smooth tracker reconnect~~ cover change with gtest unittests, clean up cmake script and code includes Sep 23, 2019

trivialfis approved these changes Sep 24, 2019

View reviewed changes

hcho3 approved these changes Sep 25, 2019

View reviewed changes

trivialfis merged commit ddcc2d8 into dmlc:master Sep 26, 2019

nateagr pushed a commit to criteo-forks/rabit that referenced this pull request Oct 9, 2019

Clean up cmake script and code includes (dmlc#106)

dd92db7

* Clean up CMake scripts and related include paths. * Add unittests.

nateagr pushed a commit to criteo-forks/rabit that referenced this pull request Oct 9, 2019

Clean up cmake script and code includes (dmlc#106)

bdb3be9

* Clean up CMake scripts and related include paths. * Add unittests.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cover change with gtest unittests, clean up cmake script and code includes #106

cover change with gtest unittests, clean up cmake script and code includes #106

chenqin commented Sep 17, 2019 •

edited

trivialfis left a comment

hcho3 commented Sep 18, 2019

chenqin commented Sep 18, 2019 •

edited

chenqin commented Sep 18, 2019

trivialfis commented Sep 19, 2019

chenqin commented Sep 19, 2019

chenqin commented Sep 19, 2019

trivialfis left a comment

chenqin commented Sep 20, 2019

trivialfis commented Sep 21, 2019

trivialfis commented Sep 21, 2019

chenqin commented Sep 22, 2019

chenqin commented Sep 23, 2019

CodingCat left a comment

chenqin commented Sep 23, 2019

chenqin commented Sep 23, 2019

chenqin commented Sep 24, 2019

trivialfis Sep 24, 2019

hcho3 Sep 25, 2019 •

edited

trivialfis Sep 24, 2019 •

edited

trivialfis Sep 24, 2019

chenqin Sep 24, 2019 •

edited

trivialfis commented Sep 24, 2019

trivialfis commented Sep 26, 2019

trivialfis commented Sep 26, 2019

		@@ -0,0 +1,20 @@
		# code copied from https://crascit.com/2015/07/25/cmake-gtest/

cover change with gtest unittests, clean up cmake script and code includes #106

cover change with gtest unittests, clean up cmake script and code includes #106

Conversation

chenqin commented Sep 17, 2019 • edited

trivialfis left a comment

Choose a reason for hiding this comment

hcho3 commented Sep 18, 2019

chenqin commented Sep 18, 2019 • edited

chenqin commented Sep 18, 2019

trivialfis commented Sep 19, 2019

chenqin commented Sep 19, 2019

chenqin commented Sep 19, 2019

trivialfis left a comment

Choose a reason for hiding this comment

chenqin commented Sep 20, 2019

trivialfis commented Sep 21, 2019

trivialfis commented Sep 21, 2019

chenqin commented Sep 22, 2019

chenqin commented Sep 23, 2019

CodingCat left a comment

Choose a reason for hiding this comment

chenqin commented Sep 23, 2019

chenqin commented Sep 23, 2019

chenqin commented Sep 24, 2019

trivialfis Sep 24, 2019

Choose a reason for hiding this comment

hcho3 Sep 25, 2019 • edited

Choose a reason for hiding this comment

trivialfis Sep 24, 2019 • edited

Choose a reason for hiding this comment

trivialfis Sep 24, 2019

Choose a reason for hiding this comment

chenqin Sep 24, 2019 • edited

Choose a reason for hiding this comment

trivialfis commented Sep 24, 2019

trivialfis commented Sep 26, 2019

trivialfis commented Sep 26, 2019

chenqin commented Sep 17, 2019 •

edited

chenqin commented Sep 18, 2019 •

edited

hcho3 Sep 25, 2019 •

edited

trivialfis Sep 24, 2019 •

edited

chenqin Sep 24, 2019 •

edited