Make Bazel the default build system #3898

pcmoritz · 2019-01-30T05:26:20Z

What do these changes do?

This switches the build system from CMake to Bazel for developers.

The wheels, valgrind tests and Jenkins are currently still run with CMake and will be switched in follow up PRs.

Related issue number

AmplabJenkins · 2019-01-30T06:22:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11291/
Test FAILed.

AmplabJenkins · 2019-01-30T21:03:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11306/
Test PASSed.

AmplabJenkins · 2019-01-30T23:51:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11321/
Test FAILed.

AmplabJenkins · 2019-01-31T00:25:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11319/
Test FAILed.

AmplabJenkins · 2019-01-31T00:49:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11326/
Test FAILed.

AmplabJenkins · 2019-01-31T00:49:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11327/
Test FAILed.

AmplabJenkins · 2019-01-31T01:29:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11332/
Test FAILed.

AmplabJenkins · 2019-01-31T02:45:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11329/
Test PASSed.

AmplabJenkins · 2019-01-31T02:52:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11330/
Test FAILed.

AmplabJenkins · 2019-01-31T03:36:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11333/
Test PASSed.

AmplabJenkins · 2019-01-31T03:47:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11334/
Test FAILed.

AmplabJenkins · 2019-01-31T03:47:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11335/
Test FAILed.

AmplabJenkins · 2019-01-31T04:43:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11338/
Test PASSed.

guoyuhong · 2019-01-31T05:39:49Z

It look like the Java dynamic lib build should be also added to the bazel building process.

pcmoritz · 2019-01-31T05:56:15Z

@guoyuhong: Yes, can you contribute this in a separate PR? I'm not too familiar with Java build systems, but Bazel should support it well.

guoyuhong · 2019-01-31T07:12:11Z

@guoyuhong I'm not familiar with Bazel... I will have a try.

AmplabJenkins · 2019-01-31T23:11:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11363/
Test FAILed.

AmplabJenkins · 2019-01-31T23:24:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11364/
Test FAILed.

AmplabJenkins · 2019-02-01T00:51:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11365/
Test FAILed.

AmplabJenkins · 2019-02-01T00:51:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11370/
Test FAILed.

guoyuhong · 2019-02-01T02:04:55Z

@pcmoritz Do you have time to take a look at #3918 ?

AmplabJenkins · 2019-02-01T02:23:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11366/
Test PASSed.

AmplabJenkins · 2019-02-01T02:27:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11368/
Test FAILed.

AmplabJenkins · 2019-02-01T03:28:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11374/
Test FAILed.

AmplabJenkins · 2019-02-01T04:19:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11375/
Test PASSed.

AmplabJenkins · 2019-02-01T04:27:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11378/
Test FAILed.

AmplabJenkins · 2019-02-01T04:56:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11377/
Test PASSed.

AmplabJenkins · 2019-02-04T23:08:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11551/
Test FAILed.

AmplabJenkins · 2019-02-08T02:56:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11673/
Test FAILed.

AmplabJenkins · 2019-02-21T20:41:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12203/
Test FAILed.

AmplabJenkins · 2019-02-21T21:27:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12204/
Test FAILed.

AmplabJenkins · 2019-02-21T22:02:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12205/
Test FAILed.

AmplabJenkins · 2019-02-21T22:29:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12206/
Test FAILed.

AmplabJenkins · 2019-02-22T01:24:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12212/
Test FAILed.

pcmoritz · 2019-02-22T01:56:29Z

Jenkins retest this please

AmplabJenkins · 2019-02-22T03:59:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12215/
Test FAILed.

robertnishihara · 2019-02-22T07:03:23Z

build.sh

+  # The following line installs pyarrow from S3, these wheels have been
+  # generated from https://github.com/ray-project/arrow-build from
+  # the commit listed in the command.
+  pip install --target=$ROOT_DIR/python/ray/pyarrow_files pyarrow==0.12.0-RAY \


will find wrong pip in some cases

robertnishihara · 2019-02-22T07:06:41Z

build.sh

+  # generated from https://github.com/ray-project/arrow-build from
+  # the commit listed in the command.
+  pip install --target=$ROOT_DIR/python/ray/pyarrow_files pyarrow==0.12.0-RAY \
+              --find-links https://s3-us-west-2.amazonaws.com/arrow-wheels/bf0f3a15e0d6583cfaedbca508627b06746fa41a/index.html


In this current form this drops support for 3.4, so we should either build Arrow wheels for 3.4 or discuss that.

robertnishihara · 2019-02-22T07:09:53Z

@pcmoritz can you file an issue describing the C++ issues that we run into with the Arrow version in this PR?

robertnishihara · 2019-02-22T07:11:17Z

build.sh

+  # generated from https://github.com/ray-project/arrow-build from
+  # the commit listed in the command.
+  pip install --target=$ROOT_DIR/python/ray/pyarrow_files pyarrow==0.12.0-RAY \
+              --find-links https://s3-us-west-2.amazonaws.com/arrow-wheels/bf0f3a15e0d6583cfaedbca508627b06746fa41a/index.html


Either in this PR or a subsequent one, it will be important to document how to update the Arrow version.

AmplabJenkins · 2019-02-22T08:59:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12222/
Test FAILed.

AmplabJenkins · 2019-02-22T19:48:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12234/
Test FAILed.

AmplabJenkins · 2019-02-22T22:00:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12235/
Test FAILed.

AmplabJenkins · 2019-02-22T23:04:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12240/
Test FAILed.

AmplabJenkins · 2019-02-22T23:56:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12243/
Test FAILed.

AmplabJenkins · 2019-02-23T00:05:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12242/
Test FAILed.

robertnishihara · 2019-02-23T19:59:23Z

Progress on #2887.

guoyuhong mentioned this pull request Jan 31, 2019

Add bazel build for JNI code #3918

Merged

pcmoritz added 2 commits February 21, 2019 16:38

cleanup

8c2aec3

update

70ae8dd

robertnishihara reviewed Feb 22, 2019

View reviewed changes

robertnishihara mentioned this pull request Feb 22, 2019

Update arrow version to fix plasma bugs #4127

Merged

pcmoritz mentioned this pull request Feb 22, 2019

Note for Ray backend developers on the Python toolchain #4129

Closed

pcmoritz force-pushed the bazel-setup branch from 4cc4961 to c908280 Compare February 22, 2019 22:06

use the right pip

cb7888a

pcmoritz force-pushed the bazel-setup branch from c908280 to cb7888a Compare February 22, 2019 22:08

robertnishihara approved these changes Feb 23, 2019

View reviewed changes

robertnishihara merged commit ba52caf into ray-project:master Feb 23, 2019

robertnishihara deleted the bazel-setup branch February 23, 2019 19:59

robertnishihara mentioned this pull request Feb 23, 2019

Convert build system to Bazel. #2887

Closed

pcmoritz mentioned this pull request Feb 25, 2019

Update Arrow to arrow-one-mmap-file-on-b228489 ray-project/arrow-build#1

Closed

Make Bazel the default build system #3898

Make Bazel the default build system #3898

Conversation

pcmoritz commented Jan 30, 2019 • edited

What do these changes do?

Related issue number

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

guoyuhong commented Jan 31, 2019

pcmoritz commented Jan 31, 2019 • edited

guoyuhong commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

guoyuhong commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 4, 2019

AmplabJenkins commented Feb 8, 2019

AmplabJenkins commented Feb 21, 2019

AmplabJenkins commented Feb 21, 2019

AmplabJenkins commented Feb 21, 2019

AmplabJenkins commented Feb 21, 2019

AmplabJenkins commented Feb 22, 2019

pcmoritz commented Feb 22, 2019

AmplabJenkins commented Feb 22, 2019

robertnishihara Feb 22, 2019

Choose a reason for hiding this comment

robertnishihara Feb 22, 2019

Choose a reason for hiding this comment

robertnishihara commented Feb 22, 2019

robertnishihara Feb 22, 2019

Choose a reason for hiding this comment

AmplabJenkins commented Feb 22, 2019

AmplabJenkins commented Feb 22, 2019

AmplabJenkins commented Feb 22, 2019

AmplabJenkins commented Feb 22, 2019

AmplabJenkins commented Feb 22, 2019

AmplabJenkins commented Feb 23, 2019

robertnishihara commented Feb 23, 2019

pcmoritz commented Jan 30, 2019 •

edited

pcmoritz commented Jan 31, 2019 •

edited