Updated CircleCI tests to cap number of OpenMP threads #407

christopherbunn · 2020-02-21T19:27:59Z

Removed cap on XGBoost version. Instead, version 1.0.0 is excluded from requirements.txt so that Python 3.5 compatibility is maintained.

Resolves #401

docs/source/changelog.rst

dsherry · 2020-02-25T07:37:51Z

@christopherbunn @jeremyliweishih hmm why are we doing this? I thought #402 fixed this.

christopherbunn · 2020-02-25T14:40:58Z

@christopherbunn @jeremyliweishih hmm why are we doing this? I thought #402 fixed this.

That PR excluded anything above and including 1.0.0. Since then, XGBoost has released 1.0.1 which is supposed to fix the Python 3.5 errors. This PR only excludes 1.0.0.

codecov · 2020-03-02T21:58:50Z

Codecov Report

Merging #407 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #407   +/-   ##
=======================================
  Coverage   98.26%   98.26%           
=======================================
  Files         104      104           
  Lines        3292     3292           
=======================================
  Hits         3235     3235           
  Misses         57       57

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e9acab...7568a3c. Read the comment docs.

dsherry

Oh weird, I just saw this review was marked as "pending". Not sure if my comments showed up or not. Sorry about that!

dsherry · 2020-02-25T17:04:16Z

.circleci/config.yml

+            mkdir build
+            cd build
+            cmake ..
+            make -j4


Why do we have to build xgboost from source instead of using pip? This makes me nervous, because our tests should mirror our user install experience--do we have to ask people to build xgboost from source now?

dsherry · 2020-02-27T10:26:24Z

requirements.txt

@@ -5,7 +5,7 @@ scikit-learn>=0.21.3,!=0.22
 dask[complete]>=2.1.0
 numpy>=1.16.4
 pandas>=0.25.0,<1.0.0
-xgboost>=0.82,<1.0.0
+xgboost>=0.82,!=1.0.0


dsherry

Thanks @christopherbunn. Left a comment about the pytest verbosity thing, otherwise LGTM!

Sorry again about the delayed review... it was probably because I was on my phone at the time.

dsherry · 2020-03-03T19:47:53Z

Makefile

@@ -20,11 +20,11 @@ test:

 .PHONY: circleci-test
 circleci-test:
-	pytest evalml/ -n 8 --doctest-modules --cov=evalml --junitxml=test-reports/junit.xml --doctest-continue-on-failure
+	pytest evalml/ -n 8 --doctest-modules --cov=evalml --junitxml=test-reports/junit.xml --doctest-continue-on-failure -v


Any particular reason to keep this, or did you just forget to remove this after debugging?

It spits out the individual names for testing. Since it's more descriptive about what test is being done imo it's better to keep it so that if we need to see what tests specific are passing in CircleCI. However, I can understand that it might look cluttered. Take a look in one of the checks and lmk what you think.

Oh cool. That's a great idea! I like it. Thanks.

In the future let's keep separate changes like this in separate PRs instead of lumping them into one. That'll keep it easy to look at our merge history and understand exactly what changed and when it changed. Having separate smaller PRs like that also makes it easier to tell where a particular test failure is coming from. That sound good?

christopherbunn · 2020-03-03T20:12:35Z

@dsherry For some reason it doesn't pass still on the XGBoost 1.0.1 release on CircleCI. Locally on my machine and in the CircleCI CLI (which runs in the Docker container that they use), it runs within 5-8 minutes. However, it times out on CircleCI's machines. I'm not exactly 100% sure why this is the case.

After some digging the best explanation I have is based on the fact that CircleCI runs multiple Docker containers on their machine at a time. It could be possible that XGBoost is configured to grab all of the cores of the machine (I think it's 36 cores?) when n_jobs=-1 rather than the 8 cores that it has in Docker. Since it doesn't have access to all of the cores, the workers have the potential to fail, then restart (and loop in a cycle).

~~I'm going to test this by fixing n_jobs in XGBoost only to confirm but I definitely wanted to post an update on this.~~

UDPATE: tried the above and did not work, I'm going to adjust the CircleCI timeout time then look at the resulting Junit XML file to see what tests exactly are taking up all of the time.

dsherry · 2020-03-03T20:54:31Z

@christopherbunn ah got it. Weird. Good digging. Any idea why this would start happening for xgboost 1.0.1 but not for our previous version? Good idea to test fixing n_jobs and see if that helps.

Hmm, I had assumed circleci's docker setup would limit the number of cores available to the process, so that something running in a circleci docker container could only take so many CPUs. Is that not the case?

And do you know how long the timeout is?

christopherbunn · 2020-03-03T21:17:35Z

@christopherbunn ah got it. Weird. Good digging. Any idea why this would start happening for xgboost 1.0.1 but not for our previous version? Good idea to test fixing n_jobs and see if that helps.

Hmm, I had assumed circleci's docker setup would limit the number of cores available to the process, so that something running in a circleci docker container could only take so many CPUs. Is that not the case?

I assumed the same thing as well. However, since. This might be a stretch but I looked though Anglea's joblib error since I thought it might be related. In the CircleCI output here, you can see that the Parallel class is initiated with 26 jobs (i.e. Parallel(n_jobs=26)) despite the fact that the Docker container is set to use 8 cores. The 26 value most likely matches the EC2 instance they're using to run the tests. I'm guessing it's because Docker on macOS runs containers in a VM (which would restrict core counts) whereas Docker for Linux isn't a VM.

I'm pretty sure the same Parallel library is used in XGBoost but I would need to double check.

And do you know how long the timeout is?

I bumped up the timeout to 30 minutes. Definitely not a long term solution, but I wanted to be able to generate a JUnit report so that we can take a look at what tests ran the longest.

I've attached a pdf of the results (since Github for some reason doesn't support attaching XML or HTML files to comments 😬). It ran the entire test suite in ~21 minutes, which is significantly greater than the 5-8 minutes it would typically run.

junit.xml - Junit Test Report.pdf

christopherbunn · 2020-03-10T18:15:30Z

So after some digging I found a workaround for our CircleCI issues. Because it looks like CircleCI does not report the number of CPUs available in a container when nproc is called, XGBoost is most likely overallocating the actual number of threads available. The library that handles CPU thread parallelism, OpenMP, has the ability to set an upper limit through an environment variable.

For the CircleCI tests, I set the cap to be 8 threads since that's the current size of our Docker container. It looks like it runs without timing out as of now. I think this issue should be specific to CircleCI only and most regular machines should have the ability to correctly report the number of threads available to them.

dsherry · 2020-03-10T18:21:24Z

.circleci/config.yml

@@ -12,6 +12,8 @@ executors:
    working_directory: ~/evalml
    docker:
      - image: circleci/python:<< parameters.python_version >>
+        environment:
+          OMP_NUM_THREADS: 8


dsherry

@christopherbunn oh that's great! Excellent digging :) have you verified this for our case? Either way, I suppose it can't hurt to try.

dsherry · 2020-03-10T18:24:12Z

@christopherbunn do you happen to know where the circleci doc is for OMP_NUM_THREADS ? I can't find it at first google.

I wonder if this is the same root cause as the failures in #167 / #441

dsherry · 2020-03-10T18:25:00Z

I do see that sklearn sets this parameter in their circleci config, which is promising.

dsherry · 2020-03-10T18:26:37Z

Aha, I think I found it: https://www.openmp.org/spec-html/5.0/openmpse50.html

Found root cause of CircleCI errors

jeremyliweishih previously requested changes Feb 21, 2020

View reviewed changes

docs/source/changelog.rst Outdated Show resolved Hide resolved

dsherry mentioned this pull request Feb 21, 2020

Update to XGBoost 1.0.2 #401

Closed

christopherbunn force-pushed the 401_remove_xg_cap branch 2 times, most recently from e3b8516 to 3da7f69 Compare March 2, 2020 21:42

dsherry reviewed Mar 3, 2020

View reviewed changes

dsherry approved these changes Mar 3, 2020

View reviewed changes

dsherry reviewed Mar 3, 2020

View reviewed changes

christopherbunn force-pushed the 401_remove_xg_cap branch 2 times, most recently from a862668 to 0adba96 Compare March 3, 2020 20:43

dsherry changed the title ~~Excluded XGBoost 1.0.0 from requirements.txt~~ [WIP] Excluded XGBoost 1.0.0 from requirements.txt Mar 3, 2020

dsherry changed the title ~~[WIP] Excluded XGBoost 1.0.0 from requirements.txt~~ [WIP] Exclude XGBoost 1.0.0 from requirements.txt Mar 3, 2020

dsherry changed the title ~~[WIP] Exclude XGBoost 1.0.0 from requirements.txt~~ [WIP] XGBoost: update to v1.0.1, and disallow v1.0.0 Mar 3, 2020

christopherbunn added 3 commits March 10, 2020 12:50

Restored ability to use xgboost 1.0.1

32a7e87

Testing to cap OpenMP threads as an env variable

987c08b

Added back verbose and updated dependencies

659ee6e

christopherbunn force-pushed the 401_remove_xg_cap branch from 3dc98b2 to 659ee6e Compare March 10, 2020 16:53

christopherbunn added 2 commits March 10, 2020 12:55

Updated changelog and removed xgboost 1.0.0 exclusion

e66a68a

Removed boto from dependency check

7568a3c

christopherbunn changed the title ~~[WIP] XGBoost: update to v1.0.1, and disallow v1.0.0~~ Updated CircleCI tests to cap number of OpenMP threads Mar 10, 2020

dsherry reviewed Mar 10, 2020

View reviewed changes

dsherry approved these changes Mar 10, 2020

View reviewed changes

christopherbunn merged commit e66ac68 into master Mar 10, 2020

This was referenced Mar 10, 2020

Flaky unit tests: joblib parallel ValueError #167

Closed

[WIP] Allow numpy.random.RandomState for random_state #441

Closed

dsherry deleted the 401_remove_xg_cap branch March 27, 2020 04:38

dsherry mentioned this pull request Mar 31, 2020

Support numpy.random.RandomState objects #530

Merged

angela97lin mentioned this pull request Apr 1, 2020

Make release for v0.8.0 #565

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated CircleCI tests to cap number of OpenMP threads #407

Updated CircleCI tests to cap number of OpenMP threads #407

christopherbunn commented Feb 21, 2020 •

edited

Loading

dsherry commented Feb 25, 2020

christopherbunn commented Feb 25, 2020

codecov bot commented Mar 2, 2020 •

edited

Loading

dsherry left a comment

dsherry Feb 25, 2020

dsherry Feb 27, 2020

dsherry left a comment •

edited

Loading

dsherry Mar 3, 2020

christopherbunn Mar 3, 2020

dsherry Mar 3, 2020

christopherbunn commented Mar 3, 2020 •

edited

Loading

dsherry commented Mar 3, 2020 •

edited

Loading

christopherbunn commented Mar 3, 2020

christopherbunn commented Mar 10, 2020

dsherry Mar 10, 2020

dsherry left a comment

dsherry commented Mar 10, 2020 •

edited

Loading

dsherry commented Mar 10, 2020

dsherry commented Mar 10, 2020

Updated CircleCI tests to cap number of OpenMP threads #407

Updated CircleCI tests to cap number of OpenMP threads #407

Conversation

christopherbunn commented Feb 21, 2020 • edited Loading

dsherry commented Feb 25, 2020

christopherbunn commented Feb 25, 2020

codecov bot commented Mar 2, 2020 • edited Loading

Codecov Report

dsherry left a comment

Choose a reason for hiding this comment

dsherry Feb 25, 2020

Choose a reason for hiding this comment

dsherry Feb 27, 2020

Choose a reason for hiding this comment

dsherry left a comment • edited Loading

Choose a reason for hiding this comment

dsherry Mar 3, 2020

Choose a reason for hiding this comment

christopherbunn Mar 3, 2020

Choose a reason for hiding this comment

dsherry Mar 3, 2020

Choose a reason for hiding this comment

christopherbunn commented Mar 3, 2020 • edited Loading

dsherry commented Mar 3, 2020 • edited Loading

christopherbunn commented Mar 3, 2020

christopherbunn commented Mar 10, 2020

dsherry Mar 10, 2020

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

dsherry commented Mar 10, 2020 • edited Loading

dsherry commented Mar 10, 2020

dsherry commented Mar 10, 2020

christopherbunn commented Feb 21, 2020 •

edited

Loading

codecov bot commented Mar 2, 2020 •

edited

Loading

dsherry left a comment •

edited

Loading

christopherbunn commented Mar 3, 2020 •

edited

Loading

dsherry commented Mar 3, 2020 •

edited

Loading

dsherry commented Mar 10, 2020 •

edited

Loading