CI test speed improvement #3851

lowener · 2021-05-12T13:03:42Z

Closes #3786.
Some redundant combinations are removed and I reduced the number of samples on some unit tests that are run very often.
test_umap.py::test_umap_fit_transform_trust is the test that takes the most time in CI.

Local speed-up:

test_umap.py: From 273s to 79s
test_dbscan.py: From 14s to 2s
dask/test_nearest_neighbors.py::test_compare_skl: From 113s to 31s

In total, the expected saved time is 288s (~5 min) on the pipeline locally.

lowener · 2021-05-12T16:42:32Z

Summary of top 40 tests times in CI before/after those improvements

Before

53.95s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[8-50]
53.93s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[random_state2-50]
20.30s call     cuml/test/test_arima.py::test_integration[float64-key6-data6]
17.76s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[8-50]
17.75s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[random_state2-50]
17.12s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[random_state2-25]
17.12s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[8-25]
15.74s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[blobs-categorical]
13.38s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
13.19s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
12.01s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_array]
12.01s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_array]
9.97s call     cuml/test/dask/test_nearest_neighbors.py::test_batch_size[100-10-10-1000]
8.56s call     cuml/test/test_arima.py::test_integration[float64-key7-data7]
7.80s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-classification]
7.59s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-regression]
7.56s call     cuml/test/test_benchmark.py::test_real_algos_runner[UMAP-Supervised]
7.34s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[blobs-euclidean]
7.28s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_cudf]
6.98s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[8-21]
6.98s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[random_state2-21]
6.50s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_array]
6.18s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
6.16s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
5.84s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_array]
5.82s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[8-25]
5.82s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[random_state2-25]
5.80s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_array]
5.56s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_array]
5.48s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_array]
5.38s call     cuml/test/dask/test_pca.py::test_pca_fit[dataframe-67-20-1000]
4.99s call     cuml/test/test_random_forest.py::test_rf_regression_float64[large_reg0-datatype0]
4.90s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-random-0.5-float32]
4.87s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-random-0.5-float32]
4.84s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-cyclic-0.5-float32]
4.79s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-cyclic-0.5-float32]
4.63s teardown cuml/test/dask/test_datasets.py::test_make_classification[float64-F-23-1234-20-4-4-False-100-1000]
4.59s teardown cuml/test/dask/test_base.py::test_getattr
4.58s teardown cuml/test/dask/test_random_forest.py::test_rf_broadcast[False-False-regression]
4.58s teardown cuml/test/dask/test_coordinate_descent.py::test_elastic_net_default[64-column_info2-500000-float64]
4.51s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
= 20246 passed, 7418 skipped, 631 xfailed, 77 xpassed, 22045 warnings in 1491.80s (0:24:51) =

After

53.92s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[50-random_state4]
20.08s call     cuml/test/test_arima.py::test_integration[float64-key6-data6]
17.59s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[50-8]
12.74s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
12.58s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
11.28s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_array]
11.19s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_array]
9.34s call     cuml/test/dask/test_nearest_neighbors.py::test_batch_size[100-10-10-1000]
8.49s call     cuml/test/test_arima.py::test_integration[float64-key7-data7]
7.79s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-classification]
7.62s call     cuml/test/test_benchmark.py::test_real_algos_runner[UMAP-Supervised]
7.55s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-regression]
7.05s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_cudf]
6.97s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[21-random_state2]
6.97s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[21-8]
6.14s call     cuml/test/dask/test_pca.py::test_pca_fit[dataframe-67-20-1000]
6.09s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_array]
5.84s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
5.76s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[25-random_state3]
5.76s call     cuml/test/test_umap.py::test_umap_transform_reproducibility[25-8]
5.75s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
5.63s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_array]
5.62s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_array]
5.22s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_array]
5.13s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_array]
4.99s call     cuml/test/test_random_forest.py::test_rf_regression_float64[large_reg0-datatype0]
4.85s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-random-0.5-float32]
4.79s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-cyclic-0.5-float32]
4.77s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-cyclic-0.5-float32]
4.77s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-random-0.5-float32]
4.63s teardown cuml/test/dask/test_datasets.py::test_make_classification[float64-F-23-1234-20-4-4-False-100-1000]
4.54s teardown cuml/test/dask/test_base.py::test_getattr
4.54s teardown cuml/test/dask/test_random_forest.py::test_rf_broadcast[False-False-regression]
4.53s teardown cuml/test/dask/test_coordinate_descent.py::test_elastic_net_default[64-column_info2-500000-float64]
4.45s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
4.40s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
3.86s call     cuml/test/dask/test_tsvd.py::test_pca_fit[dataframe-data_info0]
3.86s call     cuml/test/dask/test_coordinate_descent.py::test_lasso[True-4-column_info0-50-cyclic-0.001-float32]
3.67s call     cuml/test/test_arima.py::test_integration[float64-key5-data5]
3.57s call     cuml/test/test_pickle.py::test_regressor_pickle[False-data_size0-MBSGDClassifier-float32]
3.55s call     cuml/test/test_random_forest.py::test_rf_host_memory_leak[large_clf0-classification]
= 20214 passed, 6738 skipped, 633 xfailed, 75 xpassed, 22049 warnings in 1280.59s (0:21:20) =

dantegd · 2021-05-12T19:33:02Z

@lowener there seems to still be one particular test that is on the slower side, do you think it could be tackled in this PR?

53.92s call     cuml/test/test_umap.py::test_umap_fit_transform_reproducibility[50-random_state4]

lowener · 2021-05-14T20:07:29Z

I moved this test to a stress test and introduced a unit test that is much faster (with parameters [50-None] it's around 2 seconds).
I did this for test_umap_fit_transform_reproducibility and test_umap_transform_reproducibility which was taking 17.59s.

dantegd

Had one quick question

dantegd · 2021-05-20T15:23:29Z

python/cuml/test/test_umap.py

@@ -230,7 +230,7 @@ def test_umap_fit_transform_trust(name, target_metric):
        data = wine.data
        labels = wine.target
    else:
-        data, labels = make_blobs(n_samples=5000, n_features=10,
+        data, labels = make_blobs(n_samples=500, n_features=10,


Just had one quick question, have you seen any issues of reducing the data so much? i.e. it would be nice to run the test a number of times to be sure no flakiness is being introduced

I saw no failures with 100 retries of this test.

JohnZed

Looks good to me, but agreed we should run like 100 times in a row (you can use https://pypi.org/project/pytest-repeat/) to make sure they are not flaky.

JohnZed · 2021-05-20T16:26:10Z

python/cuml/test/test_umap.py

@@ -230,7 +230,7 @@ def test_umap_fit_transform_trust(name, target_metric):
        data = wine.data
        labels = wine.target
    else:
-        data, labels = make_blobs(n_samples=5000, n_features=10,
+        data, labels = make_blobs(n_samples=500, n_features=10,


dantegd · 2021-05-26T00:07:08Z

rerun tests

dantegd · 2021-05-26T00:07:17Z

@gpucibot merge

codecov-commenter · 2021-05-26T04:45:39Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.06@3e89f04). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-21.06    #3851   +/-   ##
===============================================
  Coverage                ?   85.55%           
===============================================
  Files                   ?      226           
  Lines                   ?    17357           
  Branches                ?        0           
===============================================
  Hits                    ?    14850           
  Misses                  ?     2507           
  Partials                ?        0

Flag	Coverage Δ
dask	`48.84% <0.00%> (?)`
non-dask	`77.57% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e89f04...9a6a223. Read the comment docs.

Closes rapidsai#3786. Some redundant combinations are removed and I reduced the number of samples on some unit tests that are run very often. `test_umap.py::test_umap_fit_transform_trust` is the test that takes the most time in CI. Local speed-up: - `test_umap.py`: From `273s` to `79s` - `test_dbscan.py`: From `14s` to `2s` - `dask/test_nearest_neighbors.py::test_compare_skl`: From `113s` to `31s` In total, the expected saved time is `288s` (~5 min) on the pipeline locally. Authors: - Micka (https://github.com/lowener) Approvers: - John Zedlewski (https://github.com/JohnZed) - Dante Gama Dessavre (https://github.com/dantegd) URL: rapidsai#3851

lowener added 2 commits May 11, 2021 22:20

Test parameter simplification

9933e71

fix style

bc392a1

lowener requested a review from a team as a code owner May 12, 2021 13:03

github-actions bot added the Cython / Python Cython or Python issue label May 12, 2021

lowener requested a review from JohnZed May 12, 2021 13:03

lowener added tests Unit testing for project improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels May 12, 2021

Add copyright

8251f83

lowener added 2 commits May 14, 2021 21:07

Reducing test_umap param size

8c300e8

Changed umap unit and stress param

9a9554d

dantegd added the 4 - Waiting on Reviewer Waiting for reviewer to review or respond label May 17, 2021

lowener changed the title ~~CI speed improvement~~ CI test speed improvement May 18, 2021

dantegd requested changes May 20, 2021

View reviewed changes

dantegd mentioned this pull request May 20, 2021

[BUG] Speed up test_umap (single-GPU) #3200

Closed

JohnZed approved these changes May 20, 2021

View reviewed changes

dantegd added 4 - Waiting on Author Waiting for author to respond to review and removed 4 - Waiting on Reviewer Waiting for reviewer to review or respond labels May 21, 2021

Merge branch 'branch-21.06' into 020-ci-test-improvement

9a6a223

lowener requested a review from dantegd May 25, 2021 14:57

lowener added 4 - Waiting on Reviewer Waiting for reviewer to review or respond and removed 4 - Waiting on Author Waiting for author to respond to review labels May 25, 2021

caryr35 added this to PR-WIP in v21.06 Release via automation May 25, 2021

dantegd approved these changes May 27, 2021

View reviewed changes

v21.06 Release automation moved this from PR-WIP to PR-Reviewer approved May 27, 2021

rapids-bot bot merged commit 474e2e7 into rapidsai:branch-21.06 May 27, 2021

v21.06 Release automation moved this from PR-Reviewer approved to Done May 27, 2021

lowener deleted the 020-ci-test-improvement branch May 27, 2021 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI test speed improvement #3851

CI test speed improvement #3851

lowener commented May 12, 2021 •

edited

lowener commented May 12, 2021 •

edited

dantegd commented May 12, 2021

lowener commented May 14, 2021

dantegd left a comment

dantegd May 20, 2021

JohnZed May 20, 2021

lowener May 25, 2021

JohnZed left a comment

JohnZed May 20, 2021

dantegd commented May 26, 2021

dantegd commented May 26, 2021

codecov-commenter commented May 26, 2021

CI test speed improvement #3851

CI test speed improvement #3851

Conversation

lowener commented May 12, 2021 • edited

lowener commented May 12, 2021 • edited

dantegd commented May 12, 2021

lowener commented May 14, 2021

dantegd left a comment

Choose a reason for hiding this comment

dantegd May 20, 2021

Choose a reason for hiding this comment

JohnZed May 20, 2021

Choose a reason for hiding this comment

lowener May 25, 2021

Choose a reason for hiding this comment

JohnZed left a comment

Choose a reason for hiding this comment

JohnZed May 20, 2021

Choose a reason for hiding this comment

dantegd commented May 26, 2021

dantegd commented May 26, 2021

codecov-commenter commented May 26, 2021

Codecov Report

lowener commented May 12, 2021 •

edited

lowener commented May 12, 2021 •

edited