Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce time for reproducibility and DGI save/load tests #1721

Merged
merged 2 commits into from Jun 25, 2020

Conversation

huonw
Copy link
Member

@huonw huonw commented Jun 23, 2020

The time for the 3.x pytest testing step seems to be bimodal at the moment: usually around 18 minutes, but sometimes hitting the 30 minute timeout. This may be due to the agent running out of burst credits, and thus running slower.

This causes problems both with the time it takes for the build to run and with the build failing due to the timeout. Cutting the test times will make development easier.

In https://buildkite.com/stellar/stellargraph-public/builds/5012, the 3.7 and 3.8 steps take about 30 minutes, but get in just under the timeout. Based on the JUnit XML files there, the slowest tests are:

3.7:

class name time
layer.test_deep_graph_infomax test_dgi_save_load[HinSAGE] 189.596
reproducibility.test_graphsage test_link_prediction[False] 104.020
reproducibility.test_deep_graph_infomax test_dgi[True-APPNP] 48.177
layer.test_hinsage test_hinsage_save_load 45.856
layer.test_deep_graph_infomax test_dgi[False-HinSAGE] 39.722
reproducibility.test_graphsage test_link_prediction[True] 39.340
reproducibility.test_graphsage test_nai[True] 38.290
layer.test_graphsage test_graphsage_save_load 34.116
test_ensemble test_ensemble_init_parameters 32.852
test_ensemble test_evaluate_link_prediction 32.474
reproducibility.test_graphsage test_unsupervised[True] 31.503
test_ensemble test_predict 28.937
layer.test_knowledge_graph test_save_load[RotH] 27.562
test_ensemble test_evaluate 27.181
reproducibility.test_deep_graph_infomax test_dgi[False-APPNP] 26.203
reproducibility.test_deep_graph_infomax test_dgi[True-GAT] 25.441
reproducibility.test_deep_graph_infomax test_dgi[False-GAT] 24.703
test_ensemble test_predict_link_prediction 23.889

3.8:

class name time
layer.test_deep_graph_infomax test_dgi_save_load[HinSAGE] 196.829
reproducibility.test_graphsage test_link_prediction[False] 77.198
reproducibility.test_graphsage test_nai[True] 53.483
layer.test_hinsage test_hinsage_save_load 45.713
reproducibility.test_deep_graph_infomax test_dgi[True-APPNP] 43.655
layer.test_deep_graph_infomax test_dgi[False-HinSAGE] 39.480
layer.test_graphsage test_graphsage_save_load 38.373
reproducibility.test_graphsage test_link_prediction[True] 36.478
test_ensemble test_evaluate_link_prediction 32.547
test_ensemble test_evaluate 32.126
test_ensemble test_ensemble_init_parameters 31.851
test_ensemble test_predict 27.299
layer.test_knowledge_graph test_save_load[RotH] 27.001
reproducibility.test_deep_graph_infomax test_dgi[True-GAT] 26.443
test_ensemble test_predict_link_prediction 23.212
test_ensemble test_BaggingEnsemble_fit 22.931
reproducibility.test_deep_graph_infomax test_dgi[False-GAT] 22.925
reproducibility.test_graphsage test_nai[False] 22.844

Clearly saving/loading a HinSAGE Deep Graph Infomax model is extremely slow, as are all of the reproducibility tests. This PR removes the HinSAGE DGI test save-load test, and reduces the repeats for the reproducibility ones to just two: the reference model and then one repeat. (The reproducibility tests are quite flaky at the moment, so it's kinda a waste to spend a lot of CPU time on them.)

@codeclimate
Copy link

codeclimate bot commented Jun 23, 2020

Code Climate has analyzed commit 6634af0 and detected 0 issues on this pull request.

View more on Code Climate.

@huonw huonw requested a review from timpitman June 25, 2020 02:33
Copy link
Contributor

@timpitman timpitman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks 4 to 5 minutes faster 🚀

@huonw huonw merged commit 59e3309 into develop Jun 25, 2020
@huonw huonw deleted the bugfix/reduce-test-time branch June 25, 2020 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants