New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Significant start slowdown on MSVC x64/Debug after upgrade to v1.68.0 #163

Closed
k15tfu opened this Issue Sep 17, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@k15tfu

k15tfu commented Sep 17, 2018

Hi!

I'm experiencing a very bad problem: I have a one boost test executable which contains over 50.000 test cases (data test cases mostly), and after upgrade to the latest Boost.Test the startup time has significantly increased from <3sec up to ~7min.

@raffienficiaud

This comment has been minimized.

Member

raffienficiaud commented Sep 17, 2018

Can you post the test module (at least the init part)? I'll have a look

@k15tfu

This comment has been minimized.

k15tfu commented Sep 17, 2018

main.cpp

// BOOST_TEST_MODULE should:
// 1. be defined before any inclusion directive to Unit Test Framework headers
// 2. be defined exactly for one compilation unit of your test module
#define BOOST_TEST_MODULE Common

// Include header for `BOOST_TEST_MODULE`.
#include <boost/test/included/unit_test.hpp>

In other files we just include boost/test/unit_test.hpp and then define test cases by BOOST_AUTO_TEST_CASE. After that the main.o is linked together with foo_test.o (& Co.) into the mentioned .exe file.

@ww898

This comment has been minimized.

ww898 commented Sep 17, 2018

Hi there,
I have the same issue in my project. My investigation shows that method traverse_test_tree() works very slow https://github.com/boostorg/test/blob/develop/include/boost/test/impl/framework.ipp#L1199. Most time it spends in test_suite::add() on BOOST_TEST_SETUP_ASSERT() https://github.com/boostorg/test/blob/develop/include/boost/test/impl/test_tree.ipp#L248. The startup time increased in 2.5 times to 20min.

Another issue is test filters apply after collecting all tests. It means that command line option --run_test can't decrease the initialization tests time.

@raffienficiaud

This comment has been minimized.

Member

raffienficiaud commented Sep 17, 2018

@ww898 I see now why it is so. Out of curiosity, how much time does it take in release?

@ww898

This comment has been minimized.

ww898 commented Sep 17, 2018

@raffienficiaud Well, I measured the execution time for one test which was chosen by the command line option --run_test:

MSVC v14.13.26128 boost v1.66.0 boost v1.68.0 in times slower
Debug 55s,048ms 20m:50s,339ms 22.71
Release 34s,174ms 1m:43s,055ms 3.01
@raffienficiaud

This comment has been minimized.

Member

raffienficiaud commented Sep 17, 2018

Thanks! That is quite a lot indeed!
I delayed the evaluation of the duplicate test cases, which seems to be the hotspot. It is still under tests, but you may want to check the branch topic/GH-163-slowdown-at-startup .

@k15tfu

This comment has been minimized.

k15tfu commented Sep 18, 2018

Thank you guys for help!

@ww898

This comment has been minimized.

ww898 commented Sep 18, 2018

@raffienficiaud Thank you very much for topic/GH-163-slowdown-at-startup branch:

MSVC v14.13.26128 boost v1.66.0 boost v1.68.0 boost v1.68.0 + branch
Debug 55s,048ms 20m:50s,339ms 53s,500ms
Release 34s,174ms 1m:43s,055ms 34s,909ms
@raffienficiaud

This comment has been minimized.

Member

raffienficiaud commented Sep 18, 2018

Thanks for testing! I'll merge once I fix this visibility issues

raffienficiaud added a commit that referenced this issue Sep 23, 2018

Merge branch 'topic/GH-163-slowdown-at-startup' into next-internal
* topic/GH-163-slowdown-at-startup:
  Change log
  Delaying the check on duplicate names during the test initialization

raffienficiaud added a commit that referenced this issue Sep 24, 2018

Merge branch 'topic/GH-163-slowdown-at-startup' into next-internal
* topic/GH-163-slowdown-at-startup:
  Change log
  Delaying the check on duplicate names during the test initialization
@raffienficiaud

This comment has been minimized.

Member

raffienficiaud commented Sep 24, 2018

Merged to develop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment