Ensure that we don't kick off SolutionCrawler's Reanalyze functionali…

[mavasani]

…ty if solution crawler is turned off with the option InternalSolutionCrawlerOptions.SolutionCrawler.

This is needed for our internal RPS performance tests which intend to turn off solution crawler for certain scenarios to reduce noise from it.

[JoeRobich]

LGTM

[genlu]

Are we disabling solution crawler in RPS?

[CyrusNajmabadi]

Doesn't this make RPS results very suspect? i.e. tey could show 'all clear', but the real state of hte system could be "massive regression" depending on the impact of solution crawler...

[sharwell]

@CyrusNajmabadi The RPS tests are not performance tests. The are functional tests covering a subset of base cases users are likely to observe the first time they use the IDE on small projects. If you treat them as product performance tests, the gaps are more chasm-like.

Solution crawler is not disabled for the manual performance/scaling tests.

[CyrusNajmabadi]

The RPS tests are not performance tests.

That's the first time i've ever heard that :) It's always been pitched as a 'performance regression system' to me.

[mavasani]

Tagging @MichaelMaio to add more context.

[genlu]

The RPS tests are not performance tests.

That's the first time i've ever heard that :) It's always been pitched as a 'performance regression system' to me.

I guess this depends on whom you ask :P

[MichaelMaio]

RPS stands for "Regression Prevention System". We do use it for performance testing. We turned off Solution Crawler in RPS because it seemed to cause significant noise in the tests, making it difficult to notice regressions in other features. However, the new batch of RPS tests (aka "modern tests") have a new SolutionCrawler test that explicitly turns that feature on so we will be testing VS with it enabled, and so can catch regressions that exceed the noise level.