Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Limit the number of concurrent tests in integration.go #6655
Not always. In one run it had
In general, we use
Yes, it's not feasible to detect the load on the machines, and it varies. I was suggesting that maybe we could pass a fixed number for travis/shippable configuration, but allows developers to use as many cores as their machine have locally. I'll test with a fixed number locally and measure the performance difference. If it's okay, I'll just set it to a static number.
Apr 10, 2015
referenced this pull request
Apr 10, 2015
This goal of the PR is not to reduce the running time (latency), and it would not. It is meant to reduce the load on system since all the tests are sharing the same apiserver, scheduler, etc (assuming shippable/travis wouldn't schedule more tasks onto one machine). If we keep adding tests, then we'll have to scale the timeout values along with it, which is a pain. I think we should have a separate performance suite similar to our e2e to track performance regression. Integration tests should focus on testing the correctness because (1) we don't control the testing infrastructure (2) it is too expensive as it runs for every PR and a lot of time and efforts are wasted in triaging the failure (3) even if we can catch some bugs, it is hard to reproduce the case locally (4) it makes people ignore the tests results.
That said, it is possible that there is a performance regression (or even deadlock) causing the timeout problem. The PR was to stop wasting all developers' time. I will file an issue for the potential performance problem. Feel free to chime in or revert this PR if you think having all the tests running is more valuable :)