MINOR: Drop wasteful 'allow-failures' builds; simplify build matrix #4556
Conversation
Our build time is an impediment: frequently we're slowed down waiting for test runs to complete. In this context, running allowed-failure builds that everyone ignores is just a waste. I'd love to see HHVM work, but that's going to require that someone actually gets the build to pass. At that point, we can add it back into the build matrix. I'm working on a PHP7 branch that will hopefully also fix our nightly builds. Once I get that going, I'll add those two back, but not have 'allow failures' Even if we decide, at some point in the future, that supporting nightly doesn't make sense, I see the *only* value of Travis is maintaining our green tick. We either care about builds or we don't: I've never seen a 'grey area' build result in anything other than embarrassment and technical debt. Let's decide what matters to us and treat it like it matters to us: it's part of the build or not.
|
Yeah, unless we get clear metrics about number of failed tests in allowed_failures, I'm inclined to agree. Number of failed tests over time could prevent us from breaking builds on HHVM even more than already, but that's not an easy metric to track from Travis. I don't think its particularly wasteful in terms of our overall build time since Travis runs all builds concurrently. But it's wasting Travis' server resources. It's a free resource for us, but we should care about the sustainability of Travis anyway :) |
I've heard about this in theory. In practise I've only seen it work when there's a dedicated test manager (Phil B), and even then it was a soul-destroying uphill battle. Note: I'm inclined to take the same approach with stuff like trailing whitespace: fix it all and then treat any violation as a build failure. |
It's important to test different databases. It's important to test different versions of PHP. But we don't need to test all possible versions of each tested in a cartesian product. This patch ensures that there are 4 builds that cover: - PHP 5.4, 5.5, 5.6 - MySQL, PostgreSQL and SQLite - PDO and each db-specific connector - Behat and non
sminnee
changed the title
MINOR: Drop wasteful 'allow-failures' builds; simplify build matrix
Our build time is an impediment: frequently we're slowed down waiting for
test runs to complete. In this context, running allowed-failure builds
that everyone ignores is just a waste.
I'd love to see HHVM work, but that's going to require that someone
actually gets the build to pass. At that point, we can add it back into
the build matrix.
I'm working on a PHP7 branch that will hopefully also fix our nightly
builds. Once I get that going, I'll add those two back, but not have
'allow failures'
Even if we decide, at some point in the future, that supporting nightly
doesn't make sense, I see the only value of Travis is maintaining our
green tick. We either care about builds or we don't: I've never seen a
'grey area' build result in anything other than embarrassment and
technical debt.
Let's decide what matters to us and treat it like it matters to us:
it's part of the build or not.