Introduced a smoke testing phase on Travis to run SQLite tests and CS checks first

[morozov]

Sometimes, the CS verification is the only failing stage of the build (especially, if CS is not verified locally before submitting the patch). In this case, after fixing the CS issue, it'd be very convenient to get immediate feedback instead of waiting 15-20 minutes until all test builds finish first.

A solution could be to introduce a "smoke testing" stage which would precede the testing one and include the following:

1. Testing against SQLite to find non-platform-specific issues since it's the fastest platform.
2. Code style verification.

[Majkl578]

Now, where would we put PHPStan and Roave BC whateveritscalled checker?

[morozov]

@Majkl578 depending on how important and fast they are — either at the beginning of the build or at the end. If they are fast and required, they will go into the smoke testing since there's no point in checking platform compatibility if the build is not going to be accepted due to quality or BC concerns.

[Majkl578]

One more thing: I'm afraid this will cause issues with Scrutinizer. When the initial stage fails, further coverage runs will not be executed, thus Scrutinizer will be waiting for them infinitely (well, until timeout = 3600 seconds). :/ Not sure how to handle this.

[morozov]

Not sure how to handle this.

Do we need to handle this? If the build fails, the coverage is irrelevant. Isn't it the same as a test run fails and doesn't send any coverage results to Scrutinizer?

[Majkl578]

It will block all other builds for an hour.

Isn't it the same as a test run fails and doesn't send any coverage results to Scrutinizer?

Yes, something to still take care of, see the Tip here: https://scrutinizer-ci.com/docs/tools/external-code-coverage/#changes-in-third-party-configuration

[morozov]

So, it looks like at the end of the code style check job, it if fails, we need to call ocular which will cancel the build on Scrutinizer. Gitter seems down.

[Majkl578]

Yes. We are currently doing that (in after_script): https://travis-ci.org/doctrine/dbal/jobs/373140364#L1014
I am just not sure whether one call like that is sufficient when 17 is expected. I guess we will have to try or we will find out. :)

Gitter seems down.

Yes. 😢 Slack is up though. \o/

[morozov]

We still can solve this problem if we collect coverage only nightly. Coverage is not taken into account to validate the pull request and is not displayed anywhere on GitHub.

Usually, we do a decent job by requiring the needed tests from contributors. And if we didn't, then instead of just a metric, we'd need something better like "differential coverage" to make sure all code changes are covered by tests.

[morozov]

While the way to solve this is still unknown and it's not a high priority, sometimes it's just annoying. You need to wait for 15 minutes and burn 2 machine-hours on Travis alone to see that you have a code style violation.

[morozov]

I am just not sure whether one call like that is sufficient when 17 is expected. I guess we will have to try or we will find out. :)

@Majkl578 the Scrutinizer support said that once the platform receives an ocular call without a file, it recognizes it as a test failure and will proceed without code coverage.

[Ocramius]

👍