Consider adding Chrome stable to Travis matrix

[patrickhulce]

Context: Recently CSS usage broke Lighthouse master on Chrome stable since the protocol method wasn't added until v56 (#1473).

It would be great to have Travis automatically check against these types of errors. Not sure how we would want to handle the failures/tests though since we would still want to ensure the smoke tests for canary were run.

[brendankenny]

Simplest solution: smoke tests take a combined ~1.5 minutes on travis right now. We could easily download Chrome latest and Chrome stable and just double that up. Obviously that's a lot of duplication of work, though, and only going to get more painful as the smoke tests grow.

[patrickhulce]

Oh @brendankenny I took that as a given, I meant more how do we handle the failures that result because stable lacks the feature support. Example: the CSS usage gatherer will fail on stable as expected and set a debug string which won't pass the expectations file. Should we maintain separate expectations, mark as OK to fail, etc?

[wardpeet]

what do you guys think about 2 travis checks? I for linting/unit tests, ... & 1 for smokehouse (stable and canary)?

[brendankenny]

for these differences between stable and canary, I think we're just going to have to go back to being stricter about not landing new audits or driver commands until supported by Chrome stable, rather than having a different expectation for each version.

This will still be a bit hard with audit changes due to Chrome changes, e.g. new deprecation warnings coming in Canary (one of the errors in the log in #1573 (comment))

[patrickhulce]

fixed by #2547 ! 🎉 moving forward we'll probably want to stick to tests that only pass in stable

[XhmikosR]

Cool, I'll add it in my long to-do list to update AppVeyor soon-ish On Jun 20, 2017 22:13, "Patrick Hulce" <notifications@github.com> wrote: fixed by #2547 <#2547> ! 🎉 moving forward we'll probably want to stick to tests that only pass in stable — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1480 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAVVtXkk4l5NxnSLpGESIA87UuyGEuuxks5sGBnBgaJpZM4LmAP8> .