[Enhancement] Ember-try & parallel travis-ci scenario tests for addons

[mike-north]

Thanks to @kategengler's add-on, inspired by @ef4's work for liquid-fire, we can easily test apps and addons against multiple versions of ember.js

This PR enhances the existing ember-try support for addons with a default configuration for testing against:

• 1.10 "default" -- may require some modification to ember-try
• components/ember#release
• components/ember#beta
• components/ember#canary

Additionally, the default travis-ci configuration will test these scenarios in parallel, while allowing builds to pass, even if tests fail using ember-canary. Benefits I see in this are:

• Developers get feedback on specific builds easily
• Test jobs are potentially run in parallel, depending on travis-ci worker node availability
• Apps and addons are tested against canary by default, with no downside to the developer. Build times are no longer than they otherwise would be, test failures in canary are tolerated
• Using the component/ember channels for scenarios instead of numbered versions allows the release team to get feedback on new releases (including canary builds) without any action required by addon and app developers

TODO

• Updates to .travis.yml
  • Rename environment variable to EMBER_TRY_SCENARIO
  • script: ember try $EMBER_TRY_SCENARIO test
  • Append build matrix
    • Append env matrix
  • Ensure that config/ember-try.js is in place for addons

[mike-north]

Including this scenario feels like ember-cli would have the opinion of encouraging 1.10 compatibility. I'd like some specific feedback on whether people feel this is appropriate

[rwjblue]

This seems somewhat odd to me, why would we advocate for testing an app against multiple versions of Ember?

This change makes makes each version tested in separate Travis builds (which is nice for sure), but we are already testing with ember try:testall by default for newly generated addons (via the npm test script).

[mike-north]

Ah, ok. I'll go ahead and update this so that it only affects addons. There's still a big advantage to running individual ember try $SCENARIO test in parallel as opposed to a single serial critical path.

[rwjblue]

I think we should stick to just checking the channels and the "default" (which may require some tweaks in ember-try).

Also, can we make the environment variable EMBER_TRY_SCENARIO (to me a bit more explicit)?

[rwjblue]

I submitted ember-cli/ember-try#21 to make the default config more in line with this.

Could you add the "default" scenario in the config/ember-try.js (like from my PR above) instead of ember-1.10?

[rwjblue]

This seems somewhat odd to me, why would we advocate for testing an app against multiple versions of Ember?

Welp, apparently it helps if I read the nice description you wrote. Sorry about that.

---

I left an inline comment/tweak, but other than that I am 👍 (and sorry for not paying close enough attention).

[alexdiliberto]

Overall 👍 for Addons, However I think opting-in to testing against Canary may be a little much. I would go with Beta and Release...I could be in the minority though

[mike-north]

@alexdiliberto would you want to test your addon against canary if build times were no longer, and it didn't get in the way of your development workflow?

[rwjblue]

I think we should definitely encourage (with allowed failures) tests against canary. It would serve to give you advanced warning of upcoming issues that you will face, but I'm not 100% opposed to testing apps only against release and beta channels (but keeping addons with all three).

[alexdiliberto]

I'm definitely on board with the premise here, @rwjblue You mentioned allowed failures, I didn't consider that, I think that would work well here. I didn't want everyone to go crazy when their tests fail for some little thing on canary and travis kept complaining

[stefanpenner]

Im +1 for add-ons.

I would love to consider this for apps as well, it would be valuable to see if you app works on the current version, the latest beta and maybe the latest canary? Of-course this seems like an opt-in thing? But it may help catch issues early, also help developers future proof there apps.

[mike-north]

ok this is working now, with the exception of:

• ember-try understanding the default version

[rwjblue]

See my example in the referenced PR you m can implement the ember-try config file now.

[rwjblue]

TL DR add default with no dependencies in the config file.

[mike-north]

Thanks @rwjblue. Just ran it locally and all scenarios are run and complete successfully. Travis works properly with the generated .yml file, so I think we're good to go

[mike-north]

PR build was failing due to stale npm cache, and is passing now

[rwjblue]

👍

[stefanpenner]

rad :)