Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support component integration tests #38

Merged
merged 1 commit into from
May 3, 2015

Conversation

ef4
Copy link
Contributor

@ef4 ef4 commented Apr 8, 2015

This implements template-driven component integration tests. See previous discussion in #25.

TestModuleForComponent now offers both a unit test mode (which is unchanged from the present behavior) and an integration test mode (which subsumes previous ModuleForIntegration).

The default is integration test mode, because I am 100% convinced that is the way most people should be testing most components most of the time. We switch to unit test mode if you provide an explicit needs argument, or say unit: true, or say integration:false.

In integration mode, tests are template-driven, giving you full and intuitive access to all features of components, including passing them block templates, yielding block params, and action handling.

People with existing component unit tests that already have a needs parameter (which I expect is most of them) will not be affected by this change. But people with component unit tests and no needs parameter will get failing tests, until they add unit: true to their module. This is a very low-cost upgrade pain, so I think it's worth it for achieving a simpler API that nudges people toward the right choice by default.

Edited to add: after feedback, I changed this to not be a breaking change. Unit test mode is still the default, but we will hit you with a deprecation warning unless you add an explicit needs or unit:true, which will protect you from a future change that makes integration mode the default.

I have tested this change with both ember-mocha and ember-qunit, and no changes are actually required in either of them to use this functionality. We will want to update their ember-cli blueprint generators however, so that they can generate both unit and integration tests for components, and so they place them under an appropriate path (like tests/integration instead of tests/unit).

this.container = isolatedContainer(this.needs);
var isolated = isolatedContainer(this.needs);
this.container = isolated.container;
this.registry = isolated.registry;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the isolated case consistent with the integrated case, which already provides this.registry.

@rwjblue
Copy link
Member

rwjblue commented Apr 8, 2015

I have a few tiny comments, but the approach looks good to me.

@ef4 ef4 force-pushed the component-integration-tests branch from 87b4972 to 0cec7a2 Compare April 8, 2015 01:24
@ef4
Copy link
Contributor Author

ef4 commented Apr 8, 2015

Based on feedback from @rwjblue I have made this a less jarring upgrade. For now we still default to unit test mode, but will fire a deprecation warning if you are in danger of getting the new behavior when we finally do switch the default.

@ef4 ef4 force-pushed the component-integration-tests branch from 0cec7a2 to 6e19bd1 Compare April 8, 2015 01:30
ef4 added a commit to ember-animation/liquid-fire that referenced this pull request Apr 8, 2015
We can merge this to master when my PR lands and releases. emberjs/ember-test-helpers#38
@rwjblue
Copy link
Member

rwjblue commented Apr 8, 2015

LGTM.

@dgeb - Any objections?

@dgeb
Copy link
Member

dgeb commented Apr 9, 2015

I was never crazy about the name TestModuleForIntegration, but it clearly provide a free-form template sandbox into which you can render any elements. You can integration test a single component or multiple components acting together. You can even test multiple configurations of the same component. None of that really goes away with these changes, but the naming gives the appearance that template-driven integration tests are tied to a single component.

This is not just an issue with component integration tests. The integration flag on our modules allows for sharing of an application instance's registry/container, but still leads the developer to focusing tests on a single thing: model, component or otherwise.

I think I'd rather see us introduce integration test modules that provide the semantics to test actual integrations. Perhaps IntegrationTestModule, ModelIntegrationTestModule, and ComponentIntegrationTestModule? The basic IntegrationTestModule could provide access to just an ApplicationInstance, the ModelIntegrationTestModule could also automatically provide a store, and the ComponentIntegrationTestModule could provide a free-form template. None of them would be tied to a single subject however.

@ef4 - I really appreciate the hard work you've done to enable these integration capabilities. I would like to make sure we expose these capabilities in the most useful light to users via the higher level libs (ember-qunit and ember-mocha) and through the blueprints in ember-cli.

@ef4
Copy link
Contributor Author

ef4 commented Apr 9, 2015

I'm not 100% clear on what your position is on this.

I realize "unit" and "integration" mean different things in different
places, and I don't necessarily care which interpretation we pick as long
as we're consistent.

But it is clear that testing against public API is better than testing
against private API. And the main public API components expose is via
templates.
And it is also clear that component tests -- even the existing ones -- are
actually testing integration between your component and many other moving
parts: htmlbars, ember, any built-in helpers or keywords, the long list of
things inside isolatedContainer. That is the kind of integration I'm
talking about.

If we continue to generate component "unit" tests by default when people
run ember g component, I think we are actively leading them down the
wrong path, by encouraging them to test against private API and by not
giving them easy access to all the template features like blocks and block
params.

@stefanpenner
Copy link
Member

in general i like to define various testing levels the following way:

  • unit: algorithmic complexity + no collaborators (public collaborators)
  • integration: low-level interaction between collaborators (mocks for foreign collaborators)
  • acceptance: black-box signals sent as the agent without internals leaking forward (exceptions include faking/mocking edge interfaces – adapters)

I believe components in ember, describe the JS/template that controls a chunk of template hierarchy. As such I do not believe today one can purely unit test this. It would have to be some adapted unit or more truthfully an integration mode of sorts.

We should explore a future where this is possible, I believe we can improve it, but pragmatically speaking the path of least resistance today (and short term) is likely more permissive component testing helpers.

This problem is shared by WebComponents and demonstrates that they aren't as isolated as people think. Their nested components are hidden, style and structure don't leak, but the implementation isn't actually encapsulated as at this phase, how to load/initialize and setup their descendants leaks forward.

a concrete example we can build a discussion around:

<my-cool-component>
  <!-- public and external -->
</my-cool-component> 
<!-- my-cool-component.html -->
<my-internal-component />
<input/>
<!-- my-internal-component.html -->
<my-other-internal-component />

When testing: my-cool-component we must introduce both my-internal-component and my-other-internal-component to the environment, or testing becomes impossible.

We do have the flexibility today to provide a mock my-internal-component that the my-component tester controls, but I believe the most common need we have is testing non-mock collaborators strung up in their natural template hierarchy .

@dgeb
Copy link
Member

dgeb commented Apr 9, 2015

@ef4 Sorry I was not clear. I am not disagreeing, but seeking consensus on what we consider integration tests to be in ember.

From a pragmatic standpoint, I agree that the kind of integration test in this PR should be the default for testing a component. As @stefanpenner points out, there is no way to purely unit test components in ember today.

My point above is that there is room for a more general purpose layer of "integration" tests here. Integration tests that are focused on testing the interaction of multiple components together, multiple models together, or perhaps other aspects of an application instance. TestModuleForIntegration provides a template-driven sandbox for this kind of general integration test, albeit with an awkward name.

So really I'd like to come to an agreement on semantics to allow for space for general integration tests. I raised this here because this PR removes TestModuleForIntegration.

I agree with @stefanpenner's take on testing levels. So how do we translate those levels to these unit-ish integration tests of components? In which directory are these component test blueprints going to be placed? They seem to be the duck-billed platypus of testing.

@ef4
Copy link
Contributor Author

ef4 commented Apr 9, 2015

Ah, I understand better now.

I guess we could still offer some more generic kind of integration test,
which would have pretty much the same implementation but a different name.

I'm hesitant though. More kinds of tests pushes more choice burden onto
users. Especially since we also still have acceptance tests.

In an Ember 2.0 world, everything is a component, so at every layer of your
app there is some component for which you can say moduleForComponent. I
think this necessarily covers all the multi-component interactions that
matter to your app, since they must occur within some enclosing component,
and that gives you a good place to put the integration test.
On Apr 9, 2015 5:19 PM, "Dan Gebhardt" notifications@github.com wrote:

@ef4 https://github.com/ef4 Sorry I was not clear. I am not
disagreeing, but seeking consensus on what we consider integration tests to
be in ember.

From a pragmatic standpoint, I agree that the kind of integration test in
this PR should be the default for testing a component. As @stefanpenner
https://github.com/stefanpenner points out, there is no way to purely unit
test
components in ember today.

My point above is that there is room for a more general purpose layer of
"integration" tests here. Integration tests that are focused on testing the
interaction of multiple components together, multiple models together, or
perhaps other aspects of an application instance. TestModuleForIntegration
provides a template-driven sandbox for this kind of general integration
test, albeit with an awkward name.

So really I'd like to come to an agreement on semantics to allow for space
for general integration tests. I raised this here because this PR removes
TestModuleForIntegration.

I agree with @stefanpenner https://github.com/stefanpenner's take on
testing levels. So how do we translate those levels to these unit-ish
integration tests of components? In which directory are these component
test blueprints going to be placed? They seem to be the duck-billed
platypus of testing.


Reply to this email directly or view it on GitHub
#38 (comment)
.

@ef4
Copy link
Contributor Author

ef4 commented Apr 15, 2015

I would like to help find consensus so we can land this feature.

It sounds like everyone supports the idea of TestModuleForComponent having an integration mode. The remaining discussion is around whether we should also have another kind of test that is more sandbox-like, without implying it's tied to a single component.

The two issues are separable. They got tied together because I eliminated TestModuleForIntegration -- which never managed to ship to users because we could never find a good UI for it. But to clarify: this PR does not foreclose the possibility of also offering more sandbox-like tests. In fact, I don't think such tests would even need to be implemented separately here in ember-test-helpers. TestModuleForComponent is a sufficiently general primitive that ember-qunit can use it to expose moduleForSnippet or ember-mocha could use it to expose describeScenario, etc. We are still free to design that API.

So I propose merging the uncontroversial feature (integration-mode for components), instead of waiting until we can design this other possible thing (sandbox tests). I suspect that design will not be trivial, which is why TestModuleForIntegration has been languishing.

return {
container: container,
registry: registry
};
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wary of changing the return value of isolatedContainer here, especially given the likelihood of the container/registry reform work proceeding very soon (emberjs/rfcs#46). In other words, I don't want developers who are using isolatedContainer to need to refactor twice in a short time. Maybe the question should be whether isolatedContainer is really used outside of ember-mocha and ember-qunit?

@ef4 @rwjblue thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that only ember-mocha and ember-qunit would be affected, but I could be wrong.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can just implement isolatedRegistry instead. isolatedContainer can keep the same public API, but get reimplemented in terms of isolatedRegistry.

(While also maintaining compatibility for old Embers that don't have a registry.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect! Thanks in advance @ef4 👍

@dgeb
Copy link
Member

dgeb commented Apr 15, 2015

@ef4 sorry for stalling on this. I am fine with merging once we resolve the isolatedContainer issue.

@ef4
Copy link
Contributor Author

ef4 commented Apr 15, 2015

No problem -- this is definitely an area where having a clear story is important.

@danfinlay
Copy link

Does this include scaffolding the needs relationships for model tests? I was referred here because I suggested that feature over at Ember-CLI.

@ef4
Copy link
Contributor Author

ef4 commented May 2, 2015

Finally got around to making the last requested change, so that isolatedContainer maintains the same public API. This should be ready to go.

@ef4
Copy link
Contributor Author

ef4 commented May 2, 2015

@FlySwatter I don't think you actually need this PR to get the behavior you want in a model test. You should already be able to pass integration: true to get a model test that doesn't need an explicit needs list, which is an appropriate way to integration test a model and its dependencies.

We do still need some clear documentation around that, and possibly some new blueprint generators.

@ef4
Copy link
Contributor Author

ef4 commented May 2, 2015

I suspect the can helper is getting manually registered in an application
initializer. Initializers don't run for these tests, and we don't really
want them to, because booting a whole app is more expensive.

That's one reason it's easier to stick with the naming conventions.
Unfortunately helpers without a dash are not auto discovered.
On May 2, 2015 5:31 PM, "James Alexander Rosen" notifications@github.com
wrote:

When using ember-can https://github.com/minutebase/ember-can's {{if
(can "view thing")}} helper in a component's template along with this
branch (and integration: true), I get

Uncaught Error: Assertion Failed: A helper named 'can' could not be found

Yet if I pause right before the this.render call in my component test,
then this.registry.lookup('helper:can') is the right helper!

All the other lookups seem to be working. It's possible that the problem
lies in how that library defines the helper. Or it could be the way this
isolated resolver works.


Reply to this email directly or view it on GitHub
#38 (comment)
.

@jamesarosen
Copy link

@ef4 great catch on the can helper. I may register some things myself in the tests. It's possible that many of my component tests should really be integration tests.

@stefanpenner
Copy link
Member

babel will likely start making it so that we can pre-compile easily via string templates: see: https://gist.github.com/stefanpenner/38a36298d04b29bbce5f

This would allow us to do this, without having to include the pre-compiler.

dgeb added a commit that referenced this pull request May 3, 2015
Support component integration tests
@dgeb dgeb merged commit 40ad0bf into emberjs:master May 3, 2015
@dgeb
Copy link
Member

dgeb commented May 3, 2015

Thanks for all your work (and patience) @ef4! :shipit:

@rwjblue
Copy link
Member

rwjblue commented May 3, 2015

@stefanpenner - That gist looks nice, but still seems that it would use runtime compilation (not build time). Unless you are saying there are additional hooks added in babel.

(we could of course write a babel transform that does this for us though)

@stefanpenner
Copy link
Member

@stefanpenner - That gist looks nice, but still seems that it would use runtime compilation (not build time). Unless you are saying there are additional hooks added in babel.

it would detect the string template handler, and pre-compile at build time. This allows valid language syntax to be used, but as an optimization (for both perf and dep size) it would be done at build time rather then runtime.

@ef4
Copy link
Contributor Author

ef4 commented May 3, 2015

@jamesarosen there does seem to be something wrong with the action handling. I would have expected your example to work. I will investigate further.

@ef4
Copy link
Contributor Author

ef4 commented May 3, 2015

I figured out the event handling problem. It's caused by doing render during the setup instead of during the test. If you move the render call into the test itself the problem goes away.

ember-qunit does a fairly wacky thing at the start of every test that breaks event handling for any view created before that point. So you can't render in setup/beforeEach.

@ef4
Copy link
Contributor Author

ef4 commented May 3, 2015

There is an existing ember-qunit issue about this behavior. https://github.com/rwjblue/ember-qunit/issues/86

@technomage
Copy link

I am all for this feature. I have borrowed @ef4 version from his liquid-fire repo and it is significantly better than the current unit tests for components.

Not having initializers run in general, I understand, but there should be a clean way to spin up ember data in this environment. When components replace controllers, there needs to be a way to test the component/data interface without reimplementing Ember Data for each test.

@mgenev
Copy link

mgenev commented May 4, 2015

@technomage

Not having initializers run in general, I understand, but there should be a clean way to spin up ember data in this environment. When components replace controllers, there needs to be a way to test the component/data interface without reimplementing Ember Data for each test.

I agree so much!

@pangratz
Copy link
Member

pangratz commented May 5, 2015

I played around with babel and created a babel-htmlbars-precompile plugin, which replaces a tagged template string with the precompiled HTMLBars template:

// babel-htmlbars-compiler/index.js
var compiler = require('./ember-template-compiler');

module.exports = function(babel) {
  var t = babel.types;

  return new babel.Transformer('babel-htmlbars-precompile', {
    TaggedTemplateExpression: function(node) {
      if (t.isIdentifier(node.tag, { name: "hbs" })) {
        var quasis = node.quasi.quasis;
        var template = quasis.map(function(quasi) {
          return quasi.value.cooked;
        }).join("");

        return "Ember.HTMLBars.template(" + compiler.precompile(template) + ")";
      }
    }
  });
};

which in turn can be used within Ember-CLI:

// Brocfile.js
var EmberApp = require('ember-cli/lib/broccoli/ember-app');

var app = new EmberApp({
  babel: {
    plugins: ['babel-htmlbars-precompile']
  }
});

module.exports = app.toTree();

and this allows to write for example a component test like this:

import { moduleForComponent, test } from 'ember-qunit';

moduleForComponent('my-component');

test('it renders', function(assert) {
  var component = this.subject({
    hello: "world",
    layout: hbs`huhu {{hello}}`
  });

  // Renders the component to the page
  assert.equal(this.$().text().trim(), "huhu world");
});

@stefanpenner is this something you had in mind with your gist https://gist.github.com/stefanpenner/38a36298d04b29bbce5f?

@rwjblue
Copy link
Member

rwjblue commented May 5, 2015

@pangratz - Perfect!

@pangratz
Copy link
Member

pangratz commented May 5, 2015

Alright, I've published the inline precompile stuff as EmberCLI addon: ember-cli-htmlbars-inline-precompile.

@ballPointPenguin
Copy link

Reading threads like this reminds me why I love the Ember community. 😍

seanpdoyle added a commit to seanpdoyle/ember-test-helpers that referenced this pull request Jun 8, 2015
The outcome of [emberjs#38][38] is undocumented. 

Raising a deprecation warning for calls to `this.subject` is a good start, 
but could be improved greatly.

[38]: emberjs#38
@Emerson
Copy link

Emerson commented Jul 13, 2015

Would love to see a few robust examples of this in action. I'm trying to do some pretty basic stuff and I'm not sure the best way of approaching the problem. As a simple example:

test('submit-button can be disabled', function(assert) {
  this.render(hbs `{{submit-button disabled=true}}`);

  assert.ok(this.$('button:disabled').length, 'It rendered a disabled submit button');
 // Great! This works.

  // Now I want to change the disabled property to false and assert that the button 
  // is no longer disabled. What's the best way of doing that?
});

I'm sure there is an easy answer to this question (just call render again?), but some more real world examples would be useful for those of us trying to migrate our tests over to this style.

Also interested in how we pass more complicated data into our components (arrays, objects, etc...).

Love the PR - thanks!

Update: rerendered the component and it works, but is that a new instance of the component? Or rerender of the original...

@rwjblue
Copy link
Member

rwjblue commented Jul 13, 2015

This should do the trick:

test('submit-button can be disabled', function(assert) {
  this.set('currentDisabledValue', true);

  this.render(hbs `{{submit-button disabled=currentDisabledValue}}`);

  assert.ok(this.$('button:disabled').length, 'It rendered a disabled submit button');

  this.set('currentDisabledValue', false);

  assert.equal(this.$('button:disabled').length, 0, 'It switched to disabled');
});

Links:

@Emerson
Copy link

Emerson commented Jul 13, 2015

Ahh - makes good sense! Thanks @rwjblue

seanpdoyle added a commit to seanpdoyle/ember-test-helpers that referenced this pull request Dec 9, 2015
The outcome of [emberjs#38][38] is undocumented. 

Raising a deprecation warning for calls to `this.subject` is a good start, 
but could be improved greatly.

[38]: emberjs#38
seanpdoyle added a commit to seanpdoyle/ember-test-helpers that referenced this pull request Dec 9, 2015
The outcome of [emberjs#38][38] is undocumented.

Raising a deprecation warning for calls to `this.subject` is a good start,
but could be improved greatly.

[38]: emberjs#38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.