ember data testing helpers

[fivetanley]

No description provided.

[igorT]

Where does this.server come from? Is it just a mock api? Hopefully using pretender/etc have much nicer apis than this

[fivetanley]

this is how you override with pretender, and sinon. ic ajax has defineFixture but it is often not granular enough. this.server here is a new Pretender

[kategengler]

I'm super excited by the potential of having these helpers.

As an example from the wild, in the test suite on my main app, we extensively create records in the store rather than mock api requests. We do this to avoid testing ember-data and to also make writing tests very simple.

We use this helper:

Ember.Test.registerHelper( 'create', function ( app, objType, properties ) {
  Ember.run(function () {
    properties = Ember.$.extend({ id: createGuid()}, properties );
    store( app ).push( objType, properties );
  });
  return store( app ).getById( objType, properties.id );
});

Combined with several factories for our different models. We're working to refactor our app and tests to share the same factories.

The only places we mock the actual API requests is for places we save or reload models.

Here is an example of one of our tests:

test( 'it shows the current monthly amount when upgrading plans', function () {

  var subscription = make( 'subscription', { planInterval: 'annually', planName: 'Basic Plan (annual)', planAmount: 180 } );
  make( 'organization', { subscription: subscription });

  visit( '/settings/plan' );

  andThen(function () {
    assertExists( ".spec-current-price:contains('$15')" );
  });

});

make is a wrapper around create that uses the appropriate factory based on the model type we are creating.

[mitchlloyd]

Forgive me if I'm way behind the curve here. I figure I'm just missing a lot of history and context.

Why is this:

getRecordById('user', 1).set('hasAccess', true);

Better than this?

store.getById('user', 1).set('hasAccess', true);

In other words, why do we need a new API layer to interact with the store in tests?

I've read your example code, and it seems to go from "mocking ajax requests is noisy" to "interacting directly with the store is cleaner". Maybe another example would help others less in the know understand.

[fivetanley]

My main concern would be with managing the store. We can't define a store property with dynamic getter (that could instantiate the store if it did not exist and throw errors outside a test) on window because we have to maintain compatibility with ES3 environments. So when does the store get set up? Who sets it up?

having helpers methods that know how to find the store seems easier than trying to create this static "store" property on window, but I could be wrong. We could also make you call store() as a function every time but that feels a little off too.

[mitchlloyd]

Thanks, I understand the concern better now.

Running off of your example code, isn't the store setup when the application is created? In today's integration testing we do something like this:

var App;
module('An Integration test', {
 setup: function() {
    App = startApp();
  },
  teardown: function() {
    Ember.run(App, App.destroy);
  }
});

After that, using the forbidden __container__ seems to work fine:

var store = App.__container__.lookup('store:main');

And then we can go on our merry way using the store. In this case, the main question is "how do we easily get access to the store?" I've been doing something like: lookup(app, 'store:main') but I'm sure someone could think of something nicer. @kategengler's store(app) looks good to me.

I suppose this.store() is what test helpers already use, but if this is an integration test and the store is already setup, I don't see the harm in just setting store on the test context if convenience is the goal.

In any event I'd rather call a function like store() than use a special API for the public methods on store.

Things get a little more challenging when we're not starting and resetting the app and instead we're doing some kind of integrated unit testing with the store and models. I think we need the store to create a model at all, right? The ember-test-helpers give us test-module-for-model which does all the work to get a store in place for model testing. Things like moduleForModel in ember-qunit do have a this.store() available.

For this case I'd rather ask "how can I create a model without the store?" rather than "how can I get an instantiated store right now?" I've got a friend who's always telling me that if something is hard to test, maybe that's because my code is hard to use. Is it weird that the creation of an object is so coupled to a Service (or soon to be Service) like the store? I actually would like a public API to create a model outside of the store so that I could choose push it onto the store later or just throw it away. This covers an issue that I see often where people create an unsaved model for a form, have the user hit the back button, and then see this blank model in a list. Also testing just got a lot easier.

If you made it this far thanks for reading :) Kind of talking out of my ass at the end there.

[stefanpenner]

assumes a single store

[fivetanley]

is multiple stores a real thing people do? why would you want to do it?

[pjmorse]

I've done that. The app had a common data set universal across the app (think reference articles) which was delivered as fixtures at app load time, and user-specific data which was managed with an API adapter (think margin comments).

[stefanpenner]

@fivetanley more then 1 app on the page.

[kategengler]

@stefanpenner In the multiple apps on the page case, wouldn't these helpers be able to get the store that belongs to the particular app easily enough? Helpers registered with registerTestHelper or registerAsyncHelper are passed the app as an argument.

[mitchlloyd]

Also the RFC is talking about replacing the Fixture Adapter and avoiding something like Pretender, but the example shows a case where neither would be needed because the store is going to hit the cache. What happens when the store tries to reach out to the network? This is where Pretender would actually come into play.

I can see why FixtureAdapter isn't a real solution for integration tests and it seems reasonable to remove it, but I just don't understand the purpose of these helpers in the context of this proposal yet.

[fivetanley]

What happens when the store tries to reach out to the network? This is where Pretender would actually come into play.

Totally. @igorT brought this up over IM. It sounds like we could rewire internals to make a helper more useful. Right now, it seems like integration testing while stubbing out XMLHttpRequest may be the best option as an interim solution.

If store.push had more hooks like, being able to tell you a record was deleted by the server, etc, Ember Data would be more awesome as well. I think that's what @igorT was bringing up in our IM conversation.

[xcambar]

I'm throwing my (quite long) 2 cents here because I've had to address some of the issues pointed out by @fivetanley in this RFC and wanted to share some thoughts we've had in my company.

Some context and opinion:

Fixtures should not be actual payloads, they should represent your data model

We're using our Ember app so QA, Sales or whoever else can use the app offline and with different datasets. We found it much easier to generate JSON datasets than having to start a dedicated instance of our server with a specifically filled database. Be it for development, testing or demo, a use case is just a JSON away.
But for non-API-oriented people, it's much easier to define a (small) "world" than payloads. And it makes the fixtures actually much more readable.

To address this, we added to FixtureAdapter the wrapFixture method (to be extended for your own purpose) so it is the responsability of your (extended) FixtureAdapter to build the payloads for the requested Fixtures. A bit of a Serializer work, but it can and should be refactored anytome soon.

Existing fixtures should be immutable (loading should be idempotent)

@fivetanley is absolutely right on this one. It's a real pain to see the fixtures modified by Ember Data.
So we fixed it in our fork, just to be sure that 2 consecutive loads return the same data. As an aside, we didn't like/want to define our fixtures in App.Model.FIXTURES so we PR'd it. Small win, but win nonetheless.

The bigger win is that with a dedicated getter/setter instead of a reference to an object, we can generate (and control) our fixtures, simulate updates of the database and so on...

You should not have to fixture the whole world or should be in the know when a relationship is missing

It's the counterpart of not providing payloads: you can miss data. We added a fixtureNotFound method that allows you to know precisely what's required to your app but missing, and potentially build it with your preferred factories/generators.

FixtureAdapter should help you build payloads

Most of the work is related to building the actual payload. As previously stated, it requires a Serializer of some sort to build the payload for the requested data (which is handled by wrapFixture).

The most interesting part is that now, in the environments where FixtureAdapter is in play, the app continues to use our production {Application,Model}Serializer, which is impossible with the current FixtureAdapter. Needless to say, we feel much more comfortable this way.

I hope it was not to long nor boring nor pointless, but as glad as I am to see something is happening to FixtureAdapter, I thought some more input could hopefully be useful.

The code is located here.

[michaelrkn]

One thing I've struggled with is how to write good integration tests for error states. For example:

test('it displays errors if the input is invalid', function() {
  visit('/posts/new');
  fillIn('input.title', 'My new post');
  fillIn('input.body', 'too short');
  click('button.submit');

  andThen(function() {
    assertExists(".error");
  });
});

I hope that any new approach handles this type of case.

Here's a suggestion of how it could look:

test('it displays errors if the input is invalid', function() {
  store.stub('post', 'create', { errors: { body: ['is too short.'] } });

  visit('/posts/new');
  fillIn('input.title', 'My new post');
  fillIn('input.body', 'too short');
  click('button.submit');

  andThen(function() {
    assertExists(".error");
  });
});

Another case could be updating an existing record:

test('it displays errors if the input is invalid', function() {
  store.createRecord('post', { title: 'My new post', body: 'This is a great post.' });
  store.stub('post', 'update', { errors: { body: ['is too short.'] } });

  visit('/posts/3/edit');
  fillIn('input.body', 'This fails server-side length validation.');
  click('button.submit');

  andThen(function() {
    assertExists(".error");
  });
});

Or, it may make more sense to set the desired outcome on the object itself:

test('it displays errors if the input is invalid', function() {
  var post = store.createRecord('post', { title: 'My new post', body: 'This is a great post.' });
  post.stub('update', { errors: { body: ['is too short.'] } });

  visit('/posts/3/edit');
  fillIn('input.body', 'This fails server-side length validation.');
  click('button.submit');

  andThen(function() {
    assertExists(".error");
  });
});

By default, the stub could be a successful server response, eliminating the need to specify the response in most cases:

test('it displays a success message if the input is valid', function() {
  visit('/posts/new');
  fillIn('input.title', 'My new post');
  fillIn('input.body', 'This is a great post.');
  click('button.submit');

  andThen(function() {
    assertExists(".success");
  });
});

[fivetanley]

I think @samselikoff's ember-cli-mirage ended up working better for us in the long term. I don't have the bandwidth to get this RFC over. If someone else wants to do this, they could make an addon or open a new PR by copying/pasting/editing this RFC.

[samselikoff]

By the way it is in my roadmap for Mirage to add the ability to generate ED models from your Mirage factories in integration & unit tests. Hopefully that will address some more of these points.