Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

End-to-end testing #255

Closed
sindresorhus opened this Issue Aug 2, 2012 · 71 comments

Comments

Projects
None yet
Owner

sindresorhus commented Aug 2, 2012

As discussed in #200 and #134. It would be useful to have some automated tests contributors could run to make sure their submission adhere to our spec.

I have to be honest. I really don't like Selenium. The last time I used it, it was painfully slow and hard to work with.

I've found a project called Casper.js which scripts PhantomJS to do various stuff, including Selenium like testing. This looks like exactly what we need. I'm going to research this further, but just wanted to throw it out there. Thoughts?

Side question: What is the benefit of having End-to-end testing if every new framework is required to come with unit tests?

Owner

addyosmani commented Aug 2, 2012

The only distinction I can see between end to end tests and unit tests is:

  • End-to-end tests allow us (the project owners) to know that all of our applications conform to our specs
  • Unit tests allow developers to see how unit tests for applications written using a framework can be put together

I have to say, I'd almost prefer that we had some sort of framework agnostic app spec that we could write which the applications has to pass but also which demonstrated unit tests sufficiently.

Owner

sindresorhus commented Aug 2, 2012

End-to-end tests allow us (the project owners) to know that all of our applications conform to our specs
Unit tests allow developers to see how unit tests for applications written using a framework can be put together

Thought about this a bit. The obvious reason for doing end-to-end testing, even though apps should come with unit tests, is that submissions may contain faulty unit tests.

I have to say, I'd almost prefer that we had some sort of framework agnostic app spec that we could write which the applications has to pass but also which demonstrated unit tests sufficiently.

This was discussed in #134. There was some concern about it not being agnostic enough.

marcenuc commented Aug 2, 2012

Has anyone tried Selenium2? It uses WebDriver API, which is becoming a W3C specification. This new API is much nicer than the obsolete RemoteControl API, and speed should be much better, because it works by making direct calls to the browser using each browser’s native support for automation.

Bindings are available for all major languages, including Javascript.

I dont know if there are better solutions, but I think that whatever solution you will choose, it must have similar features:

  • completely implementation agnostic, because the same exact specs must be run for every implementation. This is needed to ensure that all implementations have the same (minimum) specified behaviour. Integration and unit tests will differ between implementations, then they can't be used.
  • work on all major browsers (even tablets).

I'm interested in working on some end to end tests for TodoMVC, possibly in multiple implementations (selenium, casper.js, etc), depending on interest levels.

Before I dive in I have a few questions about the end goal of these tests and the intended usage.

First, the proposed solutions of selenium and casper js are typically used for automated testing, meaning they are run by an external continuous integration server whenever a code change is detected. In practice, I'm not sure how this works for open source projects such as TodoMCV? Who hosts this server and how would contributors interact with it before submitting their Todo Apps? Alternatively, contributors could run their own instance of selenium or phantom js locally or on a remote server like Amazon ec2. In my opinion, this is a lot to ask of an app contributor.

Another option is that there is one central end-to-end testing server and you allow contributors to submit an app without testing it first and then once submitted it is tested against the end to end tests.

Lastly, we could create an end-to-end test that runs in the browser and performs simple conformance tests like validating the html to make sure that it matches the template, adding a todo, deleting a todo, etc. This would need a bit more thought and conversation to see if this would be sufficient. At least contributors could run the end-to-end test in their browser without requiring a separate test server.

Thoughts?

Owner

addyosmani commented Aug 18, 2012

@colinjschmidt First, thanks for your interest in helping with the project :)

Regarding your first point about contributors needing to setup their own instance of selenium/phantom locally or on a remote server, if we did end up going for a selenium/casper setup, we would probably need to provide a server of some sort ourselves and manage access.

Your second suggestion about a central end-to-end testing server where contributors could submit an app without testing is something I like the idea of. If we're asking people to follow our specifications and perhaps write unit tests, this process will just serve as validation that they're really done everything we've asked and there's (hopefully) not much more to ask of them. What would be needed from our end to get something like this setup?

The last suggestion of being able to run tests in the browser for conformance might be the most ideal from an app authors perspective, strictly because they can verify everything prior to sending over a pull request. Are there any benefits to your second option (which would require a centralized server) and the last option?

@addyosmani Thanks for the reply. I agree with your recommendations.

I think there is a benefit of having both an external integration server and a browser based smoke test that app authors could easily run before submitting a pr.

The integration server can run end-to-end tests on all todo apps, as well as things like jshint to ensure that coding conventions are followed if that is also desirable. The nice thing about the integration server is that tests would be rerun automatically whenever there was a new code commit.

I'm going to spend a little bit of time on this today to see if I can get a prototype working.

Owner

addyosmani commented Aug 18, 2012

Thanks @colinjschmidt :) I look forward to hearing how the prototyping goes.

Owner

sindresorhus commented Aug 18, 2012

selenium or phantom js locally or on a remote server like Amazon ec2. In my opinion, this is a lot to ask of an app contributor.

Selenium yes, but PhantomJS is really easy to install. brew install phantomjs.

I think there is a benefit of having both an external integration server and a browser based smoke test that app authors could easily run before submitting a pr.

Would this be the same tests, only the former is run in PhantomJS while the latter in browser?

@sindresorhus - Yes, ideally these should be the same tests run through PhantomJS in a CI environment or in the browser for app authors.

I spent some time today looking at how testing was accomplished for Modernizr. I understand that this is a little bit different since we're dealing with apps instead of a library, but I think we can employ a similar solution.

Modernizr uses the qunit test framework in combination with PhantomJS. Also, it is taking advantage of http://travis-ci.org for automation.

I'm proposing a solution that uses Jasmine tests (with the HTMLRunner) to output the test results to the browser. In addition, we could use travis-ci with PhantomJS to run the same tests in the headless browser and grab the test results from the html to ensure that apps continue to pass the end to end tests as they are updated and refactored.

This way an app author can include the jasmine.js, jasmine-html.js and a TodoMVC functional test script at the bottom of their app and should see test results printed right in their browser. If everything passes, then they would remove these scripts from their app and submit it for approval.

Travis-ci is a hosted continuous integration for open source projects and using it would eliminate the need to host and manage our own server.

@addyosmani, @sindresorhus - Are you guys open to an implementation like this?

Owner

sindresorhus commented Aug 19, 2012

I'm not sure about using a unit testing lib for end-to-end testing. As discusses in earlier issues (links at the top), there were concerns about being able to create something that worked reliably with every framework. I think I have to agree with that.

Though whatever way we go. I like the idea of automating with Travis.

Sidenote: Mocha > Jasmine

I've spent the last few days working with phantomjs with some limited success. Currently I am injecting scripts into the browser after the page loads and using javascript to test the app functionality. To me this is a good middle ground solution that lets app authors add the same scripts while developing to ensure their apps pass the common spec.

As @sindresorhus mentioned, I'm using a unit test framework to run the tests, make assertions and output the results to the browser. IMO this has to be done for simplicity and a unit test framework works great here. I've currently been using jasmine, only because it was packaged in the repo already. I'm happy to use mocha or whatever else is recommended.

Here are a few of the assertions I have added:

describe('The #new-todo input field', function() {

    it('contains the placeholder text \'What needs to be done?\'', function() {...}

    it('adds a todo by pressing the enter key', function() {...}

    it('regains focus after adding a new todo', function() {...}
}

The main challenge that I am encountering is firing the enter keypress event via javascript doesn't seem to work consistently across the apps. It would probably be easier if the field was wrapped in a form element and the submit event was used instead. Ofcourse the keypress event would submit the form as well.

What do you guys think?

Owner

sindresorhus commented Aug 24, 2012

I've spent the last few days

Thanks for looking into this. Really appreciate it :)

IMO this has to be done for simplicity and a unit test framework works great here.

The problem with that is that I don't think it will work on every type of app. Some have different HTML structure. Some do events differently. You have string based templating which injects everything on every little change, and you have DOM-based templating which only changes what has changed. How do you synthesize user interaction with the DOM.

I urge you to look at our earlier discussion about it in #134 and #200.

The main challenge that I am encountering is firing the enter keypress event via javascript doesn't seem to work consistently across the apps. It would probably be easier if the field was wrapped in a form element and the submit event was used instead. Ofcourse the keypress event would submit the form as well.

Thats a very good argument for why using a unit testing lib for this is a bad idea. From earlier threads by @IgorMinar :

End to end tests pretend to be a user interacting with the application. These tests don't care about the internal implementation, internal state of the application or how the individual components are assembled. These tests look something like "When I click on textbox 'newTask', enter text 'new task 1' and hit enter, does the browser add a new visual element for the task with text 'new task 1'".

Have you tried using something like Casper.js as mention in the issue description? I'm inclined to think that is our best option.

Thanks for the feedback. I'll definitely reread the earlier discussions to make sure I better understand the objectives.

I disagree that the problems I've experienced with the keypress event are an argument against a using unit test library, but rather a good argument against using javascript to simulate user actions in the browser. In theory I could use the same unit test library, but invoke the events natively using phantom, selenium, etc and still read the assertion results printed by the the unit test lib. Casper.js has its own assertion library, but since phantom lacks it's own assertion library it actually recommends using jasmine or qunit.

http://code.google.com/p/phantomjs/wiki/TestFrameworkIntegration

With that said, I have looked into Casper.js and stand alone phantomjs and there is a problem. There is no way to invoke a keypress event, other than injecting jquery into the page, because Phantom hasn't implemented any keystroke related events yet. See the threads below.

https://groups.google.com/forum/?fromgroups=#!topic/casperjs/0c9Updl7uTM
http://code.google.com/p/phantomjs/issues/detail?id=492&q=keyboard

I'm happy to continue down the route and try out casper.js if you'd like. I am excited by it's possibilities and its a good learning opportunity whether or not we opt for that solution.

@sindresorhus - After reading the related threads more closely, I realized that I'm going down the same path that stas went down a few months ago with the addition of automating the tests using phantom and travis-ci.

stas/todomvc@f1084f9

It looked like Stas's implementation received mixed responses. For fun, I'm going to try using his "cross-app" tests with phantom and travis to see if I get any better results.

Owner

sindresorhus commented Aug 24, 2012

@colinjschmidt Cool, as I don't have much experience with End-to-end testing, I'm open to consider any solution that can be proven to work reliably ;)

Owner

addyosmani commented Aug 25, 2012

I'm happy to continue down the route and try out casper.js if you'd like. I am excited by it's possibilities and its a good learning opportunity whether or not we opt for that solution.

Absolutely. Please feel free to (and thanks for spending your time on this).

@colinjschmidt In case it helps, maybe you'd like to loop @stas in on the cross-app setup you're setting up - he might have some related suggestions that could assist.

I'd love to see it working in conjunction with Phantom and Travis. That would rock.

@addyosmani - Thanks! I think we'll be able to get something working with that setup.
@stas - I am using your end-to-end app spec as a starting point. For some reason, I'm unable to get the enter keypress to work even in the browser. I'm confident, however, that it's something I'm doing wrong. Hoping you're willing to take a look?

Contributor

stas commented Aug 25, 2012

Hey @colinjschmidt welcome aboard!
I'm really happy to see interest in this from others.

About the issue, you're doing fine, I couldn't figure it either, simply because, apparently, handling submit event per se, is done differently in almost every framework.

Imho, I would skip any further work in this direction and just focus on writing some functional tests with casper.js. This way we make everyone happy.

What you say team?

Thanks @stas. I see what you mean about the submit event.

I've got the initial specs running for all of the architecture-example apps using PhantomJS and have this hooked up with Travis CI.

You can see it here:

http://travis-ci.org/#!/colinjschmidt/todomvc/builds/2273549

Notice that the current build is failing, because I'm unable to create a todo item in most of the apps using client-side javascript.

I will probably branch my fork of the project to try the same thing with CasperJS, but I expect similar results. This is because you can't currently send keystrokes in PhantomJS or CasperJS, but instead can only inject client-side javascript to simulate a keypress. This is essentially what I'm doing now, but wouldn't take long to try in casper.

One advantage of my current implementation over casper, is that you can test a single app directly in a browser with the same results. This makes setup and debugging easier.

One thing to note, is that many of the apps are binding different events such as keypress, keyup or submit. Perhaps this should be standardized in the app spec? To account for this I'm triggering keydown, keypress, and keyup with limited success.

It starting to look like our options are to change the app spec to use a form tag and require all apps to bind to that form's submit event to create a todo. This would certainly make the app testable using CasperJS. Alternatively we could move to a Web Driver implementation (Selenium) that interacts with the browser like a user, instead of using client-side js to simulate events.

What do you guys recommend?

Member

passy commented Aug 29, 2012

Wow, this is really impressive! 👍

Suggesting one of the key* events to capture in the spec could be a good idea, but I don't think enforcing it would make a lot of sense. The frameworks using some kind of data binding, usually do this behind the scenes and some of them don't even provide a way to specify which event to use.

Contributor

stas commented Aug 29, 2012

Imho casper.js is not worse in any way from Selenium, since it is
basically a "driver" itself. So I'm pretty sure it should solve our
problems.

Also casperjs should be agnostic to what framework is being used.

În data de Wed, 29 Aug 2012 19:49:42 +0300, Colin Schmidt
notifications@github.com a scris:

Thanks @stas. I see what you mean about the submit event.

I've got the initial specs running for all of the architecture-example
apps using PhantomJS and have this hooked up with Travis CI.

You can see it here:

http://travis-ci.org/#!/colinjschmidt/todomvc/builds/2273549

Notice that the current build is failing, because I'm unable to create a
todo item in most of the apps using client-side javascript.

I will probably branch my
fork
of the project to try
the same thing with CasperJS, but I expect similar results. This is
because you can't currently send keystrokes in PhantomJS or CasperJS,
but instead can only inject client-side javascript to simulate a
keypress. This is essentially what I'm doing now, but wouldn't take
long to try in casper.

One advantage of my current implementation over casper, is that you can
test a single app directly in a browser with the same results. This
makes setup and debugging easier.

One thing to note, is that many of the apps are binding different events
such as keypress, keyup or submit. Perhaps this should be standardized
in the app spec? To account for this I'm triggering keydown, keypress,
and keyup with limited success.

It starting to look like our options are to change the app spec to use a
form tag and require all apps to bind to that form's submit event to
create a todo. This would certainly make the app testable using
CasperJS. Alternatively we could move to a Web Driver implementation
(Selenium) that interacts with the browser like a user, instead of using
client-side js to simulate events.

What do you guys recommend?


Reply to this email directly or view it on GitHub:
addyosmani#255 (comment)

@passy - Thanks. I understand your point and would rather not require this level of specificity across apps.

I could argue the case for a form tag being semantic markup and you get the added benefit of built in form submission on enter keypress. Also, I've seen discussion of adding non-javascript mvc frameworks, like django. In this case, I'm sure we'd see more todos using a form tag around the input field.

I'm surprised that some frameworks may not allow specific event binding.

@stas - I'm going to try CasperJS now, but so far I haven't seen a method to invoke a native keypress. There are click events and form submits, but no keypress. As a workaround people are injecting javascript into the app to fire keypress events. See the thread below for more info:

https://groups.google.com/forum/?fromgroups=#!topic/casperjs/0c9Updl7uTM

Member

passy commented Aug 30, 2012

@colinjschmidt - The default in casperjs for input fields seems to be setting the value and firing the change event. I guess that won't help you much.

https://github.com/n1k0/casperjs/blob/6e0300c5cb8b3bd554a8610de8e12209ae69c4b1/modules/clientutils.js#L500

Owner

sindresorhus commented Aug 30, 2012

One thing to note, is that many of the apps are binding different events such as keypress, keyup or submit. Perhaps this should be standardized in the app spec? To account for this I'm triggering keydown, keypress, and keyup with limited success.

I absolutely agree in principle, but there are some differences in frameworks that makes this unfeasible. Though, I do think we should suggest something in the App spec. How about keyup ?

I could argue the case for a form tag being semantic markup and you get the added benefit of built in form submission on enter keypress.

Not really, input without a form is perfectly valid. I chose this to limit the amount of markup as much as we could, and it didn't really offer any benefit seeing as the apps require JS. Using Django as a backend won't change this, since the backend would only be a API to the front-end framework.

I'm going to try CasperJS now, but so far I haven't seen a method to invoke a native keypress. There are click events and form submits, but no keypress. As a workaround people are injecting javascript into the app to fire keypress events. See the thread below for more info:

Just fyi, synthesizing key events are broken in webkit (@addyosmani u fix? :P), they always dispatch key code as 0...

But, looks like the key event support landed in PhantomJS 2 months ago and will likely be in the next release (1.7?). Just ignore the add new todo test for now, and we can come back to it when it's available.

Owner

sindresorhus commented Oct 1, 2012

@colinjschmidt ⬆️, and PhantomJS 1.7 was just released with keyboard support! :) Will you have a chance to continue work on this?

http://ariya.ofilabs.com/2012/09/phantomjs-1-7-blazing-star.html

Owner

sindresorhus commented Oct 22, 2012

@colinjschmidt ping. Just wondered if you were still interested in doing this? It would be really nice to have it unit tested! :)

I've started a new job that is demanding a lot of my time. I saw that phantom now supports keyboard events, which should make things easier. I'm still interested in the project, but am hesitant to set any estimates or expectations.

On Oct 22, 2012, at 4:42 AM, Sindre Sorhus notifications@github.com wrote:

@colinjschmidt ping. Just wondered if you were still interested in doing this? It would be really nice to have it unit tested! :)


Reply to this email directly or view it on GitHub.

Owner

sindresorhus commented Oct 22, 2012

@colinjschmidt No problem at all, we're all feeling the time pressure. Congrats on the new job btw. Just do it when you got a chance ;)

Member

passy commented Oct 23, 2012

I would really like to help out with this. Unfortunately, for the next couple of weeks I'm very short on time as well. As soon as the foundation for this is set, it should be fairly straightforward to contribute test cases to it.

Owner

sindresorhus commented Oct 25, 2012

@passy Awesome! Whenever you have the time would be great :)

Contributor

jsoverson commented Dec 8, 2012

@colinjschmidt, @passy, is this still being worked on? I'm working a few grunt tasks with casper/phantom and could spend some time on this if it's almost there.

I didn't see BusterJS mentioned anywhere here, it's beta but could be the perfect fit: http://docs.busterjs.org/en/latest/

Owner

sindresorhus commented Dec 9, 2012

@devinrhode2 How does it compare to Casper?

I don't know much about Casper, you're probably better off digging into
Buster yourself

-Devin http://zerply.com/DevinRhode2
http://zerply.com/devinrhode2

On Sat, Dec 8, 2012 at 4:43 PM, Sindre Sorhus notifications@github.comwrote:

@devinrhode2 https://github.com/devinrhode2 How does it compare to
Casper?


Reply to this email directly or view it on GitHubhttps://github.com/addyosmani/todomvc/issues/255#issuecomment-11165890.

Member

passy commented Dec 11, 2012

@jsoverson Unfortunately, it still doesn't look good time-wise for me. Grunt tasks sound super useful, though. I wrote a few lines with casper some months ago here to try the new sendEvent APIs which work really well.

Contributor

MathieuLorber commented Dec 27, 2012

While testing Watir on another project, I wondered if there was plans for such tests for TodoMVC, then I found this thread. So I'm a bit late, but that's clear that end to end tests would have a great value, and would make a perfect validator for new (and current) frameworks implementations. And they can also validate compiled frameworks, like GWT and Dart =). Unit and integration tests are more likely to be parts of each implementations, testability is an important point of comparison. But it's quite hard to make them a mandatory requirement... (hmm well... I try to work on them asap for Dart impl ;)...).

BTW, after reading this thread, I played a little bit with CasperJS, starting up with @passy work (https://github.com/passy/todomvc/blob/822e5acb93d5135d4db08ddc5071339c68506240/tests/casperjs/tests.js). The tool is great =). No problems anymore with key events... I wrote a first scenario which aims to completely validate the complete & toggle-all checkboxes, and the left item count (+ the clear-completed button, but not in a way I'd describe as "complete" for the moment) : https://github.com/MathieuLorber/todomvc/blob/casperjs/tests/test.js . My mistakes on Dart implementation helped me to write this scenario, but I probably missed possible bugs...

Following @colinjschmidt, I watched on Travis-CI to have a CasperJS build - it works well ! See the conf file https://github.com/MathieuLorber/todomvc/blob/casperjs/.travis.yml and the build itself : https://travis-ci.org/MathieuLorber/todomvc

If you think it's the good way to go, a test scenario should be written for each "function" : as this scenario tests the "complete" function, another one should test remove buttons, location state, localstorage... It would be cool to have a script which print a table of tests results : each scenario * each implementation. Actually a lot of current implementations do not satisfy current requirements.

Owner

sindresorhus commented Dec 31, 2012

@MathieuLorber Thanks for working on this :)

Looking at how succinct the tests are, I think Casper is the way to go.

@addyosmani @passy @colinjschmidt @stas What do you guys think?

Member

passy commented Dec 31, 2012

I would also go with Casper. The API makes the tests really easy to read and write. Having a table of all test results across the different implementations sounds like a good idea.

Could the tests be run with http://ci.testling.com? Then cross browser
functionality could be verified.

Testling doesn't have mobile browsers, but BrowserStack does.

On Monday, December 31, 2012, Sindre Sorhus wrote:

@MathieuLorber https://github.com/MathieuLorber Thanks for working on
this :)

Looking at how succinct the tests are, I think Casper is the way to go.

@addyosmani https://github.com/addyosmani @passyhttps://github.com/passy
@colinjschmidt https://github.com/colinjschmidt @stashttps://github.com/stasWhat do you guys think?


Reply to this email directly or view it on GitHubhttps://github.com/addyosmani/todomvc/issues/255#issuecomment-11773550.

-Devin, http://Zerply.com/DevinRhode2

Sent from Gmail Mobile

Contributor

stas commented Dec 31, 2012

👍 for Casper.js and Travis-CI integration.
If anyone started working on this please leave a message with a link to the repo.

Thanks.

Contributor

MathieuLorber commented Dec 31, 2012

As I wrote early, a complete (hopefully) test for complete & toggle-all buttons + left items left count : https://github.com/MathieuLorber/todomvc/blob/casperjs/tests/test.js & the Travis-CI "build" : https://travis-ci.org/MathieuLorber/todomvc/builds/3847221

Owner

addyosmani commented Dec 31, 2012

Nice work @MathieuLorber!

+1 for Casper.js from me.

Member

passy commented Jan 2, 2013

@MathieuLorber I played around with your code and one problem seems to be that some of the applications don't have the #todo-count element in the DOM from that start. I rewrote one of your helper functions to also accept an empty string if the expected count is 0: passy/todomvc@40034a0

Is that safe to assume?

I actually found a bug while running this against Backbone, because the counter doesn't seem to update if you complete a Todo. Looks like your test cases are already helping!

Contributor

MathieuLorber commented Jan 3, 2013

A lot of implementations do not pass my first test scenario sample. In most cases the expected DOM is not exactly the same. Some implementations simply do not have all features (for instance : all/active/completed anchors in GWT/Meteor/Vanilla impls, hiding remove all button when 0 todos are complete in Meteor impl...). Sometimes it's a bug, as Backbone one =).

I didn't look deeper for what was wrong in Backbone or others, thank you @passy. But it will clearly arise some questions about initial DOM and its modifications, and how the framework manage the DOM (for example, GWT "standard practices" use an almost empty embedding html page). CasperJS has a waitFor function which should be used (I noticed Dart sample sometimes broke the test because the application wasn't "ready").

BTW, I think it will be necessary to accept some specific modifications of the tests for some implementations (with the modified tests in a implementation/tests directory), but it's not a problem if they are justified. A "hide" function could use css display: none in a framework and remove the element from the DOM in another framework...

Do we continue in this direction ?

If so, I can write some others scenarios, but I do not refuse some help =). Who would be in ?

Owner

sindresorhus commented Jan 7, 2013

Do we continue in this direction ?

Yes. I would normalize wherever it makes sense, allow exception if they're well reasoned.

Contributor

MathieuLorber commented Jan 8, 2013

@devinrhode2 testling & browserstack does not seem to have free plans

BTW, most of the work consists in writing relevant scenarios. It will probably be easy to translate them to another test stack later.

testling is completely free for open source, see here:
http://ci.testling.com/

-Devin http://zerply.com/DevinRhode2
http://zerply.com/devinrhode2

On Tue, Jan 8, 2013 at 4:48 AM, Mathieu Lorber notifications@github.comwrote:

@devinrhode2 https://github.com/devinrhode2 testling & browserstack
does not seem to have free plans

BTW, most of the work consists in writing relevant scenarios. It will
probably be easy to translate them to another test stack later.


Reply to this email directly or view it on GitHubhttps://github.com/addyosmani/todomvc/issues/255#issuecomment-11992539.

Yet Another free for open source testing solution: http://saucelabs.com/opensource
Sauce Labs supports more browsers than testling too

Contributor

MathieuLorber commented Jan 12, 2013

I didn't see that Testling was free for OSS. But I think we should focus on writing some tests with CasperJS and investigate later for cross browsers tests. One again, the harder part is writing scenarios. Moreover, we can try to write - even partially - "lib agnostic" scenarios (I'm optimistic about doing this with CasperJS and helper functions).

I continued to dig into the first CasperJS, wrote other (shorter) tests, hope to show something soon here. I'm not sure yet about the possibilities for localstorage tests.

@addyosmani @sindresorhus a first detail is not clear : when you double-click an item, should the input text be selected or the cursor positioned at end of the input ? Spine reference app selects all, but I remember having a reason to change that behaviour for Dart impl. Should the test accept both behaviours in a first time (implementations differ a lot on this point) ?

Owner

sindresorhus commented Jan 12, 2013

when you double-click an item, should the input text be selected or the cursor positioned at end of the input ?

The app spec is always right. Some of our implementations might not be.

Text should not be selected. Implementation should only accept those that does this correctly. Which will make it easier for us to identify which that doesn't pass.

Contributor

stas commented Jan 12, 2013

I'm not sure yet about the possibilities for localstorage tests.

I don't think we should bother about that. There's another ongoing discussion about app unit-tests which should address this issue.

Contributor

MathieuLorber commented Jan 22, 2013

A little update ; I haven't much time so I did not work as much as I wanted on this =/

I began to wrote other scenarii. They are just early drafts for the moment, do not even take a look at them.

I wrote a ruby script to run them all on a first arbitrary selection of frameworks. A Travis-ci result can be seen here : https://travis-ci.org/MathieuLorber/todomvc/builds/4226478 . The KO are more test implementation problems than framework implementation ones (history & storage are not even true tests). First problem, results do not seem to be stable. Even on my computer...

@stas unit tests & end-to-end tests, even more on this project, should not have the same aim. End-to-end ones should validate implementations behaviour. Unit tests "validate code". On this project, shouldn't they be, as the implementation itself, part of the demonstration ? (note, if I'm over-confident in my words about the project, it is probably more clumsiness with english...). Moreover storage is typically something you may want to test "integrated" ( true unit tests should use an mocked api), even if integration tests are generally less exhaustive than unit ones.

BTW, I try to continue to work on scenarii as soon as I can...

Owner

sindresorhus commented Jan 22, 2013

@MathieuLorber Thanks for the update. Exciting to see it run in Travis. Can't wait to deploy this thing :)

Member

passy commented Jan 22, 2013

@MathieuLorber I really like your approach of splitting the tests up into categories. This would even allow for testing routing only in apps that implement it.

Contributor

MathieuLorber commented Feb 24, 2013

I worked on the tests today. Edit & history & "count" scenarios are almost ok. It's very difficult to write global tests and handle some implementations behaviours without impacting others. Implementations are really different for some "details". I have still problems - with angular especially.
TODO :

  • a lot of cleaning
  • find a elegant way to permit specific changes by frameworks
  • find a elegant way to make a "not implemented" status
  • i'm a in-code TODO generator =)

Good news I hope, tests results seem more consistent (same results on Travis & my computer).

https://travis-ci.org/MathieuLorber/todomvc/builds/5022790

Contributor

MathieuLorber commented Feb 24, 2013

I just took a look at EmberJS & GWT generated DOM... Not sure it's going to be easy =). BTW, I'll try to handle specificities with more helper functions. And each framework could provide its specific helper functions extension.

Hi Everyone
I coded this test with casperjs ( first code with it ), take a look and tell me what you think about. It works for knockoutjs,backbone and angularjs, hope fully it is correct.

https://github.com/pasindud/todo_test

Contributor

MathieuLorber commented May 29, 2013

Hi.

Difficult to me to find enough time for that... @pasindud, would you dare to work on the code i already produced ? It's there : https://github.com/MathieuLorber/todomvc/tree/casperjs/tests . If so, maybe we could talk about split the work between us. I'll take a deeper look at your test also.

Anybody else can help ?

I would love to help in any way I can, 👍

Owner

sindresorhus commented May 30, 2013

That's great!

@devinrhode2 @stas @stephenplusplus might be interested too.

Member

stephenplusplus commented May 31, 2013

Thank you for doing this, guys! This is a new world for me, but looks quite amazing. Like everyone else, free time is becoming hard to find, but I will try to dedicate some time to this this weekend.

Owner

sindresorhus commented Aug 18, 2013

@pasindud still interested? Anything we can do to help you get started?

Yes @sindresorhus do have any feedback on this good or bad https://github.com/pasindud/todo_test, and me and @MathieuLorber were talking about setting a date and doing some devs on a weekend

Contributor

MathieuLorber commented Aug 19, 2013

We talked about it with @pasindud last week !
I'll have time for that in the coming days/weeks =)

Owner

sindresorhus commented Aug 21, 2013

@MathieuLorber @pasindud that's great to hear.

Following are so obectives we came up with @MathieuLorber

  1. make test ok which all fmks
  2. find a elegant way to permit to each fmk to have its own helper function
  3. find a elegant way to permit to a given fmk to deactivate a given test
    cause not all fmks implement all functions
  4. a "debug"
Contributor

MathieuLorber commented Oct 29, 2013

Ok, I work in small steps, it's difficult to take time to progress.
I worked on :

  • a capture tool to "debug" a scenario with a given framework
  • the launching script, to activate this "debug mode" or not, and to possibly be verbose
  • a first storage test

I'm not sure several scenarii is the most pertinent thing to do. I'm wondering if it wound not be better to have one scenario, with optional assertions (to remove assertions about storage in a non-storing sample, for instance). When I'm thinking about the best scenario to test storage, or counters, or whatever, it's allways the same actions I wanna do.

I'm still on it...

Contributor

MathieuLorber commented Nov 10, 2013

Hi. I finally updated my previous code. Making several scenarios to have different features tests became irrelevant.
I was doing allways the same things to test the different features =]. So I rewrote a unique scenario, which aims to test all features awaited in the application - it's not the case yet.

So for the moment we lost the table of results as it can be seen here : https://travis-ci.org/MathieuLorber/todomvc/builds/5022790 . We could get it back by 'typing' the assertions (in the assertion message for example), and process the casperjs output log. But I'm no expert in pipelining, so it's not done yet. I'd like to be also able to generate a html report using the screenshots done by casper. If someone want to help on that...

The test is divided in 3 layers : the test, the assertions "high-level" functions (for instance assertDisplayedItemsCount), and the low-level functions. The target is to be able to test all the samples by just overriding low-level functions when needed. A first example is currently used for the backbone test : the "normal" behaviour for testing storage (storage is tested) is doing json.parse().length, but for backbone the main localstorage entry is a list of ids separated by a comma. When a assertion is simple (an element is visible or not...) and does not need different implementations, I tried just to use a simple casperjs assertion in the test.

The syntax to launch the test is now :

./test.rb
    for all frameworks
./test.rb angularjs backbone dart
    for just these 3 ones - note that just backbone & dart are ok as i'm writing (maybe jquery but i'm no sure)
./test.rb -d
    debug => take screenshots for the moment, more to come I hope...
./test.rb -v
    verbose => displays casperjs output instead of the clean result table

You can ./test.rb -dv backbone

Note that phamtomjs and OS X Mavericks do not play nice together for the moment (ariya/phantomjs#11418), I currently add "2> /dev/null" at end of line for running test on my laptop (beware hidding true problems =]).

So, the todolist for the continuation :

  • test all the features (reviews welcome !)
  • be able to process the casperjs output (help welcome !)
  • I wrote a lot of TODOs in the code

When that's ok, I will definitely need help to implement the low-level functions for each of the existing todomvc implementations ! This will obviously need some rewriting of the test code/high-level functions, but nothing complicated. I'll make a how-to wiki page what when I reach that point or when someone ask for it...

Owner

sindresorhus commented Nov 11, 2013

@MathieuLorber thanks for the update :)

Member

ColinEberhardt commented Jan 19, 2014

OK, so I decided to give this a go. I am currently using Selenium through the new WebDriverJS interface, together with Mocha. You can see the work in progress here:

https://github.com/ColinEberhardt/todomvc/tree/browser-testing/browser-tests

The tests are intended to be a direct reflection of the TodoMVC specification. I have currently implemented 19 out of 27 tests:

TodoMVC
No Todos
  ✓ should hide #main and #footer (99ms)
New Todo
  ✓ should allow me to add todo items (825ms)
  ✓ should clear text input field when an item is added (481ms)
  ✓ should trim text input (618ms)
  ✓ should show #main and #footer when items added (581ms)
Mark all as completed
  ✓ should allow me to mark all items as completed (1218ms)
  ✓ should allow me to clear the completion state of all items (1400ms)
  ◦ complete all checkbox should update state when items are completed / cle      1) complete all checkbox should update state when items are completed / cleared
Item
  ✓ should allow me to mark items as complete (1338ms)
  ✓ should allow me to un-mark items as complete (1315ms)
  2) should allow me to edit an item
  3) should show the remove button on hover
Editing
  4) should hide other controls when editing
  5) should save edits on enter
  6) should save edits on blur
  7) should trim entered text
  8) should remove the item if an empty text string was entered
  9) should cancel edits on escape
Counter
  ✓ should display the current number of todo items (672ms)
Clear completed button
  ✓ should display the number of completed items (1247ms)
  ✓ should remove completed items when clicked (1253ms)
  ✓ should be hidden when there are no items that are completed (1165ms)
Persistence
  ✓ should persist its data (3713ms)
Routing
  ✓ should allow me to display active items (1189ms)
  ✓ should allow me to display completed items (1220ms)
  ✓ should allow me to display all items (1387ms)
  ✓ should highlight the currently applied filter (1226ms)


  18 passing (1m)
  9 failing

I was on a plane when I did most of this, so couldn't google when I hit a problem with double-click for edit!

Let me know your thoughts / feedback?

I should have this completed in a few days time when I can tidy up / squash and create a PR if people think this is a good idea.

Member

ColinEberhardt commented Feb 3, 2014

I'd say this is done now :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment