Tests: Add intern support #450

apsdehal · 2015-05-29T19:18:08Z

Shift all the tests to use QUnit interface by intern
Add grunt tasks for intern both for local and ci.
Modify deps according to the needs.

TODO:

Create separate Grunt tasks test:unit and test:functional
Create test:ci task and make sure it runs properly on Travis
Stop testing if any of the tests fails
Allow local browser testing using connect
Support all OSes and Browsers as mentioned in Globalize capabilities
Add es5-shim html test files
Support node matrix tests.
Include mobile browsers. support for mobile browsers / phonegap? #278

arschmitz · 2015-05-30T02:08:06Z

test/intern.js

+  environments: [
+    { browserName: "chrome" },
+    { browserName: "internet explorer", version: [ "11", "10" ] },
+    { browserName: "firefox", version: "34" }


this should include all the supported browsers see https://github.com/jquery/globalize/blob/master/browserstack.json

One additional comment: When testing IE8, we need to include ES5 shim (see unit-es5-shim.html and unit-es5-shim.html).

rxaviers · 2015-05-30T10:10:14Z

The first time I ran grunt test-local, everything was executed just fine. Although, for the second and further times, I got the below (even having included stop-selenium-server that @arschmitz commented above [1]).

>> Error: Timeout waiting for selenium to start.  Check if an instance of selenium is already running.

After manually killing the live process, I was able to run it again.

$ ps aux | grep selenium
xavier   16619  1.7  2.1 4350708 84856 pts/19  Sl   07:04   0:03 java -jar node_modules/grunt-selenium-server/selenium-server-standalone-2.45.0.jar -Dwebdriver.chrome.driver=/tmp/globalize/node_modules/chromedriver/lib/chromedriver/chromedriver 

$ kill 16619

1:

diff --git a/Gruntfile.js b/Gruntfile.js
index 900c6a0..ee17768 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -477,6 +477,9 @@ module.exports = function( grunt ) {
                                        systemProperties: {}
                                }
                        }
+               },
+               "stop-selenium-server": {
+                       dev: {}
                }
        });

rxaviers · 2015-05-30T10:14:30Z

Gruntfile.js

@@ -480,10 +508,19 @@ module.exports = function( grunt ) {

 		// TODO fix issues, enable
 		// "jscs:dist",
-		"test:functional",


There are two things I don't want to lose:

Having local tests run on the default task.

Having unit tests run before building (requirejs, ...) and having functional tests run before minification (uglify).

arschmitz · 2015-05-30T11:21:48Z

@rxaviers ill checkout the stop selenium server part i asked him to add this i use it on other projects.

rxaviers · 2015-05-30T13:04:17Z

👍

apsdehal · 2015-06-06T18:56:03Z

It turns out intern doesn't support ie8 in unit tests, so I have removed it for now.

jzaefferer · 2015-06-17T14:25:50Z

@rxaviers are you happy with the latest changes? Do you want me to review this as well?

rxaviers · 2015-06-22T14:01:29Z

Gruntfile.js

+			}
+		},
+		"stop-selenium-server": {
+			dev: {}


Is this (selenium + chromedriver) a requirement for having tests to run on browserstack? Or is it a requirement for having tests to run on Chrome locally?

Isn't 'browserstack.selenium_version': '2.45.0' configuration sufficient? (more info).

This is for running locally on chrome. Normally to run locally like this with intern you would need to download selenium, download, chromedriver start selenium with chrome driver then run the tests. This downloads everything with npm and starts and stops the server.

Thanks @arschmitz. So, @apsdehal let's remove this change please. More details in (1.2) of this comment.

@rxaviers What your talking about in your comment below is rather different. This is for testing via webdriver locally it very much aids in contributors being able to do with with out the steps i list above browserstack is great but most contributors wont have access to this.

Intern supports 3 ways of running tests.

Via Selenium with webdriver
Via Node with a runner
Via the browser directly via client.html

I think contributors should be able to easily do all three of these locally for proper testing this addresses # 1 what you ask for in the comment below addresses # 3

rxaviers · 2015-06-22T15:01:14Z

@apsdehal you have done a great work so far. But, before continuing with my review, I think it would be good to align with you the goals I have in mind for testing below.

Do you think is it feasible to address them on this PR?

@apsdehal @arschmitz @jzaefferer Does anyone see anything missing in the below (or that could be performed in a better way)?

Local testing

1.1. Local Node.js

Local Node.js testing is useful during development. They are fast to run and
helps developer to know that unit and functional tests are passing for his local
Node.js environment.

The tests should be exposed via two Grunt tasks:
- grunt test:unit for running local Node.js unit tests.
- grunt test:functional for running local Node.js functional tests.
These tests should be executed by CI. If any fails, there's no need to waste
time running them on the various browsers (item 2 below).

I think it's benefical changing what we currently do (run tests via phantomjs)
and use intern instead due to its test coverage support. Sidenote: it's crazy
thinking we currently run these tests via phantomjs instead of using plain
Node.js.

1.2. Local browser

Local browser testing is useful during development. They allow for manual
inspection on a specific browser.

Developer should be able to run these tests by simply pointing any browser at:
- http://localhost:9000/test/unit.html for unit tests.
- http://localhost:9000/test/functional.html for functional tests.
- http://localhost:9000/test/unit-es5-shim.html for unit tests (including es5 shim).
- http://localhost:9000/test/functional-es5-shim.html for functional tests (including es5 shim).
I see no need for changing what we currently have.
Matrix testing
2.1. Matrix Node.js testing

Currently, this is missing. Can we have intern to execute the tests in a matrix of different Node.js environments (e.g., 0.10, 0.11, etc)?

These tests should be exposed via a Grunt task:

grunt nodeMatrixTest (better names are welcome)

2.2. Matrix browsers and OSes testing

Automated testing to ensure all browsers listed here (and the various OSes where they run on) are supported. I heard intern doesn't support automating the tests for IE8, which could be mitigated by us by running them manually (via 1.2).

These tests should be exposed via a Grunt task:

grunt browsersMatrixTest (better names are welcome)

These tests should be executed by CI only in case if the faster grunt test (1.1) passes.

I think running them through intern vs. browserstack is a great solution.
Missing: CLDR matrix testing

Globalize in theory works with any CLDR >= 26. But, we only test it using the latest CLDR. Can we automate CLDR matrix testing? I believe this is similar to 2.1.

arschmitz · 2015-06-22T15:20:17Z

@rxaviers

Local Node.js testing is useful during development. They are fast to run and
helps developer to know that unit and functional tests are passing for his local
Node.js environment.

This is part of intern already. Just need the tasks set up

Automated testing to ensure all browsers listed here (and the various OSes where they run on) are supported. I heard intern doesn't support automating the tests for IE8, which could be mitigated by us by running them manually (via 1.2).

Intern only supports IE8 for functional testing which you do not do ( or likely need ) with globalize there is no way manual or otherwise to run ie8 tests with intern. Chi does not support ie8 and is the assertion library used by intern. ES5 shim will not fix this.

How important do you consider testing against IE8 at this point it will no longer be supported in 6 months and you dont support it with out the use of es5 shim anyway. As a side not both ui and mobile have talked about the current versions we are getting ready to release being the last to support IE8.

There is intern geezer edition that supports ie8 but i don't think it supports qunit interface and has a lot of other disadvantages as well we may be able to use it just for ie8 though and add the interface for qunit i don't have any experience with geezer.

Developer should be able to run these tests by simply pointing any browser at: ...

It already works they are just redirects now all tests are run from a single html page within the node modules intern directory configuration options are passed to it to tell it what tests to run.

Matrix testing

The goal of this PR was just to really get globalize setup with intern for cross browser testing other sorts of matrix testing seem like they could be added later as you have time.

rxaviers · 2015-06-22T16:59:58Z

for functional testing which you do not do ( or likely need )

For clarification, yeap Globalize doesn't need the browser interaction simulation that intern calls functional. But, Globalize does have functional tests (i.e., tests that ensure the distributed built Globalize works as expected), which can run using what intern calls unit testing.

About IE8... Like I said, it can be tested manually in the meanwhile (while it's still supported) by using what I described in the item 1.2 above. I'm not concerned that we cannot automate its testing using intern.

It already works they are just redirects now all tests are run from a single html page within the node modules intern directory configuration options are passed to it to tell it what tests to run.

I want to keep things simple. In other words, introducing intern for 1.2 introduces an unnecessary complexity.

The goal of this PR was just to really get globalize setup with intern for cross browser testing other sorts of matrix testing seem like they could be added later as you have time.

I agree. Therefore, the items described in 2.1 and 3 could be ignored on this PR.

arschmitz · 2015-06-22T18:05:39Z

About IE8... Like I said, it can be tested manually in the meanwhile (while it's still supported) by using what I described in the item 1.2 above. I'm not concerned that we cannot automate its testing using intern.

Unless you are suggesting on keeping actual QUnit duplicating ALL the tests and having 2 complete and duplicated test setups im not sure what you mean by this?

I want to keep things simple. In other words, introducing intern for 1.2 introduces an unnecessary complexity.

What added complexity the fact that when you click the link it redirects?

- Local - General

- Contains all the files to be tests for unit

- Explicitly set intern/main in config paths

- This will test globalize using phantom js in following envs: - 0.10 - 0.11 - 0.12 - Travis Stable - This will also test globalize on browsers in nodejs 0.10 version

- `grunt test:local` - Builds - Runs connect server - Navigate to `localhost:9000/test/unit.html` - Navigate to `localhost:9000/test/functional.html`

apsdehal · 2015-10-11T13:10:04Z

Fail fast option (to fail if any of tests fail) will take a little time.
theintern/intern#413

apsdehal · 2015-10-19T19:39:12Z

@rxaviers This is ready to be tested in the current scenario, except the problem that I am not yet able to achieve 100% tests passing with browserstack because different things fail in varied scenarios on browserstack.

rxaviers · 2020-03-16T13:22:18Z

I am closing all old (and stalled) PRs

jquerybot added the CLA: Valid label May 29, 2015

apsdehal force-pushed the intern branch from b8ff7ff to 584c357 Compare May 29, 2015 23:13

arschmitz reviewed May 30, 2015
View reviewed changes

rxaviers mentioned this pull request May 30, 2015

Build: Enable automated crossbrowser testing, including PRs #381

Closed

rxaviers reviewed May 30, 2015
View reviewed changes

rxaviers reviewed Jun 22, 2015
View reviewed changes

apsdehal force-pushed the intern branch 2 times, most recently from 91ad84d to 3236106 Compare October 5, 2015 04:16

apsdehal added 8 commits October 7, 2015 18:42

Tests: Add intern's configs

14cc50e

- Local - General

Tests: Add all.js

7c34d67

- Contains all the files to be tests for unit

Chore: Add intern to Gruntfile and packages.json

3f2b556

Tests: Currency tests shift to intern

4a50a68

Tests: Shift all the rest of tests to intern

55c1398

Chore: Add requirejs dep

5ca0c1e

Gruntfile: Fix indent

108089d

Tests: Add intern all.js file for functional tests

10de6c4

apsdehal added 11 commits October 7, 2015 18:42

Build: Upgrade intern's version

cca0656

Tests: Fix intern's load timeout error

6060056

- Explicitly set intern/main in config paths

Tests: Set proxyOnly and waitSeconds options

90ecf3a

Tests: Remove proxyOnly option

4c0fb55

Tests: Enable node matrix testsing

733e59f

- This will test globalize using phantom js in following envs: - 0.10 - 0.11 - 0.12 - Travis Stable - This will also test globalize on browsers in nodejs 0.10 version

Tests: Fix travis yaml file

f62485e

Tests: Make test.sh executable

3258502

Build: Fix gruntfile style error

a770e63

Build: Updates according to bumped intern version

ea33a17

Build: Add test:local task to Gruntfile

6ceb476

- `grunt test:local` - Builds - Runs connect server - Navigate to `localhost:9000/test/unit.html` - Navigate to `localhost:9000/test/functional.html`

Tests: Add functional, unit es5 shim test html files

9c0ee39

apsdehal force-pushed the intern branch from 34f0c50 to 9c0ee39 Compare October 7, 2015 13:45

This was referenced Oct 10, 2015

Testing automation #235

Open

support for mobile browsers / phonegap? #278

Closed

Android 2.3, functional tests fail #350

Closed

Document how to run cross browser tests using browserstack #432

Closed

rxaviers added Current Sprint new feature labels Oct 10, 2015

apsdehal added 5 commits October 11, 2015 19:37

Tests: Add config for all supported browsers

68d9c8e

Tests: Update intern config for safari

aa79f0f

Tests: Update intern config for browsers

bebb641

Tests: Fix jshint errors

aa13fbe

Tests: Update browsers

a8177dd

rxaviers mentioned this pull request Jan 19, 2016

package.json: peerDependency no longer works #548

Open

JSFOwner removed the CLA: Valid label Nov 2, 2016

rxaviers removed the Current Sprint label Nov 18, 2016

rxaviers closed this Mar 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests: Add intern support #450

Tests: Add intern support #450

apsdehal commented May 29, 2015

arschmitz May 30, 2015

rxaviers May 30, 2015

rxaviers commented May 30, 2015

rxaviers May 30, 2015

arschmitz commented May 30, 2015

rxaviers commented May 30, 2015

apsdehal commented Jun 6, 2015

jzaefferer commented Jun 17, 2015

rxaviers Jun 22, 2015

arschmitz Jun 22, 2015

rxaviers Jun 22, 2015

arschmitz Jun 22, 2015

rxaviers commented Jun 22, 2015

arschmitz commented Jun 22, 2015

rxaviers commented Jun 22, 2015

arschmitz commented Jun 22, 2015

apsdehal commented Oct 11, 2015

apsdehal commented Oct 19, 2015

rxaviers commented Mar 16, 2020

Tests: Add intern support #450

Tests: Add intern support #450

Conversation

apsdehal commented May 29, 2015

arschmitz May 30, 2015

Choose a reason for hiding this comment

rxaviers May 30, 2015

Choose a reason for hiding this comment

rxaviers commented May 30, 2015

rxaviers May 30, 2015

Choose a reason for hiding this comment

arschmitz commented May 30, 2015

rxaviers commented May 30, 2015

apsdehal commented Jun 6, 2015

jzaefferer commented Jun 17, 2015

rxaviers Jun 22, 2015

Choose a reason for hiding this comment

arschmitz Jun 22, 2015

Choose a reason for hiding this comment

rxaviers Jun 22, 2015

Choose a reason for hiding this comment

arschmitz Jun 22, 2015

Choose a reason for hiding this comment

rxaviers commented Jun 22, 2015

arschmitz commented Jun 22, 2015

rxaviers commented Jun 22, 2015

arschmitz commented Jun 22, 2015

apsdehal commented Oct 11, 2015

apsdehal commented Oct 19, 2015

rxaviers commented Mar 16, 2020