Kapow! #2670

Merged
merged 168 commits into from Nov 15, 2012

Conversation

Projects
None yet
4 participants
Owner

6a68 commented Nov 1, 2012

Node tests not 100% solid (test automation crew still cranking) but want to get this mainlined sooner than later.

6a68 and others added some commits Oct 3, 2012

@6a68 @shane-tomlinson 6a68 kapow! py tests gone 08ce680
@6a68 @shane-tomlinson 6a68 copy all code from 6a68/meh and update .gitignore a469f7f
@6a68 @shane-tomlinson 6a68 minor renaming to better match unit tests 7111137
@6a68 @shane-tomlinson 6a68 add info to README a05d86b
@6a68 @shane-tomlinson 6a68 primary health check working wut wut cd9f800
@6a68 @shane-tomlinson 6a68 secondary health check done, code looking fairly clean f3fa542
@6a68 @shane-tomlinson 6a68 add test to create secondary user and verify in second browser 7e9a59b
@6a68 @shane-tomlinson 6a68 Extract browser/saucelabs setup to testSetup.
  - still need to extract args from cmd line and inject into testSetup.startup().
  - now you can get the default args by doing browser.newSession(testSetup.sessionOpts)
    or by doing browser.newSession(testSetup.mergeOpts({othercapabilities})
    but it would be even better if you didn't have to remember to do that.
    - OTOH I don't want to hard-code connections between wd and testSetup.
  - added default capabilities to sauce-platforms file
561b7f9
@6a68 @shane-tomlinson 6a68 update tests to use testSetup, not wd.remote c37ba99
@6a68 @shane-tomlinson 6a68 Finish new-user tests, add multiple browsers to testSetup
  - we have primary account creation in all of persona.org, 123done, mfb
  - we have secondary account creation in 123done, mfb, and persona.org,
    including two-browser back-and-forth jumping in mfb and 123done
  TODOs:
    - primary tests seem to break on verification step?
    - figure out why primaryVerify button sticks on first click
f8c4aaa
@6a68 @shane-tomlinson 6a68 update other tests to use new testSetup syntax 08c8cca
@6a68 @shane-tomlinson 6a68 returning-user test: first version done db5e326
@6a68 @shane-tomlinson 6a68 add sauce ID logging to testSetup, fix typo 6acca10
@6a68 @shane-tomlinson 6a68 simplify personatestuser API by moving defaults into it d931bad
@6a68 @shane-tomlinson 6a68 WIP public-terminals test; localStorage hack works.
  - localStorage hack works in sauce IE8 and local FF.
  - TODO: now that we have the setup done, figure out which cases to test.
b8cc8d1
@6a68 @shane-tomlinson 6a68 add testSetup.setup method for streamlining fixture setup b6fedd3
@6a68 @shane-tomlinson 6a68 Add test runner info to README. Thx @ozten. d5b151f
@6a68 @shane-tomlinson 6a68 Update tests to use testSetup.setup and actually work f1fdca3
@6a68 @shane-tomlinson 6a68 add sauce test running info to README 9652ac5
@6a68 @shane-tomlinson 6a68 cleaning up readme markdown 82c1af7
@shane-tomlinson shane-tomlinson Make the change-password-test run. a60108b
@jrgm @shane-tomlinson jrgm - fix the backward order of arguments to wd.prototype.newSession. (We…
… weren't

  actually running any browser but the default FF/Vista one as a result. Now
  that we are trying to run Windows, we don't seem to be able to. But that's
  next to fix.)
- fix some accidental globals and shadows
- use underscore.extend to reduce boilerplate
- remove unused testSetup.mergeOpts method
- node-inspector doesn't like one-liner functions
fa2a773
@jrgm @shane-tomlinson jrgm - use lowercase alphanum for restmail email until GH-2551 is fixed
- add back the "click twice" oddity for verifyWithPrimaryButton
d0f8ef4
@shane-tomlinson shane-tomlinson Starting on the reset password test. fe2ef8f
@shane-tomlinson shane-tomlinson If there is no error defined but the response code is not 200, pass t…
…he response body as the error.
3198fea
@shane-tomlinson shane-tomlinson Change the reset-password-test to use restmail. a6f5a86
@jrgm @shane-tomlinson jrgm give the test a name, and default to chrome/vista which does not requ…
…ire a version number
e3fbba0
@shane-tomlinson shane-tomlinson Adding a couple of wrapper functions, not all done yet.
Add lib/user.js with:
* verifyEmail
* getVerifiedUser
56523cd
@jrgm @shane-tomlinson jrgm only set 'proxy = { proxyType: 'direct' }' for Opera. IE doesn't like…
… it. And add a 'persona' tag to the job
e17095a
@jrgm @shane-tomlinson jrgm add a test to run the frontend /test/ via sauce 9eb1f0b
@jrgm @shane-tomlinson jrgm fix the in-browser javascript for IE8 with _.map() a2fbdff
@jrgm @shane-tomlinson jrgm add some mobile devices; update a comment 7a25761
@jrgm @shane-tomlinson jrgm pull out details on which qunit tests failed (if any) a5fdda7
@shane-tomlinson shane-tomlinson After a user resets their password, make sure they can log in to the …
…dialog using the new password.
8daf532
@shane-tomlinson shane-tomlinson Add remove-email tests
Add a helper to wd-extensions - delay.
Start to standardize CSS selectors.
1cc9d44
@shane-tomlinson shane-tomlinson add the removeEmail helper function. 50c4f43
@6a68 @shane-tomlinson 6a68 final attempt for now at markdown readme cleanup 90dc457
@6a68 @shane-tomlinson 6a68 Fix change-passwd test; ensure testSetup waits till personatestusers …
…are ready.

  * Added kriskowal's Q library to synchronize personatestuser waits.
  * Discovered testSetup calls need to be wrapped in an initial vows vow for
    synchronization to happen in cases where personatestuser is sluggish.
6336c9f
@shane-tomlinson shane-tomlinson Add checks to make sure deleted emails no longer have accounts.
* Add wd.prototype.wclear to enable clearing of functions.
9162d00
@6a68 @shane-tomlinson 6a68 Update all tests to synchronize the testSetup call
  The most natural way to do this seemed to be wrapping testSetup.setup() calls
  inside an initial vow. This makes it kinda sloppy if we just want to run
  certain specific tests within a file, but there are other TODOs related to
  enabling that, so it's not breaking anything that was working before.
c6c3bd6
@6a68 @shane-tomlinson 6a68 Add runner to ensure no vow keys are ever duplicated
  TODO: when we have a commandline test runner, we'll want to register and
  run tests via that command-line tool, and not inline at the foot of each test
e8b44e3
@shane-tomlinson shane-tomlinson Start work on the cancel account test cd1b7bb
@6a68 @shane-tomlinson 6a68 Update tests to use runner, not vowsHarness abdc4f0
@shane-tomlinson shane-tomlinson Adding scripts/run-all.js which allows all the tests to be run. f9b48cd
@shane-tomlinson shane-tomlinson Run the tests through vows instead of through node directly. Report s…
…tdout results to a file.

* Add 3 Reporter types, StdOutReporter, StdErrReporter and FileReporter
347aa24
@shane-tomlinson shane-tomlinson Woot! Add a deploy script which copies the SauceLabs crets to the rem…
…ote server. Selenium tests are started on Sauce against the ephemeral instance.
afd05ef
@shane-tomlinson shane-tomlinson update the dialog.emailInput and dialog.existingPassword selectors to…
… go along with the changes to remove duplicate IDs
133a374
@shane-tomlinson shane-tomlinson Fix the myfavoritebeer URL if using an ephemeral instance. c0c31ca
@jrgm jrgm fix the default testname when running under vows, and hoist this out …
…of wd-extensions.js where it didn't belong
5d73513
@jrgm jrgm You can temporarily force a local browser run with `PERSONA_NO_SAUCE`…
…. If you do this, make sure `PERSONA_BROWSER` is set to something that can be run locally.
d438d06
@jrgm jrgm Merge pull request #2653 from jrgm/kapow-stuff
fix testname when running under vows; use PERSONA_NO_SAUCE to run locally
e4982fd
@jrgm jrgm add a primary to secondary, mostly stealing work already done 3f998c1
@jrgm jrgm a different approach to clicking button#verifyWithPrimary, that saves…
… 20 seconds half the time; also pass done to browser.quit() to avoid a race at shutdown that leaves sauce jobs unfinished
5f09b9d
@jrgm jrgm Merge pull request #2654 from jrgm/kapow-adding-pri-sec
Kapow adding primary email to account with secondary
b74ec64
@jrgm jrgm create an account with primary and add a primary to it 4720575
@jrgm jrgm Merge pull request #2655 from jrgm/kapow-pri-pri
create an account with primary and add a primary to it
848c23d
@shane-tomlinson shane-tomlinson Using a new awsbox which supports local scripts to copy sauce credent…
…ials to ephemeral instances and start sauce tests.
c84179f
@jrgm jrgm always pass done to browser.quit. Otherwise the sending DELETE sessio…
…n is in a race with process quit, and when that loses, the session lives on for 90 seconds and is marked as error
4c14c2e
@jrgm jrgm Merge pull request #2657 from jrgm/kapow-browser-quit-then-done
always pass done to browser.quit()
b898fe8
@6a68 6a68 to preserve 0.6 node compat, check for existsSync on path and fs 1ea93b9
@shane-tomlinson shane-tomlinson Change the awsbox config, localhooks has been renamed to local_hooks.…
… Update awsbox sha in package.json
d6b65f6
@shane-tomlinson shane-tomlinson Add the PERSONA_BROWSER environment variable to the sauce.json config…
… file, use this on the remote server to specify which tests should be run.
44850f8
@jrgm @6a68 jrgm always pass done to browser.quit. Otherwise the sending DELETE sessio…
…n is in a race with process quit, and when that loses, the session lives on for 90 seconds and is marked as error
35ce983
@6a68 6a68 to preserve 0.6 node compat, check for existsSync on path and fs a283932
@6a68 6a68 Report Errors correctly by coercing them to string 98a7390
@6a68 6a68 Pass asserts into the done callback always.
  There is a weird mismatch in that assert() throws AssertionErrors if an
  assertion fails--yet node works by continuation passing, so that errors
  thrown outside the callback chain aren't handled gracefully.

  In our case, uncaught AssertionErrors cause vows to throw up its hands in
  despair, and not write anything to the XML results file. Since blank files
  aren't valid XML, this causes Jenkins to fail the whole test run on parser
  error grounds. What we want is for it to descriptively fail the build, by
  reading and reporting properly on which tests failed.

  Hopefully confining asserts to the initial error param in the callback chain
  fixes the error reporting headaches.
6ded220
@6a68 6a68 Merge branch 'kapow-assert-inside-callbacks' into kapow 87ddda1
@6a68 6a68 make sauce labs tests public by default 4c76d73
@lloyd @6a68 lloyd if .waitFor hits a timeout before event waiting for occurs, invoke co…
…mplete callback with an error
fe2e63a
@6a68 6a68 Use local version of asserts which return text, not Errors, on failure 38b2345
@6a68 6a68 break out of chains when an error occurs, using lloyd's wd patch acd8678
@6a68 6a68 Suppress JSON test output, so that Jenkins gets parseable XML output 3aadbb7
@shane-tomlinson shane-tomlinson A small cleanup for how output is displayed. e0de19c
@shane-tomlinson shane-tomlinson Upgrade awsbox to 0.3.3 baeee81
@shane-tomlinson shane-tomlinson Setting up travis to call test_selenium, which starts the test proces…
…s on testy.
07857bf
@shane-tomlinson shane-tomlinson exit with a code of 1 if there are is an </error> tag found, 0 otherw…
…ise.
dd4913d
@shane-tomlinson shane-tomlinson fix the typo 340dd9f
@shane-tomlinson shane-tomlinson Make sure at least one test suite is run but that there are no errors. 2c3db3b
@shane-tomlinson shane-tomlinson Write directly to stderr and stdout instead of using the util functio…
…nality. This results in fewer newlines being added to the output.
2120755

lloyd was assigned Nov 1, 2012

Owner

6a68 commented Nov 1, 2012

odd that the merge isn't automagic. working on rebasing the branch now.

Owner

6a68 commented Nov 1, 2012

ok, that looks better. @lloyd, do your thang :-)

@jrgm jrgm commented on the diff Nov 2, 2012

.gitignore
/automation-tests/results
-/automation-tests/credentials.yaml
-/automation-tests/sauce.yaml
-automation-tests/persona_server/results/*
-/automation-tests/chromedriver.log
-/.coverage_data
-/cover_html
@jrgm

jrgm Nov 2, 2012

Member

Those last two .coverage_data and cover_html were added to dev in PR-2272 and should not be removed. But if it's a hassle to rebase again, don't worry; they'll eventually get added back in some other PR.

@6a68 6a68 Pass extra arg to vows so it reports errors as valid XML.
  For an unknown reason, utils.waitFor causes vows to asplode when element
  lookup times out--vows reports these failures using the console reporter,
  instead of the xunit reporter. FWIW the relevant snippet seems to be
  https://github.com/cloudhead/vows/blob/38817c1b2670db45e982b52b30ae044e6c683130/lib/vows.js#L204

  To work around this, we ask vows to spawn each test in a separate process,
  via the "-i" option, and it reports failures correctly as well-formed XML.
b923c70
Owner

6a68 commented Nov 2, 2012

hell yes. vows finally giving jenkins something to work with. couple of broken tests, but breakage now leads to well-formed XML. we'll get to fixing the tests next :-)

Contributor

lloyd commented Nov 2, 2012

ok, debating whether landing in dev right now is the right thing to do. The only pre-requisites I would consider hard are:

  1. an easy way to run all tests all browsers
  2. test stability (90% of the time all tests all browsers succeeds)
  3. jenkins integration (daily?)

This is a subset of what is mentioned in the planning etherpad.

once 1 is done, I'd love to contribute to 2 by using it.

all that said, is there a good reason to merge into dev before these 3 top level requirements are met?

/cc @shane-tomlinson @jrgm

Contributor

lloyd commented Nov 2, 2012

yeah, and if 3 got done and we were running tests hourly using jenkins, then as soon as we have 50 runs with < 5 failures, we know we've attained 2...

lloyd added some commits Nov 2, 2012

@lloyd lloyd update vows harness to support .bailOnError behavior by default. This…
… allows us to get better error information out of vows, as well as not run tests that depend on the success of the former. BONUS: a .cleanup can be specified that will always be run. useful for shutting down saucelabs sessions rather than letting them take an additional 90s to timeout
fc6b138
@lloyd lloyd upgrade to new version of wd that fixes a bug whereby .onError handle…
…rs were not getting invoked, and improves semantics of automatic error detection so that we can ignore errors in some cases. also, allow programmatic halting of queued chain functions
36ad60a
@lloyd lloyd whitespace change, consistent indentation ce5bb3c
@lloyd lloyd for returning-user tests, quit in a cleanup handler so that we always…
… cancel the session promptly, even when it errors (means 90s less wait time when you're trying to debug a test by looking a the protocol messages that were exchanged via sauce)
cfb8384
@lloyd lloyd fix returning-user test - allow clicking of 'not my computer' to fail…
…, given that this button may not be displayed if 60s have not elapsed since account verification
0f8eb3a
Member

shane-tomlinson commented Nov 5, 2012

@lloyd - I agree with your requirements with the following caveats.

For #1, we have to define which browsers make this list and with what frequency tests are run against all browsers. Is this on checkin/push to origin, on merge, etc? What happens for developers that first push to their own repos and then submit a PR, do these checkins automatically kick off the full test set? Currently we have no way of telling travis "run Selenium tests from this repo, but not from that repo over there."

For #3, Jenkins integration against which branch - dev, stage, prod or against any old branch?

shane-tomlinson added some commits Nov 5, 2012

@shane-tomlinson shane-tomlinson Add the ability to run all tests against all supported SAUCE browsers…
… by specifying PERSONA_BROWSER=all when running run-all.js
48a8898
@shane-tomlinson shane-tomlinson Get the repo URL from the git repo and send that along with the reque…
…st. Helps establish some sort of ACL on testy-listener so that it will only run tests on repos from whitelisted users.
bdc30c7
@shane-tomlinson shane-tomlinson Up the number of concurrent runners to 30. 065d466
@shane-tomlinson shane-tomlinson Add the ability to specify the REPO_URL and REPO_SHA on the command l…
…ine when calling run-all.js. Up the number of concurrent test runners to 30.
f4756a9
@shane-tomlinson shane-tomlinson Fix getOrigin returning an origin with a \n appended to it. 86829bc
@shane-tomlinson shane-tomlinson Hook up to the process "exit" message instead of "end" message.
This fixes the problem where the results were never actually checked and travis thought all tests were passing.
f80c0bc
Owner

6a68 commented Nov 5, 2012

So, the tests look green on Jenkins with all-browsers selected: https://ci.mozilla.org/job/temporary-test-job/236/console . I'll put that job on a short cron to get some stability numbers, and update this ticket tomorrow/late tonight with details. [update--I got the env var wrong and was only running against chrome-vista, doh. more info soon.]

@shane-tomlinson, to address some of your questions:

Which browsers should we use? The browsers we worried about with our selenium tests in the past were the ones in the sauce-browsers list, excluding the mobile platforms @jrgm added, but those don't track precisely to our official list of supported browsers/platforms (see MDN). I'd say, running that set of browsers (those in sauce-browsers excluding mobile) is a good starting point.

Code proposal: we generalize the current browser info file, and pass the file name optionally on the command line. Since sauce-platforms is just an exported object, we can have a "core" file that's required by an "extras" list, to hold stuff like mobile browsers that aren't ready for primetime yet. (I'll copy this idea into the test-automation-spec etherpad so it doesn't get lost). This is a nice-to-have.

Which branches do we run against, and how often? This is mentioned at a high level in the automated testing wiki page on github.

  • We'll run all tests/all browsers against stage and prod on a long cron (every 4 or 6 hours).
  • We'll run just the health-check tests, on just one stable browser/OS combo, against stage and prod every 5 minutes, as a monitoring stopgap. (This is a temporary monitor and I'll be improving our monitoring after this crew spins down.)
  • We'll run all tests/all browsers against dev on git push, as well as on a 4 or 6 hour cron.

This is all already implemented in Jenkins on ci.mozilla.org; I just need to update the invocations to use our node tests instead of the old py tests.

What about Travis testing PRs where the branch lives on somebody's fork? I dunno. That's a good question. I guess Travis needs to give the git repo URL along with the SHA? Is that a tenable solution? Otherwise, maybe we can amend our git workflow practices to submit PRs from a branch living on mozilla/browserid.

6a68 and others added some commits Nov 5, 2012

@6a68 6a68 Temporarily remove mobile platforms from saucelabs selenium routine 5b8bf16
@shane-tomlinson shane-tomlinson Use libraries to create directories, copy objects, copy environment.
* use mkdirp, toolbelt used in testy.
* Correctly set the number of runners in run-all.js from the RUNNERS environment variable.
28f3e0e
Member

shane-tomlinson commented Nov 6, 2012

@6a68:

What about Travis testing PRs where the branch lives on somebody's fork? I dunno. That's a good question. I guess Travis needs to give the git repo URL along with the SHA? Is that a tenable solution? Otherwise, maybe we can amend our git workflow practices to submit PRs from a branch living on mozilla/browserid.

This is what I suspected, so I did that yesterday. ;) What I am trying to get at is an ACL of some sort. What I have done as a super basic measure is to create a whitelist set of repo owners whose repo's can be tested via testy. A POST request to testy passes the git repo URL as well as the sha. Testy looks at the repo URL, divines the repo owner, checks against the whitelist and proceeds.

This is a start, but it is still open to abuse as any random yahoo could POST to testy the URL of one of the approved owner's repos and kick off sauce tests.

What I would like is to whitelist a set of IP addresses so that we can check the ip address of the requester, and whitelist Travis IP addresses only or use HTTP authentication with a secret stored on travis to sign in. Dunno. The former is not yet possible as the awsbox proxy does not forward the ip address of the requester from the proxy to the main server. The latter I thought the whole reason we were doing this exercise is so we didn't have to store secrets on Travis.

shane-tomlinson and others added some commits Nov 6, 2012

@shane-tomlinson shane-tomlinson Exit with the correct status code, 0 if success, 1 if failure. 6403fc0
@shane-tomlinson shane-tomlinson Use toolbelt's copyExtendEnv function when extending the environment.…
… Fix jshint errors.
cadeae7
@shane-tomlinson shane-tomlinson Fix up the error checker when looking at the Sauce results. bf7b874
@shane-tomlinson shane-tomlinson Capture file open/write/close errors so that it does not bring the en…
…tire process down.
d4a4ecf
@shane-tomlinson shane-tomlinson Add mkdirp to the list of dependencies. 5861fba
@shane-tomlinson shane-tomlinson Only enable vista_chrome for now until we can better stablize the tests. 0d036ef
@6a68 6a68 minor edit: remove stale todo 4c9d2c4
@6a68 6a68 Use assert.fail in vows_harness, so XML will show failure/error.
  If we just throw a string, we get status 'skipped', which Jenkins doesn't
  interpret as a failure.
0107d6e
@6a68 6a68 Provide more context in the runner stderr output e85c7c3
@jrgm jrgm Optionally add tags to sauce sessions from the environment 6a8f69c
@jrgm jrgm use done as the callback for browser.quit 2e2109a
@6a68 6a68 Pass test file name to 'classname' vows xml attribute 1af3842
@6a68 6a68 Minor edit: add missing 'path' include to frontend-qunit-tests 88fd52b
@6a68 6a68 Minor edit: screwing around with Jenkins readability of stderr b046323
@shane-tomlinson shane-tomlinson Add wd.prototype.wclickIfExists which waits one second and clicks an …
…element if it exists. If not, no problem.

Use this in remove-email.js where the verifyWithPrimaryButton sometimes needs clicked twice and the "Is this your computer" can appear if the test takes longer than one minute.
31b7228
@shane-tomlinson shane-tomlinson Use wclickIfExists in place of the timeout solution when the "Is this…
… your computer" screen is shown.
92686f6
@shane-tomlinson shane-tomlinson Change the ordering of the tests so that secondary tests are run befo…
…re primary tests. This masks a timing problem that keeps occurring when trying to enter a secondary address and the password fields are not found.
4c4ec0e
@shane-tomlinson shane-tomlinson Add wd-extensions.prototype.wgetAttribute, use that in the returning-…
…user tests to check whether a radio button is selected.
b0dfda3
@shane-tomlinson shane-tomlinson Add a cleanup function so browsers are always torn down. 60e054a
@shane-tomlinson shane-tomlinson Update the timeout error message to indicate which element search tim…
…ed out.
7eaf1bb
@shane-tomlinson shane-tomlinson Adding info about how to run all tests in parallel.
Also add info on wclickIfExists, wgetAttribute, wclear and delay
0befe22
@lloyd lloyd don't run saucelabs tests at push time 876c9cc
@lloyd lloyd use optimist for command line parsing. implement --help and --platfor…
…m (backwards compatible)
6eef36e
@lloyd lloyd (automated testing) implement --list-tests 38d9e35
@lloyd lloyd (automation tests) implement --list-platforms 56e92b0
@lloyd lloyd (automation tests) implement --parallel with a default value of 10 an…
…d defer to environment for back compat
e3bf743
@6a68 6a68 re-enable non-mobile sauce platforms for stability stats gathering 4bc4b81
@6a68 6a68 Update browsers: ff13->16 on linux, win; add ie10 1c7b12b
@lloyd lloyd (automation tests) allow platform to be specified using globs, improv…
…e reporting for summary of what will be done
cd292f8
@lloyd lloyd allow persona environment to be specified on the command line, back c…
…ompat with ENV
5ae6fe2
@lloyd lloyd allow a glob to be specified to match individual tests 145a04c
@lloyd lloyd better -h for --tests 9a28ad2
@lloyd lloyd initial cleanup of test output 6c50bee
@lloyd lloyd make run-all.js friendly for local testing, add --local flag 46c14df
@lloyd lloyd merge improvements to run-all.js to make it nicer to invoke from the …
…command line

Conflicts:
	automation-tests/lib/sauce-platforms.js
563f7aa
@lloyd lloyd move minimatch down into automation-tests, that's the only place it i…
…s used and in the jenkins env we don't npm install the lower level
110d9fa
Owner

6a68 commented Nov 8, 2012

OK, so we've got 90%+ stability on all platforms except those added really recently (mac safari 5, win 8 IE10, mobile).

Suggest merging in, stabilizing those new platforms separately, and merging those fixes later.

Thoughts?

Contributor

lloyd commented Nov 8, 2012

dig it, will do a pass in the morning and merge!

On Nov 7, 2012, at 7:41 PM, Jared Hirsch notifications@github.com wrote:

OK, so we've got 90%+ stability on all platforms except those added really recently (mac safari 5, win 8 IE10, mobile).

Suggest merging in, stabilizing those new platforms separately, and merging those fixes later.

Thoughts?


Reply to this email directly or view it on GitHub.

Owner

6a68 commented on 6a6cf59 Nov 9, 2012

nice hack!

6a68 and others added some commits Nov 9, 2012

@6a68 6a68 lengthen idle-timeout so dual-session tests don't fail 8ffb12e
@lloyd lloyd more friendly console reporting for run-all.js 84091d1
@lloyd lloyd tighten up help language da93b54
@lloyd lloyd fix bug where tests not started in the first batch would not get incl…
…uded in aggregated results
b863024
@lloyd lloyd add cleanup of sauce VMs to health check tests so they don't hang for…
… 90 seconds in case of failure
94cd994
@lloyd lloyd fix cleanups so that they run properly under the vows harness, add mo…
…re cleanups to more tests so we don't leave vms lying around to timeout. also, fix proper exit codes under xunit and console output.
adff219
@lloyd lloyd remove debug message 2de1626
@lloyd lloyd the process must wait for cleanups to complete, otherwise they don't …
…actually get to make their HTTP transactions and cleanup VMs for failed tests
e6b7333
@6a68 6a68 Improve setup and teardown across all tests
  - update setup and teardown in all tests
  - testSetup now really handles fixture lifecycle:
    - init browser sessions using testSetup
    - teardown browsers using testSetup
  - use Q to handle control flow for async multi-browser teardown
    - quitIfDefined caused barfing, passing undefined args into a function call
9e4c4bd
@6a68 6a68 rearrange test-setup to more clearly separate private/public functions d315a1f
@6a68 6a68 bah, missed one stray newSession call. 61c3337
@jrgm jrgm tag test with the environment; drop the redundant persona which was u…
…seful in shared environment
0f8f115
@jrgm jrgm repair this test; gets 10/10 on vista_{chrome,firefox_16}, although t…
…his test creates a lot of sessions
605ba30
@shane-tomlinson shane-tomlinson Update post-update.js so that it exits with the same exit code as run…
…-all.js. Depend on this exit code making back to testy eventually so that test_selenium can exit with the same exit code and travis correctly marks the tests as pass/fail.
169427f
@shane-tomlinson shane-tomlinson Only enable Chrome and disable almost all tests until we have better …
…test stability.
167c9cd
@shane-tomlinson shane-tomlinson Use a GET request when requesting selenium results. 6e34a86
@6a68 6a68 Re-enabling tests that worked recently & switching active platform to…
… FF/vista
c8c3394
@6a68 6a68 Whoops, disabled the wrong test. 25ee29e
@6a68 6a68 Rework double button click in primary dialog.
  I can't repro locally, but sauce's vista/FF 16 consistently seems to
  succeed with the first click, then sadly wait forever and never make
  the second click. Trying to replace the second click with a click-if-exists.
  Hopefully this works on the sauce side.
ddbae38
@shane-tomlinson shane-tomlinson Move sauce-platforms to the config directory. fce4730
@shane-tomlinson shane-tomlinson Clean up the JSHint errors and warnings. 494baff
@shane-tomlinson shane-tomlinson Make sure all browser.chain() invocations pass an onError callback. d33e942
@shane-tomlinson shane-tomlinson Keep track of the current window so that it is possible to check whet…
…her the last opened window is still open. This prevents Firefox from hanging when checking for an element in a window that has already been closed.

When checking if an element exists, wait for 15 seconds before the check to ensure that the dialog goes away if it is going to go away.
0f93ab5
@shane-tomlinson shane-tomlinson Add wclickIfExists on the eyedee.me create account button for IE. 253c9c0
@6a68 6a68 Revert "work around aw snap in remove-email test issue #2691"
  This reverts commit 6a6cf59.

  The about:blank fix causes rampant destruction on FF and Chrome , reverting
  for the moment. We'll find a different fix when we focus on Chrome next week.
a208631
@6a68 6a68 Reorder vows in new-user-secondary. Fixes #2704.
  When I swapped the browser.newSession calls for testSetup.newBrowserSession
  calls, the vows got split in the wrong order, so that null was getting passed
  as the link to the browser.get call.
cb9bb4c
Owner

6a68 commented Nov 15, 2012

zomg so many commits! looking stable on FF 16/Vista, what else needs to happen to merge this puppy in today?

Owner

6a68 commented Nov 15, 2012

r+ from lloyd in IRC. merging.

@6a68 6a68 added a commit that referenced this pull request Nov 15, 2012

@6a68 6a68 Merge pull request #2670 from mozilla/kapow
Kapow!
cec83da

@6a68 6a68 merged commit cec83da into dev Nov 15, 2012

1 check failed

default The Travis build failed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment