test failures on windows firefox

[aclement]

Running clientServerTests, two failures on Firefox (17.0.1). These work on chrome.

Nav History tests: testHistorycrumb5 (2, 6, 8)Rerun

okay
okay
okay
failed
Expected:   

"/clientServerTests?/d:/gits/scripted/tests/client/test-resources/foo.js#{\"main\":{\"range\":[6,7],\"scroll\":15}}"

Result:     

"/clientServerTests?/d:/gits/scripted/tests/client/test-resources/foo.js#6,7"

Diff:   

"/clientServerTests?/d:/gits/scripted/tests/client/test-resources/foo.js#{\"main\":{\"range\":[6,7],\"scroll\":15}}" "/clientServerTests?/d:/gits/scripted/tests/client/test-resources/foo.js#6,7" 

Source:     

tests.testHistorycrumb5@http://localhost:7261/tests/client/navHistory/navHistoryTests.js:250
Test.run@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:136
Test.queue/run/<@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:279
process@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:1277
QUnit.start/<@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:383

okay
okay
okay
failed
Expected:   

"/clientServerTests?/d:/gits/scripted/tests/client/test-resources/bar.js#{\"main\":{\"range\":[15,25],\"scroll\":30}}"

Result:     

"/clientServerTests?/d:/gits/scripted/tests/client/test-resources/bar.js#15,25"

Diff:   

"/clientServerTests?/d:/gits/scripted/tests/client/test-resources/bar.js#{\"main\":{\"range\":[15,25],\"scroll\":30}}" "/clientServerTests?/d:/gits/scripted/tests/client/test-resources/bar.js#15,25" 

Source:     

tests.testHistorycrumb5@http://localhost:7261/tests/client/navHistory/navHistoryTests.js:264
Test.run@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:136
Test.queue/run/<@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:279
process@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:1277
QUnit.start/<@http://localhost:7261/scripts/lib/qunit/qunit-1.10.0.js:383

Nav History tests: asyncTestToggleSide (1, 15, 16)Rerun

okay
okay
expected no sub-editor
okay
okay
okay
okay
okay
okay
expected no sub-editor
okay
okay
Expected a subeditor
Source:     

.fail@http://localhost:7261/tests/client/assert.js:23
testLocation@http://localhost:7261/tests/client/navHistory/navHistoryTests.js:549
tests.asyncTestToggleSide/<@http://localhost:7261/tests/client/navHistory/navHistoryTests.js:662

okay
okay
expected no sub-editor

[aeisenberg]

I changed the title of this issue since it is not just windows where the tests are failing. I also see failures in firefox on mac. Other users have reported errors in Chrome on Mac.

I need to determine what the problematic tests are and refactor them so they are less brittle.

This is difficult since the tests are so tied to the dom and events that happen to it.

[aeisenberg]

Currently,

Nav History tests: testHistorycrumb5 (2, 6, 8)

failing for me on mac/firefox. Same error as Andy reports in original comment.

[aeisenberg]

The first problem is fixable. Looks like a slight difference between the way that firefox and chrome handle scrolling (both are behaving correctly when I try it manually). I added a switch so that the expected result is different depending on whether or not it is firefox or chrome.

I don't really like this solution since it will potentially require different test assertions for each browser. I have not tried this on phantomjs or safari, so I may need to add a switch statement for the assertion.

The second problem is more insidious. The failures occur when there is a programmatic forward or back click. It looks like this action is triggered asynchronously, but there is no way to listen for the event's completion. Currently, I have an arbitrary 1000ms setTimeout after the history.forward and history.back calls. This seems to work on chrome (mostly), but fails in firefox presumably because the browser takes longer to perform the action.

if there is no forward/back event that I can look for, then I should be hooking into some other event instead rather than rely on a timeout.

[aeisenberg]

The commit above makes navHistoryTests.js more reliable. I removed the setTimeouts around the history.forward and back calls. Instead, I added an extra event handler to check that the page state had correctly changed after history forward and back buttons were pressed.

This wasn't enough. I also needed to add an extra 0ms timeout call to ensure that the non-test pop state handler was added before tests started running.

Working for me now on firefox and chrome. Should test on windows, but I am feeling fairly confident about this change.

[aeisenberg]

This is working locally for me in the phantomjs build. Pushed cf2c1fa to github so that travis ci will pick it up. Let's see if it works. If so, I will close this bug.

[aeisenberg]

Yippee!! https://travis-ci.org/scripted-editor/scripted/builds/4135050