Some integration and extension tests fail when running all #3343

Closed
jasonsanjose opened this Issue Apr 4, 2013 · 7 comments

Comments

Projects
None yet
4 participants
@jasonsanjose
Member

jasonsanjose commented Apr 4, 2013

When running grunt test-integration --suite=all, the following suites see intermittent failures: Live Development, StaticServer, NodeConnection. All of these suites rely on NodeConnection. It's not clear why they pass in isolation, but fail in the overall "all" suite. Below is a sample of the failures due to timeout.

        "Live Development": {
            "id": 140,
            "name": "Live Development",
            "specCount": 12,
            "passedCount": 6,
            "failedCount": 6,
            "specs": [
                {
                    "name": "Live Development Live Development startup and shutdown should return a ready socket on Inspector.connect and close the socket on Inspector.disconnect.",
                    "description": "should return a ready socket on Inspector.connect and close the socket on Inspector.disconnect",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 5000 msec waiting for success Inspector.connectToURL"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should establish a browser connection for an opened html file.",
                    "description": "should establish a browser connection for an opened html file",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for browser to become active"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should push changes through the browser connection.",
                    "description": "should push changes through the browser connection",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for browser to become active"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should ignore query strings in linked CSS file hrefs.",
                    "description": "should ignore query strings in linked CSS file hrefs",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for browser to become active"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should push in memory css changes made before the session starts.",
                    "description": "should push in memory css changes made before the session starts",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for LiveDevelopment document"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should reapply in-memory css changes after saving changes in html document.",
                    "description": "should reapply in-memory css changes after saving changes in html document",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 2000 msec waiting for success LiveDevelopment.open()"
                    ]
                }
            ]
        },

        "StaticServer": {
            "id": 148,
            "name": "StaticServer",
            "specCount": 14,
            "passedCount": 10,
            "failedCount": 4,
            "specs": [
                {
                    "name": "StaticServer StaticServerDomain should trigger an event when a file path is requested.",
                    "description": "should trigger an event when a file path is requested",
                    "passed": false,
                    "messages": [
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'This is a file in folder 1.'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:266:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12",
                        "Error: Expected 71 to be greater than 500.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBeGreaterThan (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:269:37)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12"
                    ]
                },
                {
                    "name": "StaticServer StaticServerDomain should send static file contents after canceling a filter request.",
                    "description": "should send static file contents after canceling a filter request",
                    "passed": false,
                    "messages": [
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'This is a file in folder 1.'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:316:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12"
                    ]
                },
                {
                    "name": "StaticServer StaticServerDomain should override the static file server response with a new response body.",
                    "description": "should override the static file server response with a new response body",
                    "passed": false,
                    "messages": [
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'custom response'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:360:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at goAgain (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1832:18)"
                    ]
                },
                {
                    "name": "StaticServer StaticServerDomain should ignore multiple responses for the same request.",
                    "description": "should ignore multiple responses for the same request",
                    "passed": false,
                    "messages": [
                        "Error: Expected 2 to be 1.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:405:41)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12",
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'good response'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:410:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12"
                    ]
                }
            ]
        },
@RaymondLim

This comment has been minimized.

Show comment
Hide comment
@RaymondLim

RaymondLim Apr 4, 2013

Contributor

Not sure it's only me. I can consistently reproduce one unit test failure both on my win7 and mac 10.8.2. The unit test failing is "Live Development startup and shutdown".

Contributor

RaymondLim commented Apr 4, 2013

Not sure it's only me. I can consistently reproduce one unit test failure both on my win7 and mac 10.8.2. The unit test failing is "Live Development startup and shutdown".

@ghost ghost assigned jasonsanjose Apr 8, 2013

@njx

This comment has been minimized.

Show comment
Hide comment
@njx

njx Apr 8, 2013

Member

Reviewed. To @jasonsanjose for sprint 24.

Member

njx commented Apr 8, 2013

Reviewed. To @jasonsanjose for sprint 24.

@redmunds

This comment has been minimized.

Show comment
Hide comment
@redmunds

redmunds Apr 26, 2013

Contributor

FBNC for @jasonsanjose

Contributor

redmunds commented Apr 26, 2013

FBNC for @jasonsanjose

@jasonsanjose

This comment has been minimized.

Show comment
Hide comment
@jasonsanjose

jasonsanjose Apr 26, 2013

Member

Removed FBNC. #3628 was just a first step in re-arranging how we run tests on Jenkins. They are still failing more often than not. The new Jenkins jobs give us some more flexibility in how we run the tests and how we capture the results.

Member

jasonsanjose commented Apr 26, 2013

Removed FBNC. #3628 was just a first step in re-arranging how we run tests on Jenkins. They are still failing more often than not. The new Jenkins jobs give us some more flexibility in how we run the tests and how we capture the results.

@jasonsanjose

This comment has been minimized.

Show comment
Hide comment
@jasonsanjose

jasonsanjose Apr 26, 2013

Member

Removing Sprint 24 milestone

Member

jasonsanjose commented Apr 26, 2013

Removing Sprint 24 milestone

@njx

This comment has been minimized.

Show comment
Hide comment
@njx

njx May 21, 2013

Member

Moved to sprint 26

Member

njx commented May 21, 2013

Moved to sprint 26

@jasonsanjose

This comment has been minimized.

Show comment
Hide comment
@jasonsanjose

jasonsanjose Jun 7, 2013

Member

Too late to land for sprint 26. Punting to sprint 27.

Member

jasonsanjose commented Jun 7, 2013

Too late to land for sprint 26. Punting to sprint 27.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment