Skip to content

Loading…

Some integration and extension tests fail when running all #3343

Closed
jasonsanjose opened this Issue · 7 comments

4 participants

@jasonsanjose
Adobe Systems Incorporated member

When running grunt test-integration --suite=all, the following suites see intermittent failures: Live Development, StaticServer, NodeConnection. All of these suites rely on NodeConnection. It's not clear why they pass in isolation, but fail in the overall "all" suite. Below is a sample of the failures due to timeout.

        "Live Development": {
            "id": 140,
            "name": "Live Development",
            "specCount": 12,
            "passedCount": 6,
            "failedCount": 6,
            "specs": [
                {
                    "name": "Live Development Live Development startup and shutdown should return a ready socket on Inspector.connect and close the socket on Inspector.disconnect.",
                    "description": "should return a ready socket on Inspector.connect and close the socket on Inspector.disconnect",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 5000 msec waiting for success Inspector.connectToURL"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should establish a browser connection for an opened html file.",
                    "description": "should establish a browser connection for an opened html file",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for browser to become active"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should push changes through the browser connection.",
                    "description": "should push changes through the browser connection",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for browser to become active"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should ignore query strings in linked CSS file hrefs.",
                    "description": "should ignore query strings in linked CSS file hrefs",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for browser to become active"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should push in memory css changes made before the session starts.",
                    "description": "should push in memory css changes made before the session starts",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 10000 msec waiting for Waiting for LiveDevelopment document"
                    ]
                },
                {
                    "name": "Live Development CSS Editing should reapply in-memory css changes after saving changes in html document.",
                    "description": "should reapply in-memory css changes after saving changes in html document",
                    "passed": false,
                    "messages": [
                        "timeout: timed out after 2000 msec waiting for success LiveDevelopment.open()"
                    ]
                }
            ]
        },

        "StaticServer": {
            "id": 148,
            "name": "StaticServer",
            "specCount": 14,
            "passedCount": 10,
            "failedCount": 4,
            "specs": [
                {
                    "name": "StaticServer StaticServerDomain should trigger an event when a file path is requested.",
                    "description": "should trigger an event when a file path is requested",
                    "passed": false,
                    "messages": [
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'This is a file in folder 1.'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:266:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12",
                        "Error: Expected 71 to be greater than 500.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBeGreaterThan (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:269:37)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12"
                    ]
                },
                {
                    "name": "StaticServer StaticServerDomain should send static file contents after canceling a filter request.",
                    "description": "should send static file contents after canceling a filter request",
                    "passed": false,
                    "messages": [
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'This is a file in folder 1.'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:316:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12"
                    ]
                },
                {
                    "name": "StaticServer StaticServerDomain should override the static file server response with a new response body.",
                    "description": "should override the static file server response with a new response body",
                    "passed": false,
                    "messages": [
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'custom response'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:360:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at goAgain (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1832:18)"
                    ]
                },
                {
                    "name": "StaticServer StaticServerDomain should ignore multiple responses for the same request.",
                    "description": "should ignore multiple responses for the same request",
                    "passed": false,
                    "messages": [
                        "Error: Expected 2 to be 1.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:405:41)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12",
                        "Error: Expected '<!doctype html>\n<html data-brackets-id='5'>\n<head data-brackets-id='2'>\n<meta charset=\"UTF-8\">\n<title data-brackets-id='1'>Simple Test</title>\n<link rel=\"stylesheet\" href=\"simpleShared.css\">\n<link rel=\"stylesheet\" href=\"simple1.css\">\n</head>\n\n<body data-brackets-id='4'>\n<p data-brackets-id='3' id=\"testId\" class=\"testClass\">Brackets is awesome!</p>\n</body>\n</html>\n' to be 'good response'.\n    at new jasmine.ExpectationResult (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:102:32)\n    at null.toBe (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1194:29)\n    at path (file:///C:/Users/jasonsj/Github/brackets/src/extensions/default/StaticServer/unittests.js:410:34)\n    at jasmine.Block.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1024:15)\n    at jasmine.Queue.next_ (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1842:31)\n    at onComplete (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:1838:18)\n    at jasmine.WaitsForBlock.execute (file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2322:5)\n    at file:///C:/Users/jasonsj/Github/brackets/test/thirdparty/jasmine-core/jasmine.js:2336:12"
                    ]
                }
            ]
        },
@RaymondLim

Not sure it's only me. I can consistently reproduce one unit test failure both on my win7 and mac 10.8.2. The unit test failing is "Live Development startup and shutdown".

@jasonsanjose jasonsanjose was assigned
@njx
Adobe Systems Incorporated member
njx commented

Reviewed. To @jasonsanjose for sprint 24.

@jasonsanjose
Adobe Systems Incorporated member

Removed FBNC. #3628 was just a first step in re-arranging how we run tests on Jenkins. They are still failing more often than not. The new Jenkins jobs give us some more flexibility in how we run the tests and how we capture the results.

@jasonsanjose
Adobe Systems Incorporated member

Removing Sprint 24 milestone

@njx
Adobe Systems Incorporated member

Moved to sprint 26

@jasonsanjose
Adobe Systems Incorporated member

Too late to land for sprint 26. Punting to sprint 27.

@gruehle gruehle closed this in #4288
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.