RunPage should not use iframes for testing #195

Krinkle opened this Issue Jun 15, 2012 · 18 comments


None yet

7 participants

Krinkle commented Jun 15, 2012

Right now the client runner is the RunPage. There the browser periodically asks for new runs to run. And when it gets one, it add a temporary iframe to the document that is pointed to wherever the test suite is located (can be on a different origin).

The test framework includes our inject.js file which we use to "phone home" (through window.parent, since origin restrictions only apply when a window accesses a child frame, not the other way around).

Over time <iframe> has causes more and more trouble frames. We should consider finding a better method for this. An environment that is more reliable. Ideally in a stand-alone window. But just setting window.location isn't going to do it for us, because we need to keep track of the progress. And when the tests are finished, we need to be able to send a (pretty big) javascript object back up to the RunPage, which then submits it through AJAX to the API (again we can't cross-domain AJAX from the child frame, but we can from the parent window).

Moments ago on IRC I was bringing up the idea of using to create a popup and run the tests there. As far as controllability goes, this shouldn't be an issue. In JavaScript one can close windows that are opened by the same script (right?). But then we need an alternative way to send the javascript objects (and it needs to work in old browsers as well).

I'm not sure, but I don't think a window.parent like relationship exists through (even though the original window does somehow get the authority to close it) that we can use to do window.parent.SWARM.someMethod();.



Krinkle commented Jun 16, 2012

Can opener?

Beer opener?

Anyway, I heard of the (non-standard but widely implemented) window.opener (MDN) property, but it has different restrictions.

Contrary to frames, the relationship from the child window to the parent is restricted by the origin policy:

example of using window.opener


We're running tests on the same origin as testswarm itself, so that's what you should be testing, not opening swarm from MDN.

And the same applies to iframes. You can open them anywhere, but you can only control their content when its on the same origin. Right?

Krinkle commented Jun 16, 2012

Yes, we ("jquery") are running them from the same domain (by adding a non-standard directory to the root of the TestSwarm install and exporting all the test suites in there before submission). But that is (afaik) by no means the required situation for TestSwarm.

The exact example I did earlier (opening swarm from another domain) is backwards indeed, but that doesn't matter – the principle is the same. I could just as well have done it the other way around (opening another site from swarm). Either way, does not allow the child to access window.opener (when the origin doesn't match).

I tested it just now and it is indeed the same for frames:

accessing window.parent from a child frame

It turns out I wrote it wrong yesterday as well, which caused the confusion. We do indeed need access to the parent from the - potentially cross-domain - child (not the other way around). But, we don't need access to the entire window object. (e.g. not window.parent.SWARM.someMethod), what we need is access to postMessage. And that (duh!) has an exception to this rule (both for frames and windows):

cross-domain postMessage from a child iframe

cross-domain postMessage from a child window

There is still the matter of browsers that don't support postMessage, but we already have a fallback to that (we build a <form>, that posts cross-domain to a special receiver in TestSwarm. And that receiver, being on the same domain as the parent, can then access the appropriate method directly through window.parent.SWARM.someMethod).

With frames that works (we already do that), I wasn't sure if it would also work with a window, but it turns out it does. The reason I thought it wouldn't is because I didn't think the window.opener property would be preserved when navigating to a different location. But that works OK (at least in Chrome, lets hope it works in IE6 too):

accessing window.parent from a window that started on a different as the opener, but POSTed to the same origin

@Krinkle Krinkle was assigned Jun 25, 2012
@Krinkle Krinkle added a commit that closed this issue Jun 26, 2012
@Krinkle Krinkle RunPage: Create new windows instead iframes for the test suite.
* run.js:
 - Use instead of creating an iframe an appending to
 - Need to keep track of the window reference in a higher scope to
   be able to check its status (and close if needed) in

* RunPage: Remove now-redundant #iframes

* inject.js: Access postMessage from window.opener instead of

* Saverunpage: Fix first condition check to cast to boolean instead
  of comparing. Contrary to window.parent and,
  window.opener is only set if there is another window at play.
  So !== window would evaluate to true even if it is null/undefined
  (e.g. when viewing the page directly) and cause an exception on

* Issues:
 - Fixes #195: RunPage should not use iframes for testing
@Krinkle Krinkle closed this in 586a597 Jun 26, 2012
@Krinkle Krinkle reopened this Jun 26, 2012
Krinkle commented Jun 26, 2012

So, I just pushed the branch, and it is all working as expected (haven't found any issues yet). Except that there's one blocking issue that I don't know how to solve.

"What?", you ask? The friggin' popup blocker (duh!).

The Almighty Pop-Up Blocker

  • Chrome 19:
    • Popup: Blocked.
      It is just the window being suppressed. The window is opened in a hidden process and interacting with it works fine.
    • Focus: Medium
      No focus since it is hidden, but focus is not an issue in Chrome.
    • PASSED

  • Firefox 12:
    • Popup: Blocked.
      Completely prevented (return of is null).
      Once whitelisted, it works fine.
    • Focus (once enabled): Medium
      No focus since new windows go into a tab, and new tabs are not focused, but focus is not an issue in Firefox.
    • FAILED

  • Safari 5.1:
    • Popup: Blocked.
      Completely prevented (return of is undefined).
      No visual notification of it being prevented, no whitelist in preferences. The feature can only be turned off globally, from the preferences. Once disabled, it opens actual new windows and everything is fine.
    • Focus (once enabled): Good
      Opens a new window and that window is focussed
    • FAILED

  • Opera 11.6
    • Popup: Bocked.
      Prevented but somewhat initialized (return of is a Window object, with closed!=true, but the document is not parsed yet).
      Shows dialog to allow opening it, which works. But that first window lost its window.opener reference and window.close is inaccessible. Refreshing after allowing it works fine, but only for 1 round. The next run gets blocked again.
    • Focus (once enabled): Variable
      When manually allowed the new window goes into an immediately focussed tab. The next ones usually aren't focussed.
    • FAILED

  • IE 9
    • Popup: Blocked.
      Completely prevented (return of is null).
      Choosing "Options for this site> Always allow" makes it work. The new window goes into an immediately focussed tab. However it turns out IE only supports postMessage() for frames, not for windows. It completely lacks indicating this until you call it, it throws: SCRIPT16386: No such interface supported. Feature test for window.opener.postMessage works fine. We'll have to find a way to fallback to <form> POST-ing for all IE versions (instead of just for those without support for postMessage). See also msdn blogs.
    • Focus: Good
    • FAILED

(the '>' indicate the test result on a clean machine with the default browser settings after a cold boot (which is what BrowserStack will provide) - not the test result after interacting with whatever dialogs, resetting the (now failed) test run, and trying again).


I saw you mentioning something in IRC. Is that realistic? Otherwise we should just kill that one bad test in jQuery Core, if that's all we trying to address here. I think @dmethvin already pretty much agreed to that.

Krinkle commented Jun 29, 2012

I am sure we can and am convinced we will eventually end up doing in-host page control and direct client-server communication, but I'm not sure I have time for it in the next few weeks. Lets keep it as a future feature.

treasonx commented Oct 6, 2012

I was wondering if you could point me in the right direction to get an understanding of what troubles the iframes are causing?

I am currently developing a thirdparty javascript framework in my day job. And I do a lot of cross domain iframe communication. I am a little confused why you would need to open a new window? It seems like an iframe would be fine.

Right now in the test runner i built for my company I have a test iframe which communicates with its parent using easyXDM to support older browsers. The test runner iframe communicates with its parent yielding progress and test results. You cannot call functions directly on the iframe itself but you can pass async messages.

Krinkle commented Oct 8, 2012

Opening a new window isn't a solution either, because of popup blockers and what not. Initial research has shown it is not reliable to open a new window every now and then to run a test.

Iframes are still the way to go for now, and they're working fine (test suites for jQuery core and jQuery UI are passing fine). However there are some edge cases where Internet Explorer reports offsets wrong when run from within an iframe.

Running into this is fairly unlikely and it can be worked around.

For now we'll stick to iframes. In the long run we're likely switching to a system where the test run page becomes part of the navigational flow (as opposed to running it from a "run" page that will run the tests as children in iframes). In other words, redirecting to the test suite, running the test suite and submitting the results, and then (by the inject.js code that is inside the test runner) it will redirect back to an idle page where it will wait until the next run is sent to it (through a socket).

Anyway, that's all future talk. Unless you're hitting a problem with iframes, I'd recommend just ignoring this for now.


We're running into this now on jquery UI as well. It seems like your latest comment is the way to go @Krinkle. Redirect from the "run" page to test page, which should then submit results and redirect back to the run page.


I can work on this issue now if the pull request would be accepted. Even though the practical case of "it breaks jquery and jquery ui" are present, I'm especially on board with the philosophical case of "your test environment should match your actual environment as closely as possible", and running in an IFRAME certainly isn't.

Krinkle commented Nov 6, 2012

@mikesherov That depends on the approach you plan to take on the implementation. Feel free to post it here.


@Krinkle,well there are 2 issues, from my naive perspective...

  1. navigation: this would be handled by the process you've described: redirecting to the test suite, running the test suite and submitting the results, and then (by the inject.js code that is inside the test runner) it will redirect back to an idle page where it will wait until the next run is sent to it (through a socket).
  2. timeouts: 2 things here...
    1. inject.js now needs to be aware of the timeout, so it can post back to the swarm when a timeout happens. However, this doesn't cover the case of what happens when the test page breaks inject.js's ability to even report on the timeout.
    2. We'd need a cron that cleans up stalled runs after a while. This can be at the minute level of granularity, and can clean up on the next minute after the timeout.

I haven't yet done a deep dive of the code... are there other issues here that need to be covered other than what I've described above?

staabm commented Nov 6, 2012

Just a note: hoe do you implement the redirection?

Javascript location? Meta redirect? Http location header?

In case you choose http location, make sure you use temporary redirects, no permanent (http 301 vs. 302). Permanent redirects will be cached by some browsers without asking the server for the actual location..

Krinkle commented Nov 9, 2012

It would be JavaScript location. Because the user will be idling on the run page, and the polling (or socket push) will notify of a new run available. Redirecting from there will only redirect the asynchronous request, not the user viewport.



I am in an environment where I use testswarm to execute fonctional tests, because usual tool like Selenium do not exist. It's "pages" written for browsers shiped with TVs (HbbTVs). In half TVs (there are around 15 models in 2013), some tests fail because they are executed inside an iframe.
I would like to have the choice to execute them "fullscreen".

The last update to this issue was 10 months ago : do you have any plan to test this mode ?

Krinkle commented Sep 12, 2015

@jpvincent If you know a way to trigger fullscreen from JavaScript, do that from inside your test page – not from TestSwarm. If you try it in TestSwarm, it would make the TestSwarm wrapper page fullscreen, not the client unit test page.

TestSwarm is responsible for running tests. The user is responsible for configuring browsers and pointing them to the swarm.

Krinkle commented Sep 12, 2015

For unit testing (TestSwarm's primary purpose) iframes should not be a major inconvenience. jQuery Core, QUnit, and others are passing in it fine. We've had very few issues in all these years.

For UI testing, iframes can be more cumbersome. But I find in practice that when a functional library is doing heavy DOM interaction in a unit test means, it usually means one of four things:

  • It's testing UI components with a unit test framework.
  • It's unit testing a function by needlessly going through the DOM to trigger it – call functions directly.
  • It's not testing actually user code at all – but needless confirming that firing events results in their callback being run. (Upstream does this for you already; Add integration tests instead e.g. using webdriver).
  • It's jQuery.

But most importantly, the obvious alternative (a pop-up) is not a viable solution as outlined in previous comments (e.g. pop-up blocker). The only viable work-around is to rewrite the navigation code to use redirect management in form of a host page. I'm open to that but would recommend a separate issue for that. Or possibly a different framework instead of TestSwarm (e.g. Karma).

@Krinkle Krinkle closed this Sep 12, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment