Skip to content
This repository

RunPage should not use iframes for testing #195

Open
Krinkle opened this Issue June 15, 2012 · 16 comments

7 participants

Timo Tijhof Scott González Jörn Zaefferer James Morrin Mike Sherov Markus Staab jpvincent
Timo Tijhof
Collaborator

Right now the client runner is the RunPage. There the browser periodically asks for new runs to run. And when it gets one, it add a temporary iframe to the document that is pointed to wherever the test suite is located (can be on a different origin).

The test framework includes our inject.js file which we use to "phone home" (through window.parent, since origin restrictions only apply when a window accesses a child frame, not the other way around).

Over time <iframe> has causes more and more trouble frames. We should consider finding a better method for this. An environment that is more reliable. Ideally in a stand-alone window. But just setting window.location isn't going to do it for us, because we need to keep track of the progress. And when the tests are finished, we need to be able to send a (pretty big) javascript object back up to the RunPage, which then submits it through AJAX to the API (again we can't cross-domain AJAX from the child frame, but we can from the parent window).

Moments ago on IRC I was bringing up the idea of using window.open() to create a popup and run the tests there. As far as controllability goes, this shouldn't be an issue. In JavaScript one can close windows that are opened by the same script (right?). But then we need an alternative way to send the javascript objects (and it needs to work in old browsers as well).

I'm not sure, but I don't think a window.parent like relationship exists through window.open() (even though the original window does somehow get the authority to close it) that we can use to do window.parent.SWARM.someMethod();.

Scott González
Owner

window.opener

Timo Tijhof
Collaborator

Can opener?

Can opener

Beer opener?

Beer opener


Anyway, I heard of the (non-standard but widely implemented) window.opener (MDN) property, but it has different restrictions.

Contrary to frames, the relationship from the child window to the parent is restricted by the origin policy:

example of using window.opener

Jörn Zaefferer
Owner

We're running tests on the same origin as testswarm itself, so that's what you should be testing, not opening swarm from MDN.

And the same applies to iframes. You can open them anywhere, but you can only control their content when its on the same origin. Right?

Timo Tijhof
Collaborator

Yes, we ("jquery") are running them from the same domain (by adding a non-standard directory to the root of the TestSwarm install and exporting all the test suites in there before submission). But that is (afaik) by no means the required situation for TestSwarm.

The exact example I did earlier (opening swarm from another domain) is backwards indeed, but that doesn't matter – the principle is the same. I could just as well have done it the other way around (opening another site from swarm). Either way, window.open() does not allow the child to access window.opener (when the origin doesn't match).

I tested it just now and it is indeed the same for frames:

accessing window.parent from a child frame

It turns out I wrote it wrong yesterday as well, which caused the confusion. We do indeed need access to the parent from the - potentially cross-domain - child (not the other way around). But, we don't need access to the entire window object. (e.g. not window.parent.SWARM.someMethod), what we need is access to postMessage. And that (duh!) has an exception to this rule (both for frames and windows):

cross-domain postMessage from a child iframe

cross-domain postMessage from a child window

There is still the matter of browsers that don't support postMessage, but we already have a fallback to that (we build a <form>, that posts cross-domain to a special receiver in TestSwarm. And that receiver, being on the same domain as the parent, can then access the appropriate method directly through window.parent.SWARM.someMethod).

With frames that works (we already do that), I wasn't sure if it would also work with a window, but it turns out it does. The reason I thought it wouldn't is because I didn't think the window.opener property would be preserved when navigating to a different location. But that works OK (at least in Chrome, lets hope it works in IE6 too):

accessing window.parent from a window that started on a different as the opener, but POSTed to the same origin

Timo Tijhof Krinkle closed this issue from a commit June 26, 2012
Timo Tijhof RunPage: Create new windows instead iframes for the test suite.
* run.js:
 - Use window.open instead of creating an iframe an appending to
   #iframes.
 - Need to keep track of the window reference in a higher scope to
   be able to check its status (and close if needed) in

* RunPage: Remove now-redundant #iframes

* inject.js: Access postMessage from window.opener instead of
  window.parent.

* Saverunpage: Fix first condition check to cast to boolean instead
  of comparing. Contrary to window.parent and window.top,
  window.opener is only set if there is another window at play.
  So !== window would evaluate to true even if it is null/undefined
  (e.g. when viewing the page directly) and cause an exception on
  window.opener.SWARM

* Issues:
 - Fixes #195: RunPage should not use iframes for testing
586a597
Timo Tijhof Krinkle closed this in 586a597 June 26, 2012
Timo Tijhof Krinkle reopened this June 26, 2012
Timo Tijhof
Collaborator

So, I just pushed the branch, and it is all working as expected (haven't found any issues yet). Except that there's one blocking issue that I don't know how to solve.

"What?", you ask? The friggin' popup blocker (duh!).

The Almighty Pop-Up Blocker

  • Chrome 19:
    • Popup: Blocked.
      It is just the window being suppressed. The window is opened in a hidden process and interacting with it works fine.
    • Focus: Medium
      No focus since it is hidden, but focus is not an issue in Chrome.
    • > PASSED
  • Firefox 12:
    • Popup: Blocked.
      Completely prevented (return of window.open is null).
      Once whitelisted, it works fine.
    • Focus (once enabled): Medium
      No focus since new windows go into a tab, and new tabs are not focused, but focus is not an issue in Firefox.
    • > FAILED
  • Safari 5.1:
    • Popup: Blocked.
      Completely prevented (return of window.open is undefined).
      No visual notification of it being prevented, no whitelist in preferences. The feature can only be turned off globally, from the preferences. Once disabled, it opens actual new windows and everything is fine.
    • Focus (once enabled): Good
      Opens a new window and that window is focussed
    • > FAILED
  • Opera 11.6
    • Popup: Bocked.
      Prevented but somewhat initialized (return of window.open is a Window object, with closed!=true, but the document is not parsed yet).
      Shows dialog to allow opening it, which works. But that first window lost its window.opener reference and window.close is inaccessible. Refreshing after allowing it works fine, but only for 1 round. The next run gets blocked again.
    • Focus (once enabled): Variable
      When manually allowed the new window goes into an immediately focussed tab. The next ones usually aren't focussed.
    • > FAILED
  • IE 9
    • Popup: Blocked.
      Completely prevented (return of window.open is null).
      Choosing "Options for this site> Always allow" makes it work. The new window goes into an immediately focussed tab. However it turns out IE only supports postMessage() for frames, not for windows. It completely lacks indicating this until you call it, it throws: SCRIPT16386: No such interface supported. Feature test for window.opener.postMessage works fine. We'll have to find a way to fallback to <form> POST-ing for all IE versions (instead of just for those without support for postMessage). See also msdn blogs.
    • Focus: Good
    • > FAILED

(the '>' indicate the test result on a clean machine with the default browser settings after a cold boot (which is what BrowserStack will provide) - not the test result after interacting with whatever dialogs, resetting the (now failed) test run, and trying again).

Jörn Zaefferer
Owner

I saw you mentioning something in IRC. Is that realistic? Otherwise we should just kill that one bad test in jQuery Core, if that's all we trying to address here. I think @dmethvin already pretty much agreed to that.

Timo Tijhof
Collaborator

I am sure we can and am convinced we will eventually end up doing in-host page control and direct client-server communication, but I'm not sure I have time for it in the next few weeks. Lets keep it as a future feature.

James Morrin

I was wondering if you could point me in the right direction to get an understanding of what troubles the iframes are causing?

I am currently developing a thirdparty javascript framework in my day job. And I do a lot of cross domain iframe communication. I am a little confused why you would need to open a new window? It seems like an iframe would be fine.

Right now in the test runner i built for my company I have a test iframe which communicates with its parent using easyXDM to support older browsers. http://easyxdm.net/ The test runner iframe communicates with its parent yielding progress and test results. You cannot call functions directly on the iframe itself but you can pass async messages.

Timo Tijhof
Collaborator

Opening a new window isn't a solution either, because of popup blockers and what not. Initial research has shown it is not reliable to open a new window every now and then to run a test.

Iframes are still the way to go for now, and they're working fine (test suites for jQuery core and jQuery UI are passing fine). However there are some edge cases where Internet Explorer reports offsets wrong when run from within an iframe.

Running into this is fairly unlikely and it can be worked around.

For now we'll stick to iframes. In the long run we're likely switching to a system where the test run page becomes part of the navigational flow (as opposed to running it from a "run" page that will run the tests as children in iframes). In other words, redirecting to the test suite, running the test suite and submitting the results, and then (by the inject.js code that is inside the test runner) it will redirect back to an idle page where it will wait until the next run is sent to it (through a socket).

Anyway, that's all future talk. Unless you're hitting a problem with iframes, I'd recommend just ignoring this for now.

Mike Sherov

We're running into this now on jquery UI as well. It seems like your latest comment is the way to go @Krinkle. Redirect from the "run" page to test page, which should then submit results and redirect back to the run page.

Mike Sherov

I can work on this issue now if the pull request would be accepted. Even though the practical case of "it breaks jquery and jquery ui" are present, I'm especially on board with the philosophical case of "your test environment should match your actual environment as closely as possible", and running in an IFRAME certainly isn't.

Timo Tijhof
Collaborator

@mikesherov That depends on the approach you plan to take on the implementation. Feel free to post it here.

Mike Sherov

@Krinkle,well there are 2 issues, from my naive perspective...

  1. navigation: this would be handled by the process you've described: redirecting to the test suite, running the test suite and submitting the results, and then (by the inject.js code that is inside the test runner) it will redirect back to an idle page where it will wait until the next run is sent to it (through a socket).
  2. timeouts: 2 things here...
    1. inject.js now needs to be aware of the timeout, so it can post back to the swarm when a timeout happens. However, this doesn't cover the case of what happens when the test page breaks inject.js's ability to even report on the timeout.
    2. We'd need a cron that cleans up stalled runs after a while. This can be at the minute level of granularity, and can clean up on the next minute after the timeout.

I haven't yet done a deep dive of the code... are there other issues here that need to be covered other than what I've described above?

Markus Staab

Just a note: hoe do you implement the redirection?

Javascript location? Meta redirect? Http location header?

In case you choose http location, make sure you use temporary redirects, no permanent (http 301 vs. 302). Permanent redirects will be cached by some browsers without asking the server for the actual location..

Timo Tijhof
Collaborator

It would be JavaScript location. Because the user will be idling on the run page, and the polling (or socket push) will notify of a new run available. Redirecting from there will only redirect the asynchronous request, not the user viewport.

jpvincent

Hi

I am in an environment where I use testswarm to execute fonctional tests, because usual tool like Selenium do not exist. It's "pages" written for browsers shiped with TVs (HbbTVs). In half TVs (there are around 15 models in 2013), some tests fail because they are executed inside an iframe.
I would like to have the choice to execute them "fullscreen".

The last update to this issue was 10 months ago : do you have any plan to test this mode ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.