Reuse workers? Was: Error: Cannot queue more than 200 workers

[jzaefferer]

Here's an interesting issue I just ran into trying to use browserstack-runner to run all unit tests in jQuery UI. This gist has my configuration as well as the output I got: https://gist.github.com/jzaefferer/d79717435a5a537b2a67

This run failed pretty quickly, after it tried to launch 264 workers (12 browsers x 22 test suites), with "Error: Cannot queue more than 200 workers". I guess this is an area where the approach taken by TestSwarm, with all its overhead, is somewhat superior, where a run like this needs a minimum amount of 12 workers (maybe a few more if a worker gets killed inbetween or a test has to run again).

Maybe there is some way to reuse workers within browserstack-runner? Or a different approach to this problem?

[nakula]

ironically, we saw this yesterday. We discussed quite a lot, but couldn't find a solution :( we are going to work on it next.

Another interesting case in jquery-ui is, some cases it uses iframe to run tests - which again fail with runner as of now.

[jzaefferer]

True, though I guess you were looking at the all.html tests, which aren't as important as the ones in my gist. Also that's probably the same issue as #66.

If there is no way to redirect a worker to a different URL, other options might be:

• Extend the server to keep track of which worker needs which file and serve those. Reload the page after a testsuite is done to load the next test. Forces the server to keep track of a lot more state, per worker. Depends on the ability to reload the page, which should be possible in the injected script (on the client side).
• Queue workers locally: Launch workers up to a reasonable limit, wait for those to finish, then launch the next batch. More a workaround than a solution, since it still needs a lot of workers.

Either way, the console output will likely need even more compression. Even just two lines per worker will still output 528 lines for a full run.

[dhimil]

Redirection:

More related to extending the server, pointed out in previous comment.
One way to do that would be to do the redirection from the injected patches. Example: open http://localhost:8888/test1.html?_worker=abc&redirect=test2.html. Once all the tests in test1.html are done, js can redirect to http://localhost:8888/test2.html?_worker=abc. Runner has to keep track of all the cascaded tests and their results. This cascading style should be optional as not all users would like to use this style.

iFrame:

This is not about tests using iFrame.
A browser can open http://localhost:8888/all.html. all.html can have multiple iframes each pointing to http://localhost:8888/test1.html?_worker=test1 and http://localhost:8888/test2.html?_worker=test2. Less complex, faster test execution. But UI can break, Not sure if all test frameworks work equally inside iframes (need to check).

Queuing

Agreed, but I suppose this can be handled smartly on server side.

[jzaefferer]

Using iframes is very likely to introduce new issues. TestSwarm uses iframes and we've ran into various problems that never occur when running the testsuite directly, outside of TestSwarm's iframe.

Redirecting on the client might be an alternative to reloading. If you go down that route, you'd need very solid query string parser and formatter to not break any existing query strings.

[jzaefferer]

@dhimil @nakula any progress on this? I know its a difficult problem to solve, but currently this tool is not practical to use with more then one test suite per project.

[dhimil]

I tried one solution but did not work out well. May be I'll look into this again and try to close this.

[jzaefferer]

What approach did you try? What issues did you run into?

[dhimil]

We tried to create an internal API to reuse the same VM and send another
start request to same worker/VM (kind of), but it didn't work end to end.

On Wed, May 21, 2014 at 5:54 PM, Jörn Zaefferer notifications@github.comwrote:

What approach did you try? What issues did you run into?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-43747272
.

Dhimil Gosalia,
BrowserStack.

[jzaefferer]

@nakula @dhimil any update on this?

[jzaefferer]

This is still a major blocker for anything for more then a single html file. Compositing testsuites in a page is possible, but also very error-prone. Would be great to see this addressed. Or at least have a clear answer saying it won't happen.

[JamesMGreene]

\o/