TestSwarm Next #205

Closed
Krinkle opened this Issue Jul 2, 2012 · 10 comments

Comments

Projects
None yet
3 participants
Member

Krinkle commented Jul 2, 2012

Over the past few weeks it has come up several times that our current system is hitting the limit of its design. Although the recent rewrite for 1.0 has introduced a lot of new features, at its heart the logic is still the same.

The general idea for a possible next iteration is to incorporate a more on-demand / event-loop based service, that will be primarily invoked from the command line and written in node.

Some of the features / workflows:

  • Usable from cli and no long-term storage (e.g. no MySQL database)
  • Event driven (e.g. a plugin that listens to newly submitted jobs and does stuff)
  • Consistent and proper linking infrastructure from client to server to plugins (e.g. a browserstack plugin should be able to start browsers when it needs to, and be able to know instantly when it has connected, and when it has disconnected)
  • The client-side test framework aggregator should create an elaborate report card in JSON that is convertible to jUnit/Ant XML for Jenkins and perhaps other popular formats
  • Controlling of browser clients through direct communication between server and client (i.e. through JSONP or WebSockets), no iframes, popups or host pages.

We might consider a possible merge with Yeti even?

Member

Krinkle commented Jul 2, 2012

Quick brain dump for possible implementation of ability for browser-manager plugins to make a link between their browser worker and the testswarm client:

sclient = new swarm.ScheduledClient();
worker = foobar.newWorker( settings, { url: sclient.getJoinUrl() } ); // returns a url to the testswarm server with a token that allows it to connect as a pre-registered client

// wMap[worker.id] = sclient;
// cMap[sclient.id] = worker;

swarm.on( 'clientConnect', function ( client ) {
  // client.id
 ..

swarm.on( 'clientRunresult', function ( client, run ) {
  // client.id

swarm.on( 'clientDisconnect', function ( client ) {
  // client.id
Owner

jzaefferer commented Jul 2, 2012

One of the AngularJS developers showed me Testacular/Karma yesterday. That looks like its goes in a very similar direction to what you've outlined here: http://karma-runner.github.io/

Contributor

jayarjo commented Jul 2, 2012

Why don't you simply take Yeti then? It's pretty solid already and has QUnit adapters (it's very easy to write more).

Actually I came back to TestSwarm from Yeti, since it didn't provide a way to serve dynamic server-side responses written in other languages then node js. On big part I already wrote a node module (extending bunyip) that encapsulates what you've outlined here. The only problem standing on my way was the absence of detailed results. I can add that quickly though if you merge the changes in my fork. We could also fix the json format (is it outlined anywhere - what should it be?).

Another shortcoming of Yeti that I've encountered was that it was not able to test remote test files (through absolute urls), one must have a server and client serving those test files (local to client).

Member

Krinkle commented Jul 2, 2012

@jayarjo I don't know what Yeti's response to that feature request was, but it seems like something that is not (and imho should not) be possible. The serving of other content (like test suites) is unrelated to both Yeti and TestSwarm.

At least in TestSwarm, the job submission takes a url where the test suite is served. The submitter should take care of hosting the test suite and the application it tests - independent from TestSwarm.

And in the model where a dedicated server is created on-the-fly (as in Yeti), this separation is even more enforced. Whether or not Yeti has its server running should have nothing to do with whether the url to the test suite works. This also prevents having a situation in which your test server and swarm server have to be completely compatible in every aspect. And it allows you to test the application while TestSwarm/Yeti is not active and browse around and then submit the url for testing etc.


I've been in contact with a developer at Yeti as of a few weeks, and also mentioned Yeti in the opening thread. So, yes, "taking Yeti" is an option. Although we do need some changes for it to be usable for jQuery projects (which we can hopefully do upstream).

Contributor

jayarjo commented Jul 2, 2012

Some tests, at least in our case (we maintain file uploader) require exchanging information with the server. And the language of choice for the server-side historically turned out to be php. There's no easy way to run such a combined test suite on Yeti. The only choice is rewrite it in node js (not sure if this will work on Yeti though - but should).

So does this mean that you guys basically are going to drop TestSwarm, after all the efforts? :)

Contributor

jayarjo commented Jul 2, 2012

Here's pull request: #206

Member

Krinkle commented Jul 2, 2012

@jayarjo We (where "we" is Wikimedia, not jQuery) use TestSwarm for the continuos integration of MediaWiki's javascript test suites. Those test suites are executed on pages generated by PHP. As I described above, this works perfectly fine and it will work in Yeti as well. The main point is to not try to put it all in directory but keep it separate. You likely already have a apache set up on both your workstation as well as the testing server. Node can just run along side that and sets up its own server independently while running tests.

Anyway, back on topic: The whole point on this thread is to get insight into what we need, want, have and how that shapes into the future. We haven't decided anything yet.

Contributor

jayarjo commented Jul 4, 2012

To my knowing Yeti can only serve html files and only from it's internal http server. In case you guys know how to make it load anything else or maybe do a relay through Apache, we would be glad if you share it with us, cause it really solves all our problems then.

But right now TestSwarm is in much better form (thanks to @Krinkle) then when I initially forked it, and it requires very slight modifications to make it compatible with console mode (do the test, provide results, save nothing). And I think it should be brought to that state, no matter will you continue to maintain it or drop in favor of anything else. Converting to a node js module, for me personally doesn't bear any sense, as there already are some very good tools (including Yeti), and it would be more logical to make them better then introduce something written from scratch.

Member

Krinkle commented Dec 18, 2013

To give an update on this, this is what I'm thinking is our direction now. This isn't an official statement, just a general update on where I'm headed given the current information and discussions we've had so far.

TestSwarm will not be rewritten as a "nodejs library that runs fully on the command line without a long-running web server". The way TestSwarm currently works may not be ideal for every possible scenario, but it also has a good side.

TestSwarm has a place

It offers many features that are unique to the way it works. If you don't have an existing continuous integration portal (like Jenkins), or if your project has a really strong focus on browsers (e.g. not a web app of which you'd like to test the front-end, but something like jQuery or jQuery UI) then TestSwarm can function is a separate portal focussed only on browsers. A dedicated interface like the following can be quite appealing: http://swarm.jquery.org/project/qunit

screen shot 2013-12-18 at 22 02 35

(see also the clients and projects dashboards http://swarm.jquery.org/ and http://swarm.jquery.org/projects)

Such interface could be constructed on a separate server with information out of your Jenkins or Travis install, but the point is TestSwarm has this. And if that's all you need, it does this very well.

Aside from presentation, it also comes with certain features that might be appealing to you. Such as:

  • Create, delete and reset jobs from the web interface.
  • Retroactively reset past builds or runs, and have it replace the results (while still preserving original results by permalink).
  • Open your browser swarm up to public joining and have your own machines idle, or close it down and set up something like testswarm-browserstack to manage the swarm for you.
  • Access the test suites directly and run them in your own browser without submitting results back to TestSwarm.

Regarding resetting builds or runs, though this shouldn't be needed an ideal environment, we've learned it can be quite useful in practice as browsers will mess up eventually. Thus requiring you to rerun the same build again. No matter how hard you try, client browsers will pollute results or otherwise cause failures not caused by your software (even when you use BrowserStack or SauceLabs, they too sometimes make mistakes in their virtual machine images causing it to open the wrong browser version, or suddenly disconnect, or accidentally minimise the window etc.). In a continuous integration environment like Travis or Jenkins you would presumably just kick off a new build (adding an extra entry to your build history). But it can be quite nice having a clean project history that no longer exposes internal issues with the continuous integration infrastructure once they are resolved.

Next?

I don't know at this time how long TestSwarm will remain an actively supported project by jQuery or by myself. For the short to medium term, it will remain actively supported. Certainly as long as it is still the primary and only cross-browser testing tool in use for the jQuery projects.

However as mentioned last year, jQuery is interested in making it easier to run cross-browser tests locally as well as from Jenkins or Travis without having to deal with a long-running server or database. For that I was previously looking at Yeti, I'm currently looking into Karma (internal jquery/infrastructure#212). If and when jQuery makes the switch, TestSwarm will continue to exist. At least personally I'll ensure maintenance. But for active feature development it might come to rely more on support from the community 🌟, but that's never been an issue.

Thanks for everything and have a good 2014!

-- Krinkle
:bowtie:

Krinkle added the declined label Mar 14, 2017

Member

Krinkle commented Mar 14, 2017 edited

Closing per #155 (comment)

Krinkle closed this Mar 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment