Simulate external servers with Node.js+Javascript instead of (Apache|mongoose|…)+PHP

[gibson042]

This will allow us to set up and tear down the server with grunt and track all dependencies in package.json, but requires reimplementing our PHP scripts and probably (hopefully?) also establishing a mechanism for communicating the listening port to QUnit.

[markelog]

There is complication about that - testswarm requires ability to execute tests on any previous commit not just the current one. Otherwise it would done a long time ago.

@Krinkle have more info about that.

[markelog]

Or just move away from testswarm, as it was suggested by the infra team, browserstack-tunnel or karma our candidates for the replacement.

But those requires some good amount of time to do it, at least karma does, which is my preference.

[ChrisAntaki]

Intern looks like an option as well jquery/testswarm@dd6002f

[markelog]

Does intern offers qunit support? I was under impression it has its own framework.

/cc @csnover

[csnover]

You could certainly load QUnit into it if you wanted (I have heard of some people dumping Mocha in there), but it’s a kind of inception if you do since it comes with quite robust unit testing built-in :)

I’ve actually got a mostly-working branch with a full* QUnit API compatible interface so, with only a small amount of work to address the TODOs (probably only a day or two of work?), you should be able to drop existing QUnit tests into Intern by wrapping them with define([ 'intern!qunit'], function (QUnit) { }) and have them Just Work.

(And then, fingers crossed!, I can convince team QUnit to come over & help me get cool new stuff into Intern more quickly :)))

* Well it was, prior to the most recent QUnit release. Now it is slightly outdated, but it should be easy to update, since from what I could see the QUnit update was adding some features that Intern already has so it’s easy to map them

[Krinkle]

Brain dump:

Assumption: This dynamic server would be auto-started by Grunt, and uses automatic port choosing in order to run concurrently with other services on the same computer (e.g. projects developers run locally, as well as concurrent builds in Jenkins). Alternatively, we'd block this on migrating to Travis CI (no port clash) and/or pick unique ports for each git branch in each git repo.

---

As long as TestSwarm is used (unless you're okay with ajax tests not running in TestSwarm), we can't have entry points not automatically served by common installations of Apache and nginx (hence PHP was used).

Any server set up by Grunt won't exist by the time TestSwarm distributes the test urls to BrowserStack. These test urls point at the not-so-static "static" web server serving build artefacts from Jenkins jobs for upto 30 days at unique subdirectories (html/js/css and php, served by nginx). It can also be re-ran from TestSwarm at any point in the future (within 30 days).

Even if we'd remove the re-run feature in TestSwarm, QUnit would still have to know about the hostname and port the server is on. And the url would have to be publicly exposed to BrowserStack. Though of course, this wouldn't be an issue if it was using an ssh tunnel to BrowserStack. But I'm not sure that is feasible in the current testswarm-browserstack setup.

[gibson042]

For the purposes of CI generally (as opposed to local development) and TestSwarm in particular, I imagine we'd still use the stable servers for builds.jenkins.jquery.com/jquery/{build}/… URLs, they'd just defer to grunt-managed node.js servers for dynamic responses. There are many ways to skin this cat, but here's one example:

• On inbound request to a build URL, the PHP handler verifies that the corresponding port is known
  • ...and if not, spawns the build's grunt test-server and then checks output after a little while to discover it (with suitable failure handling, perhaps kill the spawned process and respond 404)
• PHP gets its responses from forwarding the request to that port (with suitable failure-to-connect handling, perhaps decrement available tries and replay the original request)
• The test-server grunt task leverages grunt-contrib-connect and possibly also node-proper-lockfile, something like:

// Connect to already-running server listening at port specified in the runfile
tryConnecting();
function tryConnecting() {
  try {
    port = fs.readFileSync( runfile );
    req = http.get({ port: port || "invalid" }, function() {
      console.log( "Server running at port " + port );
    });

    // On failure, try to _start_ a server
    req.on( "error", tryStarting );
  } catch ( err ) {
    tryStarting( err );
  }
}

function tryStarting( err ) {
  console.warn( err.message );

  // Lock the runfile for our use
  lockfile.lock(
    runfile,
    {
      // Consider existing locks stale after 7 seconds
      stale: 7000,

      // Try acquisition for up to 7 seconds (2x backoff per retry)
      { retries: 3, minTimeout: 1000 }
    },
    function( err, release ) {
      // Got the lock
      if ( !err ) {
        var keepAlive = grunt.config( "connect.testServer.keepAlive" ),
          onCreate = grunt.config( "connect.testServer.onCreateServer" );

        // Server creation listener
        onCreate.push(function( server, connect, options ) {
          console.log( "Server running at port " + options.port );

          // Write out the runfile and register a shutdown listener to clear it
          fs.writeFileSync( runfile, options.port + "" );
          server.on( "close", function() {
            fs.writeFile( runfile, "" );
            release();
          });

          // Schedule a deferable future shutdown
          if ( keepAlive ) {
            grunt.config( "connect.testServer.shutdown" ) =
              setTimeout( server.close.bind( server ), keepAlive );
          }
        });

        // Start the server daemon (only self-terminating by the above)
        grunt.task.run( "connect:testServer:keepalive" );

      // If we lost a race for the lock, try connecting to the winner
      } else if ( err.code === "ELOCKED" ) {
        console.warn( "Couldn't get lock on " + runfile );
        tryConnecting();

      // Bail on any other error
      } else {
        throw err;
      }
    }
  );
}

[csnover]

Since I was summoned already and so am seeing replies on this, I would just also point out that Dojo core has a JSGI server for mock services set up with its Intern test suite, so you don’t need to do much work again, you could just take that code and adapt it for your own use. It’s used to do the same thing the jQuery PHP test scripts are used for (testing Dojo’s I/O code). The services are in the tests/services directory.

[Krinkle]

Regarding incompatibility between TestSwarm (which independently and outside the Grunt/Jenkins run context) and Travis CI / local development, we can use the isSwarm variable that already exists in jQuery core unit tests to make any decisions that need to vary. (E.g. to change the url paths slightly, in case we don't want the .php suffix for the Node.js version of these endpoints)

More likely though, from what I can tell there isn't any need for the urls to vary. We could easily implement the same urls both ways.

The Ajax tests are already written with the expectation that they are served over HTTP (e.g. any static server will work, and if served over file:/// the Ajax tests are already skipped). The tests make requests to the PHP files using relative urls.

We'd configure out Node.js server to similarly serve all files statically, with a non-static router (that has higher precedence) for our Ajax server mocks. E.g. route /data/ajax/:script.php to a JavaScript based handler that will vary according to the script name and query parameters accordingly.

Once that is fixed, we can run the unit tests normally with QUnit/Karma/Grunt, e.g. defaulting to Headless Chrome, and we can run them Travis CI in at least Headless Chrome and Firefox (pre-installed). In addition, once they pass in e.g. Headless Chrome, we can go run them in various other browsers using Saucelabs or BrowserStack as various other jQuery projects do already.

All this can happen whilst the tests will continue to work in TestSwarm as well. (If desired)