Warp API for easy parallel execution of javascript code
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
build Add link to examples in the repo (#34) Feb 16, 2019
doc Fix mistakes, add video-link (#33) Feb 15, 2019
example Sync README and code (#29) Feb 14, 2019
test Initial setup Jan 21, 2019
.eslintignore Initial setup Jan 21, 2019
.eslintrc.json Initial setup Jan 21, 2019
.gitignore Initial setup Jan 21, 2019
.gitlab-ci.yml Allow publishing on all branches Feb 1, 2019
.jest.config.json Initial setup Jan 21, 2019
LICENSE Initial setup Jan 21, 2019
README.md
package-lock.json Add prerelease management when publishing (#22) Feb 12, 2019
package.json renamme flow(s) in thread(s) (or warp thread(s)) (#21) Feb 14, 2019
warp.js Fix typo in getCurrentNumWarpThreads (#23) Feb 12, 2019

README.md

Warp Logo

Warp

Warp is an API allowing easy parallel execution of JavaScript code, at the same physical time as the main JavaScript thread and as other threads.

// runs `f` in parallel
warp.call(fnct, <args>, cb);

Discover Warp in this short YouTube video:

Warp Principle

Installing

Warp module is an API and is not providing support for execution. You will need to acquire a Warp runtime to use it.

To add Warp API to your project:

$ npm install warp

How to use it

Warp offers two main functions starting a warp task: warp.call and warp.callAsPromise. We say that they perform a warp call. By extension, we also say that code warps a function or that the function is warped.

Here is a very simple illustrative code sample where a CPU intensive function (computePrimeFactorization) is executed in parallel by a warp call keeping the event loop free for any other processing. If you want to see complete code, browse the repository.

The first example uses a callback-based function, whereas the second uses a Promise-based function.

With callback

const warp = require('warp');

function computePrimeFactorization(n, done) {
  // Compute f
  done(f);
}

warp.call(computePrimeFactorization, 9007199254740991,
    (a) => console.log(`Prime factors of ${a.value} are ${a.factors}`));

Prints out

Prime factors of 9007199254740991 are 6361,69431,20394401

With promise

const warp = require('warp');

function computePrimeFactorization(n) {
  // Compute f
  return f;
}

let a = await warp.callAsPromise(computePrimeFactorization, 9007199254740991);
console.log(`Prime factors of ${a.value} are ${a.factors}`);

Prints out

Prime factors of 9007199254740991 are 6361,69431,20394401

How it works

Warp Principle

When you warp call a function, the execution context (i.e. everything reachable from the function scope) is captured, then this context and the warped function are sent to a distinct and parallel warp thread (see label A in the drawing).

Once the warp task is finished, the parallel context is captured and sent back to the main JavaScript thread (see label B in the drawing). Then, in an event task of the main thread, the context is updated, and:

  • If the warp call is call, the callback is executed in the main thread, with the arguments given to the callback in the parallel warp thread.
  • If the warp call is callAsPromise, the Promise returned by callAsPromise is resolved or rejected with the value or the exception obtained at the end of the parallel warp thread.

Scope

The warped functions can read and modify anything reachable from their scope.

In the following sample, the variable o is defined as a free variable (outside of the function). The warped function can read it, and also modify the object. Modifications made to o in the parallel warp thread are applied and visible in the main thread.

const warp = require('warp');

let object = {};
function computePrimeFactorization(n) {
  // Compute f
  o.result = f;
}

await warp.callAsPromise(computePrimeFactorization, 9007199254740991);
console.log(`Prime factors of ${object.value} are ${object.factors}`);

Prints out

Prime factors of 9007199254740991 are 6361,69431,20394401

Shared resources and concurrency

Warp offers no new synchronization or semaphore mechanism. If you need to manage concurrent access to a value, you'll have to keep the modifying code in the main thread.

In the following example, we compute several prime factorizations in parallel.

const warp = require('warp');

let primeFactorizations = [];

function computePrimeFactorization(index, n) {
  // Compute f
  primeFactorizations[index] = f;
}

let promises = []
for(let i = 0; i < 8; i++) {
  promises.push(warp.callAsPromise(computePrimeFactorization, i, 9007174991 + i));
}

await Promise.all(promises);

console.log(`Prime factorizations: %j`, primeFactorizations);

Prints out

Prime factorizations: [{"value":9007174991,"factors":[17,131,4044533]},{"value":9007174997,"factors":{"0":59,"1":109,"2":1400587,"value":9007174993,"factors":[12071,746183]}},{"0":59,"1":109,"2":1400587,"value":9007174993,"factors":[12071,746183]},{"0":2,"1":2,"2":2251793749,"value":9007174994,"factors":[2,37,53,2296577]},{"value":9007174995,"factors":[3,5,7,13,41,227,709]},{"value":9007174997,"factors":{"0":59,"1":109,"2":1400587,"value":9007174993,"factors":[12071,746183]}},{"value":9007174997,"factors":{"0":59,"1":109,"2":1400587,"value":9007174993,"factors":[12071,746183]}},{"value":9007174998,"factors":[2,3,3,3,19,43,204161]}]

But you need to be carefull about parallel access to shared variables or properties.

For instance, the following example shows bad usage of a shared value causing indetermism. There, you don't know in which order the additions will be made, but it is guaranteed that each fnct sees the initial value of v as 4. This means you don't know what the final value of v will be, you just know it will be either 7 or 8.

const warp = require('warp');

let v = 4;
function fnct(num) {
  let inc = num;
  v += inc;
  return;
}

let promises = []
promises.push(warp.callAsPromise(fnct, 3));
promises.push(warp.callAsPromise(fnct, 4));

await Promise.all(promises);

console.log(`Value: ${v}`);

To obtain the correct behavior in this example, you'll need to apply the changes to the shared variable in the main thread by using the .then of the promise.

const warp = require('warp');

let v = 4;
function fnct(num) {
  let inc = num;
  return inc;
}

let promises = []
promises.push(warp.callAsPromise(fnct, 3).then((inc) => v += inc));
promises.push(warp.callAsPromise(fnct, 4).then((inc) => v += inc));

await Promise.all(promises);

console.log(`Value: ${v}`);

Prints out

Value: 11

This way it'll end up with 11 as the final value.

Definitions

Parallel thread

A parallel thread is a JavaScript execution thread distinct from the main JavaScript execution thread. The main thread and some parallel warp threads are running at the same physical time. Warp threads don't necessarily belong to the same process. It can come from the fork of a new process or even in a remote machine.

Warp task

A warp task is started by a warp call (a call to warp.call or warp.callAsPromise). The task is executed in a parallel thread. It starts by executing the given function and continues until the end of the task is detected.

The end of the task is detected when:

  • an exception is thrown by the task

or

  • If the task was started by warp.callAsPromise:

    • the function given to callAsPromise has returned
    • And, if its returned value is a Promise, it is fulfilled or rejected.
  • If the task was started by warp.call

    • The callback is called,
    • And the function given to call has returned.

Note that this doesn't mean that return was executed, it means that the execution has returned to its caller. For example in the following code the end is detected during the line a = await 3 when await yields and a is still 2.

let a = 0;
async function fnct(done) {
  a = 1;
  done();
  a = 2;
  a = await 3; // End is here at `await` before assignation of `a`
  return;
}

Warning: If the task were to attempt to execute code after the end of execution were detected (e.g. with a setImmediate, setTimeout, ...), then the behavior would be unspecified. This means that if the function given to call is starting any asynchronous processing, calling the callback must be the very last thing done by the task.

Execution context

An execution context is referring to everything accessible from the scope of the considered function at a given point of the execution. This includes

  • parameters of the call,
  • local, free and global variables accessed by the function called or by all functions potentially called directly or indirectly,
  • any reachable objects (objects, arrays, functions, symbols) from the above defined parameters/variables.