Need a way to serialize closure

[adambom]

Take memoized fibonacci:

Where we store computed values in closure scope...

var p = new Parallel([0, 1, 2, 3, 4, 5, 6]),
    fib = (function () {
      var memo = {};  

      return function (n) {
        if (n < 2) {
          return 1;
        };

        memo[n] = memo[n] || fib(n - 1) + fib(n - 2);

        return memo[n];
      }
    })(),
    log = function () { console.log(arguments); };

p.map(fib).then(log)

Fails with the error, memo is not defined

[Sebmaster]

That's not possible since it would

• require a shared state across workers
• require us to transmit all scope variables (I don't think there's even a possibility to read the available scopes)

The only thing which might be possible would be cache variables, but these can already be defined in the function itself with spawn() and don't make any sense in map (because there's no shared state).

[adambom]

Yeah, I guess that makes sense. Oh well. I guess we'll have to deal with it.

[foo123]

Hi, not sure if i need to open a new thread, maybe this is related

The problem is i want to use parallel.js inside an object instance

for this library:

https://github.com/foo123/HAAR.js

The code is this right now (not in repository)

        if (this.Parallel)
        {
            // needs parallel.js library (included)
            // parallelize the detection, using map-reduce (only map needed here)
            // works same in Nodejs (using child processes)
            var scales=[], parallel;
            for (var sc=baseScale; sc<=this.maxScale; sc*=scale_inc) scales.push({self:this, scale:sc});
            parallel=new this.Parallel(scales);
            parallel.require(doParallel,endParallel).map(doParallel).then(endParallel);
        }
        else//if (this.async)
        {
            // else detect asynchronously using fixed intervals
            this._timeinterval = setInterval(function() {  self.detectAsync()  }, this._interval);
        }


function doParallel(s) {detectParallel.call(s.self, s.scale);};
function endParallel(s){console.log(s); /*detectEnd.call(s[0].self);*/};

Trying to pass the object instance (this) as part of data, since it can not be passed as part of a callback
Yet getting this error (Firefox)

DataCloneError: The object could not be cloned.

i guess this refers to the "this" reference, any way to bypass this. The main idea is to parallelize a process
(face recognition), which depends on a specific class instantiation (ie each instance can have its own parallel computation) and the callbacks and data are relative to "this".

Any ideas/help welcome

Nikos

[Sebmaster]

You can only transmit parameters to webworkers which are supported by the structured cloning algorithm. That means you'd have to create your instance of the face detection library in the parallel function too.

Remember this: You cannot access any variables which are not directly in your parameter list inside your spawned/mapped functions.

[foo123]

Thanks for the fast response,i checked afterwards the types of objects supported by the structured cloning algorithm,

Object is supported but a general class (Function) is not supported.

Was trying to avoid re-instantiating the whole class a-new in each worker.

Do you have, by any chance, any idea of how to best parallelize the task?

Thanks again,
Nikos

[Sebmaster]

Since I know nothing about face detection I'll have to pass on algorithmic questions, however if you really need your instance (and can't rewrite it) you'll have to re-instantiate.

[foo123]

It's OK, thinking about using generic functions (without "this" reference) and use all the necessary data (which are arrays, objects or canvas) as parameters.

Then if async computation is used, these generic functions can be called with this.data, else with the worker data

The detection data are usually large (~1MB of data in an object structure), does this have any side-effects?
The face-detection library works also in Nodejs (using node-canvas) will this have any side-effects?

In any case, once this is complete, it could also be an example of a real-world js application using parallel.js.

[Sebmaster]

Ah, I see, you have to initialize the detection library with some pre-initialized data?
The size of the data shouldn't make that much of a diffference, although you could try to put it into its own script and use require() it. This avoids the stuctured cloning which happens when you pass it as a parameter.

Regarding node: node has an additional restriction in that you can't use the full extent of structured cloning and you're restricted to types which are convertible to JSON. It might be possible to extend parallel with some special handling for ArrayBuffers, but I haven't looked at that yet.

[foo123]

Yes, exactly, the detection is done using data which represent the feature to be detected (same as openCV)
these data can be large (~0.5-1MB)

These exist usually in a separate js file which the user loads (depending on what is supposed to be detected)

The easiest is just to clone this whole object, i am not sure if it is possible to determine the js file that contains the data (as it is user-defined), if that is the case it might be better to just require the file itself

i am goin to try my luck ;)

Thanks

[foo123]

Finally added parallel.js support in HAAR.js (up to now seems fine, except some results might be slightly different when in parallel)

https://github.com/foo123/HAAR.js

Cheers