Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

explore acceleration w WebAssembly #400

Closed
jywarren opened this issue Oct 11, 2018 · 51 comments

Comments

Projects
None yet
5 participants
@jywarren
Copy link

commented Oct 11, 2018

WebAssembly seems like a way we might speed up pixel-by-pixel operations.

https://developer.mozilla.org/en-US/docs/WebAssembly

But would it suffer from the same issues as webworkers, i.e. a cost of passing the image data over?

@jywarren

This comment has been minimized.

Copy link
Author

commented Oct 11, 2018

Also note this way of writing WebAssembly:

https://github.com/ballercat/walt

@gitmate

This comment has been minimized.

Copy link

commented Oct 11, 2018

GitMate.io thinks the contributor most likely able to help you is @ccpandhare.

A possibly related issue is #237 (Explore OpenCV.js module).

@tech4GT

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

@jywarren Even I was thinking we can write the pixel manipulation code in c and create a js binding to it, is that the same as web assembly? I mean the code shows up as native code.

@gitmate gitmate bot added the enhancement label Oct 11, 2018

@publiclab publiclab deleted a comment from gitmate bot Nov 1, 2018

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 1, 2018

Interesting -- well, the WebAssembly approach means it'd be runnable at (near?) native speed in the browser, too! I think it could get really fast -- maybe not GPU speed, but maybe closer to video speed. For a speed reference we'd like to get closer to, see the Infragram WebGL demo: https://publiclab.github.io/infragram/

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 1, 2018

Walt seems like a good way to easily write webassembly code!

Provide a thin layer of syntax sugar on top of .wat text format. Preferably porting as much of JavaScript syntax to WebAssembly as possible. This improved syntax should give direct control over the WebAssembly output. Meaning there should be minimal to none post optimization to be done to the wast code generated. The re-use of JavaScript semantics is intentional as I do not wish to create a brand new language.

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 8, 2018

Also noting imagemagick has been compiled for webassembly... Wow!

https://github.com/KnicKnic/WASM-ImageMagick/blob/master/Readme.md

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 8, 2018

I think we could write a version of Pixelmanipulation following this example!

https://github.com/ballercat/walt/wiki/Walt-In-5-Minutes

...but would we be able to run a JavaScript function inside? I'm not sure about that. Would the function have to be compiled as well?

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 8, 2018

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 8, 2018

This is a great area for testing code snippets!

https://ballercat.github.io/walt/

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 11, 2018

Just asked over at the Walt project if their compiler runs in the browser. Otherwise we could look at precompiling these at build time.

@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 11, 2018

So I think if the simple pixel manipulation function we pass into the PixelManipulation module is close enough to Walt syntax, which is actually WebAssembly, but "looks like" javascript... so if the inside of a function like this could be compiled almost directly, that'd be interesting:

function changePixel(r, g, b, a){
var val = (options.brightness)/100.0
r = val*r<255?val*r:255
g = val*g<255?val*g:255
b = val*b<255?val*b:255
return [r , g, b, a]
}

By comparison, here is a section of walt code:

  for (i = 0; i < total; i += 1) {
    particle = i * particleByteSize;
    particle = {
      x: width / 2.0,
      y: height / 2.0,
      vx: ((i * 7 % 200) - 100) / 100.0,
      vy: ((i * 19 % 200) - 100) / 100.0
    };
  }
const x: i32 = 2;
export function echo(): i32 {
  const x: i32 = 42;
  return x;
}
@jywarren

This comment has been minimized.

Copy link
Author

commented Nov 13, 2018

Walt folks responded to say that walt does indeed compile in the browser, on the fly! Awesome.

@Mridul97

This comment has been minimized.

Copy link

commented Dec 31, 2018

I was reading the above discussions and I think we can implement pixel manipulation code in c and then compile it into wasm file and then add it into the project. After that we can instantiate the WebAssembly module and use the functions written in pixel manipulation file, which is in c, in the js files!
cc @jywarren @tech4GT

@jywarren

This comment has been minimized.

Copy link
Author

commented Jan 2, 2019

@jywarren jywarren added the project label Jan 24, 2019

@jywarren jywarren added the performance label Feb 5, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 1, 2019

Can I work on this issue?

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 1, 2019

@HarshKhandeparkar

This comment has been minimized.

Copy link
Member

commented Mar 2, 2019

@jywarren @publiclab/is-reviewers nectarJs claims that it can compile js to native OS binaries for all platforms. This can be used to compile the lib for CLI maybe? It says that it can also create webassembly from js code. (It is under development though)

NectarJs github page

webpage

@Divy123

This comment has been minimized.

Copy link

commented Mar 4, 2019

@jywarren your thoughts on nectarJs please?

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 4, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 4, 2019

@jywarren in that case may I proceed with nectarJs for applying the same to our PixelManipulation.js ?

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 4, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 4, 2019

Excited for this.
Thanks!

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 4, 2019

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 5, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 8, 2019

@Divy123 Divy123 self-assigned this Mar 12, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 12, 2019

@jywarren I looked into WASM, made these two little projects,
https://github.com/Divy123/canvas-wasmvas
https://github.com/Divy123/prime-with-wasm

I suggest we can proceed with writing native C code and compiling that with emscripten and utilizing the genrated glue JS files in our code which I feel can be quite handy.

Your opinion please!!

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 12, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 12, 2019

Ya I looked into but perhaps feel this can be more easy to go with as then we don't have the overhead of a library.
Yes we have to keep wasm files in src/webassembly and keep it in dist!!
What do you think?

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 12, 2019

@Divy123

This comment has been minimized.

Copy link

commented Mar 15, 2019

Sure !!

@Divy123

This comment has been minimized.

Copy link

commented Mar 22, 2019

@jywarren just one thing that if you can suggest apart from PixelManipulation.js what are some other areas that we can apply wasm to include in our code base like can we add to different modules as well?
Thanks!!

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 22, 2019

@tech4GT

This comment has been minimized.

Copy link
Member

commented Mar 24, 2019

Yes, this is absolutely true, the heavy lifting is done by pixelManipulation for almost all modules, so that should give us a significant boost in performance!

@HarshKhandeparkar

This comment has been minimized.

Copy link
Member

commented Mar 24, 2019

What abt gpu.js which I am trying to implement for PixelManipulation? It will also work in node and will process the pixels on gpu. Can that also be compiled to wasm? It uses a node library headless-gl(will use in the future). Currently it uses the canvas api.

@Divy123

This comment has been minimized.

Copy link

commented Mar 24, 2019

Hi Harsh!
Can you explain a bit of your work here with gpu.js?

@HarshKhandeparkar

This comment has been minimized.

Copy link
Member

commented Mar 24, 2019

You can have a look at issue #849. Essentially it will manipulate all the pixels at the same time in parallel on a gpu.

@jywarren

This comment has been minimized.

Copy link
Author

commented Mar 24, 2019

@Divy123

This comment has been minimized.

Copy link

commented May 27, 2019

@jywarren @tech4GT I have been through PixelManipulation.js file and what I am able to understand that the whole code here can not be converted as there are lots of things like get-pixels and save-pixels that have to be passed to our C code. So can I get some help regarding exactly what parts of the code do we want to convert to C code or the whole PixelManipulation.js code?

Thanks !!

@Divy123 Divy123 referenced this issue May 27, 2019

Merged

Fix overlay #1082

@Divy123

This comment has been minimized.

Copy link

commented May 27, 2019

@jywarren can you please help on this?

@jywarren

This comment has been minimized.

Copy link
Author

commented May 27, 2019

Also noting @MargaretAN9 is interested in this!

Hmm. I think this may be a good question to ask @HarshKhandeparkar for a bit more info here. I had hoped we could take everything /between/ get-pixels and save-pixels and run that in webassembly.

@Divy123

This comment has been minimized.

Copy link

commented May 27, 2019

Ok sure!! Thanks!

@MargaretAN9 can we collaborate on this??
Also this is my this week's target.
@HarshKhandeparkar any insights?

@jywarren this is my task for this week. So should I start working on this or wait for Harsh's reply?

@jywarren

This comment has been minimized.

Copy link
Author

commented May 28, 2019

@Divy123

This comment has been minimized.

Copy link

commented May 28, 2019

@jywarren can I just send here a PR with a working demo and then proceed with everyone's suggestions. Meanwhile I can make preparations for the next weeks as well to speed up the things there??

@jywarren

This comment has been minimized.

Copy link
Author

commented May 28, 2019

@Divy123

This comment has been minimized.

Copy link

commented May 28, 2019

Thanks a lot Jeff!! 😄

@Divy123

This comment has been minimized.

Copy link

commented May 29, 2019

Screenshot from 2019-05-30 01-26-27
I have made a wasm file from wasm fiddle and here is my C code

int getPixels(int , int , int );

int setPixels(int, int, int, int);

int * changePixel(int, int, int, int, int, int);

void paceOp();

void consoleLog(int);

void manipulatePixel(int width, int height, int inBrowser, int test){
  consoleLog(100);
  for (int x = 0; x < width; x++) {
        for (int y = 0; y <height; y++) {
          int* pixel =changePixel(
            getPixels(x, y, 0),
            getPixels(x, y, 1),
             getPixels(x, y, 2),
             getPixels(x, y, 3),
            x,
            y
          );
          setPixels(x, y, 0, pixel[0]);
          setPixels(x, y, 1, pixel[1]);
          setPixels(x, y, 2, pixel[2]);
          setPixels(x, y, 3, pixel[3]);

          if (!inBrowser && !test) paceOp();
        }
      }
}

I was trying to convert this part of code for testing from PixelManipulation.js file


for (var x = 0; x < pixels.shape[0]; x++) {
      for (var y = 0; y < pixels.shape[1]; y++) {
       let pixel = options.changePixel(
         pixels.get(x, y, 0),
         pixels.get(x, y, 1),
          pixels.get(x, y, 2),
             pixels.get(x, y, 3),
           x,
           y
       );

        pixels.set(x, y, 0, pixel[0]);
        pixels.set(x, y, 1, pixel[1]);
          pixels.set(x, y, 2, pixel[2]);
          pixels.set(x, y, 3, pixel[3]);

          if (!options.inBrowser && !process.env.TEST) pace.op();
         }
       }

After adding the wasm file, the code looks like:

const imports = {
         env:{
           changePixel:options.changePixel,
           getPixels: pixels.get,
           setPixels: pixels.set,
           paceOp: require('pace').op,
           consoleLog:console.log // for testing 
         }
       }
      const inBrowser = (options.inBrowser)?1:0;
      const test = (process.env.TEST)?1:0
       WebAssembly.instantiateStreaming(fetch('./program.wasm'), imports)
           .then(wasm => {
      wasm.instance.exports.manipulatePixel(pixels.shape[0],pixels.shape[1],inBrowser, test)
           });

But I think the wasm file is not getting recognized, is it some bundler issue?
As of now I have kept it in src/no_module but is there something else I need to consider??
Am I required to include it in dist?If yes, then how to? I am not accustomed to it.

@jywarren @tech4GT @HarshKhandeparkar need your help here!!
Its something urgent. Please help !

@jywarren

This comment has been minimized.

Copy link
Author

commented May 29, 2019

Hi @Divy123 -- can you share a PR link, so we can take a look at how you've gotten it set up? Help us reproduce your error so we can help you solve it! I've also pinged folks in the weekly check-in to see if anyone there can help.

@Divy123

This comment has been minimized.

Copy link

commented May 29, 2019

Thanks a lot @jywarren !!
Making a PR now

@Divy123 Divy123 referenced this issue May 29, 2019

Merged

Using wasm to accelerate PixelManipulation.js #1093

4 of 4 tasks complete
@Divy123

This comment has been minimized.

Copy link

commented May 29, 2019

@jywarren here is the link:
#1093

Please look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.