-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Passing a ton of data around #1
Comments
Thanks for your time 😄
Yes I know. I have mentioned in my blog post as something to be done. Probably coming weekend ;)
Not very worried about that now. This is just an experiment to look into WASM. Interactivity comes later.
Interesting, can you point me to some documentation ? |
Hmm .. unfortunately it becomes slower by using canvas/webgl. Because now I have to pass an even bigger img.Pix array to the browser. That alone takes around 1.5s. The canvas rendering just takes 30ms. See my webgl branch Ignore the inverted image for now. Some webgl issue. |
Thanks for the very nice proof of concept. I had a bit more time to experiment with it. I have slightly modified the worflow to push the |
Can we make sure the webgl option is on a branch too. I also have some ideas to speed it up but am rushed right now for time. |
Yes it addresses the Edit List concept nicely. You can go the full monty and distribute tiles too for each image. |
@yml -
Brilliant idea. Can you just modify the code not to resize the target image ? That's going to a different class of optimization which I don't want to do right now. Please send a PR in your convenience. Thanks for your time. @gedw99 -
Yep, it already is. https://github.com/agnivade/shimmer/commits/webgl. I would like to apply @yml's patch first to master though. |
@agnivade I am not sure exactly which part you want in the PR. I have done 4 things in this branch
If you don't work on the resized image you are going to increase the amount of data that is transferred between "js world" and "wasm world". You are source image is going to cross the boundary 2 times instead of one. |
Ah I did not think that you would have to preview the image back in the browser. That was dumb of me. I will look into ways we can get the event in browser, as well as DOM. So that we can show the preview directly in browser, as well as get the data in "wasm world" directly. I see that
Yes, I know. Resizing the image is a simple way to reduce data transfer. For now, I just want to experiment with basic ways to optimize the workflow without affecting image size or quality. (Like directly writing to a canvas, which did not yield any positive results) But feel free to keep experimenting and let me know if you find something interesting :) |
I was able to load the image directly in wasm as well as preview it in the browser. We simply can make both callbacks live side-by-side. The browser side callback can read from the file and set the img url. The wasm side callback can read and then set the image base64 string. But most probably this doesn't do what we think it does. i.e. it does not save any data transit from browser to wasm land. The moment we do a Conceptually thinking, there is no shared memory here. Anything that needs to be loaded in wasm needs to be initialized first, and therefore the image data needs to be transferred anyways. I don't see any way around it. Happy to be proven wrong. |
@agnivade I keep using this ticket to continue the discussion :-) I went ahead and did more experimentation around image processing : https://github.com/yml/wasmBild I am curious to hear about your thoughts on the approach taken in the repo above. |
Yes, please do report with your findings here :) It seems like you are injecting html content with a template so that you don't have to change in 2 places when you add a new transformation. Looks like this might be helpful for big apps with lots of moving pieces. I am however, still interested in performance optimizations. On a cursory look at your code, it seems that you are still getting the |
@agnivade I only insert in the DOM the resized images and all the transformation are applied to the resized images. This makes the app very interactive even if you chain the transformations. Ideally I would like to to be able to render the app server and client side, sharing the template would be a good first (baby) step. |
You are reading the image here - See my comment #1 (comment). |
I agree with you that this call transfer the full image resolution from JS to the WASM land. But as far I can tell further manipulation only transfer the |
Right. I don't have any further ideas as of this moment. Unless we can transfer the data in binary in some way .. The canvas/webgl approach that I tried needed 32bits for every pixel which bloated the data transfer size even more. We need some way to directly transfer data in binary and read it in the browser. |
Did some more investigation. The whole brunt of time is actually going in the image processing. If I just calculate the time it takes to transfer the base64 data, it is actually 2-3ms. The compute time itself is around 200-300ms. I believe data transfer is not the bottleneck here. And the compute entirely happens in wasm which is not under my control. I will keep looking, but since the data transfer is so tiny part of the entire time, I wonder if it is worth spending more effort in optimizing this. |
I wonder if this app could also trigger the behavior describe in that ticket:
This is why working on the resized image makes such a difference. As long as the memory is properly garbage collected and that there is no leak, I don't think that there is much improvement that can be done until JS and WASM can share some memory to avoid the copy back and forth. |
Hmm .. I don't think so. I am just passing strings, not slices. It might trigger, but it is certainly not the primary bottleneck in this app. Agree with your other observations. |
Going to close this because data transfer is not the main bottleneck as I see it. We can open further issues to optimize other areas of this. Thank you everyone for your input. |
It's throwing alot of base64 in and out from Dom land to WASM VM land.
If you use a canvas / webgl on the out it should be more efficient.
Also what about web workers to make it stay interactive.. passing the base64 as a message post is same impact as you have now.
I would also do an edit list. Edit list is where you key up the mutations as a list and in interactive mode it just does the operations on an image 10% of the size.
Then when your happy with the effect the mutations have done you tell it to do it again on the real full size image.
This is what all special effects editors do.
The text was updated successfully, but these errors were encountered: