[Bug] image segmentation very slow

[leonardoFu]

I'm trying to use transform.js with MetaAI SAM, it cost above half a min to segment a 540p picture.

How to reproduce
Steps or a minimal working example to reproduce the behavior

Expected behavior
A clear and concise description of what you expected to happen.

Maybe faster?

Logs/screenshots

Environment

• Transformers.js version: 1.4.2
• Browser (if applicable): Edge 112.0.1722.34
• Operating system (if applicable): MacOS 13.2.1
• Other:

Additional context
Add any other context about the problem here.

get warning from console:
-nodewas not found. Usingonnxruntime-webas a fallback. We recommend installingonnxruntime-node` as it generally improves performance (up to 5X).

Source Code

import { pipeline } from '@xenova/transformers';

(async function main() {
performance.mark('segmentation');
let segmenter = await pipeline('image-segmentation', 'facebook/detr-resnet-50-panoptic');
let img = 'img'
let outputs = await segmenter('https://cdn.pixabay.com/photo/2017/02/20/18/03/cat-2083492__340.jpg');
performance.mark('segmentation-end');
console.log(performance.measure('segmentation-elapse', 'segmentation', 'segmentation-end'));

console.log(outputs);

const cat = outputs.find(out => out.label === 'cat');
const image = new ImageData(cat.mask.width, cat.mask.height);
const canvas = document.createElement('canvas');
canvas.width = cat.mask.width;
canvas.height = cat.mask.height;
const ctx = canvas.getContext('2d');
cat.mask.data.forEach((d, index) => {
const offset = index * 4
if (d === 255) {
image.data.set([255, 99, 71, 255], offset)
} else {
}
})
ctx.putImageData(image, 0, 0);
document.documentElement.append(canvas);
})()

[kungfooman]

Usually the first thing I do with performance issues is to look at the exact functions: https://developer.chrome.com/docs/devtools/performance/

Can you post what you see there? It samples everything and you get a rather clear picture where it could be optimized.

[xenova]

Right, so the main issue is that the model is currently running on the CPU (instead of the GPU). This is because:

1. onnxruntime-web does not yet have the webgpu backend support (follow the PR here)
2. When webgpu is supported, only Chrome currently has it enabled by default (with other browsers coming soon)

---

TLDR: wait a week or two until webgpu is supported! 🤣

[xenova]

One thing I do find strange, however, is that it thinks you're running in Node? Somehow, the process global variable is defined. I tried in Edge, but I did not encounter that issue. Perhaps it's because you're running Edge with MacOS?

[leonardoFu]

Usually the first thing I do with performance issues is to look at the exact functions: https://developer.chrome.com/docs/devtools/performance/

Can you post what you see there? It samples everything and you get a rather clear picture where it could be optimized.

[leonardoFu]

One thing I do find strange, however, is that it thinks you're running in Node? Somehow, the process global variable is defined. I tried in Edge, but I did not encounter that issue. Perhaps it's because you're running Edge with MacOS?

I don't know where process is defined (not from user code), maybe because of the parcel bundler? but use process may not be a exact way to confirm the runtime environment. How about add process.versions.node ？

In this case, process is defined like:

[kungfooman]

Bundlers can introduce all kinds of issues, that's why I changed my personal projects to use ES6 import / import-maps syntax, which is supported by modern browsers. And for older browsers you can still build single-file-bundles, so at least the bundle process is not messing with you developer experience (a simple ctrl+r vs. calling starting bundle/watch commands).

[xenova]

Bundlers can introduce all kinds of issues, that's why I changed my personal projects to use ES6 import / import-maps syntax, which is supported by modern browsers. And for older browsers you can still build single-file-bundles, so at least the bundle process is not messing with you developer experience (a simple ctrl+r vs. calling starting bundle/watch commands).

Do you suggest the project migrate to ES6 syntax? I've been wanting to do it for a while to be honest.

[kungfooman]

Do you suggest the project migrate to ES6 syntax? I've been wanting to do it for a while to be honest.

I feel a bit uncertain, because onnxruntime only ships with ES5 files e.g. here:

https://github.com/microsoft/onnxruntime/blob/main/js/web/lib/onnxjs/backends/webgl/utils.ts

I found some JSFIDLE ES6 files here, but I don't know how to make use of them yet:

await import("https://cdn.jsdelivr.net/npm/onnxruntime-web@1.14.0/+esm")

I didn't read/test too much so far due to lack of time... usually you can just import ES5 files in ES6, but ONNX is maybe a bit more challenging/sophisticated (on account of different backends like WASM), so simply assumptions can be wrong.