Problem when using mediapipes from inside a worker loaded as a module

[martenrichter]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Web

MediaPipe Tasks SDK version

master branch

Task name (e.g. Image classification, Gesture recognition etc.)

Image classification but should apply to all tasks

Programming Language and version (e.g. C++, Python, Java)

Javascript

Describe the actual behavior

When loading the code I get an importScript exception

Describe the expected behaviour

That it also works from workers loaded as module

Standalone code/steps you may have used to try to get what you need

It is straightforward to reproduce. Just try to load it from a worker, which is loaded as a module, so something like this:

import { ImageSegmenter, FilesetResolver } from '@mediapipe/tasks-vision'
  async initSegmenter() {
    return ImageSegmenter.createFromOptions(
      await FilesetResolver.forVisionTasks(
        new URL(`../node_modules/@mediapipe/tasks-vision/wasm`, import.meta.url)
          .pathname
      ),
      {
        baseOptions: {
          modelAssetPath: new URL(
            './models/selfie_segmenter_landscape.tflite',
            import.meta.url
          )
        },
        canvas: this.canvas,
        outputCategoryMask: true,
        runningMode: 'VIDEO'
      }
    )
  }

If your worker is loaded as a worker, e.g.:

AVMediaPipe.workerInternal_ = new Worker(
  new URL('./avmediapipeworker.js', import.meta.url),
  {
    type: 'module'
  }
 )

You will get an error saying that importScripts is not supported.
Note, as framework with bundlers such as vite, convert you commonJS module to ESM module, converting the worker to commonJS is not an option.

### Other info / Complete Logs

```shell
The origin is here:
https://github.com/google/mediapipe/blob/fde195e4a86298c12c300811b3ae239fd7b6238c/mediapipe/web/graph_runner/run_script_helper.ts.template#L13

I also tried to patch it and send a PR by replacing the first part of the code with:

if (typeof importScripts === 'function') {
     try {
        importScripts(scriptUrl.toString());
    } catch (error) { 
        if (error instanceof TypeError) {
          // importScripts does not work in module workers
          const module = await eval('import')(scriptUrl.toString());
          //@ts-ignore
          self.ModuleFactory = module.ModuleFactory
        } else {
          throw error
        }
    }
  } else {

However, I had terrible luck, as the typescript compiler in the project transpiles everything to commonJS and replaces the import with resolve, which is a dead end. (At least after a few hours, I did not find a suitable workaround.)
What is the reason for compiling with TypeScript to CommonJS and converting it via rollup to an ES module, anyway?
So, I am here asking for help or guidance on proceeding. Changing the overall compiling workflow and supplying a patch without asking would be a bad style.

[kuaashish]

Hi @martenrichter,

Could you please confirm whether the issue has been resolved on your end, or if you still require assistance from our end?

Thank you!!

[martenrichter]

Well, yes and no.
I managed to patch the library and solve the issue by using a custom build of the package.
The patch is the PR listed in this issue.
Your colleague said he will integrate the changes manually, which makes sense as most changes come from upstream.
I just tried an install (npm install @media-pipe/tasks-vision), and the version seems to be unchanged, not including the necessary changes. However, I have not checked all recent commits.

So, the problem persists, and I think a fix would benefit other users. (Of course, I will be happy to help.)

[kuaashish]

Hi @martenrichter,

Thank you for confirming. I will bring this to the attention of our team. Please allow us some time, and we will update you through the same thread once we have any further information available.

[martenrichter]

Hi @kuaashish,
Sure, no problem; I have no hurry, as it works for me...

[arcinston]

any update on this ,
trying to use tasks-vision on web worker