Model:body-segmentation browser freezes for ~7-9 seconds in initial run

[Exlord]

Hi, I am not sure if this is a bug so am posting it here.
This line take ~7-9 seconds to run and freezes the browser when running for the first time, then it just takes ~100ms.

bodySegmenter.segmentPeople(video);
video is a HTMLVideoElement

Running on chrome 107 with webgl backend.
The tf and backend and the segmenter has already loaded and is ready before running this.

import * as bodySegmentation from '@tensorflow-models/body-segmentation';
import * as tf from '@tensorflow/tfjs-core';
import '@tensorflow/tfjs-backend-webgl';
import '@tensorflow/tfjs-backend-wasm';

const model = bodySegmentation.SupportedModels.MediaPipeSelfieSegmentation;
const bodySegmenterConfig: MediaPipeSelfieSegmentationTfjsModelConfig = {
  runtime: 'tfjs', //mediapipe or 'tfjs'
  modelType: 'general', // or 'landscape'
};

let bodySegmenter: BodySegmenter;

tf.ready().then(() => {
  log('backend ready  :', tf.getBackend()); // <--------------webgl
  bodySegmentation.createSegmenter(model, bodySegmenterConfig).then((value) => {
    log('segmenter created');
    bodySegmenter = value;
  });
});

    "@tensorflow-models/body-segmentation": "^1.0.1",
    "@tensorflow/tfjs": "^4.0.0",
    "@tensorflow/tfjs-backend-wasm": "^4.0.0",
    "@tensorflow/tfjs-backend-webgl": "^4.0.0",
    "@tensorflow/tfjs-converter": "^4.0.0",
    "@tensorflow/tfjs-core": "^4.0.0",

------------------
System Information
------------------
Operating System: Windows 10 Enterprise N 64-bit (10.0, Build 19043) (19041.vb_release.191206-1406)
System Manufacturer: ASUS
BIOS: 1401 (type: UEFI)
Processor: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz (12 CPUs), ~2.9GHz
Memory: 32768MB RAM
Available OS Memory: 32686MB RAM
Page File: 26483MB used, 8250MB available
DirectX Version: DirectX 12
User DPI Setting: 96 DPI (100 percent)
System DPI Setting: 96 DPI (100 percent)
DWM DPI Scaling: Disabled
DirectX Database Version: 1.0.8
DxDiag Version: 10.00.19041.2075 64bit Unicode

---------------
Display Devices
---------------
Card name: NVIDIA GeForce RTX 3060
DAC type: Integrated RAMDAC
Device Type: Full Device (POST)
Device Status: 0180200A [DN_DRIVER_LOADED|DN_STARTED|DN_DISABLEABLE|DN_NT_ENUMERATOR|DN_NT_DRIVER] 
Device Problem Code: No Problem
Driver Problem Code: Unknown
Display Memory: 28484 MB
Dedicated Memory: 12142 MB
Shared Memory: 16342 MB
Current Mode: 1920 x 1080 (32 bit) (60Hz)

Any points or ideas ?

[rthadur]

The first detection is slow since the backend are caching the shader programs, the prediction after the first one should much faster and stable, for example after the model is loaded, you can run a warming up step, to feed the model prediction call with all zero tensor. tf.zeros([tensorShape]).

[qjia7]

It's similar issue with #6970. There proposed several solutions to mitigate it you can try 1) add tf.env().set('WEBGL_USE_SHAPES_UNIFORMS', true) to enable WEBGL_USE_SHAPES_UNIFORMS to reduce the compiled shaders 2) use ENGINE_COMPILE_ONLY to enable parallel compilation. See here

[Linchenn]

Thank you Jiajia for redirecting the solution!

@Exlord for your case, you could use tf.env().set('WEBGL_USE_SHAPES_UNIFORMS', true); when tf is ready as Jiajia mentioned. Also using ENGINE_COMPILE_ONLY may have obvious perf gain. Specifically, you could add the following codes before your first inference:

    tf.env().set('ENGINE_COMPILE_ONLY', true);
    bodySegmenter.segmentPeople(video);
    tf.backend().checkCompileCompletion();
    tf.backend().getUniformLocations();
    tf.env().set('ENGINE_COMPILE_ONLY', false);

If it still does not work, I could help you tell a look at your codes.

[Exlord]

Tanks everyone , It worked like a charm.
Just 1 more question, Is there TS typing's for the backends?
TS2339: Property 'checkCompileCompletion' does not exist on type 'KernelBackend'.
Something like (tf.backend() as WebglBackend).checkCompileCompletion(); ?

[Linchenn]

For WebGL backend, it's MathBackendWebGL (class MathBackendWebGL extends KernelBackend) and you could use it as tf.MathBackendWebGL.

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please @mention us if this needs more attention.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[Exlord]

Hi Again,
Slight Problem ...
These changes breaks the detection in android ... The entire webcam screen is detected as the person, If I comment these then it works again in android but with a huge initial loading time...
Any advice on how to get the android to work again ?

This is my current code :

tf.ready().then(() => {
  log('backend ready  :', tf.getBackend());
  bodySegmentation.createSegmenter(model, bodySegmenterConfig).then((value) => {
    log('segmenter created');
    bodySegmenter = value;
    // https://github.com/tensorflow/tfjs/issues/7026#issuecomment-1311101591
    if (tf.getBackend() === BACKEND_WEBGL) tf.env().set('WEBGL_USE_SHAPES_UNIFORMS', true);
  });
});


export async function getPersonSegmentationWebGL(video: HTMLVideoElement) {
  // https://github.com/tensorflow/tfjs/issues/7026#issuecomment-1311101591
  tf.env().set('ENGINE_COMPILE_ONLY', true);
  const p = bodySegmenter.segmentPeople(video);
  // @ts-ignore
  tf.backend().checkCompileCompletion();
  // @ts-ignore
  tf.backend().getUniformLocations();
  tf.env().set('ENGINE_COMPILE_ONLY', false);
  segmenterWarmedUp = true;
  return p;
}

export async function getPersonSegmentation(video: HTMLVideoElement) {
  if (!bodySegmenter) return null;

  if (segmenterWarmedUp || tf.getBackend() !== BACKEND_WEBGL) return bodySegmenter.segmentPeople(video);
  else return getPersonSegmentationWebGL(video);
}

[Linchenn]

Could you provide the codes to reproduce the error? @Exlord

[Exlord]

Hi @Linchenn sorry for the late response, this project was put on hold for a while, now I am working on it again.
I have posted the code that couses the problem in the my last comment ...

#7026 (comment)

If I don't use this code it works but with a huge initial loading time.