[Web] Image Segmenter GPU delegate produces incorrect categories on iOS Safari

[vin-ni]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

iOs 18.7.1

Mobile device if the issue happens on mobile device

iPhone 12 Pro

Browser and version if the issue happens on browser

Safari

Programming Language and version

Javascript

MediaPipe version

@mediapipe/tasks-vision@0.10.22-rc.20250304 (NPM package for Web)

Bazel version

N/A (using NPM package, not building from source)

Solution

Image Segmenter (Selfie Multiclass)

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

N/A (Web/JavaScript implementation)

Xcode & Tulsi version (if issue is related to building for iOS)

N/A (Web/JavaScript implementation, but bug manifests on iOS Safari)

Describe the actual behavior

When using the ImageSegmenter with the selfie_multiclass_256x256 model on iOS Safari with GPU delegate, the segmentation categories are scrambled: Actual categories on iOS Safari + GPU: Category 0: Background ✓ (correct) Category 1: Neck/Body ✗ (should be Hair) Category 2: Face (~12% of pixels) ✗ (should be Body-skin) Category 3: Clothes (~30% of pixels) ✗ (should be Face-skin) Category 4: Possibly Hair ✗ (should be Clothes) Category 5: Accessories ✓ (correct) This causes the head extraction to fail, showing the person's t-shirt instead of their face.

Describe the expected behaviour

Categories should match the model specification and desktop browser behavior: Expected categories (Desktop Chrome/Safari + GPU, iOS Safari + CPU): Category 0: Background Category 1: Hair Category 2: Body-skin Category 3: Face-skin Category 4: Clothes Category 5: Accessories (Others)

Standalone code/steps you may have used to try to get what you need

import { FilesetResolver, ImageSegmenter } from '@mediapipe/tasks-vision' // Load WASM files const vision = await FilesetResolver.forVisionTasks( 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.22-rc.20250304/wasm' ) // Create segmenter with GPU delegate (causes bug on iOS Safari) const imageSegmenter = await ImageSegmenter.createFromOptions(vision, { baseOptions: { modelAssetPath: 'https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_multiclass_256x256/float32/latest/selfie_multiclass_256x256.tflite', delegate: 'GPU', // ⚠️ This causes category scrambling on iOS Safari }, runningMode: 'IMAGE', outputCategoryMask: true, outputConfidenceMasks: false, }) // Capture image from camera/upload const imageData = /* your ImageData from canvas */ // Create canvas and run segmentation const canvas = document.createElement('canvas') canvas.width = imageData.width canvas.height = imageData.height const ctx = canvas.getContext('2d') ctx.putImageData(imageData, 0, 0) const result = await imageSegmenter.segment(canvas) const mask = result.categoryMask.getAsUint8Array() // Count categories to see the scrambling const categoryCounts = new Map() for (let i = 0; i < mask.length; i++) { const category = mask[i] categoryCounts.set(category, (categoryCounts.get(category) ?? 0) + 1) } console.log('Category distribution:', categoryCounts) // On iOS Safari + GPU: Shows wrong categories (clothes as category 3 instead of face) // On iOS Safari + CPU: Shows correct categories

Other info / Complete Logs

Browser Console Output (iOS Safari + GPU):
Category distribution: {
  total: 307200,
  categories: [
    { category: 0, pixels: 123456, percentage: "40.20%" },  // Background
    { category: 3, pixels: 93095,  percentage: "30.31%" },  // ⚠️ Clothes (wrong!)
    { category: 2, pixels: 36455,  percentage: "11.87%" },  // Face (wrong!)
    { category: 1, pixels: 28934,  percentage: "9.42%" },   // Neck/Body (wrong!)
    { category: 4, pixels: 18234,  percentage: "5.94%" },   // Hair (wrong!)
    { category: 5, pixels: 7026,   percentage: "2.29%" }    // Accessories
  ]
}
Browser Console Output (iOS Safari + CPU or Desktop + GPU):
Category distribution: {
  total: 307200,
  categories: [
    { category: 0, pixels: 123456, percentage: "40.20%" },  // Background
    { category: 3, pixels: 93095,  percentage: "30.31%" },  // ✅ Face-skin (correct!)
    { category: 1, pixels: 36455,  percentage: "11.87%" },  // ✅ Hair (correct!)
    { category: 2, pixels: 28934,  percentage: "9.42%" },   // ✅ Body-skin (correct!)
    { category: 4, pixels: 18234,  percentage: "5.94%" },   // ✅ Clothes (correct!)
    { category: 5, pixels: 7026,   percentage: "2.29%" }    // ✅ Accessories (correct!)
  ]
}

[HarishPermal]

Hi @vin-ni,

Thanks for reporting 🙂, give me some time to setup the environment to reproduce it from my end. and i'll get back to you with updates.