[Feature request] Manually set ByteBuffer and Bitmap in mlkit selfie segmentation

[pawaom]

What's your feature request? Please describe.
Can we set Manually ByteBuffer and Bitmap in mlkit selfie segmentation

Mobile environment
both

Additional context
We are currently using this code

private Segmenter segmenter;
    private ByteBuffer maskBuffer = ByteBuffer.allocate(0);
    private int maskWidth = 0;
    private int maskHeight = 0;
    BitmapUtils bitmapUtils = new BitmapUtils();
    Bitmap maskBitmap;
    ProcessedListener listener;
    public SegmentHelper(ProcessedListener mlistener) {
        this.listener = mlistener;
    }

    public void init() {
        SelfieSegmenterOptions options =
                new SelfieSegmenterOptions.Builder()
                        .setDetectorMode(SelfieSegmenterOptions.SINGLE_IMAGE_MODE)
                        .build();
        segmenter = Segmentation.getClient(options);
    }

    /**
     * Processes a bitmap and informs the listener on success
     */
    public void processImage(Context mContext, Bitmap image) {
        InputImage input = InputImage.fromBitmap(image, 0);
        segmenter.process(input).addOnSuccessListener(SegmentationMask -> {
            maskBuffer = SegmentationMask.getBuffer();
            maskWidth = SegmentationMask.getWidth();
            maskHeight = SegmentationMask.getHeight();
            listener.imageProcessed();

        })
                .addOnFailureListener(e -> {
                });
    }

    /**
     * Highlights the detected background of the segmentation mask as a green overlay
     */
    public Bitmap generateMaskImage(Bitmap image) {
        maskBitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), image.getConfig());
        for (int y = 0; y < maskHeight; y++) {
            for (int x = 0; x < maskWidth; x++) {
                int bgConfidence = (int) ((1.0 - maskBuffer.getFloat()) * 255);
                maskBitmap.setPixel(x, y, Color.argb(bgConfidence, 0, 0, 0));
            }
        }
        maskBuffer.rewind();
        return bitmapUtils.mergeBitmaps(image, maskBitmap);
    }

every thing is done by MLKIT, can we set the bytbuffer and Create a bitmap out of it Manually , since some times MLKIT misses some parts of the image we want to capture.

[ai-plays]

At the moment, the Segmenter API doesn't accept a Bitmap or ByteBuffer for the segmentation results.

since some times MLKIT misses some parts of the image we want to capture.

Could you elaborate this issue a bit more, maybe with an example?
Also, could you help me understand why accepting a Bitmap/ByteBuffer for result would solve this?

[ai-plays]

Close this due to no reply. Feel free to re-open if you are still working on this.

[pawaom]

The best example is on the mlkit website itself

Compare these two images

https://developers.google.com/ml-kit/images/vision/selfie-segmentation/selfie3_mask.jpg

and

https://developers.google.com/ml-kit/images/vision/selfie-segmentation/selfie3.jpg

We see the wine glass and jar kept on the Table is part of the Selfie segmentation aswell, if there is a possibility to edit these manually may be using touch to remove that region from the mask , it can be useful.

The idea is there should be ability to edit the mask

[ai-plays]

I see. For the touch-and-edit behavior, it is more like a UI level thing and out of the scope of ML Kit detector.

Some idea for you to consider: you can use a Bitmap variable to store the current mask. The initial value of such a mask should come from the Ml Kit segmenter. Then you implement UI behavior to update the mask variable based on touch event.

One better experience I can think about is that ML Kit segmentation API takes in a list of points that you want to make sure to be segmented in the foreground and another list of points that you want to make sure to be segmented in the background, then the detector produces the final smooth segmentation meeting the requirement. However, this is not directly supported by ML Kit at the moment.

You might be able to achieve similar behaviors with current API. Idea: the output of the current segmentation API is not binary (0 or 1). Instead, it is a confidence score in range [0.0, 1.0]. You can use different threshold in different area of an image to construct the final binary segmentation (foreground v.s. background). The threshold of each area can be determined by the list of including and excluding points as mentioned above.