Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The input packet MPImage returned by onResult after detectAsync call is corrupted #5017

Open
definability opened this issue Dec 10, 2023 · 2 comments
Open

The input packet MPImage returned by onResult after detectAsync call is corrupted #5017

definability opened this issue Dec 10, 2023 · 2 comments
Assignees
@schmidt-sebastian
Labels
platform:android Issues with Android as Platform stat:awaiting googler Waiting for Google Engineer's Response task:hand landmarker Issues related to hand landmarker: Identify and track hands and fingers type:bug Bug in the Source Code of MediaPipe Solution

Comments

@definability
Copy link

definability commented Dec 10, 2023 •
edited

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Android 11

Mobile device if the issue happens on mobile device

Redmi Note 8T

Browser and version if the issue happens on browser

No response

Programming Language and version

Kotlin

MediaPipe version

0.10.8

Bazel version

6.1.1

Solution

hand_landmarker

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

Android Studio 2023.1.1, Android SDK 30, Android NDK 21

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

The resulting input packet contains a horizontally stretched RGBA image with vertical stripes

Describe the expected behaviour

The resulting input packet must contain an identical image to the input one if it was RGBA/RGB

Standalone code/steps you may have used to try to get what you need

Other info / Complete Logs

No response
@definability definability added the type:bug Bug in the Source Code of MediaPipe Solution label Dec 10, 2023
@definability
Copy link
Author

Here is the example image (from the tests from the examples repository)
expected-bitmap

Here is the resulting input packet:
actual-bitmap

definability added a commit to definability/mediapipe that referenced this issue Dec 10, 2023
@definability
Android. Process ARGB and Grayscale input packets
9925dc0 
Issue google-ai-edge#5017
points to the issue of incorrect bitmap conversion in
Android when RGBA or Grayscale images are used.
As far as the library allows the creation of ALPHA8 bitmaps,
it is not enough to use only `getBitmapFromRgb` and `getBitmapFromRgba`,
so the new `getBitmapFromAlpha` method is needed.
Otherwise, existing users relying on the 8-bit input but not using the
input packet would catch errors despite their code operating as
intended.

Add correct processing of images with 1, 3, and 4 channels
to `FaceDetector`, `FaceLandmarker`, `FaceStylizer`,
`GestureRecognizer`, `HandLandmarker`, `HolisticLandmarker`,
`ImageClassifier`, `ImageEmbedder`, `ImageSegmenter`,
`InteractiveSegmenter`, `ObjectDetector`, and `PoseLandmarker`.
If the number of channels is not 1, 3, or 4,
throw MediaPipeException with `INVALID_ARGUMENT` status code.

Implement methods and functions needed to convert single-channel images
to RGBA bitmaps.
The native function uses `memset`
to set all bytes of the resulting buffer to zeros
and then copies the alpha values of the input image to the alpha values
of the corresponding output pixels.

Perform a slight refactoring:
use `operator[i]` instead of `*(array + i)` for clarity.
definability added a commit to definability/mediapipe that referenced this issue Dec 10, 2023
@definability
Android. Process ARGB and Grayscale input packets
ecbc4e0 
Issue google-ai-edge#5017
points to the issue of incorrect bitmap conversion in
Android when RGBA or Grayscale images are used.
As far as the library allows the creation of ALPHA8 bitmaps,
it is not enough to use only `getBitmapFromRgb` and `getBitmapFromRgba`,
so the new `getBitmapFromAlpha` method is needed.
Otherwise, existing users relying on the 8-bit input but not using the
input packet would catch errors despite their code operating as
intended.

Add correct processing of images with 1, 3, and 4 channels
to `FaceDetector`, `FaceLandmarker`, `FaceStylizer`,
`GestureRecognizer`, `HandLandmarker`, `HolisticLandmarker`,
`ImageClassifier`, `ImageEmbedder`, `ImageSegmenter`,
`InteractiveSegmenter`, `ObjectDetector`, and `PoseLandmarker`.
If the number of channels is not 1, 3, or 4,
throw MediaPipeException with `INVALID_ARGUMENT` status code.

Implement methods and functions needed to convert single-channel images
to RGBA bitmaps.
The native function uses `memset`
to set all bytes of the resulting buffer to zeros
and then copies the alpha values of the input image to the alpha values
of the corresponding output pixels.

Perform a slight refactoring:
use `operator[i]` instead of `*(array + i)` for clarity.
definability added a commit to definability/mediapipe that referenced this issue Dec 10, 2023
@definability
Android. Process ARGB and Grayscale input packets
4c21321 
Issue google-ai-edge#5017
points to the issue of incorrect bitmap conversion in
Android when RGBA or Grayscale images are used.
As far as the library allows the creation of ALPHA8 bitmaps,
it is not enough to use only `getBitmapFromRgb` and `getBitmapFromRgba`,
so the new `getBitmapFromAlpha` method is needed.
Otherwise, existing users relying on the 8-bit input but not using the
input packet would catch errors despite their code operating as
intended.

Add correct processing of images with 1, 3, and 4 channels
to `FaceDetector`, `FaceLandmarker`, `FaceStylizer`,
`GestureRecognizer`, `HandLandmarker`, `HolisticLandmarker`,
`ImageClassifier`, `ImageEmbedder`, `ImageSegmenter`,
`InteractiveSegmenter`, `ObjectDetector`, and `PoseLandmarker`.
If the number of channels is not 1, 3, or 4,
throw MediaPipeException with `INVALID_ARGUMENT` status code.

Implement methods and functions needed to convert single-channel images
to RGBA bitmaps.
The native function `AlphaToRGBA` uses `memset`
to set all bytes of the resulting buffer to zeros
and then copies the alpha values of the input image to the alpha values
of the corresponding output pixels.

Perform a slight refactoring:
use `operator[i]` instead of `*(array + i)` for clarity.
@definability definability mentioned this issue Dec 10, 2023
Open
@definability
Copy link
Author

Proposed fix: #5018.

P.S. Does anybody know how to build a jar library like com.google.mediapipe:tasks-vision to test the changes easily?
P.P.S. Is there a way to launch the unit tests https://github.com/google/mediapipe/tree/master/mediapipe/tasks/javatests/com/google/mediapipe/tasks?

@kuaashish kuaashish assigned kuaashish and unassigned ayushgdev Dec 11, 2023
@kuaashish kuaashish added platform:android Issues with Android as Platform task:hand landmarker Issues related to hand landmarker: Identify and track hands and fingers labels Dec 11, 2023
@kuaashish kuaashish added the stat:awaiting googler Waiting for Google Engineer's Response label Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@schmidt-sebastian schmidt-sebastian

Labels
platform:android Issues with Android as Platform stat:awaiting googler Waiting for Google Engineer's Response task:hand landmarker Issues related to hand landmarker: Identify and track hands and fingers type:bug Bug in the Source Code of MediaPipe Solution
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@definability @schmidt-sebastian @kuaashish @ayushgdev