Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Landmark Coordinates in Iris Model #992

Closed
ridoy-ai opened this issue Aug 9, 2020 · 6 comments
Closed

Landmark Coordinates in Iris Model #992

ridoy-ai opened this issue Aug 9, 2020 · 6 comments
Assignees
@fanzhanggoogle

Comments

@ridoy-ai
Copy link

ridoy-ai commented Aug 9, 2020

There aren't any documentations or code samples on how to get landmark coordinates of left eye, right eye in Iris model. Unlike facemesh model, it seems the code sample didn't explain the things sofar.
@verma-akash
Copy link

As per face mesh example, i tried adding packet callback in Iris Sample like this:

processor.addPacketCallback(
                "iris_landmarks",
                (packet) -> {
                    List<LandmarkProto.NormalizedLandmarkList> irisLandmarks =
                            PacketGetter.getProtoVector(packet, LandmarkProto.NormalizedLandmarkList.parser());
                    
                });

But it is unable to parse the landmarks and crashing

2020-08-10 16:29:13.668 26111-26111/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2020-08-10 16:29:13.668 26111-26111/? A/DEBUG: Build fingerprint: 'OnePlus/OnePlus7/OnePlus7:10/QKQ1.190716.003/2007240000:user/release-keys'
2020-08-10 16:29:13.668 26111-26111/? A/DEBUG: Revision: '0'
2020-08-10 16:29:13.668 26111-26111/? A/DEBUG: ABI: 'arm64'
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: Timestamp: 2020-08-10 16:29:13+0530
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: pid: 25962, tid: 26106, name: Thread-9 >>> com.google.mediapipe.apps.iristrackinggpu <<<
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: uid: 10630
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x0 0000000000000000 x1 00000000000065fa x2 0000000000000006 x3 0000007c4f811f60
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x4 747265766e6f6320 x5 747265766e6f6320 x6 747265766e6f6320 x7 206f7420656c6269
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x8 00000000000000f0 x9 98890f085a06daf3 x10 0000000000000001 x11 0000000000000000
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x12 fffffff0fffffbdf x13 7420656c62697472 x14 0000007d4b0ee358 x15 0000000000000010
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x16 0000007d4b1ae738 x17 0000007d4b18cbe0 x18 eb9df7111f86a5ca x19 000000000000656a
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x20 00000000000065fa x21 00000000ffffffff x22 0000007c59c6d100 x23 0000007c4f814020
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x24 0000000000000008 x25 0000007c4f814020 x26 0000007c4f814020 x27 000000000000002e
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: x28 0000007c4f812280 x29 0000007c4f812000
2020-08-10 16:29:13.672 26111-26111/? A/DEBUG: sp 0000007c4f811f40 lr 0000007d4b13e404 pc 0000007d4b13e430
2020-08-10 16:29:13.713 852-25768/? E/sensors-hal: ssc_conn_event_cb:490, event[0] msg_id=776, ts=1671959750826
2020-08-10 16:29:13.713 852-25768/? E/sensors-hal: handle_sns_client_event:65, device_orient_event: ts=87077574496322, type=1
2020-08-10 16:29:13.819 858-10819/? I/WifiHAL: event received NL80211_CMD_VENDOR, vendor_id = 0x1374, subcmd = 0xd
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: backtrace:
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: NOTE: Function names and BuildId information is missing for some frames due
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: NOTE: to unreadable libraries. For unwinds of apps, only shared libraries
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: NOTE: found under the lib/ directory are readable.
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #00 pc 0000000000073430 /apex/com.android.runtime/lib64/bionic/libc.so (abort+160) (BuildId: a2584ee8458a61d422edf24b4cd23b78)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #1 pc 0000000000b11388 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #2 pc 0000000000b100b4 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #3 pc 0000000000b10630 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #4 pc 0000000000b14e3c /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #5 pc 000000000097e71c /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #6 pc 000000000089e7b8 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #7 pc 0000000000140350 /apex/com.android.runtime/lib64/libart.so (art_quick_generic_jni_trampoline+144) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #8 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #9 pc 000000000014600c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #10 pc 00000000002e3948 /apex/com.android.runtime/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+384) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #11 pc 00000000002deba8 /apex/com.android.runtime/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+892) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #12 pc 00000000005a3c08 /apex/com.android.runtime/lib64/libart.so (MterpInvokeStatic+372) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #13 pc 0000000000131994 /apex/com.android.runtime/lib64/libart.so (mterp_op_invoke_static+20) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #14 pc 00000000002b66fc /data/local/tmp/incrementaldeployment/com.google.mediapipe.apps.iristrackinggpu/dex/incremental_classes1.dex (com.google.mediapipe.framework.PacketGetter.getProtoVector+8)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #15 pc 00000000002b4c5c /apex/com.android.runtime/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEbb.llvm.12938883504528282530+240) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #16 pc 000000000059269c /apex/com.android.runtime/lib64/libart.so (artQuickToInterpreterBridge+1032) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #17 pc 0000000000140468 /apex/com.android.runtime/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #18 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #19 pc 000000000014600c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #20 pc 00000000002e3948 /apex/com.android.runtime/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+384) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #21 pc 00000000002deba8 /apex/com.android.runtime/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+892) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #22 pc 00000000005a3c08 /apex/com.android.runtime/lib64/libart.so (MterpInvokeStatic+372) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #23 pc 0000000000131994 /apex/com.android.runtime/lib64/libart.so (mterp_op_invoke_static+20) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #24 pc 000000000040f236 /data/local/tmp/incrementaldeployment/com.google.mediapipe.apps.iristrackinggpu/dex/incremental_classes1.dex (com.google.mediapipe.apps.iristrackinggpu.MainActivity.lambda$onCreate$0$MainActivity+34)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #25 pc 00000000005a3ea4 /apex/com.android.runtime/lib64/libart.so (MterpInvokeStatic+1040) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #26 pc 0000000000131994 /apex/com.android.runtime/lib64/libart.so (mterp_op_invoke_static+20) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #27 pc 000000000035a00c /data/local/tmp/incrementaldeployment/com.google.mediapipe.apps.iristrackinggpu/dex/incremental_classes1.dex (com.google.mediapipe.apps.iristrackinggpu.MainActivity$$Lambda$0.process)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #28 pc 00000000002b4c5c /apex/com.android.runtime/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEbb.llvm.12938883504528282530+240) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #29 pc 000000000059269c /apex/com.android.runtime/lib64/libart.so (artQuickToInterpreterBridge+1032) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #30 pc 0000000000140468 /apex/com.android.runtime/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #31 pc 0000000000137334 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #32 pc 0000000000145fec /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+244) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #33 pc 00000000004b103c /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #34 pc 00000000004b2488 /apex/com.android.runtime/lib64/libart.so (art::InvokeVirtualOrInterfaceWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+424) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #35 pc 0000000000398284 /apex/com.android.runtime/lib64/libart.so (art::JNI::CallVoidMethodV(_JNIEnv*, _jobject*, _jmethodID*, std::__va_list)+628) (BuildId: ced0e918261ca872f5cff4cdba80b1a9)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #36 pc 000000000088e140 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #37 pc 000000000088dee8 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #38 pc 00000000008b50f0 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #39 pc 00000000008f2308 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #40 pc 00000000008de8cc /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #41 pc 00000000008de320 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #42 pc 00000000009140d8 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #43 pc 0000000000913d08 /data/data/com.google.mediapipe.apps.iristrackinggpu/incrementallib/libmediapipe_jni.so
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #44 pc 00000000000d6b70 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: a2584ee8458a61d422edf24b4cd23b78)
2020-08-10 16:29:13.831 26111-26111/? A/DEBUG: #45 pc 0000000000074eac /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: a2584ee8458a61d422edf24b4cd23b78)

@mehedi28
Copy link

I managed to get iris landmarks by following below steps:

  1. add processor.addPacketCallback() in MainAcitivity (iris) like this:
processor.addPacketCallback(
                    "iris_landmarks",
                    (packet) -> {
                        byte[] landmarksRaw = PacketGetter.getProtoBytes(packet);
                        try {
                            NormalizedLandmarkList landmarks = NormalizedLandmarkList.parseFrom(landmarksRaw);
                            if (landmarks == null) {
                                Log.v(TAG, "[TS:" + packet.getTimestamp() + "] No iris landmarks.");
                                return;
                            }
                            // Note: If eye_presence is false, these landmarks are useless.
                            Log.v(
                                    TAG,
                                    "[TS:"
                                            + packet.getTimestamp()
                                            + "] #Landmarks for iris: "
                                            + landmarks.getLandmarkCount());
                            Log.v(TAG, getLandmarksDebugString(landmarks));
                        } catch (InvalidProtocolBufferException e) {
                            Log.e(TAG, "Couldn't Exception received - " + e);
                            return;
                        }
                    });
  1. add getLandmarksDebugString()
private static String getLandmarksDebugString(NormalizedLandmarkList landmarks) {
        int landmarkIndex = 0;
        String landmarksString = "";
        for (LandmarkProto.NormalizedLandmark landmark : landmarks.getLandmarkList()) {
            landmarksString +=
                    "\t\tLandmark["
                            + landmarkIndex
                            + "]: ("
                            + landmark.getX()
                            + ", "
                            + landmark.getY()
                            + ", "
                            + landmark.getZ()
                            + ")\n";
            ++landmarkIndex;
        }
        return landmarksString;
    }
  1. also do not forget to add this line as deps at android_binary in BUILD file:
    "//mediapipe/framework/formats:landmark_java_proto_lite"

@ridoy-ai
Copy link
Author

Thanks @mehedi28 for mentioning the steps.

@fanzhanggoogle fanzhanggoogle self-assigned this Aug 12, 2020
@fanzhanggoogle
Copy link

@mehedi28 Thank you for helping answer the question!

@xiezhongzhao
Copy link

I managed to get iris landmarks by following below steps:

  1. add processor.addPacketCallback() in MainAcitivity (iris) like this:
processor.addPacketCallback(
                    "iris_landmarks",
                    (packet) -> {
                        byte[] landmarksRaw = PacketGetter.getProtoBytes(packet);
                        try {
                            NormalizedLandmarkList landmarks = NormalizedLandmarkList.parseFrom(landmarksRaw);
                            if (landmarks == null) {
                                Log.v(TAG, "[TS:" + packet.getTimestamp() + "] No iris landmarks.");
                                return;
                            }
                            // Note: If eye_presence is false, these landmarks are useless.
                            Log.v(
                                    TAG,
                                    "[TS:"
                                            + packet.getTimestamp()
                                            + "] #Landmarks for iris: "
                                            + landmarks.getLandmarkCount());
                            Log.v(TAG, getLandmarksDebugString(landmarks));
                        } catch (InvalidProtocolBufferException e) {
                            Log.e(TAG, "Couldn't Exception received - " + e);
                            return;
                        }
                    });
  1. add getLandmarksDebugString()
private static String getLandmarksDebugString(NormalizedLandmarkList landmarks) {
        int landmarkIndex = 0;
        String landmarksString = "";
        for (LandmarkProto.NormalizedLandmark landmark : landmarks.getLandmarkList()) {
            landmarksString +=
                    "\t\tLandmark["
                            + landmarkIndex
                            + "]: ("
                            + landmark.getX()
                            + ", "
                            + landmark.getY()
                            + ", "
                            + landmark.getZ()
                            + ")\n";
            ++landmarkIndex;
        }
        return landmarksString;
    }
  1. also do not forget to add this line as deps at android_binary in BUILD file:
    "//mediapipe/framework/formats:landmark_java_proto_lite"

The image resolution is 960*1280, and the landmarks are normalized. My question is how to get the raw landmarks projected on the image.

@nitin26gupta
Copy link

nitin26gupta commented Sep 4, 2020 •
edited
Loading

@mehedi28 thanks for writing the steps.
but I am not seeing these logs in logcat in android studio, so where else could be see these logs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fanzhanggoogle fanzhanggoogle

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@nitin26gupta @verma-akash @fanzhanggoogle @xiezhongzhao @ridoy-ai @mehedi28 and others