Use button to switch over to another mode(PoseTrackingGpu/ObjectDetection)

[UnityDev1021]

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in Mediapipe): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Android
• MediaPipe version:
• Bazel version: 1.10.1
• Solution (e.g. FaceMesh, Pose, Holistic): PoseTrackingGpu, ObjectDetectionGpu
• Programming Language and version ( e.g. C++, Python, Java): Java

Hello, so I am currently building a project that integrates both PoseTrackingGpu and ObjectDetectionGpu. I have successfully build both the files using bazel, but now I would like to build both of them into one app and use a button to switch between the two modes. I can modify the PoseTrackingGpu graph to include both the graphs in one, but I only want one to work at a time to output a specific value and so I would like to keep both the graphs separate and switch between the 2 using a button. Is there any way to do this?

I have looked at the BUILD files but as there are 2 graphs and 1 "binaryGraphName" value, how can I switch between the 2 while the app is running, like when I press the switch button, then the current graph stops running and starts to run a new graph. Can I close the current graph and open a new one? For ex:- Open PoseTrackingGpu graph as primary, close it and start the ObjectDetectionGpu graph.

[sgowroji]

Hi @UnityDev1021, Have you tried any implementation and What is the error noticed while switching the models?

[UnityDev1021]

Hi there @sgowroji, I have not yet tried to implement anything but I am thinking of making a single graph consisting of both PoseTracking and ObjectDetection subgraph and using a SwitchConatiner to switch between the two Subgraphs, as I cannot find any other way to stop and start a new graph as of now.

The only problem is I do not know of any way to supply a value to the SwitchContainer Node at runtime(pretty new to Mediapipe). If there is a way to do it, I would really appreciate to know about it.

Also, if I run the Single graph having the 2 sub-graphs, will the 2 sub-graphs run simultaneously or only the sub-graph that is being selected by the SwitchContainer? For eg:- If I pass a boolean value of FALSE to the Switch Container, then the PoseTracking sub-graph should run and vice-versa. I don't want both the sub-graphs to run at a time to reduce the load.

If there is any alternate way to do the above things where I can just specify 2 main graphs and have the application switch between the 2, then it can also be helpful to me. Any help is appreciated.

[UnityDev1021]

@sgowroji An update. So I wrote some sample code to try out the SwitchContainerCalculator strategy but unfortunately it does not work. I made a simple TestCalculator which accepts the "select" value of the SwitchContainerCalculator and based on that outputs a String which I can then display using a TextView.

I used this code to change the input_side_packet of the SwitchContainerCalculator :-

    protected void InputSidePacket()
    {
        try {
            Packet switchSubGraph = processor.getPacketCreator().createInt32(selectGraph);
            Map<String, Packet> inputSidePackets = new HashMap<>();
            inputSidePackets.put(INPUT_SIDEPACKET_STREAM_NAME, switchSubGraph);
            processor.setInputSidePackets(inputSidePackets);
            Log.d(TAG, "Set the Side Packet");
        } catch (Exception exception) {
            Log.e(TAG, "Failed to set input side packet : ", exception);
        }
    }

I am using a button to call this method, and a TextView to check the output of the calculator. Unfortunately it gives me this error at runtime when I try to click the button:-

2021-12-06 17:11:00.495 28305-28305/? E/MainActivity: Failed to set input side packet : 
    java.lang.IllegalStateException: setInputSidePackets must be called before the graph is started
        at com.google.common.base.Preconditions.checkState(Preconditions.java:510)
        at com.google.mediapipe.components.FrameProcessor.setInputSidePackets(FrameProcessor.java:309)
        at com.google.mediapipe.apps.test.MainActivity.InputSidePacket(MainActivity.java:101)
        at com.google.mediapipe.apps.test.MainActivity$1.onClick(MainActivity.java:63)
        at android.view.View.performClick(View.java:7559)
        at android.view.View.performClickInternal(View.java:7536)
        at android.view.View.access$3600(View.java:828)
        at android.view.View$PerformClick.run(View.java:28700)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:236)
        at android.app.ActivityThread.main(ActivityThread.java:7861)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:600)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:967)

And now I am stuck again and don't know how to proceed further as I am out of ideas. I can see that the processor is responsible for calling the graph at this code :-

    processor =
        new FrameProcessor(
            this,
            eglManager.getNativeContext(),
            applicationInfo.metaData.getString("binaryGraphName"),
            applicationInfo.metaData.getString("inputVideoStreamName"),
            applicationInfo.metaData.getString("outputVideoStreamName"));

So, I was thinking of changing this at runtime. Again no idea as to how I can change it as I think that I would have to restart the whole thing including the camera and the graph, so yeah, awaiting for any help.

[sgowroji]

To reroute packets through an already running graph, "SwitchContainerCalculator" must be controlled using an input stream rather than an input-side-packet. SwitchContainerCalculator does accept input streams similar to its input side packets.

Also, a common design pattern for switching calculator options is to re-start the graph with a new configuration and/or new side-packets, so you can consider that design alternative.

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No