Tfjs pose detection not working with newer Android devices (12, 13, 14)

[expolli]

Please see repro project: https://github.com/expolli/pose-test/tree/main

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 11
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Please see repro project for details.
• TensorFlow.js installed from (npm or script link): npm
• TensorFlow.js version (use command below): 4.19.0
• Browser version: Expo 51
• Tensorflow.js Converter Version: -

Describe the current behavior

Repro app working with older devices but not with new ones

Please see repro project for details.

Describe the expected behavior

Pose detection working on all devices.

Standalone code to reproduce the issue

https://github.com/expolli/pose-test/tree/main

Other info / logs Include any logs or source code that would be helpful to

Please see repro project.

[rachelwong44]

Tensorcamera is not working with expo-camera thanks to the SDK51 update

[expolli]

Thank you for the update @rachelwong44 . Do you know of a workaround to get the barebones app working?

App was not working with Expo 50 either with newer Androids, btw.

Not sure if tfjs is using the tensorcamera internally, but app itself is not using camera at all. It is just reading the one image from the assets and processing that.

[gaikwadrahul8]

Hi, @expolli

I apologize for the delayed response and thank you for bringing this issue to our attention, could you please give it try with these Expo versions 47.0.14 and 49.0.23 and see is it working as expected or not ? meanwhile I'll discuss this issue with our internal meeting and will update you soon.

Thank you for your cooperation and patience.

[expolli]

Hei @gaikwadrahul8 ,

Installed Expo 49.0.23 and unfortunately the result was the same. Older Android working but newer not.

Old Android: LOG {"poses": [{"keypoints": [Array], "keypoints3D": [Array], "score": 0.9999972581863403}]}
New Android: LOG {"poses": []}

These were the versions used (dropped the camera as it was not needed):

    "@tensorflow-models/pose-detection": "^2.1.3",
    "@tensorflow/tfjs": "^4.19.0",
    "@tensorflow/tfjs-react-native": "^1.0.0",
    "async-storage": "^0.1.0",
    "expo": "^49.0.23",
    "expo-asset": "~8.10.1",
    "expo-gl": "~13.0.1",
    "expo-status-bar": "~1.6.0",
    "react": "18.2.0",
    "react-native": "0.72.10",
    "react-native-fs": "^2.20.0"

Had some issues using expo-asset so used fetch instead to get the image from local web server:
const response = await fetch("https://my-webserver/image.jpg")

Please let me know if there is something else I can do to help in your investigations.

Thank you!

[Jersyfi]

Dear @gaikwadrahul8 , an update to the newest Expo SDK Version should be discussed instead on fucussing on old versions. With some changes it is possible to run the SDK 50 and for SDK 51 only the dependencies need to be updated first. Otherwise it isn't so complicated to update to the newest expo-camera release.
My intentions in sending pull-requests are pretty low because i know that no one will answer and release it. Is there a solution we can get progress on this?

At the moment i try to get it running with the following package.json and a custom modified @tensorflow/tfjs-react-native:

"dependencies": {
        "@hookform/resolvers": "^3.3.4",
        "@mediapipe/pose": "^0.5.1675469404",
        "@react-native-async-storage/async-storage": "1.23.1",
        "@react-native-community/datetimepicker": "8.0.1",
        "@react-native-firebase/app": "^20.0.0",
        "@react-native-firebase/auth": "^20.0.0",
        "@react-native-firebase/firestore": "^20.0.0",
        "@react-native-firebase/functions": "^20.0.0",
        "@react-native-firebase/storage": "^20.0.0",
        "@react-navigation/bottom-tabs": "^6.5.20",
        "@react-navigation/native": "^6.1.17",
        "@react-navigation/native-stack": "^6.9.26",
        "@tensorflow-models/pose-detection": "^2.1.3",
        "@tensorflow/tfjs": "^4.19.0",
        "@tensorflow/tfjs-backend-webgl": "^4.19.0",
        "@tensorflow/tfjs-backend-webgpu": "^4.19.0",
        "@tensorflow/tfjs-react-native": "^1.0.0",
        "expo": "^51.0.8",
        "expo-av": "~14.0.5",
        "expo-build-properties": "~0.12.1",
        "expo-camera": "~15.0.9",
        "expo-dev-client": "~4.0.14",
        "expo-device": "~6.0.2",
        "expo-file-system": "~17.0.1",
        "expo-gl": "~14.0.2",
        "expo-image": "~1.12.9",
        "expo-image-manipulator": "~12.0.5",
        "expo-image-picker": "~15.0.5",
        "expo-keep-awake": "~13.0.2",
        "expo-linear-gradient": "~13.0.2",
        "expo-linking": "~6.3.1",
        "expo-screen-orientation": "~7.0.5",
        "expo-splash-screen": "~0.27.4",
        "expo-status-bar": "~1.12.1",
        "expo-system-ui": "~3.0.4",
        "expo-web-browser": "~13.0.3",
        "lucide-react-native": "^0.378.0",
        "react": "18.2.0",
        "react-hook-form": "^7.51.1",
        "react-native": "0.74.1",
        "react-native-fs": "^2.20.0",
        "react-native-gesture-handler": "~2.16.1",
        "react-native-reanimated": "~3.10.1",
        "react-native-safe-area-context": "4.10.1",
        "react-native-screens": "3.31.1",
        "react-native-svg": "15.2.0",
        "react-native-ui-lib": "^7.22.0",
        "seedrandom": "^3.0.5"
    },
    "devDependencies": {
        "@babel/core": "^7.20.0",
        "@babel/plugin-proposal-nullish-coalescing-operator": "^7.18.6",
        "@babel/plugin-proposal-optional-chaining": "^7.21.0",
        "@babel/plugin-transform-arrow-functions": "^7.24.1",
        "@babel/plugin-transform-optional-chaining": "^7.24.1",
        "@babel/plugin-transform-shorthand-properties": "^7.24.1",
        "@babel/plugin-transform-template-literals": "^7.24.1",
        "@babel/preset-env": "^7.24.3",
        "@react-native/babel-preset": "*",
        "@types/react": "~18.2.79",
        "@typescript-eslint/eslint-plugin": "^6.12.0",
        "@typescript-eslint/parser": "^6.12.0",
        "eslint": "^8.45.0",
        "eslint-config-prettier": "^9.0.0",
        "eslint-config-universe": "^12.0.0",
        "eslint-plugin-react-native": "^4.1.0",
        "prettier": "3.0.0",
        "typescript": "~5.3.3"
    }