Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minicap for Android 11 #3

Closed
siva-kranthi opened this issue Jul 28, 2020 · 20 comments
Closed

Minicap for Android 11 #3

siva-kranthi opened this issue Jul 28, 2020 · 20 comments
Labels
question Further information is requested

Comments

@siva-kranthi
Copy link

What is the issue or idea you have?
stf is not working on Android 11 which requires minicap.so file

Have you tried STF?
Yes. It is not working

May I know, how we can build the so file using AOSP code. Do I need to follow exactly this minicap-shared or is there any other simple way

Right now android-r-beta-2 is available
openstf#208

@issue-label-bot
Copy link

Issue-Label Bot is automatically applying the label question to this issue, with a confidence of 0.74. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

@issue-label-bot issue-label-bot bot added the question Further information is requested label Jul 28, 2020
@koral--
Copy link
Member

koral-- commented Jul 30, 2020

@pcrepieux could you help with this?

@pcrepieux
Copy link

@Babu-SKK
We are indeed aware of the issue. This is currently due to AOSP not being fully updated. Building minicap-shared for API level 30 doesn't help for the moment because there are a couple API changes in libgui that are not available in the current AOSP source code (even on the android-r-beta-2 tag :-/). See openstf#208 (comment) to have a more detailed explanation. We are keeping an eye on this to be able to add Android 11 support as soon as possible. In the meantime, I am investigating what could be done in order not to depend on libgui which breaks minicap at nearly every release.

@varundtsfi
Copy link
Collaborator

varundtsfi commented Aug 14, 2020

Hi @pcrepieux

I am using scrcpy utility which is working fine with preview as well as on beta versions without doing anything.

I am using this for my application testing on local system.
This kind of solution we can use as a reference.

@pcrepieux
Copy link

Hi @varundtsfi
That's indeed something we are thinking about. scrcpy uses the MediaCodec and Surface APIs to encode the screen display as a H.264 stream.

One of the main advantage I see with this approach is that it doesn't require a compilation step that depends upon AOSP. Plus, H.264 would be more bandwidth efficient than a sequence of jpeg images.

The downside is that it will come at the price of reworking [a lot ?] the client side. Rendering this raw H.264 stream is managed by ffmpeg in scrcpy and they did it very well. Implementing something similar in the browser is likely to be a bit challenging.

In any case, that's an approach to investigate because things would be way more easy/stable if we could get rid of libgui dependency. So basically, I agree with you 👍

@varundtsfi
Copy link
Collaborator

varundtsfi commented Aug 15, 2020

Thanks @pcrepieux for your views on this.

I have found a similar stuff which is working top of Scrcpy with web client ws-scrcpy

It's a prototype I am looking into this. I am checking their open and closed issues. I will share my findings with you.

@man4567890
Copy link

man4567890 commented Aug 24, 2020

I copied the android 29 folder in /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/arm64-v8a/lib/ to android 30 and got from this

2020-08-24T05:22:00.276Z INF/device:support:sdk 4667 [05161FQCB00318] Supports SDK 30
2020-08-24T05:22:00.276Z INF/device:support:abi 4667 [05161FQCB00318] Supports ABIs arm64-v8a, armeabi-v7a, armeabi
2020-08-24T05:22:00.278Z FTL/device 4667 [05161FQCB00318] Setup had an error Error: At least one of these paths should exist: /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/arm64-v8a/lib/android-30/minicap.so, /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/arm64-v8a/lib/android-30/minicap.so, /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/armeabi-v7a/lib/android-30/minicap.so, /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/armeabi-v7a/lib/android-30/minicap.so, /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/armeabi/lib/android-30/minicap.so, /usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/armeabi/lib/android-30/minicap.so
    at Object.module.exports.requiredMatch (/usr/lib/node_modules/stf/lib/util/pathutil.js:33:9)
    at /usr/lib/node_modules/stf/lib/units/device/resources/minicap.js:41:23
    at SerialSyrup.ParallelSyrup.invoke (/usr/lib/node_modules/stf/node_modules/stf-syrup/lib/parallel.js:54:24)
    at /usr/lib/node_modules/stf/node_modules/stf-syrup/lib/serial.js:43:33
    at tryCatch1 (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/util.js:63:19)
    at Promise$_callHandler [as _callHandler] (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/promise.js:708:13)
    at Promise$_settlePromiseFromHandler [as _settlePromiseFromHandler] (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/promise.js:724:18)
    at Promise$_settlePromiseAt [as _settlePromiseAt] (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/promise.js:896:14)
    at Promise$_fulfillPromises [as _fulfillPromises] (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/promise.js:1041:14)
    at Async$_consumeFunctionBuffer [as _consumeFunctionBuffer] (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/async.js:74:12)
    at Async$consumeFunctionBuffer (/usr/lib/node_modules/stf/node_modules/stf-syrup/node_modules/bluebird/js/main/async.js:37:14)
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickCallback (internal/process/next_tick.js:181:9)
2020-08-24T05:22:00.279Z FTL/util:lifecycle 4667 [05161FQCB00318] Shutting down due to fatal error
2020-08-24T05:22:00.287Z INF/provider 2103 [*] Cleaning up device worker "05161FQCB00318"
2020-08-24T05:22:00.287Z ERR/provider 2103 [*] Device worker "05161FQCB00318" died with code 1
2020-08-24T05:22:00.287Z INF/provider 2103 [*] Restarting device worker "05161FQCB00318"

to this, but I still cannot open the screen


2020-08-24T06:00:33.091Z INF/device:plugins:touch 7672 [05161FQCB00318] minitouch says: "using Android InputManager"
2020-08-24T06:00:33.091Z INF/device:plugins:touch 7672 [05161FQCB00318] minitouch says: "Connection established"
2020-08-24T06:00:33.096Z INF/device:plugins:vnc 7672 [05161FQCB00318] Starting VNC server on port 7538
2020-08-24T06:00:33.098Z INF/device:plugins:browser 7672 [05161FQCB00318] Loading browser list
2020-08-24T06:00:33.115Z INF/device:plugins:browser 7672 [05161FQCB00318] Updating browser list
2020-08-24T06:00:33.117Z INF/device:plugins:mute 7672 [05161FQCB00318] Will not mute master volume
2020-08-24T06:00:33.305Z INF/device:resources:minirev 7672 [05161FQCB00318] Installing "/usr/lib/node_modules/stf/vendor/minirev/arm64-v8a/minirev" as "/data/local/tmp/minirev"
2020-08-24T06:00:33.502Z INF/device:plugins:forward 7672 [05161FQCB00318] Launching reverse port forwarding service
2020-08-24T06:00:33.532Z INF/device:plugins:forward 7672 [05161FQCB00318] Connecting to reverse port forwarding service
2020-08-24T06:00:33.552Z INF/device 7672 [05161FQCB00318] Fully operational
2020-08-24T06:01:44.004Z INF/device:plugins:group 7672 [05161FQCB00318] Subscribing to group channel "27wYow6vQ7KHx53MvmzFQw=="
2020-08-24T06:01:45.076Z INF/device:plugins:screen:stream 7672 [05161FQCB00318] Setting frame producer projection to 843x843
2020-08-24T06:01:45.076Z INF/device:plugins:screen:stream 7672 [05161FQCB00318] Requesting frame producer to start
2020-08-24T06:01:45.077Z INF/device:plugins:screen:stream 7672 [05161FQCB00318] Launching screen service
2020-08-24T06:01:45.141Z INF/device:plugins:screen:stream 7672 [05161FQCB00318] minicap says: "CANNOT LINK EXECUTABLE "/data/local/tmp/minicap": cannot locate symbol "_ZN7android21SurfaceComposerClient17getDisplayEEPNS_6VectorINS_11DisplayInfoEEE" referenced by "/data/local/tmp/minicap.so"..."
2020-08-24T06:01:45.142Z WRN/device:plugins:screen:stream 7672 [05161FQCB00318] Shell keeping minicap running ended unexpectedly
2020-08-24T06:01:47.102Z INF/device:plugins:screen:stream 7672 [05161FQCB00318] Disconnecting from minicap service
2020-08-24T06:01:47.102Z INF/device:plugins:screen:stream 7672 [05161FQCB00318] Stopping minicap service
2020-08-24T06:01:47.103Z FTL/device:plugins:screen:stream 7672 [05161FQCB00318] Frame producer had an error TimeoutError: operation timed out
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
From previous event:
    at FrameProducer._waitForPid (/usr/lib/node_modules/stf/lib/units/device/plugins/screen/stream.js:272:10)
    at FrameProducer.<anonymous> (/usr/lib/node_modules/stf/lib/units/device/plugins/screen/stream.js:81:27)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
2020-08-24T06:01:47.103Z FTL/util:lifecycle 7672 [05161FQCB00318] Shutting down due to fatal error
2020-08-24T06:01:47.110Z INF/provider 2103 [*] Cleaning up device worker "05161FQCB00318"
2020-08-24T06:01:47.110Z ERR/provider 2103 [*] Device worker "05161FQCB00318" died with code 1
2020-08-24T06:01:47.110Z INF/provider 2103 [*] Restarting device worker "05161FQCB00318"
2020-08-24T06:01:47.903Z INF/device:support:push 15968 [05161FQCB00318] Sending output to "tcp://127.0.0.1:7116"
2020-08-24T06:01:47.906Z INF/device 15968 [05161FQCB00318] Preparing device
2020-08-24T06:01:48.292Z INF/device:support:sub 15968 [05161FQCB00318] Receiving input from "tcp://127.0.0.1:7114"
2020-08-24T06:01:48.293Z INF/device:support:sub 15968 [05161FQCB00318] Subscribing to permanent channel "*ALL"
2020-08-24T06:01:48.377Z INF/device:support:properties 15968 [05161FQCB00318] Loading properties
2020-08-24T06:01:48.426Z INF/device:support:sdk 15968 [05161FQCB00318] Supports SDK 30
2020-08-24T06:01:48.427Z INF/device:support:abi 15968 [05161FQCB00318] Supports ABIs arm64-v8a, armeabi-v7a, armeabi
2020-08-24T06:01:48.652Z INF/device:resources:minicap 15968 [05161FQCB00318] Installing "/usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/arm64-v8a/bin/minicap" as "/data/local/tmp/minicap"
2020-08-24T06:01:48.653Z INF/device:resources:minicap 15968 [05161FQCB00318] Installing "/usr/lib/node_modules/stf/node_modules/minicap-prebuilt-beta/prebuilt/arm64-v8a/lib/android-30/minicap.so" as "/data/local/t
2020-08-24T06:01:48.958Z INF/device:resources:service 15968 [05161FQCB00318] Checking whether we need to install STFService
2020-08-24T06:01:49.023Z INF/device:resources:service 15968 [05161FQCB00318] Running version check
2020-08-24T06:01:49.667Z INF/device:resources:service 15968 [05161FQCB00318] Installing STFService
2020-08-24T06:01:51.357Z INF/device:resources:service 15968 [05161FQCB00318] STFService up to date
2020-08-24T06:01:51.359Z INF/device:plugins:service 15968 [05161FQCB00318] Launching agent
2020-08-24T06:01:51.868Z INF/device:plugins:service 15968 [05161FQCB00318] Agent says: "Starting minitouch agent"
2020-08-24T06:01:51.883Z INF/device:plugins:service 15968 [05161FQCB00318] Agent says: "Listening on @stfagent"
2020-08-24T06:01:51.938Z INF/device:plugins:service 15968 [05161FQCB00318] Launching service
2020-08-24T06:01:51.938Z INF/device:plugins:service 15968 [05161FQCB00318] using 'start-foreground-service' command for API 30
2020-08-24T06:01:51.943Z INF/device:plugins:service 15968 [05161FQCB00318] Agent says: "InputClient started"
2020-08-24T06:01:52.104Z INF/device:plugins:display 15968 [05161FQCB00318] Reading display info
2020-08-24T06:01:52.160Z INF/device:plugins:phone 15968 [05161FQCB00318] Fetching phone info
2020-08-24T06:01:52.180Z INF/device:plugins:identity 15968 [05161FQCB00318] Solving identity
2020-08-24T06:01:52.184Z INF/device:plugins:solo 15968 [05161FQCB00318] Subscribing to permanent channel "jG0pffP3VOKbfoELJ99pvkTpOt0="
2020-08-24T06:01:52.185Z INF/device:plugins:screen:stream 15968 [05161FQCB00318] Starting WebSocket server on port 7540
2020-08-24T06:01:52.368Z INF/device:resources:minitouch 15968 [05161FQCB00318] Installing "/usr/lib/node_modules/stf/node_modules/minitouch-prebuilt-beta/prebuilt/arm64-v8a/bin/minitouch" as "/data/local/tmp/minit
2020-08-24T06:01:52.579Z WRN/device:plugins:data 15968 [05161FQCB00318] Unable to find device data { serial: '05161FQCB00318',
  platform: 'Android',
  manufacturer: 'GOOGLE',
  operator: ',
  model: 'Pixel 4a (5G)',
  version: '11',
  abi: 'arm64-v8a',
  sdk: '30',
  product: 'bramble',
  cpuPlatform: 'lito',
  openGLESVersion: '3.2',
  marketName: 'bramble',
  display: 
   { id: 0,
     width: 1080,
     height: 2340,
     xdpi: 415.635986328125,
     ydpi: 412.75,
     fps: 60.000003814697266,
     density: 2.625,
     rotation: 0,
     secure: true,
     size: 6.236400407587315,
     url: 'ws://10.119.102.86:7540' },
  phone: { network: 'UNKNOWN' } }
2020-08-24T06:01:52.582Z INF/device:plugins:touch 15968 [05161FQCB00318] Touch origin is top left
2020-08-24T06:01:52.583Z INF/device:plugins:touch 15968 [05161FQCB00318] Requesting touch consumer to start
2020-08-24T06:01:52.583Z INF/device:plugins:touch 15968 [05161FQCB00318] Launching touch service
2020-08-24T06:01:52.605Z INF/device:plugins:touch 15968 [05161FQCB00318] Connecting to minitouch service
2020-08-24T06:01:52.621Z INF/device:plugins:touch 15968 [05161FQCB00318] Reading minitouch banner
2020-08-24T06:01:52.664Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "open: Permission denied"
2020-08-24T06:01:52.664Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Unable to open device /dev/input/event5 for inspectionopen: Permission denied"
2020-08-24T06:01:52.664Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Unable to open device /dev/input/event4 for inspectionopen: Permission denied"
2020-08-24T06:01:52.665Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Unable to open device /dev/input/event3 for inspectionopen: Permission denied"
2020-08-24T06:01:52.665Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Unable to open device /dev/input/event2 for inspectionopen: Permission denied"
2020-08-24T06:01:52.665Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Unable to open device /dev/input/event1 for inspectionopen: Permission denied"
2020-08-24T06:01:52.666Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Unable to open device /dev/input/event0 for inspectionUnable to find a suitable touch device"
2020-08-24T06:01:52.666Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "using Android InputManager"
2020-08-24T06:01:52.666Z INF/device:plugins:touch 15968 [05161FQCB00318] minitouch says: "Connection established"
2020-08-24T06:01:52.673Z INF/device:plugins:vnc 15968 [05161FQCB00318] Starting VNC server on port 7542
2020-08-24T06:01:52.674Z INF/device:plugins:browser 15968 [05161FQCB00318] Loading browser list
2020-08-24T06:01:52.692Z INF/device:plugins:browser 15968 [05161FQCB00318] Updating browser list
2020-08-24T06:01:52.693Z INF/device:plugins:mute 15968 [05161FQCB00318] Will not mute master volume
2020-08-24T06:01:52.877Z INF/device:resources:minirev 15968 [05161FQCB00318] Installing "/usr/lib/node_modules/stf/vendor/minirev/arm64-v8a/minirev" as "/data/local/tmp/minirev"
2020-08-24T06:01:53.089Z INF/device:plugins:forward 15968 [05161FQCB00318] Launching reverse port forwarding service
2020-08-24T06:01:53.112Z INF/device:plugins:forward 15968 [05161FQCB00318] Connecting to reverse port forwarding service
2020-08-24T06:01:53.132Z INF/device 15968 [05161FQCB00318] Fully operational

Does anyone have any suggestions please

@siva-kranthi
Copy link
Author

@pcrepieux android-r-beta-3 released

@varundtsfi
Copy link
Collaborator

varundtsfi commented Aug 31, 2020

But still Rotation.h is not there in ui directory. In short not completely sync. Might be on 8th September Google will release Final Android 11 with fully sync aosp. By that time we have to wait and watch.

android-r-beta-3

@WeiPeiXian
Copy link

https://github.com/WeiPeiXian/minicap/tree/master/minicap.so
Here is the minicap.so files for Android 11, may be it will be helpful
best wishes
@varundtsfi

@varundtsfi
Copy link
Collaborator

varundtsfi commented Sep 7, 2020

@WeiPeiXian
Thanks for sharing the binaries. I have tested these binaries on Xiaomi Android 11 device. These are working.

INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:248) Creating SurfaceComposerClient
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:251) Performing SurfaceComposerClient init check
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:262) Creating virtual display
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:268) Creating buffer queue
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:271) Setting buffer options
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:275) Creating CPU consumer
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:279) Creating frame waiter
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:283) Publishing virtual display BBBB:
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:289) Starting apply
INFO: (packages/apps/SYSOPT/minicap/so/src/minicap_30.cpp:291) Ending apply

@pcrepieux Here is a good information about these binaries. These binaries are working for Google Pixel 2(Android 11) Also.

Android11

@pcrepieux
Copy link

Good to know 👍 And the other good news is now that Android 11 has been officially launched, the source code is finally available 🎉
Better late than never. It is now possible to compile minicap-shared using open source code. Stay tuned.

@varundtsfi
Copy link
Collaborator

varundtsfi commented Sep 9, 2020

@pcrepieux

Here is the first android 11 r1 branch 😊😊😍

https://android.googlesource.com/platform/frameworks/native/+/refs/tags/android-11.0.0_r1

@siva-kranthi
Copy link
Author

siva-kranthi commented Sep 10, 2020

@varundtsfi @pcrepieux @WeiPeiXian I tried by placing these .so files in below folder in the deployed stf setup (running locally, no docker)

~STF/node_modules/minicap-prebuilt-beta/prebuilt/
├── arm64-v8a
│   └── lib
│   └── android-30
│   └── minicap.so
├── armeabi-v7a
│   └── lib
│   ├── android-30
│   │   └── minicap.so
│ 
├── x86 // Nothing is added in this
│   
└── x86_64 // Nothing is added in this

I tried using Android Studio Emulator, But got below error

Sep 10 10:11:43 2030027559 STF[30312]: 2020-09-10T04:41:43.797Z FTL/device:plugins:screen:stream 30751 [emulator-5554] Frame producer had an error TimeoutError: operation timed out
Sep 10 10:11:43 2030027559 STF[30312]: at ontimeout (timers.js:498:11)
Sep 10 10:11:43 2030027559 STF[30312]: at tryOnTimeout (timers.js:323:5)
Sep 10 10:11:43 2030027559 STF[30312]: at Timer.listOnTimeout (timers.js:290:5)
Sep 10 10:11:43 2030027559 STF[30312]: From previous event:
Sep 10 10:11:43 2030027559 STF[30312]: at FrameProducer._waitForPid (~STF/lib/units/device/plugins/screen/stream.js:271:10)
Sep 10 10:11:43 2030027559 STF[30312]: at FrameProducer.<anonymous> (~STF/lib/units/device/plugins/screen/stream.js:80:27)
Sep 10 10:11:43 2030027559 STF[30312]: at runCallback (timers.js:810:20)
Sep 10 10:11:43 2030027559 STF[30312]: at tryOnImmediate (timers.js:768:5)
Sep 10 10:11:43 2030027559 STF[30312]: at processImmediate [as _immediateCallback] (timers.js:745:5)
CANNOT LINK EXECUTABLE "/data/local/tmp/minicap": "/data/local/tmp/minicap.so" is for EM_ARM (40) instead of EM_386 (3)

Is it the right way of using the so files?

@varundtsfi
Copy link
Collaborator

varundtsfi commented Sep 10, 2020

No these are the aosp pre built libraries.which you have to keep inside aosp/android-30/ directory.

x86 binaries we need to build with aosp, Only these will be available but for arm these are working properly.

@siva-kranthi
Copy link
Author

I am trying to use it in the deployed STF setup (running locally, no docker). In this, we have the only minicap-prebuilt-beta folder. Please help me in using the lib directly in the STF

@doogieaus
Copy link

The problem is these binaries are ARM/ARM64, not x86 which is what your emulator is running. Create an ARM based emulator and it should work

@siva-kranthi
Copy link
Author

siva-kranthi commented Sep 10, 2020

Libs are working fine with Samsung devices. Followed same procedure as I mentioed in earlier post

@varundtsfi
Copy link
Collaborator

varundtsfi commented Sep 11, 2020

@Babu-SKK now you can use intel binaries also which is shipped by @pcrepieux in PR

https://github.com/DeviceFarmer/minicap/commits/android-11-support

@koral--
Copy link
Member

koral-- commented Sep 20, 2020

Issue seems to be resolved by #7

@koral-- koral-- closed this as completed Sep 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

7 participants