Support for accessing devices not on the machine SikuliX works on (phones, vms, remote computers, ...) #210

Open
tg44 opened this Issue Jun 17, 2016 · 57 comments

Projects

None yet

5 participants

@tg44
tg44 commented Jun 17, 2016

Hy there! I love this tool, but it has a basic problem. If you want to automate things that are not on your computer, but you have basic api to get the actual picture, and how to click on it, you still need to lose a PC just because it can't handle this type of functionality.

For example browsers or phones or vms or remote computers.

My specific case is Android gameing automation. So I have a tool which can get pictures from a phone screen from adb. This tool can handle click to (x,y), and swipe from-to actions too. (It could do pich and some other gesture too, but I want only the swipe (aka drag and drop) and touch (aka click)) at first run.

It would be pretty nice if sikuli and the IDE could use some api as screen input, and some other api to click and drag and drop and other functions.

I browsed the source, I think if I reimplement the Screen and Mouse classes that could be work. I have 2 problems with it: It's not the clean and generic way and I have no idea if it will work or not.

So basicly my questions:

  • Have you any advice about a mergable implementation, or its too complex, and an Android specific fork would be easyer?
  • If I reimplement the Screen and Mouse classes and modify the ide to be able to capture pictures from api screen instead of computer screen the run command can do its stuffs without touching the host computer?
  • Will be any headless/apiscreen support in the 2.0 version?

If I will have time and you say it's doable I really want to make a version that can handle Android devices thought adb and still enable to use the host computer,

@RaiMan
Owner
RaiMan commented Jun 18, 2016

Highly appreciated.

A template for the principal approach can be taken from the VNC implementation in the contributed package module API::edu/unh/iol/dlc.

The relevant classes are

  • VNCScreen (screen as region, capture, ...)
  • VNCRobot (mouse and keyboard)

For both there are Interface classes (IScreen, IRobot) that define the methods to be implemented.
Recently I added some docs and conveniences, which might help too having a look at (http://sikulix-2014.readthedocs.io/en/latest/screen.html#connecting-to-a-vnc-server-vncscreen)

If you give me some more detailed information about what you already have, I can give more precise instructions what to do and how.

As a start, just make a fork of this repo to work with. So I can have a look and suggest or even create pull requests.

@RaiMan RaiMan changed the title from "headless" support to Support for accessing devices not on the machine SikuliX works on (phones, vms, remote computers, ...) Jun 18, 2016
@RaiMan RaiMan added the enhancement label Jun 18, 2016
@tg44
tg44 commented Jun 26, 2016

I just started this:
https://github.com/tg44/SikuliX-2014/tree/master/API/src/main/java/org/sikuli/android
The IDE has screen capture functions to vnc?

I will working on this slowly, I think I can produce a PR in this week with a limited but working functionality.

@RaiMan
Owner
RaiMan commented Jun 28, 2016

Thanks. I will have a look the next days.

--- The IDE has screen capture functions to vnc?
no.
you have to write your own capture script or program.

@tg44
tg44 commented Jun 28, 2016

So I implemented a screen viewer function for ADB.
My only task left for a working solution is making a new button to the ide to run a macro on adb instead of screen.
Can you help me how can I tell the scriptruner to run my IRobot or IScreen instead of the build in? Where I need to modify things? (I searched where the IRobot interface is used, and didn't find in the IDE which can be a problem... )

@RaiMan
Owner
RaiMan commented Jun 29, 2016

In fact I am a bit short on time currently due to other private priorities, so you have to wait for another few days.
I hope, this is acceptable. Thank you.

@jivank
jivank commented Jul 7, 2016

@tg44 Nice work! Perhaps a radio button will do to select local screen and adb device. Also something to specify which adb device via the sikulix commandline. Looking forward to this great feature. Also does adb require root to view the screen?

@tg44
tg44 commented Jul 7, 2016 edited

@jivank if you check out my work I added "adb screen" feature where you can see your device screen in a window and make subimages as patterns with the build in screen capture tool (yes, this is a little bit tricky).
In the future it would be nice if we can select adb devices, and not just get the default one. And it would be nice too if the adb screen would be rotateable and resizeable and the sample in the window would transformed when we use the ide. But, I just want a working solution first :D
Root not needed, only problem could be, that we can show only 2 frames/sec with this method. BUT you will need a properly installed adb (or at least the driver) and maybe a working adb console if you want to use my build (in theory you dont need to start a deamon by hand, but now this feature is untested :) ).

My problems now:
The ide crashes when I try to capture from the screen .
I have no idea how can I start the script from the ide on my implementation...

@RaiMan
Owner
RaiMan commented Jul 8, 2016 edited

@tg44
I would highly appreciate if you tell me the steps, to get an environment to check your work either on Mac or on Windows:

  • what packages to install
  • what IDE to use (currently I prefer InteliJ IDEA CE)
  • any configuration hints
  • I have a Nexus7, if this is of any use for that

BE AWARE: I do not have any Android dev knowledge.
Thank you

@tg44
tg44 commented Jul 8, 2016 edited

http://forum.xda-developers.com/showthread.php?t=2588979 - this installs adb + drivers (to win)
http://stackoverflow.com/questions/31374085/installing-adb-on-mac-os-x - adb to mac
I'm using IntelliJ.
If adb is on your path adb devices will show your device (in a console). After that everything needs to be working.

Still I only could test the adb api so I have no idea if I could integrate it properly to sikulix or it has bugs.

@RaiMan
Owner
RaiMan commented Jul 8, 2016

thanks, will try ;-)

so I have no idea if I could integrate it properly to sikulix or it has bugs.

I accept that is to some extent my job.
But I still need some days.

@jivank
jivank commented Jul 8, 2016

@tg44
Take a look at VNCScreen:
https://github.com/RaiMan/SikuliX-2014/blob/5f64ab2602e2550357a2569713cbbc52b6b93e70/API/src/main/java/edu/unh/iol/dlc/VNCScreen.java

Here is how I was able to use it in the IDE:
#193 (comment)

So I suppose you can create an ADBScreen class using VNCScreen as an example. It won't work as the default screen, but at least it would be usable (and you can see if there is any bugs).

@RaiMan
Owner
RaiMan commented Jul 9, 2016

@tg44, @jivank
I just started to look at the work of @tg44

  • I am currently on Mac 10.11
  • successfully installed the ADB package with Homebrew (my standard package installer)
  • made my Nexus 7 accessible (switch on developer mode)
  • added the org.sikuli.adroid package to this project
  • made some basic tests: works :-))

Then I started to completely revise the classes and added a class ADBDevice where all the device features are handled. The ADBClient is reduced to a singleton class, with only static features (handling the connection to the adb server and managing the list of attached devices).
ADBScreen initializes to represent a device and ADBRobot delegates to the device, that is representing the related screen.

The major revision goal is to take care, that a screen-device-relation can internally be handled as a singleton, to allow to make it thread safe later, in the sense, that on one device at any time only one action can be done (which is vital for the observe feature).

Currently saying:
ADBScreen adbScreen = new ADBScreen();
gives a Region/IScreen object, that is backed by an attached device at position 0 in the devicelist (if any).

When I finish my revision tomorrow, the basic Screen and Region features will work.

for the usage in the IDE as default screen, my idea is to add a menu entry "Android" in the Tools menu, that allows to make this switch:

  • list attached devices
  • select one as default screen
  • ...??
@jchoover

@RaiMan any chance you can push a feature branch with your changes? I've played a bit with keyboard automation, using the input text/input keyevent, but I cant seem to get a proper delay for what I am trying to accomplish.

Also, with @tg44 fork, I cant get a script to work without first invoking the menu item to show the screen (otherwise the package isn't found), and then it takes 2 x script runs as the first one still isn't resolving the classes.

@RaiMan
Owner
RaiMan commented Jul 12, 2016 edited

I just pushed my revision of the package org.sikuli.android.
There is an ADBTest class, that can be run.

  • adb must be installed and useable
  • a device must be attached to USB
  • the device is waked up if needed and unlocked by a swipe up
  • a swipe left and a swipe right is done (only makes sense if the menu screen is visible)
  • you get a chance to capture something on the device's screen shown on your workstation's screen
  • the capture is then searched on the device's screen capture
  • if found, the target is tapped on the device

everything is very experimental and based on my Nexus 7 (Android 6.0.1) running on Mac OSX 10.11.

The capture is rather slow (average 5 seconds) using capture -p. Using capture (returns the raw pixel bytes) cuts down to 2 seconds and less, leaving the image conversion to the host (some 10 milliseconds): see 0169775. Nevertheless it seems to be needed, to insert many waits, to not sending requests faster, than the device changes the screen content or has finished with the last request.

I take this state as some "proof of concept" and stop the development for now, since I want to make the 1.1.1 final now.

Then I will add a development branch with version 1.1.2., where we could go further.
Then maybe I will come back to the idea of adding a feature branch for the Android stuff.

@tg44
tg44 commented Jul 12, 2016

LoL 5 sec is so slow :) My best performance with this screen capture method was 2 FPS with my Xiaomi Mi2S (Android 5.0).
The frame rate can't be higher with this method, I will consult with the android team for more ideas but this need nearly 0 config, we are using built in features with built in tools, and its totally undetectable from appcontext.

Thanks for your work, I will check it later today.

(BTW if you allow wifi debugging and connect your phone to the adb daemon with hand, you can use it without usb connection. )

@RaiMan
Owner
RaiMan commented Jul 12, 2016

Yep, this solution is surely not for game automation (as it is used frequently on workstations).

... but it is rather acceptable for some basic workflows and even some testing, that is allowed to run for some minutes or hours ...
... and that it runs out of the box (only adb must be available) is a big advantage.

About WiFi: I know and will probably try it out the next days.

BTW: I added the internal use of dumpsys to get some state and config information (display on/off, screen size), but I am aware, that this might be version dependent to some extent.

I decided to add the device specific actions (tap, swipe, ...) to the ADBScreen and directly delegate to the ADBDevice, so the ADBRobot is only needed if you use the workstation specific actions like click.

In the IDE this works for now:

import org.sikuli.android.ADBScreen as AS
use(AS()) # set as default screen 
wakeUp(2)
swipeLeft()
swipeRight()
@jivank
jivank commented Jul 14, 2016

This is great news! I suppose the last step would menu integration with the IDE, where you can select a device from ADB and a window will popup of the live screen (so you can create images from the android screen). I assume this would also be the way to integrate VNC as well.

@RaiMan
Owner
RaiMan commented Jul 14, 2016

@jivank
Exactly ;-)
I added a menu entry in the tool menu,

  • where you can see, wether a device is accessible via adb
    (currently only the first one in the device list)
  • where you can run a test against the device including capture/find/tap
  • where you can switch the image capture to address the device screen

If you want to run a script against the attached device: see the comment above.
The next days I will write something about usage, restrictions and next steps somewhere.

Be aware:
The current device capture, using the png-capture command (capture -p), is rather slow (see above). This then is the main duration influence for the find ops. I started to experiment with the so called raw capture, that promises more speed (up to 10 times faster capture), since it allows, to move the post processing of the raw pixel bytes to the host machine (where we talk about milliseconds and not seconds).

All this currently happens on Mac OSX 10.11 with my Nexus 7 (Android 6). I have no idea (and no chance to get any) how all this looks and feels on newer devices.

The stuff is available in the next nightly build 2016-07-15 (package org.sikuli.android).

Any feedback and ideas are welcome.

@RaiMan
Owner
RaiMan commented Jul 14, 2016

@jivank

... and yes, this will be implemented for the VNCScreen too later.

@RaiMan
Owner
RaiMan commented Jul 15, 2016

Capture now uses raw capture on the device, which makes it a lot faster:

see commit: 0169775

@RaiMan
Owner
RaiMan commented Jul 16, 2016

Just checked a device connection over WiFi:

At least in my test environment the capture is unacceptable slow (30 seconds).

Until someone tells me a solution for that, I will not test any further with WiFi connections.

@tg44
tg44 commented Jul 16, 2016

Raw capture is a good idea. I think the framerate is highly depends on the device screen resolution, lower res -> higher frame rate...

I had so much issues because of a lower version adb-server, I solved it but take me lot of time to find out that this is the actual problem. The sysdump is really ugly, my device is printing things to a console more than 1minute long.

Now I have issues with the ADBScreen.userCapture, I have no idea why it's not working, and currently I have no time to dig more deeper to the code. (Your exception handling and logging is killing me, you are caches exceptions and logging them to "I have no idea where". And there is no error on the console, but the app is hanging and I know that there is an exception somewhere, but I cant find it :D )
How this is working (when its working)? It's give you a new window? Or how you can capture imges from the android screen?

@jchoover

When I tried Gergo's fork it works with quirks. When I tried last night
with the latest from RaiMan, I had to modify the regex for it to detect the
screen being awake on Nox, but then it would fail to find the opencv native
libs and give an error when trying to create the Mat class.

On Saturday, July 16, 2016, Gergő Törcsvári notifications@github.com
wrote:

Raw capture is a good idea. I think the framerate is highly depends on the
device screen resolution, lower res -> higher frame rate...

I had so much issues because of a lower version adb-server, I solved it
but take me lot of time to find out that this is the actual problem. The
sysdump is really ugly, my device is printing things to a console more than
1minute long.

Now I have issues with the ADBScreen.userCapture, I have no idea why it's
not working, and currently I have no time to dig more deeper to the code.
(Your exception handling and logging is killing me, you are caches
exceptions and logging them to "I have no idea where". And there is no
error on the console, but the app is hanging and I know that there is an
exception somewhere, but I cant find it :D )
How this is working (when its working)? It's give you a new window? Or how
you can capture imges from the android screen?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#210 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmlRXJu0q-1eaKneoE9-Ysqvje0ayt_ks5qWVBVgaJpZM4I4Y5n
.

@RaiMan
Owner
RaiMan commented Jul 16, 2016

Finalized the Android support for now in this experimental state as some proof of concept:

  • IDE tool menu to run a short test on an attached device and to switch to an Android device for image capture and back to the default local screen again
  • running scripts towards an attached device with the possibility, to restrict the search region to smaller parts of the device's screen
  • on my Nexus 7 (1200 x 1920) a search on the whole screen runs less than 2 seconds, on only portions of a screen 1 second and below down to 200 milliseconds
  • click is converted to a tap and the ADBScreen already knows tap and swipe (no timing yet)
  • adb must be available on the host and must be executable from command line using adb

This is a sample script:

from org.sikuli.android import ADBScreen

Debug.on(3)

ascr = ADBScreen.start() # get the one attached device
if not ascr:
  exit(1)

use(ascr) # set as the default region  
ascr.needsUnLock = False # see comment
wakeUp(2) # take care that the display is on, wait max 2 seconds

gsearch = "gsearch.png" # to detect home screen
img = "img.png" # an icon to tap

top = newRegion(Location(0, 0), 300, 300) # top left corner
iconRow = newRegion(Location(0, 600), 1200, 300) # an icon band

for i in range(3): # just do it 3 times 
  key(ADBScreen.HOME) # tap the home button
  top.wait(gsearch, 10) # wait for home screen
  iconRow.click(img) # click icon
  wait(5) # to give time to the triggered action to process

The images are captured from the device using the capture button after having switched the default screen to the device in the tool menu.

ascr.needsUnLock = False: if set to true (the default), after waking up the display a swipe up is done to let a lock screen vanish.

@RaiMan
Owner
RaiMan commented Jul 16, 2016

@tg44 and @jchoover

I understand, that you might have problems:
be aware: it is developed on Mac and not yet tested on Windows nor on Linux.

The next nightly should make less problems.

If you give me more information, what you are doing in what environment, I might get things fixed faster.

@jchoover

I'll get more details in a couple hours, but my env is win7 x64, with x64
JVM v7 something. Using IntelliJ community edition for edits, but compiling
per the doc's with mvn.

On Saturday, July 16, 2016, Raimund Hocke notifications@github.com wrote:

@tg44 https://github.com/tg44 and @jchoover
https://github.com/jchoover

I understand, that you might have problems:
be aware: it is developed on Mac and not yet tested on Windows nor on
Linux.

The next nightly should make less problems.

If you give me more information, what you are doing in what environment, I
might get things fixed faster.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#210 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmlRUPKtczZZMNgKap7DsMk8ckQcaD6ks5qWVnXgaJpZM4I4Y5n
.

@jchoover

Exception in thread "Thread-12" java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J
at org.opencv.core.Mat.n_Mat(Native Method)
at org.opencv.core.Mat.(Mat.java:477)
at org.sikuli.android.ADBDevice.captureDeviceScreenMat(ADBDevice.java:168)
at org.sikuli.android.ADBDevice.captureDeviceScreen(ADBDevice.java:115)
at org.sikuli.android.ADBDevice.captureScreen(ADBDevice.java:102)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:216)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:208)
at org.sikuli.util.OverlayCapturePrompt.prompt(OverlayCapturePrompt.java:213)
at org.sikuli.android.ADBScreen$1.run(ADBScreen.java:271)

Exception in thread "Thread-11" java.lang.NullPointerException
at org.sikuli.ide.SikuliIDE.androidSupportTest(SikuliIDE.java:1810)
at org.sikuli.ide.SikuliIDE.access$2500(SikuliIDE.java:107)
at org.sikuli.ide.SikuliIDE$1.run(SikuliIDE.java:1780)

Is the error I am getting when running the tool menu from your latest build. Note, I still have to tweak the ADBDevice.isDisplayOn method:

String dump = dumpsys("power");
    Pattern displayOn = Pattern.compile("Display Power: state=ON|mScreenOn=true");
    Matcher match = displayOn.matcher(dump);
    if (match.find()) {
        return true;
    } else {

And I am using JDK/JRE 1.8.0_91

@jchoover

FYI, If I add the opencv core and java to the libs the ADBDevice static constructor loads, then this works!

public class ADBScreen extends Region implements EventObserver, IScreen {

  static {
    RunTime.loadLibrary("VisionProxy");
    RunTime.loadLibrary("libopencv_core248");
    RunTime.loadLibrary("libopencv_java248");
  }
@jchoover
jchoover commented Jul 17, 2016 edited

ADBDevice:
public void type(String characters) { try { device.executeShell("input text", characters); } catch (IOException | JadbException e) { log(-1, "type: %s", e); } }

ADBRobot:

  private boolean typing = false;
  private String typingBuffer;
  public void typeChar(char character, KeyMode mode) {
    if(mode == KeyMode.PRESS_RELEASE) {
      if (typing) {
        typingBuffer += "" + character;
      } else {
        device.type("" + character);
      }
    }
}
  @Override
  public void typeStarts() {
    typing = true;
    typingBuffer = "";
  }

  @Override
  public void typeEnds() {
    if(typing) {
      typing = false;
      device.type("" + typingBuffer);
    }
  }
@tg44
tg44 commented Jul 17, 2016

I found this on ADBDevice 145:
if (byte2int(imagePrefix, 0, 4) != devW || byte2int(imagePrefix, 4, 4) != devH) {
devW is 1280, devH is 720 b2i0,4 is 720, b2i4,4 is 1280 so the if is not ideal :( some device is rotated by 90°

!((byte2int(imagePrefix, 0, 4) == devW && byte2int(imagePrefix, 4, 4) == devH) || ((byte2int(imagePrefix, 0, 4) == devH && byte2int(imagePrefix, 4, 4) == devW))) is nearly working.

My main problem is that my device is rotated by 90° so I need to change W and H in some code, and because I dont have big enought screen the half of the mobile screen is never showed. Thats why I tryed to use scrollPanel instead of onscreen render :)

@tg44
tg44 commented Jul 17, 2016

The click on the ide is not working yet as expected:

import org.sikuli.android.ADBScreen as AS
use(AS()) # set as default screen 
wait("asd.png", FOREVER)
click("asd.png")

Wait is waiting for the picture appearence on the device, but click is moving the PC cursore.

@jchoover

Use "tap" instead of click.

On Sunday, July 17, 2016, Gergő Törcsvári notifications@github.com wrote:

The click on the ide is not working yet as expected:

import org.sikuli.android.ADBScreen as AS
use(AS()) # set as default screen
wait("asd.png", FOREVER)
click("asd.png")

Wait is waiting for the picture appearence on the device, but click is
moving the PC cursore.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#210 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmlRWUi5c0ZxKHErnNTsrn4KiKiIQb9ks5qWe41gaJpZM4I4Y5n
.

@RaiMan
Owner
RaiMan commented Jul 17, 2016

@tg44 and @jchoover:
ok, a comment, what I have done/fixed so far:

  • OpenCV problem: fixed
  • ADBScreen.wakeUp() now tries different patterns for version < 5 and > 4, does nothing if not possible to detect screen state (which might lead to black captures, if device sleeps)

@tg44:
I will now track down the click() problem (... and yes: tap should work)
BTW: this is very inefficient and risky:
wait("asd.png", FOREVER) # if it does not appear, you have to kill the IDE
click("asd.png") # does a costly additional search
... better:
tap(wait("asd.png", 20)) # gives up after 20 seconds or taps the found match

@tg44
I will fix the rotated problem of course ;-)

@jchoover
ADBDevice/ADBRobot: as mentioned already before: IMHO it makes more sense to implement the device specific actions in ADBDevice. I will check, what else should be delegated from Robot to Device.

@RaiMan
Owner
RaiMan commented Jul 17, 2016

watch out for the next nightly build 2016-07-18

@RaiMan
Owner
RaiMan commented Jul 17, 2016

@tg44

just checked the situation "device rotated to landscape":

as long as you take care, that it is in landscape before starting the script, everything works fine for me.

I did not check yet, what happens, if you rotate the device during a scriptrun.

use ADBScreen.reset() at the beginning of a script, so that you always start with a new device connection.

@RaiMan
Owner
RaiMan commented Jul 17, 2016

revised/added:

  • device start/init/wakeup
  • input (input text) and delegation of Region(type)
  • click() did not target the device
@jchoover

2 part comment:
The first is there is a deficiency in my type method, as spaces are misinterpreted as additional args, need to use "%s" instead of " ".
I am having issues with the subRegions, where the images get skewed. For now, I am using :

 public BufferedImage captureDeviceScreen(int x, int y, int w, int h) {
    //Mat matImage = captureDeviceScreenMat(x, y, w, h);
    Mat matImage = captureDeviceScreenMat(0, 0, devW, devH);
    BufferedImage bImage = null;
    if (matImage != null) {
      bImage = new BufferedImage(matImage.width(), matImage.height(), BufferedImage.TYPE_3BYTE_BGR);
      byte[] bImageData = ((DataBufferByte) bImage.getRaster().getDataBuffer()).getData();
      matImage.get(0, 0, bImageData);

      if (x != 0 | y != 0 | w != devW | h != devH) {
        BufferedImage subImage = bImage.getSubimage(x, y, w, h);
        log(lvl+1, "captureDeviceScreen:[%d,%d %dx%d]", x, y, w, h);
        return subImage;
      }
    }
    return bImage;
  }

But I'd like to find out the root cause and fix it. It all most seems like were off by a pixel or two, and so the scan lines get shifted. I'm running an emulator, with 480x800@160DPI.

@RaiMan
Owner
RaiMan commented Jul 19, 2016

you should update getting the fixes from yesterday including the implementation of type and the alternative input() in ADBScreen.
I do not have any problems with capturing smaller parts of a screen.

@jivank
jivank commented Aug 18, 2016

[error] java.lang.UnsatisfiedLinkError ( java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J )

I attempted to use this on Lubuntu and I get the error above. But the 2016-07-18 build on my macbook works fine.

@RaiMan
Owner
RaiMan commented Aug 18, 2016

@jivank
on linux systems you have to care for the prerequisites like OpenCV yourself.
see: http://sikulix.com/quickstart/#qs3

@jivank
jivank commented Aug 18, 2016

@RaiMan
On Lubuntu 16.04 I have run
apt install libopencv-dev tesseract-ocr

Everything shows up with the commands:
ldconfig -p
ldd -r

I ran the setup again with
options 1.1 buildv notes
and I still get this similar error when I click on click() via the IDE.

Exception in thread "Thread-10" java.lang.UnsatisfiedLinkError: org.opencv.core
.Mat.n_Mat(III)J                                                               
at org.opencv.core.Mat.n_Mat(Native Method)                                    
at org.opencv.core.Mat.<init>(Mat.java:477)                                    
at org.sikuli.android.ADBDevice.captureDeviceScreenMat(ADBDevice.java:204)     
at org.sikuli.android.ADBDevice.captureDeviceScreen(ADBDevice.java:151)        
at org.sikuli.android.ADBDevice.captureScreen(ADBDevice.java:138)              
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:148)                    
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:140)                    
at org.sikuli.util.OverlayCapturePrompt.prompt(OverlayCapturePrompt.java:213)  
at org.sikuli.android.ADBScreen$1.run(ADBScreen.java:203)
@RaiMan
Owner
RaiMan commented Aug 19, 2016

@jivank
Ok, I did not know, that you are using the experimental Android support.
This was never tested on Linux yet.

The problem here is, that one needs a library file called something like libopencv_java248.so, which bridges via JNI between the OpenCV Java API and the native OpenCV libraries.

Since you seem to know what you are doing:
Find out, wether you can get this library via apt or otherwise

Since I am currently on vacation, I have no chance to check this on my Ubuntu before beginning of September.

@jivank
jivank commented Aug 19, 2016

A quick check reveals that /usr/lib/jni/libopencv_java249.so exists. I will try 14.04 to see if it uses the older 248 binary.

@RaiMan
Owner
RaiMan commented Aug 19, 2016

You might also try to add a link libopencv_java248.so pointing to libopencv_java249.so

@jivank
jivank commented Aug 20, 2016

@RaiMan
That did not work. I also tried Lubuntu 14.04 which does have libopencv_java248.so as the default and I still get the same error.

@jchoover

On the Windows side I still had to explicitly load the dll in order for it
to work.

On Friday, August 19, 2016, Jivan Kulkarni notifications@github.com wrote:

@RaiMan https://github.com/RaiMan
That did not work. I also tried Lubuntu 14.04 which does have
libopencv_java248.so as the default and I still get the same error.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#210 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmlRcwS81rZLM8yENwCQoOZIHMi1Mtuks5qhlZQgaJpZM4I4Y5n
.

@RaiMan
Owner
RaiMan commented Aug 20, 2016

ok, I will check after beginning of September (still one week on vacation ;-)

@trigsoft

Hi. I am trying nightly build 20160719. My env is win10 64, genymotion emulator. menutoolAndroid can swipe, but when it tried capture the image and aTap it, the main IDE disappeared and never came back.

If I added the mouse action to the script, it tried to capture the screen but came back with below erros. Could you help out?
Exception in thread "Thread-12" java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J
at org.opencv.core.Mat.n_Mat(Native Method)
at org.opencv.core.Mat.(Mat.java:477)
at org.sikuli.android.ADBDevice.captureDeviceScreenMat(ADBDevice.java:204)
at org.sikuli.android.ADBDevice.captureDeviceScreen(ADBDevice.java:151)
at org.sikuli.android.ADBDevice.captureScreen(ADBDevice.java:138)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:148)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:140)
at org.sikuli.util.OverlayCapturePrompt.prompt(OverlayCapturePrompt.java:213)
at org.sikuli.android.ADBScreen$1.run(ADBScreen.java:203)

Exception in thread "Thread-14" java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J
at org.opencv.core.Mat.n_Mat(Native Method)
at org.opencv.core.Mat.(Mat.java:477)
at org.sikuli.android.ADBDevice.captureDeviceScreenMat(ADBDevice.java:204)
at org.sikuli.android.ADBDevice.captureDeviceScreen(ADBDevice.java:151)
at org.sikuli.android.ADBDevice.captureScreen(ADBDevice.java:138)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:148)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:140)
at org.sikuli.util.OverlayCapturePrompt.prompt(OverlayCapturePrompt.java:213)
at org.sikuli.android.ADBScreen$1.run(ADBScreen.java:203)

@jchoover

@trigsoft Try my suggestion:
in ADBScreen

Find the existing
RunTime.loadLibrary("VisionProxy");

and add the to libopencv lib's

static {
RunTime.loadLibrary("VisionProxy");
RunTime.loadLibrary("libopencv_core248");
RunTime.loadLibrary("libopencv_java248");
}

@trigsoft

Thanks @jchoover , I guess it is the problem as well. Downloading the project now.

@trigsoft

Hi. @jchoover your suggestion works. However, it still doesn't work for Android part. After menutoolAndroid tried capture the screen, the main IDE disappeared and never came back. What should I expect?

@jchoover

I'm by no means a java expert, in fact closer to a hack. Shouldn't have to do the menu, if you do it in script.

Have you ensured your emulator has USB debugging enabled?

You could try a script like this, to help debugging...

import org.sikuli.android.ADBScreen as AS
x= AS().start() 
x.needsUnLock = False # see comment
use(x) # set as default screen 


def ScreenShot():
    img = x.capture()
    img.save('c:\\Temp\\')


ScreenShot()

Also, if your IDE is hidden, you can use Shift+Alt+C to abort the script and return to the IDE.

@trigsoft

USB debugging is enabled as I can see swipe left and right works.

I just tried your code and got below error. Thanks anyway

[error] script [ Untitled ] stopped with error in line 12
[error] java.lang.UnsatisfiedLinkError ( java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J )
[error] --- Traceback --- error source first line: module ( function ) statement 8: main ( ScreenShot ) img = x.capture()
[error] --- Traceback --- end --------------

@jchoover

That's after applying my suggested hack/fix to the ADBScreen.java static constructor? You need to apply the fix, build, and deploy locally the private compile to be able to use it. If need be I can push all my hacks to a fork, though they are by no means the best options. I got it functional, before I hit the roadblock with ADB X'fer speeds and decided on an alternate path.

@trigsoft

Yes. I did apply your suggestion. Then I build artifacts for sikulixapi only and then copied to my previous folder.

@RaiMan
Owner
RaiMan commented Aug 23, 2016

@jchoover thanks for your evaluations and suggestions.
As mentioned: I will check and surely repair somewhen next week.

@trigsoft
trigsoft commented Sep 3, 2016

@RaiMan, have you got a chance to check the library linking problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment