Skip to content
This repository has been archived by the owner on Mar 14, 2018. It is now read-only.

Support for accessing devices not on the machine SikuliX works on (phones, vms, remote computers, ...) #210

Open
tg44 opened this issue Jun 17, 2016 · 82 comments

Comments

@tg44
Copy link

tg44 commented Jun 17, 2016

Hy there! I love this tool, but it has a basic problem. If you want to automate things that are not on your computer, but you have basic api to get the actual picture, and how to click on it, you still need to lose a PC just because it can't handle this type of functionality.

For example browsers or phones or vms or remote computers.

My specific case is Android gameing automation. So I have a tool which can get pictures from a phone screen from adb. This tool can handle click to (x,y), and swipe from-to actions too. (It could do pich and some other gesture too, but I want only the swipe (aka drag and drop) and touch (aka click)) at first run.

It would be pretty nice if sikuli and the IDE could use some api as screen input, and some other api to click and drag and drop and other functions.

I browsed the source, I think if I reimplement the Screen and Mouse classes that could be work. I have 2 problems with it: It's not the clean and generic way and I have no idea if it will work or not.

So basicly my questions:

  • Have you any advice about a mergable implementation, or its too complex, and an Android specific fork would be easyer?
  • If I reimplement the Screen and Mouse classes and modify the ide to be able to capture pictures from api screen instead of computer screen the run command can do its stuffs without touching the host computer?
  • Will be any headless/apiscreen support in the 2.0 version?

If I will have time and you say it's doable I really want to make a version that can handle Android devices thought adb and still enable to use the host computer,

@RaiMan
Copy link
Owner

RaiMan commented Jun 18, 2016

Highly appreciated.

A template for the principal approach can be taken from the VNC implementation in the contributed package module API::edu/unh/iol/dlc.

The relevant classes are

  • VNCScreen (screen as region, capture, ...)
  • VNCRobot (mouse and keyboard)

For both there are Interface classes (IScreen, IRobot) that define the methods to be implemented.
Recently I added some docs and conveniences, which might help too having a look at (http://sikulix-2014.readthedocs.io/en/latest/screen.html#connecting-to-a-vnc-server-vncscreen)

If you give me some more detailed information about what you already have, I can give more precise instructions what to do and how.

As a start, just make a fork of this repo to work with. So I can have a look and suggest or even create pull requests.

@RaiMan RaiMan changed the title "headless" support Support for accessing devices not on the machine SikuliX works on (phones, vms, remote computers, ...) Jun 18, 2016
@tg44
Copy link
Author

tg44 commented Jun 26, 2016

I just started this:
https://github.com/tg44/SikuliX-2014/tree/master/API/src/main/java/org/sikuli/android
The IDE has screen capture functions to vnc?

I will working on this slowly, I think I can produce a PR in this week with a limited but working functionality.

@RaiMan
Copy link
Owner

RaiMan commented Jun 28, 2016

Thanks. I will have a look the next days.

--- The IDE has screen capture functions to vnc?
no.
you have to write your own capture script or program.

@tg44
Copy link
Author

tg44 commented Jun 28, 2016

So I implemented a screen viewer function for ADB.
My only task left for a working solution is making a new button to the ide to run a macro on adb instead of screen.
Can you help me how can I tell the scriptruner to run my IRobot or IScreen instead of the build in? Where I need to modify things? (I searched where the IRobot interface is used, and didn't find in the IDE which can be a problem... )

@RaiMan
Copy link
Owner

RaiMan commented Jun 29, 2016

In fact I am a bit short on time currently due to other private priorities, so you have to wait for another few days.
I hope, this is acceptable. Thank you.

@jivank
Copy link

jivank commented Jul 7, 2016

@tg44 Nice work! Perhaps a radio button will do to select local screen and adb device. Also something to specify which adb device via the sikulix commandline. Looking forward to this great feature. Also does adb require root to view the screen?

@tg44
Copy link
Author

tg44 commented Jul 7, 2016

@jivank if you check out my work I added "adb screen" feature where you can see your device screen in a window and make subimages as patterns with the build in screen capture tool (yes, this is a little bit tricky).
In the future it would be nice if we can select adb devices, and not just get the default one. And it would be nice too if the adb screen would be rotateable and resizeable and the sample in the window would transformed when we use the ide. But, I just want a working solution first :D
Root not needed, only problem could be, that we can show only 2 frames/sec with this method. BUT you will need a properly installed adb (or at least the driver) and maybe a working adb console if you want to use my build (in theory you dont need to start a deamon by hand, but now this feature is untested :) ).

My problems now:
The ide crashes when I try to capture from the screen .
I have no idea how can I start the script from the ide on my implementation...

@RaiMan
Copy link
Owner

RaiMan commented Jul 8, 2016

@tg44
I would highly appreciate if you tell me the steps, to get an environment to check your work either on Mac or on Windows:

  • what packages to install
  • what IDE to use (currently I prefer InteliJ IDEA CE)
  • any configuration hints
  • I have a Nexus7, if this is of any use for that

BE AWARE: I do not have any Android dev knowledge.
Thank you

@tg44
Copy link
Author

tg44 commented Jul 8, 2016

http://forum.xda-developers.com/showthread.php?t=2588979 - this installs adb + drivers (to win)
http://stackoverflow.com/questions/31374085/installing-adb-on-mac-os-x - adb to mac
I'm using IntelliJ.
If adb is on your path adb devices will show your device (in a console). After that everything needs to be working.

Still I only could test the adb api so I have no idea if I could integrate it properly to sikulix or it has bugs.

@RaiMan
Copy link
Owner

RaiMan commented Jul 8, 2016

thanks, will try ;-)

so I have no idea if I could integrate it properly to sikulix or it has bugs.

I accept that is to some extent my job.
But I still need some days.

@jivank
Copy link

jivank commented Jul 8, 2016

@tg44
Take a look at VNCScreen:
https://github.com/RaiMan/SikuliX-2014/blob/5f64ab2602e2550357a2569713cbbc52b6b93e70/API/src/main/java/edu/unh/iol/dlc/VNCScreen.java

Here is how I was able to use it in the IDE:
#193 (comment)

So I suppose you can create an ADBScreen class using VNCScreen as an example. It won't work as the default screen, but at least it would be usable (and you can see if there is any bugs).

@RaiMan
Copy link
Owner

RaiMan commented Jul 9, 2016

@tg44, @jivank
I just started to look at the work of @tg44

  • I am currently on Mac 10.11
  • successfully installed the ADB package with Homebrew (my standard package installer)
  • made my Nexus 7 accessible (switch on developer mode)
  • added the org.sikuli.adroid package to this project
  • made some basic tests: works :-))

Then I started to completely revise the classes and added a class ADBDevice where all the device features are handled. The ADBClient is reduced to a singleton class, with only static features (handling the connection to the adb server and managing the list of attached devices).
ADBScreen initializes to represent a device and ADBRobot delegates to the device, that is representing the related screen.

The major revision goal is to take care, that a screen-device-relation can internally be handled as a singleton, to allow to make it thread safe later, in the sense, that on one device at any time only one action can be done (which is vital for the observe feature).

Currently saying:
ADBScreen adbScreen = new ADBScreen();
gives a Region/IScreen object, that is backed by an attached device at position 0 in the devicelist (if any).

When I finish my revision tomorrow, the basic Screen and Region features will work.

for the usage in the IDE as default screen, my idea is to add a menu entry "Android" in the Tools menu, that allows to make this switch:

  • list attached devices
  • select one as default screen
  • ...??

@jchoover
Copy link

@RaiMan any chance you can push a feature branch with your changes? I've played a bit with keyboard automation, using the input text/input keyevent, but I cant seem to get a proper delay for what I am trying to accomplish.

Also, with @tg44 fork, I cant get a script to work without first invoking the menu item to show the screen (otherwise the package isn't found), and then it takes 2 x script runs as the first one still isn't resolving the classes.

@RaiMan
Copy link
Owner

RaiMan commented Jul 12, 2016

I just pushed my revision of the package org.sikuli.android.
There is an ADBTest class, that can be run.

  • adb must be installed and useable
  • a device must be attached to USB
  • the device is waked up if needed and unlocked by a swipe up
  • a swipe left and a swipe right is done (only makes sense if the menu screen is visible)
  • you get a chance to capture something on the device's screen shown on your workstation's screen
  • the capture is then searched on the device's screen capture
  • if found, the target is tapped on the device

everything is very experimental and based on my Nexus 7 (Android 6.0.1) running on Mac OSX 10.11.

The capture is rather slow (average 5 seconds) using capture -p. Using capture (returns the raw pixel bytes) cuts down to 2 seconds and less, leaving the image conversion to the host (some 10 milliseconds): see 0169775. Nevertheless it seems to be needed, to insert many waits, to not sending requests faster, than the device changes the screen content or has finished with the last request.

I take this state as some "proof of concept" and stop the development for now, since I want to make the 1.1.1 final now.

Then I will add a development branch with version 1.1.2., where we could go further.
Then maybe I will come back to the idea of adding a feature branch for the Android stuff.

@tg44
Copy link
Author

tg44 commented Jul 12, 2016

LoL 5 sec is so slow :) My best performance with this screen capture method was 2 FPS with my Xiaomi Mi2S (Android 5.0).
The frame rate can't be higher with this method, I will consult with the android team for more ideas but this need nearly 0 config, we are using built in features with built in tools, and its totally undetectable from appcontext.

Thanks for your work, I will check it later today.

(BTW if you allow wifi debugging and connect your phone to the adb daemon with hand, you can use it without usb connection. )

@RaiMan
Copy link
Owner

RaiMan commented Jul 12, 2016

Yep, this solution is surely not for game automation (as it is used frequently on workstations).

... but it is rather acceptable for some basic workflows and even some testing, that is allowed to run for some minutes or hours ...
... and that it runs out of the box (only adb must be available) is a big advantage.

About WiFi: I know and will probably try it out the next days.

BTW: I added the internal use of dumpsys to get some state and config information (display on/off, screen size), but I am aware, that this might be version dependent to some extent.

I decided to add the device specific actions (tap, swipe, ...) to the ADBScreen and directly delegate to the ADBDevice, so the ADBRobot is only needed if you use the workstation specific actions like click.

In the IDE this works for now:

import org.sikuli.android.ADBScreen as AS
use(AS()) # set as default screen 
wakeUp(2)
swipeLeft()
swipeRight()

@jivank
Copy link

jivank commented Jul 14, 2016

This is great news! I suppose the last step would menu integration with the IDE, where you can select a device from ADB and a window will popup of the live screen (so you can create images from the android screen). I assume this would also be the way to integrate VNC as well.

@RaiMan
Copy link
Owner

RaiMan commented Jul 14, 2016

@jivank
Exactly ;-)
I added a menu entry in the tool menu,

  • where you can see, wether a device is accessible via adb
    (currently only the first one in the device list)
  • where you can run a test against the device including capture/find/tap
  • where you can switch the image capture to address the device screen

If you want to run a script against the attached device: see the comment above.
The next days I will write something about usage, restrictions and next steps somewhere.

Be aware:
The current device capture, using the png-capture command (capture -p), is rather slow (see above). This then is the main duration influence for the find ops. I started to experiment with the so called raw capture, that promises more speed (up to 10 times faster capture), since it allows, to move the post processing of the raw pixel bytes to the host machine (where we talk about milliseconds and not seconds).

All this currently happens on Mac OSX 10.11 with my Nexus 7 (Android 6). I have no idea (and no chance to get any) how all this looks and feels on newer devices.

The stuff is available in the next nightly build 2016-07-15 (package org.sikuli.android).

Any feedback and ideas are welcome.

@RaiMan
Copy link
Owner

RaiMan commented Jul 14, 2016

@jivank

... and yes, this will be implemented for the VNCScreen too later.

@RaiMan
Copy link
Owner

RaiMan commented Jul 15, 2016

Capture now uses raw capture on the device, which makes it a lot faster:

see commit: 0169775

@RaiMan
Copy link
Owner

RaiMan commented Jul 16, 2016

Just checked a device connection over WiFi:

At least in my test environment the capture is unacceptable slow (30 seconds).

Until someone tells me a solution for that, I will not test any further with WiFi connections.

@tg44
Copy link
Author

tg44 commented Jul 16, 2016

Raw capture is a good idea. I think the framerate is highly depends on the device screen resolution, lower res -> higher frame rate...

I had so much issues because of a lower version adb-server, I solved it but take me lot of time to find out that this is the actual problem. The sysdump is really ugly, my device is printing things to a console more than 1minute long.

Now I have issues with the ADBScreen.userCapture, I have no idea why it's not working, and currently I have no time to dig more deeper to the code. (Your exception handling and logging is killing me, you are caches exceptions and logging them to "I have no idea where". And there is no error on the console, but the app is hanging and I know that there is an exception somewhere, but I cant find it :D )
How this is working (when its working)? It's give you a new window? Or how you can capture imges from the android screen?

@jchoover
Copy link

When I tried Gergo's fork it works with quirks. When I tried last night
with the latest from RaiMan, I had to modify the regex for it to detect the
screen being awake on Nox, but then it would fail to find the opencv native
libs and give an error when trying to create the Mat class.

On Saturday, July 16, 2016, Gergő Törcsvári notifications@github.com
wrote:

Raw capture is a good idea. I think the framerate is highly depends on the
device screen resolution, lower res -> higher frame rate...

I had so much issues because of a lower version adb-server, I solved it
but take me lot of time to find out that this is the actual problem. The
sysdump is really ugly, my device is printing things to a console more than
1minute long.

Now I have issues with the ADBScreen.userCapture, I have no idea why it's
not working, and currently I have no time to dig more deeper to the code.
(Your exception handling and logging is killing me, you are caches
exceptions and logging them to "I have no idea where". And there is no
error on the console, but the app is hanging and I know that there is an
exception somewhere, but I cant find it :D )
How this is working (when its working)? It's give you a new window? Or how
you can capture imges from the android screen?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#210 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmlRXJu0q-1eaKneoE9-Ysqvje0ayt_ks5qWVBVgaJpZM4I4Y5n
.

@RaiMan
Copy link
Owner

RaiMan commented Jul 16, 2016

Finalized the Android support for now in this experimental state as some proof of concept:

  • IDE tool menu to run a short test on an attached device and to switch to an Android device for image capture and back to the default local screen again
  • running scripts towards an attached device with the possibility, to restrict the search region to smaller parts of the device's screen
  • on my Nexus 7 (1200 x 1920) a search on the whole screen runs less than 2 seconds, on only portions of a screen 1 second and below down to 200 milliseconds
  • click is converted to a tap and the ADBScreen already knows tap and swipe (no timing yet)
  • adb must be available on the host and must be executable from command line using adb

This is a sample script:

from org.sikuli.android import ADBScreen

Debug.on(3)

ascr = ADBScreen.start() # get the one attached device
if not ascr:
  exit(1)

use(ascr) # set as the default region  
ascr.needsUnLock = False # see comment
wakeUp(2) # take care that the display is on, wait max 2 seconds

gsearch = "gsearch.png" # to detect home screen
img = "img.png" # an icon to tap

top = newRegion(Location(0, 0), 300, 300) # top left corner
iconRow = newRegion(Location(0, 600), 1200, 300) # an icon band

for i in range(3): # just do it 3 times 
  key(ADBScreen.HOME) # tap the home button
  top.wait(gsearch, 10) # wait for home screen
  iconRow.click(img) # click icon
  wait(5) # to give time to the triggered action to process

The images are captured from the device using the capture button after having switched the default screen to the device in the tool menu.

ascr.needsUnLock = False: if set to true (the default), after waking up the display a swipe up is done to let a lock screen vanish.

@RaiMan
Copy link
Owner

RaiMan commented Jul 16, 2016

@tg44 and @jchoover

I understand, that you might have problems:
be aware: it is developed on Mac and not yet tested on Windows nor on Linux.

The next nightly should make less problems.

If you give me more information, what you are doing in what environment, I might get things fixed faster.

@jchoover
Copy link

I'll get more details in a couple hours, but my env is win7 x64, with x64
JVM v7 something. Using IntelliJ community edition for edits, but compiling
per the doc's with mvn.

On Saturday, July 16, 2016, Raimund Hocke notifications@github.com wrote:

@tg44 https://github.com/tg44 and @jchoover
https://github.com/jchoover

I understand, that you might have problems:
be aware: it is developed on Mac and not yet tested on Windows nor on
Linux.

The next nightly should make less problems.

If you give me more information, what you are doing in what environment, I
might get things fixed faster.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#210 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmlRUPKtczZZMNgKap7DsMk8ckQcaD6ks5qWVnXgaJpZM4I4Y5n
.

@jchoover
Copy link

Exception in thread "Thread-12" java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J
at org.opencv.core.Mat.n_Mat(Native Method)
at org.opencv.core.Mat.(Mat.java:477)
at org.sikuli.android.ADBDevice.captureDeviceScreenMat(ADBDevice.java:168)
at org.sikuli.android.ADBDevice.captureDeviceScreen(ADBDevice.java:115)
at org.sikuli.android.ADBDevice.captureScreen(ADBDevice.java:102)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:216)
at org.sikuli.android.ADBScreen.capture(ADBScreen.java:208)
at org.sikuli.util.OverlayCapturePrompt.prompt(OverlayCapturePrompt.java:213)
at org.sikuli.android.ADBScreen$1.run(ADBScreen.java:271)

Exception in thread "Thread-11" java.lang.NullPointerException
at org.sikuli.ide.SikuliIDE.androidSupportTest(SikuliIDE.java:1810)
at org.sikuli.ide.SikuliIDE.access$2500(SikuliIDE.java:107)
at org.sikuli.ide.SikuliIDE$1.run(SikuliIDE.java:1780)

Is the error I am getting when running the tool menu from your latest build. Note, I still have to tweak the ADBDevice.isDisplayOn method:

String dump = dumpsys("power");
    Pattern displayOn = Pattern.compile("Display Power: state=ON|mScreenOn=true");
    Matcher match = displayOn.matcher(dump);
    if (match.find()) {
        return true;
    } else {

And I am using JDK/JRE 1.8.0_91

@jchoover
Copy link

FYI, If I add the opencv core and java to the libs the ADBDevice static constructor loads, then this works!

public class ADBScreen extends Region implements EventObserver, IScreen {

  static {
    RunTime.loadLibrary("VisionProxy");
    RunTime.loadLibrary("libopencv_core248");
    RunTime.loadLibrary("libopencv_java248");
  }

@trigsoft
Copy link

Yes. I did apply your suggestion. Then I build artifacts for sikulixapi only and then copied to my previous folder.

@RaiMan
Copy link
Owner

RaiMan commented Aug 23, 2016

@jchoover thanks for your evaluations and suggestions.
As mentioned: I will check and surely repair somewhen next week.

@trigsoft
Copy link

trigsoft commented Sep 3, 2016

@RaiMan, have you got a chance to check the library linking problem?

@HongminHan
Copy link

I am using 1.1.1-SNAPSHOT.

Debug.on(3);
ADBScreen screen = ADBScreen.start();
screen.wakeUp(2);
ImagePath.add("images");
Region r = screen.wait("game-icon.png");
LOGGER.debug("r:{},{}",r.getX(),r.getY());
screen.aTap(r);//does not work
screen.aSwipeLeft();//worked fine

Why tap action is not worked? I just do the cmd with adb, it worked!
adb shell input tap 154 91

And is there some documents for this feature?

@RaiMan
Copy link
Owner

RaiMan commented Feb 6, 2017

@kainecy
sorry, just an experiment - not complete at all and not documented.
In doubt you have to look into the code.

@HongminHan
Copy link

I think it will be worked fine. How can i get this worked?
I am not familiar with jadb, when i debug ,i find the cmd is shell:xxxxx, there is a colon in it. It is right?

@HongminHan
Copy link

Region temp = screen.wait("battle-do.png",3.0);
screen.aSwipe(temp,temp);

I used this code to do a "tap/click" , it worked just ok, but i know it is wrong way!

@Umriyaev
Copy link

Umriyaev commented Feb 8, 2017

@RaiMan I found why aTap is not working.
The problem is in the org.sikuli.android.ADBDevice.tap:

public void tap(int x, int y) {
    try {
      device.executeShell("input aTap", Integer.toString(x), Integer.toString(y));
    } catch (IOException | JadbException e) {
      log(-1, "aTap: %s", e);
    }
  }

adb command for tapping on the screen is input tap, but you used input aTap

I tested with

public void tap(int x, int y) {
    try {
      device.executeShell("input tap", Integer.toString(x), Integer.toString(y));
    } catch (IOException | JadbException e) {
      log(-1, "aTap: %s", e);
    }
  }

and it worked well

@Umriyaev
Copy link

Umriyaev commented Feb 8, 2017

@RaiMan One problem found with org.sikuli.android.ADBDevice.input:

public void input(String text) {
    try {
      device.executeShell("input text", text);
      RunTime.pause(text.length() * inputDelay);
    } catch (Exception e) {
      log(-1, "input: %s", e);
    }
  }

Problem: There is no space between input text and the actual text to be typed.
On my local PC I changed it to the following:

public void input(String text) {
    try {
        text = text.replaceAll("\\s+", "%s");
      device.executeShell("input text ", text);
      RunTime.pause(text.length() * inputDelay);
    } catch (Exception e) {
      log(-1, "input: %s", e);
    }
  }

Also I'm replacing all the spaces with %s, otherwise adb server will raise an error.

To test them I changed org.sikuli.android.ADBTest.basicTest to the following:

private static void basicTest(ADBScreen adbs) throws FindFailed {
    log(lvl, "**************** running basic test");
    adbs.aSwipeLeft();
    adbs.aSwipeRight();
    adbs.wait(1f);
    ScreenImage sIMg = adbs.userCapture("Android");
    sIMg.save(RunTime.get().fSikulixStore.getAbsolutePath(), "android");
    Image img = new Image(sIMg);
    adbs.aTap(img);
    adbs.wait(1f);
    sIMg = adbs.userCapture("Android");
    sIMg.save(RunTime.get().fSikulixStore.getAbsolutePath(), "android");
    img = new Image(sIMg);
    adbs.aTap(img);
    adbs.wait(1f);
    adbs.aInput("Hello world");
  }

Please try them on your local PC

@RaiMan
Copy link
Owner

RaiMan commented Feb 8, 2017

@Umriyaev
great thanks.
I will add the fix.
https://bugs.launchpad.net/sikuli/+bug/1662784

@Umriyaev
Copy link

Umriyaev commented Feb 8, 2017

@RaiMan
You are welcome
I've just tested in Sikuli IDE too, both aInput and aTap are working now

Btw, you misspelled my github id at https://bugs.launchpad.net/sikuli/+bug/1662784
My id is umriyaev =))

@RaiMan
Copy link
Owner

RaiMan commented Feb 8, 2017

@Umriyaev
... misspelled: copy & paste is still not intelligent enough ;-)
Again thanks: corrected and added the tap fix.

@Umriyaev
Copy link

Umriyaev commented Feb 8, 2017

@RaiMan
Great
Glad to be helpful for such an amazing project =))

@HongminHan
Copy link

@RaiMan

When and how can i get a fixed version in maven?

@RaiMan
Copy link
Owner

RaiMan commented Feb 8, 2017

@kainecy
Hope I get it managed today. So watch the nightly page wether a new build is available.
Having setup your Maven correctly, the new snapshot should be pulled automatically on the next clean install.

@Umriyaev
Copy link

Umriyaev commented Feb 8, 2017

@RaiMan
I forgot to mention
There was one more problem at org.sikuli.android.ADBDevice.getDisplayDimension. Some devices (including Samsung galaxy s7 edge) return mDefaultViewport as an array (in my case 0th element of the array). Therefore regex pattern needs to be changed. My modified code for that method is as following:

 private Dimension getDisplayDimension() {
    String dump = dumpsys("display");
    String token = "mDefaultViewport= ... deviceWidth=1200, deviceHeight=1920}";
    Dimension dim = null;
    Pattern displayDimension = Pattern.compile("mDefaultViewport.*?=.*?deviceWidth=(\\d*).*?deviceHeight=(\\d*)");
    Matcher match = displayDimension.matcher(dump);
    if (match.find()) {
      int w = Integer.parseInt(match.group(1));
      int h = Integer.parseInt(match.group(2));
      dim = new Dimension(w, h);
    } else {
      log(-1, "getDisplayDimension: dumpsys display: token not found: %s", token);
    }
    return dim;
  }

Tested with Samsung galaxy s7 edge which returns mDefaultViewport[0] and LG V10 which returns mDefaultViewport

@RaiMan
Copy link
Owner

RaiMan commented Feb 8, 2017

very productive ;-) thanks again.

@HongminHan
Copy link

Is there a plan to support multiple device in java api?
May be:

ADBScreen screen = ADBScreen.start("device-name");

@RaiMan
Copy link
Owner

RaiMan commented Feb 9, 2017

The ADB feature is experimental and there are currently no plans for more features.

Feel free to overwrite and extend the respective classes meanwhile.

RaiMan added a commit that referenced this issue Feb 24, 2017
@pepijnve
Copy link
Contributor

pepijnve commented Mar 8, 2017

@RaiMan I had a need for authentication support in the VNC client, so I started adding that in https://github.com/pepijnve/SikuliX-2014/tree/vncauth
After running into more and more limitations of the VNC client implementation, I ended up replacing the entire VNCScreen implementation with one based on the TigerVNC Java client. Would you be interested in these changes? If so I'll make a PR for this.
Since TigerVNC is GPL licensed and not readily available on maven central, I've been thinking about replacing it with https://github.com/comtel2000/jfxvnc which is ASL2 licensed. If the license and package availability is an issue I'm happy to do that before making the PR.

@RaiMan
Copy link
Owner

RaiMan commented Mar 9, 2017

@pepijnve
Great thanks for coming up with that.
Of course is any PR welcome, that enhances SikuliX features or even adds new ones.
Since the current implementation in 1.1.x does not and will not have any regression tests, I usually take the risk of merging even larger changes after some basic evaluation (just eyes on ;-).

GPL-2.0 license as far as I understand, fits to the MIT license of SikuliX.

Since I guess that only the viewer jar (500+ MB) is needed, it would be the easiest to just add it to the SikuliX sources, like I already did with other stuff.

So I just suggest, you first make a PR for your code changes and then lets decide further, after I have understood your solution.

@pepijnve
Copy link
Contributor

pepijnve commented Mar 9, 2017

Created #247 for the VNC changes.

@mitec
Copy link

mitec commented Dec 6, 2017

Hi,
what kind of vnc server on android device are you using?
I guess, sikulix over adb is still slow compared to vnc, is it?

@RaiMan
Copy link
Owner

RaiMan commented Dec 6, 2017

@mitec
I do not use any vnc server on Android nor do I know anything about it. It is up to you, to test the SikuliX vnc client against whatever vnc server wherever.
The device access via adb is experimental.

@mitec
Copy link

mitec commented Dec 6, 2017

How do you work with the vncscreen? Are you using ios?

@RaiMan
Copy link
Owner

RaiMan commented Dec 7, 2017

@mitec
Copy link

mitec commented Dec 7, 2017

I've read the docs, but I thought you've been somehow testing the vncscreen. When I think about it more, there is also the posibility to test the vncscreen locally (vnc server is on the same machine as the viewer-client), not on android or ios.

@RaiMan
Copy link
Owner

RaiMan commented Dec 7, 2017

@mitec
as mentioned: SikuliX only offers a vnc-client based on the TigerVNC sources.

Setup the server and connecting correctly is completely up to the user of SikuliX.

My tests of the client run against a free version of RealVNC's VNC Server on macOS 10.13+ and Windows 10 64-Bit.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants