-
Notifications
You must be signed in to change notification settings - Fork 235
Support for accessing devices not on the machine SikuliX works on (phones, vms, remote computers, ...) #210
Comments
Highly appreciated. A template for the principal approach can be taken from the VNC implementation in the contributed package module API::edu/unh/iol/dlc. The relevant classes are
For both there are Interface classes (IScreen, IRobot) that define the methods to be implemented. If you give me some more detailed information about what you already have, I can give more precise instructions what to do and how. As a start, just make a fork of this repo to work with. So I can have a look and suggest or even create pull requests. |
I just started this: I will working on this slowly, I think I can produce a PR in this week with a limited but working functionality. |
Thanks. I will have a look the next days. --- The IDE has screen capture functions to vnc? |
So I implemented a screen viewer function for ADB. |
In fact I am a bit short on time currently due to other private priorities, so you have to wait for another few days. |
@tg44 Nice work! Perhaps a radio button will do to select local screen and adb device. Also something to specify which adb device via the sikulix commandline. Looking forward to this great feature. Also does adb require root to view the screen? |
@jivank if you check out my work I added "adb screen" feature where you can see your device screen in a window and make subimages as patterns with the build in screen capture tool (yes, this is a little bit tricky). My problems now: |
@tg44
BE AWARE: I do not have any Android dev knowledge. |
http://forum.xda-developers.com/showthread.php?t=2588979 - this installs adb + drivers (to win) Still I only could test the adb api so I have no idea if I could integrate it properly to sikulix or it has bugs. |
thanks, will try ;-)
I accept that is to some extent my job. |
@tg44 Here is how I was able to use it in the IDE: So I suppose you can create an ADBScreen class using VNCScreen as an example. It won't work as the default screen, but at least it would be usable (and you can see if there is any bugs). |
@tg44, @jivank
Then I started to completely revise the classes and added a class ADBDevice where all the device features are handled. The ADBClient is reduced to a singleton class, with only static features (handling the connection to the adb server and managing the list of attached devices). The major revision goal is to take care, that a screen-device-relation can internally be handled as a singleton, to allow to make it thread safe later, in the sense, that on one device at any time only one action can be done (which is vital for the observe feature). Currently saying: When I finish my revision tomorrow, the basic Screen and Region features will work. for the usage in the IDE as default screen, my idea is to add a menu entry "Android" in the Tools menu, that allows to make this switch:
|
@RaiMan any chance you can push a feature branch with your changes? I've played a bit with keyboard automation, using the input text/input keyevent, but I cant seem to get a proper delay for what I am trying to accomplish. Also, with @tg44 fork, I cant get a script to work without first invoking the menu item to show the screen (otherwise the package isn't found), and then it takes 2 x script runs as the first one still isn't resolving the classes. |
I just pushed my revision of the package org.sikuli.android.
everything is very experimental and based on my Nexus 7 (Android 6.0.1) running on Mac OSX 10.11. The capture is rather slow (average 5 seconds) using I take this state as some "proof of concept" and stop the development for now, since I want to make the 1.1.1 final now. Then I will add a development branch with version 1.1.2., where we could go further. |
LoL 5 sec is so slow :) My best performance with this screen capture method was 2 FPS with my Xiaomi Mi2S (Android 5.0). Thanks for your work, I will check it later today. (BTW if you allow wifi debugging and connect your phone to the adb daemon with hand, you can use it without usb connection. ) |
Yep, this solution is surely not for game automation (as it is used frequently on workstations). ... but it is rather acceptable for some basic workflows and even some testing, that is allowed to run for some minutes or hours ... About WiFi: I know and will probably try it out the next days. BTW: I added the internal use of dumpsys to get some state and config information (display on/off, screen size), but I am aware, that this might be version dependent to some extent. I decided to add the device specific actions (tap, swipe, ...) to the ADBScreen and directly delegate to the ADBDevice, so the ADBRobot is only needed if you use the workstation specific actions like click. In the IDE this works for now:
|
This is great news! I suppose the last step would menu integration with the IDE, where you can select a device from ADB and a window will popup of the live screen (so you can create images from the android screen). I assume this would also be the way to integrate VNC as well. |
@jivank
If you want to run a script against the attached device: see the comment above. Be aware: All this currently happens on Mac OSX 10.11 with my Nexus 7 (Android 6). I have no idea (and no chance to get any) how all this looks and feels on newer devices. The stuff is available in the next nightly build 2016-07-15 (package org.sikuli.android). Any feedback and ideas are welcome. |
@jivank |
Capture now uses raw capture on the device, which makes it a lot faster: see commit: 0169775 |
Just checked a device connection over WiFi: At least in my test environment the capture is unacceptable slow (30 seconds). Until someone tells me a solution for that, I will not test any further with WiFi connections. |
Raw capture is a good idea. I think the framerate is highly depends on the device screen resolution, lower res -> higher frame rate... I had so much issues because of a lower version adb-server, I solved it but take me lot of time to find out that this is the actual problem. The sysdump is really ugly, my device is printing things to a console more than 1minute long. Now I have issues with the ADBScreen.userCapture, I have no idea why it's not working, and currently I have no time to dig more deeper to the code. (Your exception handling and logging is killing me, you are caches exceptions and logging them to "I have no idea where". And there is no error on the console, but the app is hanging and I know that there is an exception somewhere, but I cant find it :D ) |
When I tried Gergo's fork it works with quirks. When I tried last night On Saturday, July 16, 2016, Gergő Törcsvári notifications@github.com
|
Finalized the Android support for now in this experimental state as some proof of concept:
This is a sample script:
The images are captured from the device using the capture button after having switched the default screen to the device in the tool menu.
|
I'll get more details in a couple hours, but my env is win7 x64, with x64 On Saturday, July 16, 2016, Raimund Hocke notifications@github.com wrote:
|
Exception in thread "Thread-12" java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J Exception in thread "Thread-11" java.lang.NullPointerException Is the error I am getting when running the tool menu from your latest build. Note, I still have to tweak the ADBDevice.isDisplayOn method:
And I am using JDK/JRE 1.8.0_91 |
FYI, If I add the opencv core and java to the libs the ADBDevice static constructor loads, then this works!
|
Yes. I did apply your suggestion. Then I build artifacts for sikulixapi only and then copied to my previous folder. |
@jchoover thanks for your evaluations and suggestions. |
@RaiMan, have you got a chance to check the library linking problem? |
I am using 1.1.1-SNAPSHOT.
Why tap action is not worked? I just do the cmd with adb, it worked! And is there some documents for this feature? |
@kainecy |
I think it will be worked fine. How can i get this worked? |
I used this code to do a "tap/click" , it worked just ok, but i know it is wrong way! |
@RaiMan I found why aTap is not working.
I tested with
and it worked well |
@RaiMan One problem found with
Problem: There is no space between
Also I'm replacing all the spaces with To test them I changed
Please try them on your local PC |
@Umriyaev |
@RaiMan Btw, you misspelled my github id at https://bugs.launchpad.net/sikuli/+bug/1662784 |
@Umriyaev |
@RaiMan |
When and how can i get a fixed version in maven? |
@kainecy |
@RaiMan
Tested with |
very productive ;-) thanks again. |
Is there a plan to support multiple device in java api?
|
The ADB feature is experimental and there are currently no plans for more features. Feel free to overwrite and extend the respective classes meanwhile. |
…gestions from #210 (comment) and following
@RaiMan I had a need for authentication support in the VNC client, so I started adding that in https://github.com/pepijnve/SikuliX-2014/tree/vncauth |
@pepijnve GPL-2.0 license as far as I understand, fits to the MIT license of SikuliX. Since I guess that only the viewer jar (500+ MB) is needed, it would be the easiest to just add it to the SikuliX sources, like I already did with other stuff. So I just suggest, you first make a PR for your code changes and then lets decide further, after I have understood your solution. |
Created #247 for the VNC changes. |
Hi, |
@mitec |
How do you work with the vncscreen? Are you using ios? |
@mitec see: http://sikulix-2014.readthedocs.io/en/latest/screen.html#connecting-to-a-vnc-server-vncscreen |
I've read the docs, but I thought you've been somehow testing the vncscreen. When I think about it more, there is also the posibility to test the vncscreen locally (vnc server is on the same machine as the viewer-client), not on android or ios. |
@mitec Setup the server and connecting correctly is completely up to the user of SikuliX. My tests of the client run against a free version of RealVNC's VNC Server on macOS 10.13+ and Windows 10 64-Bit. |
Hy there! I love this tool, but it has a basic problem. If you want to automate things that are not on your computer, but you have basic api to get the actual picture, and how to click on it, you still need to lose a PC just because it can't handle this type of functionality.
For example browsers or phones or vms or remote computers.
My specific case is Android gameing automation. So I have a tool which can get pictures from a phone screen from adb. This tool can handle click to (x,y), and swipe from-to actions too. (It could do pich and some other gesture too, but I want only the swipe (aka drag and drop) and touch (aka click)) at first run.
It would be pretty nice if sikuli and the IDE could use some api as screen input, and some other api to click and drag and drop and other functions.
I browsed the source, I think if I reimplement the Screen and Mouse classes that could be work. I have 2 problems with it: It's not the clean and generic way and I have no idea if it will work or not.
So basicly my questions:
If I will have time and you say it's doable I really want to make a version that can handle Android devices thought adb and still enable to use the host computer,
The text was updated successfully, but these errors were encountered: