Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iOS support #64

Open
kwv opened this Issue Aug 26, 2015 · 99 comments

Comments

Projects
None yet
@kwv
Copy link
Contributor

kwv commented Aug 26, 2015

Awesome, inspiring stuff.

TLDR; is iOS support on your roadmap?

I've been tinkering around with a solution that works on iOS devices (or simulators). The approach streams snapshots over websockets to a canvas and forwards user-generated mouse/keyboard events to Appium which replays them against a device. For a stand alone solution, with a single device, it works.

next up on the roadmap:

  • improve performance of image capture
  • support multicasting of a device
  • support streaming multiple concurrent devices (i.e. one per browser tab)
  • polish the code (env variables and such)
  • approval w/ stakeholders to release as open source

I understand its hard to comment on a solution you haven't seen. Before I get too much further down this proof-of-concept, I wanted to solicit feedback Are there any plans to support iOS in STF? Have you already been down this path and abandoned it?

proof of concept approach

image capture

To capture images, I've evaluated:

  • using libimobiledevice which uses devloper disk image to capture on the device to achieve 5-10 fps. Unfortunatly these are large files (3MB TIFFs), that are begging to be pre-processed into a PNG.
  • using QuickTime's screen mirroring; and screenshoting that (on the host) which allows ~25 fps, but won't scale beyond the first device.
  • forking xrecord to implement a QT mirroing solution that output jpegs (and would allow multiple devices)

devices

Testing against a physical device is desirable however without jailbreaking the device there are some limitations (lockscreen, home button).

The iOS simulator was able to produce highest framerates, and the closest experience to what STF provides for Android, but there is a nagging feeling it's not going to scale or meet the requirements (e.g. managed in MDM, recgnoize an internal cert chain)

Shortcomings

  • Appium is limited to the underlying device automation implementation. On iOS it is UIATarget
  • UIATarget has an api to send the application to the background for a duration. Appium wants to launch an application on startup. On a physical device this means there's no interacting with Home Screen, system properties without jailbreaking the device and getting creative. For simulators, there may be a way to mimic, but would need to figure out how to support concurrent simulators.
  • perceptable lag (~1-2 seconds) on keyboard events propagating back to the device. A short (100ms) buffer for keyboard events may help, but special keys like Delete make it interesting
  • Duplication of inventory in Selenium Grid and STF. We've got a node module that watches USB for new device connections, starts an Appium server and registeres it with a Selenium grid, but we need to federate that back into STF for discovery
@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Aug 27, 2015

Hi,

Thanks for the awesome work. We would like to have iOS support, yes. However we currently do not have the resources to develop it by ourselves, but if someone provides a reasonably functional version I may be able to port the rest of the functionality. I would be very interested in seeing what you've got so far.

Btw you mentioned you were hoping to perhaps compress TIFF frames into PNG. I think PNG is a lost cause for screen capture purposes as the compression is very very slow compared to JPG. JPG (w/ libjpeg-turbo) is the fastest method we've found so far, but over time we're planning on moving away from JPG frames to transferring raw x264 frames instead. As long as the device supports hardware encoding, this method should yield the highest FPS.

@kwv

This comment has been minimized.

Copy link
Contributor Author

kwv commented Sep 3, 2015

I'm still working on this. The code's in much better shape, and the stakeholders are listening.

Thanks for the tip on TIFF to JPG. It shaved 100ms off the image processing.

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Sep 3, 2015

Great, really looking forward to this :)

@gunta

This comment has been minimized.

Copy link
Member

gunta commented Sep 3, 2015

Amazing.

For capturing images, I get that you are using com.apple.mobile.screenshotr which firsts captures a TIFF frame and then you have to convert it to JPEG; which will not get you the max frame-rate.

Since iOS 8 there is com.apple.cmio.iOSScreenCaptureAssistant which gives you an encoded video stream. If you are able to get the stream (you may have to do some reverse-engineering) and then somehow forward that to the browser, you'll probably get the highest FPS.

On the Browser side then you could try to do any necessary processing and show it by using Media Source Extensions or WebRTC.

If you need any help in getting the approval of stakeholders, just let them know that we can add the name of your company in the credits as a big contributor on http://openstf.io, if that is the case just let us know ;)

@gunta

This comment has been minimized.

Copy link
Member

gunta commented Oct 9, 2015

@kwv any update on this? :)

@kwv

This comment has been minimized.

Copy link
Contributor Author

kwv commented Oct 9, 2015

The effort stalled. Stakeholders haven't said no, but they've shifted priorities.

I'm able to sample stills from a video stream, but it's drinking from a firehose. I was refactoring the image acquisition code to send the results back over a WebSocket (rather than diskIO), when I bumped into a hard limit of one device at a time WPO-Foundation/xrecord#1

@leandromoreira

This comment has been minimized.

Copy link

leandromoreira commented Feb 2, 2016

Thanks for such amazing work ;)

@PattabhiramaPandit

This comment has been minimized.

Copy link

PattabhiramaPandit commented Mar 19, 2016

Hi ,

Can you please share your solution?
We want to test it out for iOS devices?

@PattabhiramaPandit

This comment has been minimized.

Copy link

PattabhiramaPandit commented Mar 19, 2016

If anyone has a SD solution available please share the same. We are in dire need of getting this on iOS.

@PattabhiramaPandit

This comment has been minimized.

Copy link

PattabhiramaPandit commented Mar 19, 2016

Ooppsss!... I meant iOS.. Not SD

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Mar 20, 2016

If you feel like you need an iOS implementation faster, consider posting a bounty on this issue (see badge below). If the bounty gets large enough, perhaps we or someone else will take a look at it.

Bountysource

@PattabhiramaPandit

This comment has been minimized.

Copy link

PattabhiramaPandit commented Mar 24, 2016

Dear Sorccu, you mentioned to me that you would give me the exact quote for developing the iOS Support. Can you please give me as soon as possible?

Regards
Pattabhi

@Ashraf-Ali-aa

This comment has been minimized.

Copy link

Ashraf-Ali-aa commented May 13, 2016

@sorccu @kwv it might be worth checking this out https://github.com/phonegap/ios-deploy and https://github.com/phonegap/ios-sim for building and deploying iOS apps to device and simulators. I know STF does not currently support iOS but it would be nice to deploy apps via the web browser. The tool also supports lldb which is useful for out putting console logs.

@daluu

This comment has been minimized.

Copy link

daluu commented Jun 6, 2016

Curious to the original proposed solution that got stalled, as well as any other potential solutions. How far can you scale device support (or even simulator support)?

From what I encountered, unless I'm mistaken, just toying with Appium and physical iOS devices, there seemed to be a single device limitation per instance of XCode on Mac OS. Therefore, to reliably scale to more iOS devices (to work in parallel), one has to deploy 2 virtual machines (the max licensed amount allowed by Apple, on the physical Mac host) running Mac OS with the same XCode and Appium setup, etc. In that setup, assign a specific iOS device per VM via USB port/device allocation in the VM software, and the remaining (last) device to the physical Mac. Each instance (physical Mac, VMs) have their own XCode and Appium to control the device. So there's a max device limit of 3 iOS devices per Mac machine (using the VMs). This doesn't scale as well compared to Android, as more Mac machines would be needed to scale higher.

@vbanthia

This comment has been minimized.

Copy link
Member

vbanthia commented Jun 7, 2016

@daluu

XCode supports testing multiple real devices using UIAutomator since v6.3.2

appium/appium#273 (comment)

@Ashraf-Ali-aa

This comment has been minimized.

Copy link

Ashraf-Ali-aa commented Jun 14, 2016

this might be useful for video recording https://github.com/isonic1/flick

@codeskyblue

This comment has been minimized.

Copy link
Member

codeskyblue commented Jun 14, 2016

@Ashraf-Ali-aa This tools about iOS recording is using capture pictures and combine them to video file. maybe not fitted for stf.

@gunta

This comment has been minimized.

Copy link
Member

gunta commented Jun 14, 2016

@codeskyblue it works also on Android, should be able to be used without problems

@codeskyblue

This comment has been minimized.

Copy link
Member

codeskyblue commented Jun 14, 2016

Since stf got minicap, combine the image stream into video is a better way in my view.

@gunta

This comment has been minimized.

Copy link
Member

gunta commented Jun 14, 2016

Well its for different use cases afaik, at least currently.

@codeskyblue

This comment has been minimized.

Copy link
Member

codeskyblue commented Jun 14, 2016

@gunta I have a question, how to know ios orientation from command line.

@Ashraf-Ali-aa

This comment has been minimized.

Copy link

Ashraf-Ali-aa commented Jun 15, 2016

facebook have implemented an extension of xcuitest that allows remote interactions with ios devices and appium has adopted the code.

https://github.com/facebook/WebDriverAgent
https://github.com/appium/appium-xcuitest-driver

related links:
https://github.com/facebook/FBSimulatorControl
https://github.com/rsms/peertalk

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Jun 21, 2016

I think that's probably the reason why no one has contributed a solution
yet. It takes a huge amount of developer time and there are clearly
financial benefits for end users. In other words it would be a viable
product. Not many are willing to give that away for free.

On Tuesday, 21 June 2016, Naresh Jain notifications@github.com wrote:

any update on iOS support?
If anyone has a iOS solution available for OpenSTF please share, that will
help my company to reduce cost on devices.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#64 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAB-_bzGMKCJOp7_YKDQZARghu4O20xWks5qN8WwgaJpZM4FyyUk
.

@kwv

This comment has been minimized.

Copy link
Contributor Author

kwv commented Jun 21, 2016

@daluu I never made it past the scaling limitation.

I started down the path of spinning up VMs for each tethered device, but I started to recognize the difference between being a trailblazer and stumbling around in the woods. Defining a repeatable approach was going to be challenging, let alone coming up with a solution that would fit back into STF.

@Ashraf-Ali-aa Capturing video and sending the stream back to a browser was a problem. @codeskyblue is right, it is easier to use images over a websocket to fake a stream. Munging the QT frames into a websocket was next, but that required stronger opinions on the architecture. (Does the app collecting the screen pipe directly to the browser or through a middle agent? If its direct, what supports discovery, or starts the app?) I wanted to leverage a lot of STF's architecture decisions for this, but the approach had already diverged.

I wonder how the commercial products work, if they use an app installed on the device which functions similar to minicap. Rather than pulling from the host machine (like the approach outlined above), push from the device itself (using private APIs?). In the near term FBSimulatorControl looks very interesting.

If nothing else, it just re-enforces that @sorccu and @gunta are doing amazing work.

@HelderVSousa

This comment has been minimized.

Copy link

HelderVSousa commented Jul 5, 2016

Hello guys! Like @kwv i'm trying find a way to screencast the ios screen, i am currently try apply this code from @isonic1, this one https://github.com/isonic1/flick, to make a minicap for IOS. if you guys have some advise to me please tell me.

@HelderVSousa

This comment has been minimized.

Copy link

HelderVSousa commented Jul 6, 2016

@NareshJain91 I am dividing the problem, first i want identify the device and set it visible in stf, next i want make the screencast from ios devices! that is my main work, for now, but we want keep going until get full functionality presented in the android.
So far i am fighting to detect iphone and make screencast.Now with flick i am capable to detect the udid for all devices and take screenshots in all ios devices. I will try this path to make a solution.

@NareshJain91

This comment has been minimized.

Copy link

NareshJain91 commented Jul 6, 2016

@HelderVSousa Sure, if you need my assistance anywhere please let me know. Till now i am able to capture ScreenShots and video from iOS Devices, also i can get how many devices are connected and their UDID. But major problem i am stuck is how to give command from browser/System to iPhone, if it resolves i will be able to implement rest things, for that i have also raised the question to Developer apple forum, but most of have replied that apple do not allow it.

If you find something to give commend from mac to iOS, let me know.
Thanks

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Jul 6, 2016

They probably meant that Apple wouldn't allow you to publish an app that does that. However, you should be able to use private APIs to your heart's content if you never publish your app in the App Store.

@glolsh

This comment has been minimized.

Copy link

glolsh commented Sep 27, 2017

Hello everyone! Any news about iOS support? iOS 11 introduced screenshare option. Maybe this'll help.

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Sep 27, 2017

I don’t think anyone is actively working on it right now. Comtributions are welcome :)

@glolsh

This comment has been minimized.

Copy link

glolsh commented Sep 27, 2017

@sorccu nobody including your team? I suppose this has least priority than ability to connect emulated devices to stf. That's bad. I can't find any solutions to create own iOS-device farm.

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Sep 27, 2017

People tend to think that relatively popular open source projects have some magic behind them that somehow produces funding to the team, but that’s rarely the case. As we have no business model, “the team” quite literally makes nothing from it, aside from the occasional donation. All the time we spend on the project is free, unpaid labor, that is directly out of our own leisure time. We even buy our own devices with our own money. Considering that, don’t you think you’re being a bit unfair?

@glolsh

This comment has been minimized.

Copy link

glolsh commented Sep 27, 2017

@sorccu I'm just asking, no offence. You already made very powerful tool, no doubt. I know that I'm in no position to demand something from opensource project. About funding - I thought that you have some sponsorship from Headspin.
About priorities - It was a question about feature roadmap.

@jayzsolt

This comment has been minimized.

Copy link

jayzsolt commented Sep 27, 2017

@glolsh if you scroll up a few messages you'll see someone recommended http://kobiton.com - I've heard they provide on-prem android and ios device farm options as well, it's not free though. Wish I could figure out how they did fluent screen capturing with iPhones... :)

@jayzsolt

This comment has been minimized.

Copy link

jayzsolt commented Sep 27, 2017

@sorccu hi, if you come up with business model, plans and roadmap, i'm pretty sure you'll have more contribution than just a few bug ticket posters. Many of open source software succeeded that way, and your tool is awesome, why not making it even better? :)

@glolsh

This comment has been minimized.

Copy link

glolsh commented Sep 27, 2017

@apprayadmin thank you! Also I agree with you about "plans". If I (and other people too, I think) would be sure that this functionality is going to be implemented, I would donate some money.
upd: these guys in kobition allow only testing and autotesting, no way to develop there.

@PattabhiramaPandit

This comment has been minimized.

Copy link

PattabhiramaPandit commented Sep 27, 2017

Agree with glolsh. I think this certainly deserve funding. I will talk to my organization about this.

@rdhandapani88

This comment has been minimized.

Copy link

rdhandapani88 commented Oct 2, 2017

Team,
I just want to give one heads up here. Currently I am using Macaca App inspector for object identification in IOS. The XCTestWD agent installed in the device can emulate real device screen, right now atleast we can give an option in OpenSTF to show the connected device screen. But the problem is we cannot control the device, if we have that device control option it will be great for us to continue in IOS also in Open STF. Just check through, and it may be helpful for us.

Regarding Funding,
Will talk to my Company also. Lets rock this.

@codeskyblue

This comment has been minimized.

Copy link
Member

codeskyblue commented Oct 2, 2017

@rdhandapani88 this method has been talked before metioned as WDA

@Ashraf-Ali-aa

This comment has been minimized.

Copy link

Ashraf-Ali-aa commented Oct 4, 2017

@glolsh

This comment has been minimized.

Copy link

glolsh commented Oct 9, 2017

Looks like it can be used for showing display, not for controlling it.

@vdelendik vdelendik referenced this issue Nov 11, 2017

Closed

STF+iOS support #5

@manishPatwari

This comment has been minimized.

Copy link

manishPatwari commented Jan 12, 2018

https://github.com/manishPatwari/WebDriverAgent We have tried to improve upon the WebDriverAgent provided by Facebook.

Enhancements:

  • We have created a node server which would be able to connect to several web and device clients through web sockets.
  • Web sockets in mobile clients have introduced two way communication and improved the speed significantly
  • The screenshots of the devices are being casted in the background thread from mobile clients for optimisation towards unblocking of the actions
  • Added the functionality of supporting multiple devices (both simulator and real ones) with a lock enabled when a device is being used

NOTE: This is a POC from our side and still requires more work. You would be more than welcome to contribute towards making this even better.

Special thanks to @shubhankaryash for contributing in this

@guadaran

This comment has been minimized.

Copy link

guadaran commented Jan 12, 2018

@manishPatwari

Great work! Very impressive the performance improvements that you made upon the Facebook's WebDriverAgent.

@codeskyblue

This comment has been minimized.

Copy link
Member

codeskyblue commented Jan 13, 2018

can u provide some gif videos? recommended using LiceCap

@manishPatwari

This comment has been minimized.

Copy link

manishPatwari commented Jan 13, 2018

@codeskyblue We have recorded the video : https://www.youtube.com/watch?v=bGJAKXdfrgM

@TuHuynhVan

This comment has been minimized.

Copy link

TuHuynhVan commented Jan 15, 2018

@manishPatwari
Just Awesome! Thank you so much! I was able to launch on some Simulator but not able on real ones.
I got no device connected with real ones even the build and test OK.

If there is no devices, run the WebDriverAgentRunner testRunnerSocket
Could you possibly tell a little more about this?

I'm sorry for asking here because there is no place on the forked project to ask about that!

@TuHuynhVan

This comment has been minimized.

Copy link

TuHuynhVan commented Jan 15, 2018

@manishPatwari
The root cause (as I saw from the log) is the socket is not connected for the real ones but not sure which one triggered that...

@glolsh

This comment has been minimized.

Copy link

glolsh commented Jan 15, 2018

@manishPatwari allow Issue creation on your repo please.

@manishPatwari

This comment has been minimized.

Copy link

manishPatwari commented Jan 15, 2018

@TuHuynhVan You have to update the Socket path in this file : https://github.com/manishPatwari/WebDriverAgent/blob/master/WebDriverAgentLib/Routing/FBWebSocket.m ,

Replace "localhost" with the ip address where Node server is running.

@sorccu

This comment has been minimized.

Copy link
Member

sorccu commented Jan 15, 2018

Guys thanks for your hard work but can you please take this discussion to your repo :)

@manishPatwari

This comment has been minimized.

Copy link

manishPatwari commented Jan 15, 2018

@glolsh I have enabled issue creation and also created reference issue in my repo .
manishPatwari/WebDriverAgent#1

@glolsh

This comment has been minimized.

Copy link

glolsh commented Jan 15, 2018

@sorccu we were trying to do so:)

@rdhandapani88

This comment has been minimized.

Copy link

rdhandapani88 commented Apr 19, 2018

Team,
Entire World is waiting for this precious solution for iOS with OpenSTF. STF team can consider manishPatwari's webdriver agent implementation of mirroring and accessing connected IOs devices. I know you guys are busy with other stuff's. But to make openSTF more power ful with IOs, we all should be united. Team need to tell, what they want actually ? what is blocking. Regards.

@JamesChungZLL

This comment has been minimized.

Copy link

JamesChungZLL commented Dec 19, 2018

I'm still working on this. The code's in much better shape, and the stakeholders are listening.

Thanks for the tip on TIFF to JPG. It shaved 100ms off the image processing.

Hi kwv, awesome work!
Are you willing to share the image processing code implemented with fast ios screenshot?

@qmfrederik

This comment has been minimized.

Copy link

qmfrederik commented Dec 19, 2018

The Appium WebDriverAgent has evolved significantly since this work was started (over 3 years ago!). In particular, you probably want to look at:

  1. The MJPEG endpoint they have added. On recent iOS devices you can get a very high framerate (> 10 fps)
  2. They have implemented the actions API which gives you the ability to simulate multitouch gestures at a reasonable speed.

Not the same as minicap + minitouch on Android but much closer.

@qmfrederik

This comment has been minimized.

Copy link

qmfrederik commented Feb 28, 2019

We've released Docker images of xcuitrunner, a commercial tool which allows you to start the WebDriverAgent on iOS devices, even from Linux hosts. http://docs.quamotion.mobi/quamotion4appium/xcuitrunner-docker/ has some rough documentation on how you could use it.

I think it kind of gives you all the building blocks you need to extend OpenSTF to iOS, if you wanted to:

  • Runs on Linux
  • Screenshots and touch events (reasonably high performance) via the WebDriverAgent
  • libimobiledevice provides device management (push applications,...)

I'm just wondering, if someone wanted to integrate this into OpenSTF, how would that work? I assume we'd need some kind of middleware (I believe OpenSTF mainly uses gRPC?)? How would we register it & what APIs would that middleware need to expose?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.