ios-deploy not validating #8
Labels
experimentation
experiments with flutter and devices
Comments
|
Created an experimental project to test signing using Fastlane's match (Match is not a requirement for managing certificates and provisioning profiles, just a convenient way to experiment for now): Will try to implement on Device Farm (manually via test_spec.yml). (Secure env variables can ultimately be passed via sylph -- securely? -- when automated. Though these variables will need to be declared 'locally' -- or securely on travis, etc... -- before starting a sylph run) AFAIK there is no way to get around needing to register the Device Farm device with Apple Developer Account (barring access to Device Farm's Apple Developer Account, since there does not seem to be a secure way to do that via the API that would fit this use case. BTW: currently using public, not private devices). Looks like: BTW: if using some parts of Fastlane makes sense, Fastlane and a Fastfile, etc.., can be added automatically during a Sylph run by adding to Appium bundle and test_spec.yml without requiring to make changes to the original app project. |
|
I think I can output a testable app, packaged and signed as a .ipa. If packaging the testable app as an .ipa works, we can upload it as a usual app (not as an auxiliary) and see if the flutter tools can find it. As I recall, I think I tried this and ran into some problems running |
|
I updated Even if I get Runner.app running on my device and then bundle it as a The most likely path forward at the moment is to explore the use of the Other paths forward do not seem that straightforward:
For now I will continue down the |
|
Was able to download developer signing cert and provisioning profile using fastlane's match on a device farm mac, but am having problems managing the keychain (though keychain management is working locally and on travis). I also did a quick test of fastlane's support for auto-registering devices and it seems to be working (on a local mac). I expect this should work on the device farm mac using the attached iOS device. So it looks like getting flutter tests to run on device farm is pretty close (at least this way). If I can resolve the keychain issue on device farm mac, and some similar minor blockers related to logging in to apple developer account, I expect to be able to run which should install the .app on the device and run the test (barring any possible currently unknown device farm mac issues). BTW: credentials are auto-extracted from the local environment and reproduced in the device farm mac (similarly to travis). The signing certificate and provisioning profiles are installed correctly. The problem is adding the new keychain to the keychain search path so that can find it. The command I am using to add the keychain to the keychain path is: This works locally, but not on a device farm mac (may be a permissions problem that may require the device farm team to modify the device farm mac permissions for keychains). The following log shows the current problem with keychain (logging into apple developer account was not tested in this run). Any suggestions would be appreciated. device farm log |
|
According to: ... it looks like might have to put the kibosh on the 'run test like a developer' approach. This probably bring us back to the drawing board. I've since learned a little more about the device farm mac env. [DeviceFarm] env... the enterprise-resigned .ipa is available in on the mac. So if I can package a testable .app as a .ipa, I can retrieve the enterprise-resigned .ipa, unpackage it to a testable .app, and install it with ios-deploy. Then should start the .app for testing (with the start-for-testing args) and run the test. I think this is worth trying. Edit: So far, I'm not able to upload a testable .app packaged as a .ipa. Creating a testable .ipa is currently disabled in Even if I were able to upload a testable .app via device farm api, it looks like resigning uses the resigners related provisioning profile (which is private). ios-deploy, which One possible way around this is to package the testable .app in the Appium bundle and for device farm to provide a private call from the device farm mac to resign and install the testable .app onto the device (no change to the device farm api). Then should have everything it needs to run the test. This is a blocker on this path. |
|
At this point the paths described in comment above have been examined except: And have each resulted in a potential blocker (as described), with no discovered workarounds so far. The most viable path at the moment is to add a feature to device farm to take the testable .app (uploaded via Appium bundle), resign it, and install it on the device (no need to start it). ios-deploy seems like the most likely way to install a .app. This should be a reasonably straightforward feature to implement. The new feature would callable from the device farm mac (exclusively). |
|
Fixed a problem with: in I created flutter/flutter#34148 with the flutter team and may submit a pull request. |
|
Added building and publishing of non-testable .ipa, signed using my Apple Developer Account, to https://github.com/mmcc007/flutter_ios_build. This can be used with device farm api (though is not used in testing). It is what ultimately get's delivered to the app store for beta testing and release. The https://github.com/mmcc007/flutter_ios_build project is a demo of one way a flutter build can be automated. It can be cloned and used with another developer account locally or on travis by setting-up the secure envs to produce the same artifacts with different signatures (expects a fastlane match repo). Modified the script here to download this .ipa, together with the other artifacts, and run a test on an attached iOS device. The test runs if the matching developer accounts's cert and prov profile are installed locally. For example, the test will run on my local setup, with these artifacts, because I have the matching cert and prov profile for the testable .app installed locally. BTW: the project and script are only intended as a convenience for experimentation with device farm to provide support for developing a solution for dealing with the testable .app. Per the path I am currently pursuing, an installation service implemented by device farm would be called from the device farm mac. The testable .app would be sent to the installation service, re-signed, installed and started on the device (using ios-deploy). The installation service would return the observatory string found in the system log. A modification to Alternatively, the installation service could be exposed by extending the public facing API and optionally skipping the upload of the non-testable .ipa completely. |
|
Hi Maurice, For now though, the path of least resistance seems, to my naive eyes, to be to add a parameter to Adding an on-the-fly resigning option in Device Farm such that, from the macOS custom environment test spec file, you could call a command Thanks! |
|
@aristeia sure, it's interesting stuff, thnx! Registering devices using developer account Packaging a testable app for delivery to device farm I have not yet looked into this restriction to see if there is a workaround. I believe it is because of AppStore rules on embedding an interpreter. So given that we cannot use a testable .ipa, we are using a testable .app. flutter observatory In theory if you know the port number you can connect to this service from the host using any program in any language. Flutter provides a test package written in dart that supports sending and receiving these commands. Developers can then write integration tests to test the behavior of the app (see example). re-signing a testable .app For the sake of experimentation, I looked around for examples of how to automate the re-signing of a .app (instead of a .ipa). Found a script on fastlane that has a feature to re-sign a .app. Fastlane doesn't expose this feature (see Nevertheless, I updated the script to break-out commands to download, re-sign, and test a testable .app. Don't have another developer (or enterprise) cert and prov to test with. There is a problem with the resigning when I try to re-sign the testable .app with my own cert/prov (see output below), but at least it is some way down the path of re-signing a testable .app. @aristeia, this may be useful in experimenting with re-signing a .app with enterprise cert/prov. Here is the output from the following commands: outputstarting a testable .app This is a straightforward process in a developer environment. However, in the device farm mac environment we are assuming there is no access to the device farm's enterprise cert and provision profile. Therefore the app has to be installed and started by device farm and the observatory port made available to the device farm mac. Startup can be done using a modified The observatory port can be made available on the device farm mac as the environment variable (for example, FLUTTER_OBSERVATORY_PORT).
overall process
This requires a modification to the device farm api to accept a .app instead of a .ipa and the addition of the observatory port as an environment variable on the device farm mac. Sylph can be modified to automate this process. |
|
I looked into the restriction on creating a testable .ipa and found a workaround! See details below. This means that the overall process can be further streamlined as follows:
Tests are then started in normal manner (for example, as currently works in Sylph for android) This process requires no change to the device farm api (other than optional enhancement) and can be automated by Sylph. minimum impact
workaround details The fix involved creating a fork of flutter and making some changes: test of workaround The download of the artifacts and running of the test are automated with this script: Test should succeed. No re-signing is required if using artifacts built on your fork with your developer cert/prov. test logs ./script/ios_device_test.sh --download untagged-8ce77308e0d6a2165adb./script/ios_device_test.sh --install_ipa./script/ios_device_test.sh --test |
|
Notes about running flutter in device farm: flutter testing The solution is to build a testable .ipa that does have access to the flutter widgets using commands sent via a port (the observatory). Tests can then be written, in a somewhat traditional way, that results in generating these commands. The tests run on the a host and talk to the testable .ipa via a local port that is forwarded to the observatory port. Incidentally, the same test will run on an iOS or android device (a nice plus). The closest counterpart I can think of to a testable .ipa is perhaps Selenium WebDriver with the mobile extensions, except that the WebDriver is contained in the testable .ipa. Plus there is no need for Appium-like wrappers around vendor provided automation frameworks(XCUITest, UIAutomation, etc...) since you are already talking to the 'native' flutter widgets. flutter driver When running a test, Flutter Driver normally takes care of everything. Basically, it will run the build of the testable .ipa (really a .app) including signing using local developer (or enterprise, etc...) account, install the testable .app on the iOS device, start the app, which opens a random port on the device to receive commands (the observatory), finds the port, and finally forwards a port on the host to the observatory. Then it will start the test on the host and tell the test which local port to use. Note: the test is a standalone dart program that simply talks to a remote service (the observatory). In theory, once it knowns how to access the remote service, the test could be running anywhere on any host (macOS, linux, windows), even in any language. In practice, for iOS, it is normally running on a mac with a local device attached. Further, the test has no dependency on flutter or the device type (iOS, android, etc...) or the app it is talking to (other than the wire protocol, the commands). device farm integration Note: this testable .ipa is not intended to ever be delivered. It is intended exclusively for testing only. Which is why flutter tools disables creating a testable .ipa and only builds a testable .app (hence the fix to workaround this in my experiments. BTW: it may be possible to apply this fix on the fly, locally, without having to change flutter, which would probably be preferable). |
|
@aristeia Looks like there is a problem with installing the testable .ipa on a Device Farm device. The device farm test harness is trying to take a screenshot and erroring-out before starting the run on the mac. This means I cannot currently do test runs on the mac using the testable .ipa. Using testable .ipa: |
|
@aristeia I streamlined the testing of the testable .ipa artifact (from above). Instead of cloning the project and building the artifacts on travis etc..., it should now be possible to download and resign the testable .ipa artifact using your local certificate and provisioning profile (developer, enterprise, ad-hoc etc..) and run the test. This can perhaps facilitate further in-house testing at Device Farm. In particular, experimenting with unbundling the testable .ipa, installing the testable .app on the device, and getting the observatory port (per the overall process described above). Flutter Driver could be modified to install and start the testable .app and return the observatory port with something like: This would be run by Device Farm and the returned observatory port exported as an env var to the mac. Then to run a test on the mac, Flutter Driver could be modified to use the observatory to run a test with something like: simplified testing of testable .ipa test logs ./script/ios_device_test.sh --download untagged-8ce77308e0d6a2165adb./script/ios_device_test.sh --resign ipa "iPhone Developer: Maurice McCabe (5UG6C3962Q)" --provisioning /Users/jenkins/Library/MobileDevice/Provisioning\ Profiles/408fa202-3212-469d-916c-c7f2ae4d083a.mobileprovisiont./script/ios_device_test.sh --test |
|
Added a feature to test if uploaded .ipa is a Flutter testable .ipa. This can be used to start the process for handling Flutter tests in Device Farms (per overall process above). See updated: Usage: |
Hi Maurice, Thanks, |
|
@aristeia It worked on a re-try (this was the first time I had tried with the testable .ipa). Surprisingly ios-deploy of the unpacked $DEVICEFARM_APP_PATH (testable .ipa) get's passed the verification stage and installs. This is good news, as it may make it simpler to run a flutter test. However, I'm running into several configuration and permissions related issues. In general, the configuration of the Device Farm mac is not standard for development and building (especially when compared to a mac on Travis-CI or a local mac):
You can see some of the issues I ran into at run Edit: I tried with a different device (11.4), with more verbose output from ios-deploy with same result. It's having a hard time handling the DeveloperDiskImage.dmg. See related run. |
|
@aristeia Great news! I fixed the problem with not finding the symbols by going thru the ios-deploy source and identifying a device-specific hack. After that ios-deploy was able to start the testable .ipa! I was then able to run a flutter test as normal. The first Flutter test to run on an iOS device on Device Farm looks like: This confirms that Flutter integration tests can run on Device Farm for both iOS and Android devices. This can be integrated to Sylph using the Device Farm API and can be used in a variety of CI/CD scenarios. There is much to add to make this ready for production. But the proof of concept is there. |
|
This is really cool Maurice, nice job! I can't wait to get a look at this! |
This was referenced Jun 20, 2019
Merged
|
@aristeia Implemented the fix for iOS and added support for running Sylph in a CI environment by automatically managing the developer certificate and provisioning profile on the fly (see merges #9 and #10 for details). The resulting CI build can be seem here: Remains to do a Flutter integration test run which includes both iOS and android pools in the same run. Then start making improvements for production quality, like speed, usability, more support for developers, etc... Please let me know if you have any questions, suggestions, etc... |
|
Ran a Flutter integration test on both an iOS and Android pool of devices in a single Sylph run from a local mac and downloaded the respective artifacts for each device. Also fired off the same Sylph run on travis to demonstrate how running Flutter integration tests performs on Device Farm: |
|
@aristeia I notice fairly frequent failures in what appears to be the test harness and related areas on Device Farm. For example, had to run an identical sylph on an iOS and android device three times before getting a successful run: Third sylph run succeeded. Is this rate of failure expected? If more devices are added to each device pool, should the rate of failure be expected to increase? Or is there some way to manage these failures? |
This was referenced Jun 24, 2019
This issue was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On a macOS with an attached iOS device on Device Farms (via web interface) with an uploaded debug .app.
ios-deploy is not validating:
ios-deploy --bundle build/ios/Debug-iphoneos/Runner.app --no-wifi --justlaunch --args '--enable-dart-profiling --start-paused --enable-checked-mode --verify-entry-points'
Currently experimenting with installing provisioning profile and developer certificate, etc...
@tvolkert @aristeia
The text was updated successfully, but these errors were encountered: