Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional features #4

Merged
merged 2 commits into from Dec 20, 2016
Merged

Conversation

@sushihangover
Copy link
Contributor

sushihangover commented Dec 11, 2016

Add features to allow unit testing on simulator

* InstalliOSApplication
* UninstalliOSApplication
* LaunchiOSApplication
* TestiOSApplication

This builds upon pull request #3 & #2

Review the [unitest.cake](https://github.com/ghuntley/Cake.AppleSimulator/blob/6bd82694372f6c3eccf6052ef1b14b5385e41db3/unittest.cake) on performing xUnit/NUnit for Devices testing...

There are two core Tasks (with supporting Action-based aliases) that show the basics:

  • UnitTestingFailExample
  • UnitTestingPassExample

Unit testing on the iOS Simulator boils down to a fairly simple Task:

Task("UnitTestingFailExample")
    .IsDependentOn("RunningOnMacOS")
    .IsDependentOn("RestorePackages")
    .Does (() =>
{
    buildThisApp(
        "./src/Test.NUnit/Test.NUnit.csproj",
        "UnitTesting",
        "Clean"    
    );
    buildThisApp(
        "./src/Test.NUnit/Test.NUnit.csproj",
        "UnitTesting",
        "Build"    
    );
    unitTestApp(
        "cake.applesimulator.test-nunit",
        "./artifacts/Test.NUnit.app"
    );
});
* InstalliOSApplication
* UninstalliOSApplication
* LaunchiOSApplication
* TestiOSApplication
@sushihangover sushihangover force-pushed the sushihangover:additional-features branch from 3008989 to 6bd8269 Dec 12, 2016
@ghuntley
Copy link
Member

ghuntley commented Dec 14, 2016

Thanks for all the contributions @sushihangover. I need some more time to think about this particular PR before merging (since it surfaces public API changes). In the mean time; can you share with me some insights as to how you're using the addon? Also, all of your commits have been atomically perfect - would you like to be a co-maintainer of the repo?

@sushihangover
Copy link
Contributor Author

sushihangover commented Dec 14, 2016

Geoffrey,

  • Thanks for merging the other PRs 😎
  • Maintainer? : That would be awesome.
  • FYI : I fork'd and published as Cake.AppleSimulator.SushiHangover as I needed these additional features for actual production usage ASAP

My usage pattern of this Cake addin (w/ the additional APIs):

In a continuous deployment environment (devops-style) I am faced with the issue that all mobile devs have, unit testing on devices. You can only mock the environment so much before you have to involve the actual final deployment platform (via real devices and/or simulators|emulators).

I almost always use four layers of testing with Xamarin applications, if I have any say in it! ;-)

  1. The classic unit testing of pure .Net/Mono code via xUnit|NUnit
  • Nothing new here as long as you are doing it
  1. On Platform testing of APIs that are dependent on platform features
  • i.e. Does Cake.AppleSimulator actually have any macOS/simulator based go/no-go unit testing?
  1. UI Testing via Test Cloud/Mobile Center and/or other public or private mobile test clouds (preferably w/ Appium ;-)
  • Nothing earth shattering here except the fact that it is not done in most projects (usually a $ issue and a very bad assumption that all devices running the same OS API level respond the same)
  1. Crash and App Analytics Reporting
  • Including this on a test list and as Devops process causes very heated debates, but I live and die by it 🤕

As far as no. 2 goes, I've seen so many devs mock out platform functions in order to unit test their apis off device via no. 1 and then wonder why the platform is not responding properly. IMHO, this is such a wrong approach to testing, I get sick to my stomach when I see it done. The case of Android alarms that are not time-based deterministic due to the OS optimizating battery life and only propagating changes to a group of app that need that info in a single broadcast per cycle pattern, the fact that iOS's Background Fetch intervals are determined by actual app usage patterns, the fact that my Reactive streams of CoLocation updates are not guaranteed to come at my specify interval, or that the accuracy changes, or that it can stop at anytime the OS determines that battery level is too low or amp consumption is too high, but allows my app to continue running. I could go on for hours of what I've seen apps do and the fact that only doing 1, 3 (and maybe some 4) from the list is not enough to prevent bad reviews on the app store sites and that 50%+ of the issues I see at clients are from skipping 2, another 45% from not close looping the test creation by involving no. 4. The remaining 5% is the really interesting stuff that requires Sherlock Holmes level develop experience. I have one client that is at that magically 5% level and is now implementing AI to determine the causation of human interaction and device response to drive more test creation (by having the millions of active users in multiple apps 24 hours a day create the test fuzzing input that are inversely filtered by the problem users and their issues/analytics, and it totally awesome 😎 watching this run on a private mobile test cloud with a thousand test devices...think VS Intellitest but for devices and that it actually works). About 50% of the issues found in this process are causing additional no 2 on-platform tests to be written... the remaining 50% are usually specific device related.

To bring it back to this project and I am using it as one of a series of required criteria for submitting commits/pull requests before allowing those changes to ripple up through a chain of tiered automated unit testing that start on a small CI and mobile test cloud before allowing it to get to the full-blown test cloud. Just yesterday it blocked 6 pull requests due to iOS specific code mistakes that would have normally consumed about 3 hours of private mobile cloud time at $500/hour, over a year we are expecting it to save a minimum of 120 hours on cloud time, we are also doing with Android AVDs, and macOS VMs and are seeing a saving $200k a year in test hours.

Anything that can make platform-specific unit testing on a device or simulator|emulator easier, this project is one of those, is worth it. I even started adding it to one of my public projects RealmThread as that project is only testable on a device as Realm for C#/Xamarin does not yet run on macOS/UWP/Windows, only iOS and Android. I can CI build and release on Appveyor or Travis but without those build systems connecting to external hosted mobile devices to run something like xUnit|NUnit for Devices, I have to do my own local device testing. Running local sim based tests, will become a minimum require criteria of submitting to Appveyor|Travis, later I'll add to the Cake script take the build artifacts and push them to run the xUnit based tests on Test Cloud/Mobile Center and pull the results back to the build server for pass/failing the build/release on Appveyor|Travis. But as an open source project, keeping the cost down on the mobile device cloud testing is required and thus using Cake.AppleSimulator as a required submission criteria is needed.

Wow, way too long a reply, should a blog post... 👀

-Robert

@ghuntley
Copy link
Member

ghuntley commented Dec 20, 2016

Rob, I would love to have you as a maintainer/lead. Three perfect pull-requests (fine grained) and you are passionate about the exact reason why this library was created. To E2E test switch (i.e. platform) assemblies in a unit test application on on-device simulators to test against regressions which only show when running on-device. As you say, it's not 100% perfect there are differences in how the compiler outputs for ARM vs x86 and how the linker behaves but running unit tests on simulator catches the majority of silly regressions.

Your thoughts/logic of using this as a way to penny pinch, pounds in the cloud needs to be written up into a blog post - it's on the bleeding edge and would be a good talk at a usergroup or Xamarin Evolve nudge - I haven't seen anyone thinking or talking along these lines yet. This library was made to solve problems I have w/ReactiveUI (similar to what you experience with RealmThread) as a framework author but it's now clear that there is immense value (savings $$$) downsteam at the application level. :)

Feel free to add whatever features that you need to the library without my approval, but always please do it via a PR workflow so changes are conducted in the public arena. Check your email for an invite from GitHub, NuGet & AppVeyor.

@ghuntley ghuntley merged commit b136468 into cake-contrib:master Dec 20, 2016
1 check failed
1 check failed
continuous-integration/appveyor/pr AppVeyor build failed
Details
@ghuntley ghuntley added the feature label Feb 4, 2017
@ghuntley ghuntley added this to the 1.0.0 milestone Feb 4, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.