Kiwi 2.0 - Reboot

[allending]

Reboot

Development has been stalled for the past year. I have not been the best maintainer. Now, this changes. The admittedly lofty goal is for Kiwi to become the test library for iOS.

This is the announcement for the 2.0 plan, where the goal is refinement and focus areas based on the biggest issues with the library.

Focus areas

1. Deliberate higher level testing support (primarily UIKit components). Kiwi is pretty good at testing domain model and controller objects, but lets face it - a big part of iOS is UI and Kiwi sucks at it.
2. Streamlined installation and usage of the library. Too many people can't figure out how to configure their environment to work with Kiwi. There are also too many ways to install Kiwi.
3. Documentation enhancements to cover not only how to use Kiwi, but also what to use it for. A library is pointless if most people are using it wrong. Worse, they may think they are using it right.
4. Fixing of the most essential outstanding issues.
5. Getting the community involved more to keep the project active.

Bonus - we need better versioning and roadmapping

Anything that is not directly related to one of the 5 focus areas below will be deprioritized.

Contribute

We already have a bunch of great guys helping on Kiwi, @lukeredpath, @nheagy, and @mneorr. But lets face it - this is open source. Projects die and rot without community involvement. If you are interesting in helping on any part of Kiwi (1-5 above), contact me. We need your help.

Pull requests are absolutely welcome, but even better, open an issue so that we can discuss them first! PRs from out of the blue are very difficult to evaluate.

Feel free to participate in any discussions. Tell us about the problems you want Kiwi to solve and what you want 2.0 to be good at.

First steps I'll take

• Outstanding open issues will be triaged based on the focus areas above
• A more concrete set of issues/tasks will be derived from the focus areas above
• A proper roadmap will be made, with a clear target for a 2.0 release
• Implementation galore (with your help)

References

• http://blog.thepete.net/blog/2012/11/18/writing-ios-acceptance-tests-using-kiwi/
• http://dimsumthinking.com book

Plug

Since I need to pay the bills, if you have iOS (or Android) projects you need help with from experts, give me a ping.

[nheagy]

Looks great. I absolutely support this, and hopefully soon, will have more
time to spend on it!

• N

On Wed, Dec 5, 2012 at 10:33 AM, Allen Ding notifications@github.comwrote:

Reboot

Development has been stalled for the past year. I have not been the best
maintainer. Now, this changes. The admittedly lofty goal is for Kiwi to
become the test library for iOS.

This is the announcement for the 2.0 plan, where the goal is refinement
and focus on these areas based on the biggest issues with the library.
Focus areas

1. Deliberate higher level testing support (primarily UIKit
   components). Kiwi is pretty good at testing domain model and controller
   objects, but lets face it - a big part of iOS is UI and Kiwi sucks at it.
2. Streamlined installation and usage of the library. Too many people
   can't figure out how to configure their environment to work with Kiwi.
   There are also too many ways to install Kiwi.
3. Documentation enhancements to cover not only how to use Kiwi, but
   also what to use it for. A library is pointless if most people are using it
   wrong. Worse, they may think they are using it right.
4. Fixing of the most essential outstanding issues.
5. Getting the community involved more to keep the project active.

Bonus - we need better versioning and roadmapping

Anything that is not directly related to one of the 5 focus areas below
will be deprioritized.
Contribute

We already have a bunch of great guys helping on Kiwi, @lukeredpathhttps://github.com/lukeredpath,
@nheagy https://github.com/nheagy, and @mneorrhttps://github.com/mneorr.
But lets face it - this is open source. If you are interesting in helping
on any part of Kiwi (1-5 above), contact me. We need your help.

Pull requests are absolutely welcome, but even better, open an issue so
that we can discuss them first! PRs from out of the blue are very difficult
to evaluate.

Feel free to participate in any discussions. Tell us about the problems
you want Kiwi to solve and what you want 2.0 to be good at.
First steps I'll take

• Outstanding open issues will be triaged based on the focus areas
  above
• A more concrete set of issues/tasks will be derived from the focus
  areas above
• A proper roadmap will be made, with a clear target for a 2.0 release
• Implementation galore (with your help)

References

http://blog.thepete.net/blog/2012/11/18/writing-ios-acceptance-tests-using-kiwi/

• http://dimsumthinking.com book

  —
  Reply to this email directly or view it on GitHubhttps://github.com/allending/Kiwi/issues/176.

[supermarin]

Looking forward for this!

[dimsumthinking]

I'll try to lend a hand - thanks Allen.

The one issue I might suggest is a test runner

[jonathanpenn]

Woot!

[casademora]

I've got a few features I'd like to see added to kiwi, I'll see what I can help with...

[allending]

Thanks guys, this is going to be fun.

@dimsumthinking is there a specific feature set you would want in a test runner. Put another way as a thought experiment:

• Why do we need a test runner (what problem does it address)?
• What is the one thing it needs to do very well?

These questions are to help focus in on what to build initially.

[allending]

Adding a note here that dealing with Async APIs is something that is likely in scope as it is a pattern that is highly prevalent.

[luisobo]

I'm in!

[luisobo]

CLI runner, or even better, pluggable runners (IDE plugins, CLI, etc) would be great. Specially for running tests in a CI. Pluggable output formatters would be nice for the same purpose.

👍 for running one file or one test, it would increase developer productivity a lot

Ability to manage Kiwi configuration in some sort of SpecHelper.m:

• Adding global before* and after* will be great for configuring third party testing libraries like Nocilla
• Register custom matchers.

This last point would help to increase the community: if it's easy to plug more tools into Kiwi, more people will create these tools.

[allending]

Here we go.

The approach I'm going to take to keep things manageable is to get a more solid foundation in place. Therefore the initial work on 2.0 will focus on configuration and documentation holes/refinement.

All progress will be tracked via GitHub issues.

[shepting]

I am really hoping to have some better documentation for using Kiwi on the Mac. This could also entail updating the Podspec to have it configured out-of-the-box for Mac OS X development just as it is on iOS.

[shepting]

@luisobo You can kind of make global before and after blocks by creating specs that begin with AAA and ZZZ. Something better would definitely be appreciated.

The particular use case we used this for was moving aside a Core Data db, testing, and then moving it back.

[subdigital]

I've been using this script to run tests and get my red/green fix, however it would be nice to have something like this baked in.

Maybe this can be of use to others asking for a test runner.

[jonah-williams]

I like the sound of this. @allending can you create a "2.0" milestone for existing and new issues? It would be helpful to see which issues are seen as problems with Kiwi today, which are areas we should focus on to get a 2.0 release, and which are "nice to have" features which are not necessarily ready or high priority for 2.0. That can also serve as a nice indicator of when an issue has been sufficiently well defined to be prioritized for 2.0.

I also like the focus areas listed above. However in considering them there are a couple of things I see as core principles which I hope the project does not lose.

Kiwi runs on top of OCUnit. This is lame for a number of reasons and limits what Kiwi can do however it does have some benefits. Xcode, AppCode, CI systems, and anything else that has built support for OCUnit tests can run a project which uses Kiwi. At least for me it has been hugely valuable to be able to drop in Kiwi to get BDD syntax and a solid test double implementation on a project. Even if I can never move beyond a set of Logic Tests at least I can say with confidence that the tests will work with existing tools. As much as I would love to see better tools Kiwi has often been my default because I can get it adopted more easily than a system which requires ruby gem installs or other external configuration.

Kiwi is opinionated and designed with good practices in mind, even when those practices are not pervasive in iOS development.
I think we should advocate for installation via CocoaPods by default (its automated, cheap to repeat once you are familiar with the tool, and because projects deserve descent dependency management if they hope to use shared code).
I think we should be clear about if we are writing unit tests, integration tests, or functional tests. We could support testing asynchronous code in integration tests or we could offer patterns for unit testing asynchronous components without needed async test support. Either or both of those sound good to me but I hope we can avoid conflating the two. Unit tests which start to creep into integration tests have caused be all sorts of problems in the past.

[allending]

@jonah-carbonfive milestone created, thanks for the suggestion.

Running on top OCUnit/SenTest - I agree this is unlikely to change, unless Apple introduces a completely revamped test approach in the future.

Installation - Its between CocoaPods and a Framework at the moment. My preference is a framework because of reduced dependencies toolchain wise for end users.

Unit/integration - Do you mean better and clearer documentation?

Async (and also threaded code) - This is a tough problem. Properly testing this is basically a nightmare, I'm not even convinced we can do this reliably in the general case because of how much is left unspecified about the environment tests are running in.

[jonah-williams]

With regards to types of tests. I think better documentation is part of it for sure and that focus area #3 above covers that problem well. My concern is that many of the UI and async test cases developers have do not belong in unit tests and therefore probably cannot (or at least I argue should not) be solved with Kiwi alone. The mention of UI and async tests above made me nervous about more non-unit test features creeping into the project.

Kiwi is my preferred unit testing tool but we already have expectFutureValue creating some integration tests. I see that leading to tests which can span multiple run loop iterations or tests which can use UI components with async behavior and those lead us to write higher and higher level tests. Where and how do we draw that line to determine what belongs in Kiwi and what should be part of a different tool?

[jonathanpenn]

I'd like to echo the concern about beefing up Kiwi with knowledge about asynchronous testing. At the very least, could we punt it to solve at a later time? If the goal is to clean Kiwi up, document it, and hone down it's core competency, then maybe an asynchronous testing component can be an add-on that can be built using the core. I'm thinking something like the way rspec is broken out into pieces with rspec-core being the main part. The goal would be to delay the decision with how to solve the asynchronous testing problem and let good solutions bubble up in the community independent of Kiwi's core competency.

[frosty]

If the proposed installation solution for Kiwi is Cocoapods, then would a potential solution to your async worries be to add those components as a separate Cocoapod subspec, ala ShareKit or RestKit?

[allending]

Trying out Cocoapods. Awesome, this will be the official supported way. wonderful /cc @alloy

[alloy]

^5

[jonah-williams]

@frosty sort of, yes. If it is easy to manage our dependencies we have a bunch of options. As you suggested subspecs would allow for a core Kiwi library with several default but optional components.

That's slightly different than the scenario I had imagined. I would like to see other testing tools be able to declare Kiwi as a dependency. Thus we might have a library of UI testing tools which require and run on top of Kiwi. CocoaPods helps with this because it will manage a case where I have multiple independent tools which all share a common dependency. Imagine that I might have a project which uses Kiwi directly for unit tests but also includes an async testing library for some integration tests and a UI testing library for some functional tests. Both the async and UI testing tools might depend on Kiwi to provide a test runner or test doubles or other behaviors.
Without CocoaPods it becomes rather difficult to use several libraries which each end up trying to bundle their own copies of the same dependencies (doubly so in an iOS environment where we cannot provide dynamic frameworks).

[allending]

Closing, will incorporate comments.

@frosty @jonah-carbonfive yup, the approach will be a core library similar to what exists now, and any sort of UI specific tools will be a separate project where dependencies are managed by Cocoapods.