From 05137f97119510d0515c791f5a0e1a67e7f7e8b3 Mon Sep 17 00:00:00 2001 From: JP Simard Date: Mon, 11 Mar 2013 09:57:08 -0400 Subject: [PATCH] Removed parts 2 & 3 of testing series --- .../index.html.md | 48 ---- .../lab/ios-unit-testing/index.html.md | 219 ------------------ 2 files changed, 267 deletions(-) delete mode 100644 src/documents/lab/integrated-testing-comparison/index.html.md delete mode 100644 src/documents/lab/ios-unit-testing/index.html.md diff --git a/src/documents/lab/integrated-testing-comparison/index.html.md b/src/documents/lab/integrated-testing-comparison/index.html.md deleted file mode 100644 index bd89343..0000000 --- a/src/documents/lab/integrated-testing-comparison/index.html.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -layout: 'post-lab' -title: 'Integration Testing Frameworks Comparison' -author: 'stu' -nav: 'lab' -date: 2013-02-25 18:37 -short_url: 'mgn.tc/testframeworks' ---- -Like Unit Testing Frameworks, there are an abundance of Integration and UI testing frameworks available. We'll be focusing on three of them: [KIF](https://github.com/square/KIF), [UIAutomation](http://developer.apple.com/library/ios/#documentation/DeveloperTools/Reference/UIAutomationRef/_index.html), and [Frank](http://testingwithfrank.com/). Each has strengths and weaknesses. - -Unlike my previous [testing frameworks comparison](/lab/ios-unit-testing), this article will be much more concise, with examples being provided in a github repository instead of in the code. This should keep everyone from falling asleep mid-article. - ---- - -## KIF -KIF, or for its full name, Keep It Functional, is the current integration testing framework we use. It has an active community, and a well thought out framework created by the clever developers at Square. - -### Setup -KIF integrates incredibly easily into an application. With Cocoapods, it's literally as simple as setting up a new target, installing via cocoapods, and writing your tests. However, it's incredibly important that KIF tests stay in their own target, due to the fact that it uses undocumented API's. If these got into production code, they would be shut down by Apple almost instantly. - -KIF is also written in Objective-C. This makes it incredibly easy for developers to write tests (something we focus on here). Unlike UIAutomation and Frank, which are more suited towards typical QA analysts. - -KIF is not without it's downsides though. Due to the nature of automated testing, there have been some noted bugs in timing and finding the right element at the right time (in some cases, I've had to intentionally wait a second or two before looking for an element because it hadn't been fully loaded by the simulator yet). Also, KIF requires each tappable element to be labelled. Sometimes these labels are implicitly created by Xcode, sometimes not. It's important for both the developer and the tester to keep in communication, and keep these elements labelled and up to date, or the tests will break. - -Finally, When running KIF inside of Jenkins, The developer must provide their own script in order to catch the results from the console. For a full reference on how to get Jenkins and KIF to work well together, see [here](http://www.leonardoborges.com/writings/2012/05/03/build-automation-with-xcode-4-dot-3-kif-and-jenkins/). - -## UIAutomation -UIAutomation was introduced by Apple in Xcode 4, and is the default UI/Integration testing suite in Xcode. All tests are managed in JavaScript (which I found to be somewhat odd). - -### Setup -UiAutomation is run through the Instruments Suite. Starting it is relatively simple, in just clicking the UIAutomation button, and start writing the script. Out of the three, this setup is by far the easiest. - -Unfortunately, UIAutomation leaves a lot to be desired. It runs into the same problems as KIF in that all elements must be labelled correctly. Furthermore, with javascript syntax, each developer now must learn two languages to test the application (granted, JavaScript is a well known language). The Instruments Application is quite clunky as well. Finally, while it is possible to run UIAutomation in Jenkins, it's no easier than KIF. - -## Frank -Frank is the iOS implementation of Cucumber. Written in ruby, Frank stays almost entirely outside of the application project completely. - -### Setup -Frank is installed through the gem package manager. After that, you simply run `frank setup` and `frank build` and `frank launch` to launch to the app in the simulator. - -Writing tests is done in ruby. The syntax is ruby. If a tester enjoyed ruby, I would recommend this framework. I however, do not. - -While Frank is a fair sight better than UIAutomation, and runs with Jenkins reasonably well (probably the best out of the three), the heavy use of ruby as it's testing language is enough to throw off some objective-C developers. Furthermore, it appears that many of the things frank does can be done with Kiwi (which makes sense, given that they're both designed for BDD). To be frank (no pun intended), if a developer is using Kiwi, I see no reason to use Frank. - -## Conclusion -In conclusion, KIF appears to be the best tool for the job as it stands right now. UIAutomation may have a place if it were to allow the exiting and re-entrance of an application, it would be more useful. Frank has no place in our testing suite if we're already using Kiwi (If other developers are not using Kiwi, it would make sense to look into frank a bit more heavily). - -*This post is the third in a 3-part series on how Magnetic Bear Studios manages its quality assurance process.* \ No newline at end of file diff --git a/src/documents/lab/ios-unit-testing/index.html.md b/src/documents/lab/ios-unit-testing/index.html.md deleted file mode 100644 index 41d7575..0000000 --- a/src/documents/lab/ios-unit-testing/index.html.md +++ /dev/null @@ -1,219 +0,0 @@ ---- -layout: 'post-lab' -title: 'iOS Unit Testing' -author: 'stu' -nav: 'lab' -date: 2013-02-25 18:35 -short_url: 'mgn.tc/iosut' ---- -There are numerous types of unit testing frameworks for Objective-C. From the basic OCTest/Sentest, to BDD-style frameworks like [Kiwi](https://github.com/allending/Kiwi) and [Cedar](https://github.com/pivotal/cedar). We'll be testing three of them here. OCtest, [GHUnit](https://github.com/gabriel/gh-unit), and Kiwi. These were chosen due to popularity and support. Other options such as Cedar may be viable, but aren't as well supported. - ---- - -## Testing criteria -1. ease of use - - writing a useful test method - - ease of interpretation - - ease of automated testing -2. support for Asynchronous testing -3. Integration with Xcode - -These are rated in order of importance to me. Ease of use and Async testing are crucial. Integration with Xcode, while important, isn't as vital. - -## Testing -Each framework will run through each of the following: -* Test a basic test scenario (the temperature conversion) -* Test a basic network connection (Using AFnetworking) -* Testing compatibility with Jenkins - -## Test a basic test scenario -The first test case I ever wrote was for a temperature conversion method. It's simple, yet has enough to worry about that fringe cases will be required. - -In our Test Project's viewController, we'll have two methods. - -``` --(double)convertCelsiusToFarenheit:(double)celsius{ - double farenheit = 0; - farenheit = celsius * (9/5) + 32; - return farenheit; -} -``` - -``` --(double)convertFarenheithToCelsius:(double)farenheit{ - double celsius = 0; - celsius = (5/9) * (farenheit - 32); - return celsius; -} -``` - -Yes, these methods are a little contrived and verbose, but they'll serve for what we need. - -Looking at these, we know we'll have a few test cases to worry about: -* -40 is the same for both temperatures. We must test for this. -* 32 degrees farenheit must equal 0 celsius -* 100 degrees celsius must equal 212 farenheit. -* converting one temperature and back to the original should produce the correct number. - -If any of these test cases fail, we know there's a problem with our methods. - - -## OCunit -OCUnit is the easiest of the frameworks to set up. When setting up the project, simply check `include unit tests` in the setup box. This creates a second target that's specific to the OCTest Framework. - -### Writing Tests -First, set up your setup and TearDown Methods to create destroy a viewController Object - -``` -- (void)setUp -{ - [super setUp]; - controller = [[ViewController alloc]init]; - // Set-up code here. -} -``` - -``` -- (void)tearDown -{ - // Tear-down code here. - - [super tearDown]; - controller = nil; -} -``` - -On to the tests. All test methods must begin with "test". This is how OCUnit determines what's a test, and what isn't. - -First, we'll check if -40 equals -40: -``` --(void)testCelsiusEqualsFarenheitAtNegativeForty{ - double celsiusResult = 0; - double farenheitResult = 0; - - celsiusResult = [controller convertFarenheithToCelsius:-40]; - farenheitResult = [controller convertCelsiusToFarenheit:-40]; - STAssertEquals(celsiusResult, farenheitResult, @"these should be the same"); -} -``` - -Next, we check that 32 farenheit if 0 celsius, and vice versa -``` --(void)testThatThirtyTwoFarenheitEqualsZeroCelsius{ - double celsiusResult = [controller convertFarenheithToCelsius:32]; - double farenheitResult = [controller convertCelsiusToFarenheit:0]; - STAssertEquals(celsiusResult, 0, @"should be zero"); - STAssertEquals(farenheitResult, 32, @"should equal 32"); -} -``` - -We test that 100 celsius is 212 farenheit and vice versa -``` --(void)testThatHundredCelsiusIsTwoTwelveFarenheit{ - double farenheitResult = [controller convertCelsiusToFarenheit:100]; - double celsiusResult = [controller convertFarenheithToCelsius:212]; - STAssertEquals(farenheitResult, 212, @"should equal 212"); - STAssertEquals(celsiusResult, 100, @"should equal 100"); -} -``` - -Finally, test that the temperatures can be converted back and forth -``` --(void)testThatConvertingBackAndForthDoesntAffectResults{ - double celsius = 50; - double farenheitResult = [controller convertCelsiusToFarenheit:celsius]; - //convert back - double celsiusResult = [controller convertFarenheithToCelsius:farenheitResult]; - STAssertEquals(celsius, celsiusResult, @"should be equal"); -} -``` - -We could split these up even further (one STassert per test), but this will do for our purposes. - -Running the test suite is also really easy with Xcode. Simply change your target to, and run as test. If there isn't a scheme for the tests, it's easily added in the manage schemes window. -Overall, integrating these tests into Xcode, and writing basic tests are fairly simple and easy to do. No different than GHUnit Really. - -### Ease of interpretation: -Unfortunately, this is where OCUnit begins to faulter. If a test fails, it automatically pops up as an error inside Xcode where the assert call is. For those who are used to other frameworks like Junit, this is an annoyance as we can't determine if there are other errors that are actually affecting the code, and overall looks much more cluttered. The Console output isn't much better, with the output hard to read. - -### Automated Testing -As we'll see in most of our frameworks, automated testing is not an easy task I won't go into the full details here, as there are numerous postings about the problems with Xcode and Jenkins. The summary is that xcodebuild in Xcode 4 doesn't support unit testing on iOS (Amusingly, OSX Unit testing allowed and easy to implement). The solution is to run the tests through the simulator, then take the console output, and format it to an XML file that Jenkins can understand. This is completed most often with a script written in Ruby. It can be found [here](https://github.com/ciryon/OCUnit2JUnit). - -Resources: -http://www.raingrove.com/2012/03/28/running-ocunit-and-specta-tests-from-command-line.html -http://stackoverflow.com/questions/12557935/xcode-4-5-command-line-unit-testing - -### Asynchronous Testing -Unit Tests by design are intended to run fast. Because of this, Apple made the decision to not wait for asynchronous requests during a test, and assume they had passed. As such, any code inside the request won't actually be tested! In the world of constant server connections, this is a huge problem. There are however, some workarounds: - -* Implement a semaphore in your test case. This forces the test to wait until the semaphore is dispatched, allowing the block or asynch request to finish and be tested. -* Implement the OCMock Framework. OCMock is a testing suite in it's own right, but is most often used in tandem with a typical testing framework. OCMock has some methods in it's implementation that allow for a mocked object to "expect" a response. When this response is expected, it "invocates" the block response, and the code is tested then. - -In my opinion, either of these are a rather ugly hack. I'd rather have a framework that takes a second longer to finish a request, than implement the solutions above. - -## GHUnit -GHUnit is another popular style of Unit Testing. Apparently, before Xcode 4, adding OCunit was a pain. Compared to the power of GHUnit had, most people used GHUnit exclusively. While that has since changed, it's worth looking into GHUnit Regardless. - -### Integration -Integration is slightly more complicated than OCUnit, but using [cocoapods](http://www.cocoapods.org) makes things much easier. I originally intended to write how to set up GHUnit, but a tutorial is already available [here](http://samwize.com/2012/10/04/how-to-setup-ghunit-with-cocoapods/), and it explains it much better than I could. Be sure to check it out. - -### Writing Tests -First, create your test file inside the GH Target. Instructions can be found [here](http://gabriel.github.com/gh-unit/docs/appledoc_include/guide_testing.html) Just be sure that if you copy and paste the template, to change `GHAssertNotNull` to `GHAssertNotNil`. It appears the docs haven't been updated since the arc conversion. - -The rest of the test writing is fairly straightforwad. Pretty much the same as OCUnit. In fact, it still contains the same setup and teardown methods, as well as the same methods. The only difference is that instead of STAssert(), we use GHAssert. The syntax is the same. GHUnit though has more assert methods than OCUnit, making it easier to write tests. - -### Ease of interpretation -GHUnit's main power is easy interpretation. If these tests are being run from Xcode instead of CI, the simulator pops up with easily readable results. The console output is also quite easy to read. It either passes or fails. Nothing fancy, Just what you need. - -### Automated Testing -Automated testing is slightly different While you still have to run through the simulator, most people tell jenkins to run a build script that has a make file inside of it. This is the method used here: http://www.youtube.com/watch?v=6ycxFcIPhQg&feature=related It's much better than I can explain, so I'll just leave it here. - -Thankfully, GHUnit outputs results as a JUnit style XML file. This saves us the step of having to convert the console output to something readable like in the OCUnit step. - -### Asynchronous testing -Like OCUnit, GHunit doesn't support Asynchronous testing out of the box. The reasons why and the solutions to implement them are the same. - -## Kiwi Testing -Kiwi is a slightly different style of testing. While GHUnit and OCUnit are based around test cases that pass or fail. Kiwi follows the concept of stories or specifications that must be acted out. This is popular in BDD style testing. - -### Integration With Xcode -Kiwi, while not as easy as OCunit, is still a fair sight easier than GHUnit. A simple cocoapods install and a new TestUnit Target, and you're ready to write your tests. - -### Writing Tests -Kiwi is different syntactically speaking in comparison to OCUnit and GHUnit. Instead of setting up test methods, you set up specifications that you expect to be passed. - -``` -SPEC_BEGIN(TemperatureConverterTests) - -describe(@"basic converter", ^{ - it(@"converts farenheit To celsius", ^{ - ViewController *controller = [ViewController alloc]init]; - [[theValue([controller convertCelisusToFarenheit:-40]) should]equal:theValue(-40)]; - }); - it(@"converts celsius to farenheight", ^{ - ViewController *controller = [ViewController alloc]init]; - [[theValue([controller convertfarenheitToCelsius:-40]) should]equal:theValue(-40)]; - }); -}); - -SPEC_END -``` - -This syntax is a fair bit different. That's because with Kiwi, we're more describing stories and specifications than actual tests. - -### Ease of interpretation -The logs in Kiwi are a lot easier to read than OCUnit. With it's human like syntax when writing the specs, these descriptions are taken and output at the console. If a human needs to read these logs, they'll be able to do it as easily as GHUnit's output. - -### Automated Testing -Automated Testing in Kiwi requires a bit of work. Due to Xcode not allowing application unit tests (the same problem as OCUnit), we have to hack our way through. Some people have had success with ios-sim, others use the same method as OCUnit outlined above. - -### Asynchronous testing -Kiwi is the only framework that supports Asynchronous testing out of the box. This is due to Kiwi's BDD style testing, where it's not so much about Unit testing, but a combination of Unit Testing and Integration testing, forming a behaviour. Personally, this alone is enough to warrant a huge win for Kiwi. - -### A note about Mocking Objects -As mentioned in OCUnit, mocking objects with OCMock is a solution to asynchronous testing. However, it is useful in much more. While this was not discussed in this article, it should be noted that Kiwi supports it's own mocking implementation, and appears to be rather easy to use. - -## Conclusion -In conclusion, each of these frameworks have strengths and weaknesses, and using a hybrid approach amongst them is definitely possible, and probably needed at some point. However, although Kiwi has a bit of a harder time with Automated testing, it's strengths of Asynchronous testing, ease of interpretation, native support of mocked objects, and easy to write tests cannot be ignored. The Kiwi framework will make up the bulk of our tests, with OCUnit most likely playing a support role if it is ever needed. - -*This post is the second in a 3-part series on how Magnetic Bear Studios manages its quality assurance process.* \ No newline at end of file