Application vs. System Testing #1315

Open
jamierumbelow opened this Issue May 3, 2012 · 76 comments

Projects

None yet
@jamierumbelow
Contributor

This isn't so much of an issue but more of an open question as to where unit testing support is heading for CI3.0.

It's awesome that unit tests are now a part of CI core, and it's great that the branch has been merged in and everything's starting to look good and official. I'm excited.

As you'll all know, currently the tests are in /tests. While this is fine, that folder seems to only facilitate system tests. What about application tests? It would make sense to move the /tests directory into /system/tests and have an /application/tests directory with a sample test inside it.

Is there a framework in place for application tests? There'll need to be mocks setup, methods to override core bits of CI functionality etc.

Is there anything in there that I'm missing to allow for this yet? If not, want me to look into it?

@timw4mail
Contributor

There is CodeIgniter's unit testing library, but I'm not sure how much that's used.

@jamierumbelow
Contributor

As far as I was aware, CodeIgniter's unit testing library was being deprecated/removed. It's never used and doesn't support mocks, custom reporters or a "test environment".

...plus, surely the whole point of integrating PHPUnit into CodeIgniter was to provide core tests for the system and a platform for devs to write tests for their applications?

@timw4mail
Contributor

I wasn't aware of it being deprecated. But I do see your point. Personally, I would like to avoid adding any more folders to application if possible, but that's a personal preference.

I tend to think this is something that should be left up to the user.

@jamierumbelow
Contributor

One of the benefits of introducing unit tests to the system is that we can also introduce unit testing to the ecosystem, that is to say, start to try to grow a culture of writing tests in our applications. Only a fool would disagree that programatic testing is a good thing to do.

"Leaving it up to the user" would almost certainly manifest itself as a page hidden away in the user guide stating that CodeIgniter supports PHPUnit tests and this is how you add them; an approach that doesn't engage new users or raise any curiosity about unit testing (for proof of this statement, see aforementioned existing unit testing library). Having an application/tests directory would be obvious, apparent and direct, and would certainly encourage more people to take a look at writing unit tests.

@accentinteractive

I think an application/tests folder would be a great idea. In my opinion, a sample test is not necessary. There are no sample models either. Just as long as the docs contain a good example of setting up a test and fetching the results.

@sirfilip
sirfilip commented May 3, 2012

+1 for the application/tests with sample phpunit.xml ready to run.

@timw4mail
Contributor

Ughh...aren't there enough folders under application already?

@joelcox
Contributor
joelcox commented May 3, 2012

Adding an extra folder to application seem to be the most sensible to me. The system and application testing should be separated IMO.

@mpmont
Contributor
mpmont commented May 3, 2012

Totally agree with Joel therefore => +1 on this.

@jamierumbelow
Contributor

What's the community's view on splitting up the tests directory into subdirectories for models, controllers, libraries and helpers? I'm worried it would be TOO cluttery... but then again, it provides a great structure for testing (particularly for newbies).

P.S. I'm off for a month so won't be able to help out until I'm back (but would very much like to be a part of creating it). Could everyone who's +1ed here fight the good fight for me while I'm away? :)

@joelcox
Contributor
joelcox commented May 4, 2012

I would like an '/application'-like directory structure, but I think there are more things to consider. While we're focussing on unit tests here, there are other sort of tests you might want to write like integration tests (in which folder should these go if we adhere to the typical '/application' directory structure?), browser based tests (using something Selenium) or behavior tests (for BDD, using something like Cucumber).

There are so many sort of tests that it's almost impossible to dictate a directory structure, but we could document some best practices in the user guide.

@toopay
Contributor
toopay commented May 4, 2012

Current unit testing only covering CodeIgniter API (DB, Library, Core and helpers) under tests/codeigniter, and it still incomplete. Bear in mind that the current unit testing suite, also should cover :

  • Application tests (at least the default "welcome" controller).
  • Package tests.

Using tests/application will have benefits : 1). It can be run alongside with system test 2). It can easily use the current mocks object, ref : tests/mocks 3) More clean and logical, in directory structure, instead put it under application/test.

The only potential issues, will be package test, IMO.

@mpmont
Contributor
mpmont commented May 4, 2012

I think we should have a test folder for application and other for system. However tests like the ones @joelcox mentioned could be just mentioned in the Documentation, and not actually implemented.

As for application test folder structure, I think it would be nice to have the separation @jamierumbelow mentioned. I think that would help everyone that is getting started with tests. Having all the tests in one application folder can make things messy, just imagine the amount of files that folder can have.

@jamierumbelow
Contributor

@joelcox I'd forgotten about integration and functional tests, thanks. Maybe then copy what Rails does with application/tests/integration, application/tests/functional and application/tests/unit? There should certainly be recommendations, a convention that we should try to follow.

@toopay I really really don't like adding another top-level folder. I'd much rather have system/tests and application/tests.

1). We can use a phpunit.xml file to configure the system so that system tests can be easily run alongside application tests anyway. Besides, this isn't really an issue since developers aren't going to want to test CodeIgniter (unless they're working on CI itself). It's the application they write in CodeIgniter they're interested in testing.
2). Again, using a phpunit.xml file to setup a bootstrap script would be simple and would allow app tests to use the system testing framework, no matter where the files are located.
3). I feel application/tests and system/tests is more clean and logical than tests/application and tests/codeigniter. After all, we have application/libraries and system/libraries, application/language and system/language...

As far as packages go, it makes more sense for the tests to go in the package/tests since a package is essentially a modular application folder.

Laravel handles testing in a really clever way. Using its artisan system, it automatically generates a phpunit.xml file based on what the user wants to test. Could we add something similar as a default controller (accessed through the CLI)?

$ php index.php test models
$ php index.php test package_name
$ php index.php test integration

If the user wasn't interested in using tests, they could simply remove the test.php controller and be done with it.

@benedmunds
Contributor

If sparks is going to be integrated (when is this scheduled for BTW?) then tests could be initiated from sparks similar to using artisan in laravel.

I definitely agree with the application/tests directory. Most importantly so it can gain exposure to the community and promote best practices.

@philsturgeon
Contributor

@benedmunds I'm not sure when sparks is getting integrated, its one of the last things on my mental checklist for 3.0. Any input @katzgrau @seejohnrun @derekjones ?

For me, tests/application is backwards. We are not distributing tests with codeigniter for the application, we are considering adding tests to your application that can be run with the rest.

Some would argue that this should be left in the hands of the developer so they can handle their own testing and have it totally seperate from the CI stuff. This would avoid bloat and make it much easier to fit it in however they like. It would also avoid the inevitable "I have X different applications running on the same CI folder and it's not called "application" bootstrapping issue that we will likely run into by having the main phpunit.xml assume that application/ is "where it's at".

Not sure of the best action for that but I do know that packages should be tested by themselves. I think the people in charge of making sparks should be dealing with stability, not the folks using it. Having tests run out of packages and displaying a travis-ci green light would be a very nice feature for Sparks IMO.

@toopay
Contributor
toopay commented May 4, 2012

@jamierumbelow @benedmunds tests folder is an "optional" folder. I doubt it is necessary to bundle them into downloadable version at http://codeigniter.com/downloads/. Not everyone think and work with TDD method. The separation between codebase(system and application) and the tests cases, also mean that these two things is decoupled. If you look into test bootstrap, it clearly mean you could put the tests folder anywhere, and just point the constant to your application and system paths. This also mean, the tests itself could be developed, outside the repo (using Git submodule, for example). I will hardly consider the other way around, of how we manage the directory structure, for this reason :)

Regarding package tests, i think about give the spark (as an official "package" distribution platform), the ability to hook into the current unit test, but maybe @katzgrau could answer whether this possible or not, with current spark logic.

And +1 for adding more functionalities to spark, as console interface for CodeIgniter.

@jamierumbelow
Contributor

@toopay See, one of the best things about bringing testing into CodeIgniter is so that we can encourage newer developers to start writing tests and using TDD. It should be a part of application for this very reason. TECHNICALLY, models are optional, so by that logic application/models should be an "optional" folder.

@philsturgeon @toopay I think we have to be careful to remember that devs aren't going to want to run the CI tests alongside their application tests much. If we do our jobs properly, the CodeIgniter tests will pass. We want to provide the structure and framework to encourage developers to test their applications, not CI core itself.

@philsturgeon Using sparks as an automation / test runner would allow us to dynamically generate the phpunit.xml file depending on the dev's intent. If they want to run the tests for a specific application in a specific folder, we generate a config file on the fly for it. If they're building a package and want to run a package's tests, we generate a config file. If they want to run everything, we generate a config file. It's a really nice, dynamic way of doing it.

@derekjones
Contributor

Not wanting to derail this discussion I just wanted to point out that TDD, as a specific development philosophy, should not be part a native of CodeIgniter. The framework does not favor nor encourage any particular development philosophy.

@philsturgeon
Contributor

@jamierumbelow "I think we have to be careful to remember that devs aren't going to want to run the CI tests alongside their application tests much." right, which is why many would say it's a better idea to keep out of it altogether.

Two bits that are agreed:

Spark devs (and the system around it) should be responsible for testing the sparks.
CodeIgniter devs should be responsible for testing codeigniter.

But surely by that logic application devs should be responsible for testing their applications, and CodeIgniter only giving them "one-true-way" to do it is not helpful.

Are you just taking things from rails and trying to implement them in CI again? ;-)

@jamierumbelow
Contributor

@philsturgeon Phillip, I'm trying to encourage more people to write tests, because I know that the code we write, programs we ship and professional service we as developers give to our clients will be better for it. Rails/Ruby has a brilliant testing philosophy and it's due to the inherent nature of testing in the framework.

I'm not saying there should be one true way, and I certainly believe that whatever implementation it takes it should be flexible. I do believe application developers should be responsible for testing their applications, and I also believe it's the framework's responsibility to make it easy for them by providing the setup: with the runners in place, mocked out classes readily available etc.

@derekjones CodeIgniter strongly recommends the usage of MVC, so why can't it strongly recommend the usage of tests? I'm not suggesting it enforce test driven development, but it should certainly recommend testing as a whole.

Before MVC we wrote spaghetti, template-y Rasmus-y code. Before the query builder we wrote driver-specific database interactions. Hell, before Sparks we distributed our code through ZIP files on our blogs, the forums and the wiki. Testing is merely the next evolution of a framework helping improve the quality of code, and it's irresponsible to disregard it.

@derekjones
Contributor

@jamierumbelow correct, you have identified what I was getting at, that TDD and testing/tests in CI are not the same thing. We should have the latter, we should not push the former. Also, CI doesn't "strongly" recommend MVC. It's not required, you don't even have to use models, or even views for that matter if one so chose. It's there, but implemented in a way that can be interpreted and used in any number of ways that reflect a developer's design philosophy and preferences.

@philsturgeon
Contributor

@jamierumbelow: I say this in a very friendly way as I always enjoy meeting you for beer BUT: get off your high horse. The CodeIgniter community doesn't need you or any of us holding its hand to become better programmers through rigid implementations. We do that through documentation, tutorials, guides, screencasts, etc but not through code.

Rails is a kitchen sink framework, which is the polar opposite from the CodeIgniter philosophy. Testing the core makes sense, we need to make sure our code is stable. Testing an application? that's a job for somebody else!

We can offer tutorials and guides, and as @toopay has already pointed out it's very possible to do with the Bootstrap.php that exists in there already, but baking it in to be part of the actual framework? I am extremely unertain about wether that is something that needs to be done.

@derekjones
Contributor

but baking it in to be part of the actual framework? I am extremely unertain about wether that is something that needs to be done.

App testing packages sound like good Sparks to me as well, @philsturgeon. Entirely different ballgame from unit tests for the framework.

@yoosuf
yoosuf commented May 4, 2012

CI is a simple and powerful PHP framework, Please don't try to make it as yet another ZEND

Thank you

@derekjones
Contributor

+1 @eyoosuf :)

@jamierumbelow
Contributor

@philsturgeon How is implementing testing support into the core any different from integrating Sparks as a better way to distribute packages / libraries, or having a query builder as a better way of interacting with the database, or having MVC as a better way of structuring code?

I am not suggesting a rigid implementation. I'm suggesting a flexible implementation with sensible defaults. And I don't think I'm out of line by suggesting that a framework has a responsibility to promote best practices! Like it or not, there are certain conventions that the majority of CodeIgniter's users follow. And why? Because it's part of the code. Because it's inherent to the design and structure of the framework. Because the framework provides a conducive environment and atmosphere for those conventions.

@derekjones @philsturgeon I wasn't suggesting that we demand users write tests. I wasn't suggesting that we write tests for them. All I'm suggesting is that, like I said above, we provide a good environment to make it easier for people to do the right thing and write tests.

From the moment of downloading CI it's obvious that CodeIgniter inherently recommends the usage of MVC. There are controller, model and view folders. There is a separation of code across the layers in tutorials, documentation, screencasts, all the things Phil mentioned in the community. Sure, CI doesn't demand you use models or views, but it's pretty frikken obvious that it's recommended, commonplace and conventional. Testing should be treated the same. It should be a recommended best practice, and the framework should facilitate that.

@philsturgeon
Contributor

Thank you Jamie. Your points have been put forward, lets get some other opinions.

@derekjones
Contributor

Sure, CI doesn't demand you use models or views, but it's pretty frikken obvious that it's recommended, commonplace and conventional

And look at any two CI apps developed by two different developers, and I almost guarantee you they are using the paradigm differently.

How is implementing testing support into the core any different from integrating Sparks as a better way to distribute packages / libraries

Sparks is not a feature of the framework, it is a method of extending the framework. The entire purpose of bringing it into the fold is so that the framework could rightly have a more skeptical view of incoming features into the core. jQuery, Cart, Trackbacks, etc. would never have been added to the framework if Sparks existed from the beginning.

@joelcox
Contributor
joelcox commented May 4, 2012

I think @derekjones puts it nicely.

@jamierumbelow correct, you have identified what I was getting at, that TDD and testing/tests in CI are not the same thing. We should have the latter, we should not push the former.

That's what CI is all about, at least for me. Enable the developers, but not force them.

@seejohnrun
Contributor

We have the branch with the sparks integration ready for review and merging in whenever the time is right:
https://github.com/EllisLab/CodeIgniter/tree/feature/sparks

@jamierumbelow
Contributor

And look at any two CI apps developed by two different developers, and I almost guarantee you they are using the paradigm differently.

Let me repeat. I'm not suggesting we demand people write tests or we demand what way they write whatever code they do. I'm just saying that it is a framework's responsibility to promote best practices and if the inclusion of an application/tests folder helps that, than what damage is it going to cause?

@philsturgeon
Contributor

No it's really not Jamie, it is a web frameworks job to help people develop a web application.

Those applications can be tested in a multitude of ways. One of those is to write some tests... maybe a "tests/" folder in the root. And those tests should use some sort of test framework like... say PHPUnit.

Oh wait; all of that is already completely possible exactly how things are at the moment.

@yoosuf
yoosuf commented May 4, 2012

+1 @philsturgeon

it is a web frameworks job to help people develop a web application.

@jamierumbelow
Contributor

@philsturgeon @derekjones The overriding truth is that if the framework makes it easier for devs to write tests, without demanding it, being overly obtuse or obstructing devs (as long as you can do exactly what you do without writing tests), that's surely surely surely a really good thing?! I don't see what the issue is here; you all seem to be making me out to be this fascist dictator demanding everybody write tests instantly. I'm not saying that we enforce it. I'm just saying that making it easier for developers cannot possibly be a bad thing,

@philsturgeon
Contributor

Right, so put them in the "tests/" folder and modify the existing bootstrap.

I don't think PHPUnit has been sitting around thinking "One day, I hope CodeIgniter integrates us into their core so EVERYONE will get to learn about us".

@toopay
Contributor
toopay commented May 4, 2012

@philsturgeon no need to touch or doing any drastically move, to the bootsrap.

The only thing missed from current unit testing was : documentation. The explanation here, is far out-of-date. Most tasks like :

  • Clean up naming conventions
  • Figure out config stuff
  • Figure out database testing

were done.

The explanation for running the test also only available for Linux users, which may lead to big trouble for Windows/Mac users. I want to update it, but my poor english grammar will only make it worst :D

Soon i'm done, completing the database tests, i will try to write a simple application test, for the default "welcome" controller. And since PHPUnit could integrated nicely with somthing like selenium, clarification for @mpmont , @joelcox idea actually practical and could be implemented.

@jamierumbelow
Contributor

Right, so put them in the "tests/" folder and modify the existing bootstrap.

Phil, you said, mere hours ago that "tests/application is backwards". Why the change of heart? What on EARTH is wrong with having a system/tests directory to test the system and an application/tests directory to test the application code?

@philsturgeon
Contributor

When we started this conversation I was unsure, I made that clear. The point of these discussions is so that opinions can be formed and during the course of this little talk it has become more and more clear that this is not something that we need to worry about. Every reason you have put forward for wanting to integrate application unit tests is already possible in the existing structure.

I have used Rails before. It's very nice. This is not Rails.

@abfan1127

I support any integration of PHPUnit. I think Jamie's system/tests and application/tests is a great idea, especially if we can run just system tests or just application tests. Great idea!

@adamgriffiths

So, when we were talking about test being integrated into CI 3.0, it's just for the core and not for the application? Or…?

@zackkitzmiller

I'm a huge +1 for application/tests.

It doesn't really make any sense to mix app and system testing.

@chartjes
chartjes commented May 6, 2012

Phil Sturgeon asked me to come over and comment here instead of on Twitter.

I was told "favouring one methodology over the other is not how CI works" and I think that is a pretty shitty attitude to take. Of course, I am biased because I believe in using testing frameworks and building testable applications.

What is the problem? Why all the fear of deciding "hey, let's make it easier for those who want to use TDD to build apps using CI". Help a grumpy programmer sick of shitty apps in any language without documentation or tests understand how providing PHPUnit hooks for testing application code is a huge problem.

@philsturgeon
Contributor

@chartjes thanks for posting on here, Twitter is useless for this sort of conversation.

Straight off the bat you are acting like I have said unit testing is bad. That is really not the case, unit tests are great and I'd
hope everyone uses them for their applications along with other testing methods.

It is my job as a CodeIgniter Engineer to act as a gatekeeper for feature requests and pull requests. That means I am sat on the fence, trying to make a decision about the suggestion and sometimes to get to that decision I have to play devils advocate. For that I should not expect abuse, it is part of how this works. If I blindly merged everything or didn't argue then we'd have a horrible framework.

The point in this thread where the conversation moved to being a somewhat heated discussion was when @jamierumbelow started saying the same thing as you Chris:

I'm trying to encourage more people to write tests, because I know that the code we write, programs we ship and professional service we as developers give to our clients will be better for it.

This to me just seems like a patronising "Go on son, remember to clean your teeth or they'll fall out"! We all know that if we don't clean our teeth our applications are going to develop cavities in the future and we'll wish we'd done it in the first place, but CodeIgniter has never tried to be your mother and force you to do things, even when they are a good idea.

Making Unit Testing easier is what we should be talking about, and that is the only part of Jamies argument I suggest we bare in mind.

Now there are multiple points in this thread being discussed at the same time:

CI has unit test support

Absolutely. There /tests folder contains optional tests that you can remove from your file system if you don't want them there. This means we can make sure CI is well tested without forcing anything on the end user.

CI should allow application tests

Well... it does. CodeIgniter's usual lack of ties to any specific framework mean you can use whatever to do whatever. We don't require jQuery/MooTools/etc officially, but all of them can be used easily with CodeIgniter. I've often heard people asking to integrate AJAX and I STILL don't know what that means, but it doesn't at any point mean you can't use AJAX.

Make it easy by adding mocks, etc

Those are available in the tests/ folder already. You can already create tests/application and do it in there if you like. If we add more support then we are infecting the application folder with test logic, and this test logic is for a specific framework. If the community is decided that we ONLY care about PHPUnit then that is perhaps something that can be lived with, but that may not be the case and we can make the jump to support it.

Only supporting unit testing

As @jamierumbelow and @joelcox have been discusisng, only supporting TDD is a bad call (this is what I was refering to @chartjes). While unit tests are good, what about integration tests, selinum and all that jazz. Do we then add in /application/tests/integration, /application/tests/unit and /application/tests/functional like Jamie suggested? How do we decide which systems to support then?

In one fel move we'd go from "CodeIgniter works with whatever testing system you like because it is entirely agnostic" to "You have to use PHPUnit, Seliunium, etc and hack it to work with anything else". How is that a good thing?

Make a application/tests folder

Some have said that they don't want an extra folder in application. I'd suggest this is a non-issue, as by default this could not be there, or it could easily be deleted if you don't want it.

What I DO suggest about that is that we would need to configure phpunit.xml (in the root I expect) to look there specifically, and if you have multiple applications or non-standard application names you will need to configure it to do so. If that is fine, then great, bootstrapping becomes easier - but there will be no way to automatically detect where your application actually lives.

System does it, so why not application

The current tests are heavily Travis-CI orientated, meaning you can only run them if you have SQLite, Postgres, etc set up. This means we need to have different bootstrap logic set up for system and application, meaning it's not as easy as "just make it work for both" as some of you are suggesting.

Talking about this is a requirement so don't just shout +1 at me. This is not some sort of Roman Gladiator fight that gets decided with a thumbs up or thumbs down. We need an actual poroposal before we can vote on anything.

@chartjes
chartjes commented May 6, 2012

So let's be clear about what I wanted to point out as a desirable outcome from my point of view:

As someone who has written applications using CI I feel it is important to make it as easy as possible to write unit tests for their CI applications using the dominant unit testing tool available, which is PHPUnit

I wasn't picking on you Phil, and I apologize if it came across that way.

Deciding that CI will make things easier for folks to use PHPUnit is one option for sure. I see no other real viable competitor to PHPUnit for unit testing.

In my opinion, adding support for PHPUnit is not turning CI into Zend Framework (like someone earlier suggested) but instead another step along the path to making CI better for all types of developers. Tossing that away out of fear of making things complicated would make me sad.

Sent from my iPhone

On 2012-05-06, at 5:56 PM, Phil Sturgeonreply@reply.github.com wrote:

@chartjes thanks for posting on here, Twitter is useless for this sort of conversation.

Straight off the bat you are acting like I have said unit testing is bad. That is really not the case, unit tests are great and I'd
hope everyone uses them for their applications along with other testing methods.

It is my job as a CodeIgniter Engineer to act as a gatekeeper for feature requests and pull requests. That means I am sat on the fence, trying to make a decision about the suggestion and sometimes to get to that decision I have to play devils advocate. For that I should not expect abuse, it is part of how this works. If I blindly merged everything or didn't argue then we'd have a horrible framework.

The point in this thread where the conversation moved to being a somewhat heated discussion was when @jamierumbelow started saying the same thing as you Chris:

I'm trying to encourage more people to write tests, because I know that the code we write, programs we ship and professional service we as developers give to our clients will be better for it.

This to me just seems like a patronising "Go on son, remember to clean your teeth or they'll fall out"! We all know that if we don't clean our teeth our applications are going to develop cavities in the future and we'll wish we'd done it in the first place, but CodeIgniter has never tried to be your mother and force you to do things, even when they are a good idea.

Making Unit Testing easier is what we should be talking about, and that is the only part of Jamies argument I suggest we bare in mind.

Now there are multiple points in this thread being discussed at the same time:

CI has unit test support

Absolutely. There /tests folder contains optional tests that you can remove from your file system if you don't want them there. This means we can make sure CI is well tested without forcing anything on the end user.

CI should allow application tests

Well... it does. CodeIgniter's usual lack of ties to any specific framework mean you can use whatever to do whatever. We don't require jQuery/MooTools/etc officially, but all of them can be used easily with CodeIgniter. I've often heard people asking to integrate AJAX and I STILL don't know what that means, but it doesn't at any point mean you can't use AJAX.

Make it easy by adding mocks, etc

Those are available in the tests/ folder already. You can already create tests/application and do it in there if you like. If we add more support then we are infecting the application folder with test logic, and this test logic is for a specific framework. If the community is decided that we ONLY care about PHPUnit then that is perhaps something that can be lived with, but that may not be the case and we can make the jump to support it.

Only supporting unit testing

As @jamierumbelow and @joelcox have been discusisng, only supporting TDD is a bad call (this is what I was refering to @chartjes). While unit tests are good, what about integration tests, selinum and all that jazz. Do we then add in /application/tests/integration, /application/tests/unit and /application/tests/functional like Jamie suggested? How do we decide which systems to support then?

In one fel move we'd go from "CodeIgniter works with whatever testing system you like because it is entirely agnostic" to "You have to use PHPUnit, Seliunium, etc and hack it to work with anything else". How is that a good thing?

Make a application/tests folder

Some have said that they don't want an extra folder in application. I'd suggest this is a non-issue, as by default this could not be there, or it could easily be deleted if you don't want it.

What I DO suggest about that is that we would need to configure phpunit.xml (in the root I expect) to look there specifically, and if you have multiple applications or non-standard application names you will need to configure it to do so. If that is fine, then great, bootstrapping becomes easier - but there will be no way to automatically detect where your application actually lives.

System does it, so why not application

The current tests are heavily Travis-CI orientated, meaning you can only run them if you have SQLite, Postgres, etc set up. This means we need to have different bootstrap logic set up for system and application, meaning it's not as easy as "just make it work for both" as some of you are suggesting.

Talking about this is a requirement so don't just shout +1 at me. This is not some sort of Roman Gladiator fight that gets decided with a thumbs up or thumbs down. We need an actual poroposal before we can vote on anything.


Reply to this email directly or view it on GitHub:
#1315 (comment)

@philsturgeon
Contributor

I didn't feel picked on and hopefully my reply wasn't too grumpy either. I just want to get this conversation to the best logical conclusion.

hopefully a few people can weigh in with useful suggestions on the implementation so we can take it forward. @jamierumbelow has buggered off to South America for a month, so the rest of us have to carry this on.

@philsturgeon
Contributor

@chartjes Extra note, looks like Ed has his own testing system FUnit, which is based on QUnit. He's making your "Only PHPUnit" call more tricky. Deathmatch on the next podcast? https://github.com/funkatron/FUnit

@narfbg
Contributor
narfbg commented May 7, 2012

+1 on @eyoosuf

Just to be clear - I'm not against unit testing, it's a good thing. But lately almost all activity on the repository has been tied to unit tests. It's a useful tool to have, but it's just that - a tool, and if we remove all comments block from the system/ code, tests/ will already have a third of it's size.
Yes, I'm aware that it's optional, it doesn't add any real overhead, etc. But while there's indeed a need for a way to automatically test code in the core, please - let's just stop at this point. As @philsturgeon has pointed out - if we go that route, there's way too much scenarios to be covered. If not anything else - we need to focus more on CodeIgniter's quality and new functionalities.

I'd vote with two hands for some decent documentation, tutorials or a spark (if it's possible) for application/ unit testing, but I just don't think it belongs in the default distribution.

@chartjes
chartjes commented May 7, 2012

Ed and I already discussed this but not on the podcast. My conclusion was that Ed is a dipshit for not using PHPUnit. :p

Sent from my iPad

On 2012-05-07, at 2:54 AM, Phil Sturgeonreply@reply.github.com wrote:

@chartjes Extra note, looks like Ed has his own testing system FUnit, which is based on QUnit. He's making your "Only PHPUnit" call more tricky. Deathmatch on the next podcast?


Reply to this email directly or view it on GitHub:
#1315 (comment)

@philsturgeon
Contributor

Ha, good to know! He can shove FUnit up his AUnit.

Emailing from my iPhone 4S like a BOSS!

On 7 May 2012, at 06:59, Chris Hartjesreply@reply.github.com wrote:

Ed and I already discussed this but not on the podcast. My conclusion was that Ed is a dipshit for not using PHPUnit. :p

Sent from my iPad

On 2012-05-07, at 2:54 AM, Phil Sturgeonreply@reply.github.com wrote:

@chartjes Extra note, looks like Ed has his own testing system FUnit, which is based on QUnit. He's making your "Only PHPUnit" call more tricky. Deathmatch on the next podcast?


Reply to this email directly or view it on GitHub:
#1315 (comment)


Reply to this email directly or view it on GitHub:
#1315 (comment)

@toopay
Contributor
toopay commented May 7, 2012

and if we remove all comments block from the system/ code, tests/ will already have a third of it's size.

Its a consequence of having unit-testing. A series of test agains API from one class, will be bigger that the class/file itself. Something like swiftmailer going crazy on their test-suite size.

If not anything else - we need to focus more on CodeIgniter's quality and new functionalities.

There is a direct relation, between having test-suite and all of these holla-mumbo-jumbo code-coverage, with code quality. It is also ensure any new functionalities not break the current one.

@philsturgeon
Contributor

Agreed. Before we can even consider large changes we need to have existing code well tested with high code coverage, so we can see it break and make sure it's fixed.

Emailing from my iPhone 4S like a BOSS!

On 7 May 2012, at 07:26, Taufan Adityareply@reply.github.com wrote:

and if we remove all comments block from the system/ code, tests/ will already have a third of it's size.

Its a consequence of having unit-testing. A series of test agains API from one class, will be bigger that the class/file itself. Something like swiftmailer going crazy on their test-suite size.

If not anything else - we need to focus more on CodeIgniter's quality and new functionalities.

There is a direct relation, between having test-suite and all of these holla-mumbo-jumbo code-coverage, with code quality. It is also ensure any new functionalities not break the current one.


Reply to this email directly or view it on GitHub:
#1315 (comment)

@yoosuf
yoosuf commented May 7, 2012

sorry about Out of the chapter

@philsturgeon i like your email signature

@ttessin
ttessin commented May 10, 2012

Just tuning into this thread. I have been using PHPUnit/Selenium now for quite a while in our CI based web application.
We started with Selenese (xml) and converted it to PHPUnit. We reduced test time from 5+ hours to less than 45 minutes using parallel threads. I have since started writing plain PHPUnit tests for our 'bolt on' libraries and included pear libraries, and just started playing with PHPUnit tests for controllers/models/libraries. My directory structure has /system, /application, and /tests all at the same level. /tests has the following subdirs: selenium (all PHPUnit/Selenium tests), phpunit (vanilla PHPUnit unit tests) and api (home grown command line api tests). phpunit is further subdivided into models, controllers, and libraries and ciunit(fooStack). We have multiple phpunit.xml type files (some of them are auto-generated in the build to do parallel test runs).

We can work with system/tests and application/tests if that is the decision. However having our application test code in a /tests directory allows me to zip it to a selenium driver server and farm out the test threads. /tests/selenium can run independently of the CI code base so the selenium driver server doesn't need to have the rest of the app code checkout.

@mastacheata

This discussion obviously stopped about half a year ago. Did you come to an agreement of any kind behind the scenes or did this just get lost over the summer?

Nevertheless I'd like to offer my opinion on the Application Testing stuff:
You all discussed about whether and how many subfolders to create in application/tests/...
I would appreciate an all empty application/tests folder or at most a very basic PHPUnit Testcase covering the default controller. There should not be any hierarchy like in the system part.

This would lead to too many different possiblities.
People can't even agree where controller starts and model ends or how many intelligence may sit in the view.
(Is a loop inside a view already too much, should validation be performed in controller or in model etc.)

Even though some of the professional users or even dedicated amateurs would really like to see Test Coverage in all possible areas I doubt that you will convince the majority to use it. Keep in mind everyone is complaining about missing or insufficient documentation since the beginning of computers and even before that.

CI should not force users to do tests, it should make it easy enough for people to want to do testing.
This is also not about which Test Framework is possibly better. PHPUnit was already chosen for Unit Testing the CI-Framework itself, so we should stick to that and offer the facilities to easily extend this to the Application.
How to extend that on the application should then be explained to a greater extent in the User Guide.

Another matter was that the current Test Coverage is mostly achieved with the help of the Travis Continuous Integration System. It is rather safe to assume that this is not available to most users. I wouldn't mind the System Test Coverage to stick to that, but it must definitely be optional for Application Testing.

@alexbilbie
Contributor

CI won't force users to do tests and it's never been suggested it will.

As Phil has said before, we need to bolster the existing unit tests and add tests for classes that haven't yet been covered so that the core is water tight before we can go any further on this

@mastacheata

Sorry, maybe I chose the wrong words there. What I meant was not so much that this was ever thought to enforce tests on the user. I meant it more like it's a bad idea to have a too tight path for testing the application part. (And that was at least discussed here even though it was mostly discarded)

I can fully understand that you want to focus on getting the base of codeigniter covered by tests first before deciding on any final way for application testing. Nevertheless collecting ideas and opinions should still be fine even while the core stuff is far from finished.

@cryode
Contributor
cryode commented Oct 24, 2012

From what I gather here (and agree with), zero unit testing will be included with the stable production release of CodeIgniter - system, application, or otherwise. Unit testing is being used to aid in development of 3.0, not as a released feature. Application unit testing should be encouraged via the user guide and/or a Spark. Anyone interested in contributing to the guide or a spark should do just that.

@alexbilbie
Contributor

@cryode to clarify there will be system unit tests in CI3, there might not be a way of doing application tests yet

@cryode
Contributor
cryode commented Nov 6, 2012

Won't that force users to make sure a particular testing suite is installed / available?

@alexbilbie
Contributor

Does it actually matter if /system is tested with PHPUnit?

When it comes to /application testing I imagine there will be a number of hooks into CI so if you want to test in Behat you can or you can stick with PHPUnit

@cryode
Contributor
cryode commented Nov 7, 2012

I don't really know if it matters or not; I have limited experience with unit testing. I'm legitimately curious.

@igravity

Why all this fuss about application-level testing? If need be, CI has somewhat embraced Composer. Add this to your application and test using Codeception. The CI tests are CI's tests; completely orthogonal to an app's test (a point over-hammered). But this begs the question though… what does CodeIgniter mean by "Application Tests" in the test/ folder?

@ivantcholakov
Contributor

@jamierumbelow

"One of the benefits of introducing unit tests to the system is that we can also introduce unit testing to the ecosystem, that is to say, start to try to grow a culture of writing tests in our applications. Only a fool would disagree that programatic testing is a good thing to do."

I am that fool to disagree.

Jamie, some time ago I forked your project codeigniter-base-model. I liked it, of course. But I sensed your approach "Where are your tests?", it kills motivation. So, I worked on the forked project and use it for production purpose. If you make a comparison, you will see, that your original project is lagging behind.

People ask new features, and they do not have them, because you decided to stick with TDD methodology. You even removed a feature (multiple database connections), just because you was not able to test it.

I bet on progress, not on tests.
-1 from me.

I agree with @derekjones that TDD is a specific methodology, and the framework should not promote it.

P.S.: A humble advice: Author of a piece of software != author of the corresponding tests.

@stgeneral
Contributor

Here is a way how I've solved the problem of tests in existing CodeIgniter app https://github.com/stgeneral/CodeIgniter-TestSuite. Solution was performed on CodeIgniter 3.0 (master develop branch snapshot from Oct-Nov 2013).

@narfbg
Contributor
narfbg commented Jan 31, 2014

master is not 3.0.

@stgeneral
Contributor

Sorry, @narfbg. I've missed that default branch is develop. Fixed my previous comment.
By the way, the ideas used in my suite very likely will work for 2.1 too.

@greenwizard88

Anything that promotes more unit testing is good in my book.

I'm a fan of Application/tests and System/tests, if anyone cares.

@bgeesaman

I have just been charged with refactoring a good chunk of a CI 2.2 codebase with ~47K loc find ./applicaiton -name '*.php' | grep -v "third_party" | xargs wc -l and zero tests. I came across this thread in looking for the best method for integrating a unit testing framework so I can write tests to cover certain areas before I start refactoring safely. The built-in CI unit testing library is insufficient, and despite their best efforts, community implementations are rife with self-admitted "hacks" to get things working.

I find it extremely suspicious that the "core" is unit tested with PHPUnit, but extending that structure to the "application" space is considered undesirable and would go against the core tenets of CI to not "force" you into any methodology. Picking a primary testing framework and making it usable for the application side in NO WAY means I have to actually write test files or even run them. It doesn't force you into TDD or test-after-the-fact. But what it does do is make sure that there is a common and supportable way to hook into the framework to run tests that other testing frameworks can leverage if one liked those instead. As it is now, that interface doesn't exist. Any testing framework I shoe-horn in can't be guaranteed any sort of compatibility going forward. That is why I believe the decision to refrain from making the built-in CI test framework full featured (fixtures, mocks, environment handling, web drivers, etc) or even providing a somewhat legitimate integration path for external testing frameworks is so short-sighted.

This example is one of the main reasons frameworks with the tenets of "remove any friction for the developer to create and refactor quality code" are thriving and frameworks that haven't adopted critical features into their core approach are on life support. I consider having a full-featured testing framework as a core feature to be "removing friction". I'm spending days/weeks trying to bolt testing on to this codebase after the fact, but all I can think of is "What would it take to rewrite this all in ____ framework? It's probably less effort and a better long-term decision."

+1 for picking one, integrating it, documenting its usage, and treating it like a first-class citizen. application/tests and system/tests makes sense as an approach for clean separation.

@kenjis
Contributor
kenjis commented Jul 21, 2015

I wrote a bridge tool called ci-phpunit-test for CodeIgniter 3.0 application testing.
http://kenjis.github.io/ci-phpunit-test/

And I sum up request list for CI 3.x, just for your information.
https://github.com/kenjis/ci-phpunit-test/blob/master/docs/RequestsToCodeIgniter.md

+1 for system/tests and application/tests

@jagandecapri

@kenjis I am using the bridge tool that you wrote. Pretty neat. It will be nice if it is included in CI core and further improved. 👍

@kenjis
Contributor
kenjis commented Aug 5, 2015

@jagandecapri Thanks!

@ghost
ghost commented Sep 25, 2015

@kenjis +1. Works fine, well documented and add a good tool for developers

@feryardiant
Contributor

+1 for @kenjis, actually I'm not using your bridge but I do aggreed that Codeigniter should be testable.

Writing tests for CodeIgniter 3.0 application has troublesome. Because CodeIgniter has some code which is untestable or difficult to test.

I'm using this one and had same problem.

@kenjis
Contributor
kenjis commented Sep 26, 2015

@kfern-github Thanks!

@feryardiant all the problems stated in the request list have been solved (or there is workaround) in my ci-phpunit-test.

@feryardiant
Contributor

@kenjis great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment