swat integration tests for httpd cookbook

[melezhik]

I have just created an integration tests for httpd cookbook using swat - DSL for rapid web tests development.

To run them:

$ kitchen test swat

[melezhik]

Here is couple successful and none successful verify examples with swat:

• successful (debian) - https://travis-ci.org/melezhik/httpd/jobs/102547224#L2009
• none successful (ubuntu) - https://travis-ci.org/melezhik/httpd/jobs/102547225#L2233

[melezhik]

Hi! Any news here?

[iennae]

@melezhik,

Apologies for not responding sooner. When I go to look up swat from the link you've included it doesn't have a lot of information. Can you provide more info about this test suite? It looks pretty interesting, I've just never used sparrow before.

[melezhik]

Hi Jennifer! Thanks for answer. Ok, to make it more clear

1. swat is web integration tests tool , it allows to write blackbox test scenarios on http level, using curl to make http requests and then analyzing server response, like check if we have successful http status code, matching content against some patterns and so on, a detailed information could be found here - http://swatpm.org , recently I made some updates to it and added hello world example and FAQ page.
2. sparrow is a tool to install packaged swat test scenraios to run tests somewhere else on your environment - like rubygems for ruby or cpan for perl. So sparrohub is central repository for swat test suites ... - https://sparrowhub.org cosiders different applications where we could have a specific tests suites like apache, nginx, tomcat, jboss, whatever . Someone write a specific test suite for a specific software, application and upload it to sparrowhub. Then others can install this suite and run it. Simple! Reusable test scenarios. For the time being all the suites presented on sparrohub are wtitten by me, but I seek for new community members.
3. Ok , not lets talks about chef. As chef cookbooks often deal with application installing and configuration for various software swat tests could be good it for post converging checks. Like, if apache response on given request ? Ot if we could see apache server version at response header? And if this version is equal to required? Or if basic authentication works fine and we succeed in sending request with user/password and have a proper content returned? And so om, a lot of use cases. I found a quite "pure" solution for such a checks using serverspec's wrapper around curl command or the same approach inside busser-bash bash scripts. As swat test suite make a difference when encapsulate all the complexity of check logic inside a swat plugin could be run by sparrow tool ( via busser-bash transport )
4. swat / sparrow dependencies are installed with Sparrow cookbook ( not that many as all depends on perl , curl + a few cpan modules )

This pull request is like example where one can see how swat could be used as blackbox testing toll in cookbook development process ( chef / test kitchen )

Let me know if you need some more information ....
Thanks!

PS or else ... we could talk somewhere in chef-dev irc channel ...

[melezhik]

Hi @iennae - any updates on this? Thanks 😄

[lamont-granquist]

I'm sorta 👎 on this...

Not because its a horrible idea in principle, but because this cookbook desperately needs a high level of maintenance going forwards, and its needs to be kept simple enough and use common enough chef tooling that anyone familiar with the Chef ecosystem can use it. Pulling in a tool like swat now means that maintainers need to be familiar with that tool in order to contribute and maintain these tests after you've gone off and gotten interested in something else, which is a barrier to entry. Its not a bad idea to show off using swat tests in a cookbook, but I don't think its a particularly good idea to put them in this cookbook....

[iennae]

@melezhik

Thank you so much for taking the time to show swat in use. As @lamont-granquist mentioned, this is not the right cookbook to commit this work though. I think if you could create a sample cookbook that uses swat that would be really useful for you to add to the supermarket as an example of how to use swat. We try not to create additional requirements for folks to learn in order to contribute.

Jennifer

[melezhik]

Hi @lamont-granquist ! It sounds like you miss my point at all. I am not going to use swat with this cookbook , idea is to use swat as integration tests tool for cookbook developing, could you you see the difference ? Well to talk about chef Eco system , I am not going to make a flame here , but what wrong with using other tools together with chef ? Like for example some officially maintained opscose cookbooks use a bash as integration test tool , so what ? :)

[melezhik]

Hi @iennae , I already did it, an example of usage of swat like black box testing tool in test kitchen infrastructure for httpd cookbook, no need to create another examples, please take a look more carefully at the code I am trying to merge. :-)

I am not trying to force you or something , but before you close this request ( well you already did it ) , I am just trying to help you understand clearly what I am talking about.

[melezhik]

Not because its a horrible idea in principle,

And one more thing to add here, @lamont-granquist , before calling someone's ideas horrible it is good to explain why you consider such an ideas as horrible, in other way you sound not respectful.

Sorry, probably I missed a context of your phrase, as I can see you mean "that this idea is not horrible ... or it is not horrible in principle?", Am I right? Just please clear what you meant. Anyway see my other remarks on your previous comment.

Thanks

[lamont-granquist]

I'm saying this is a great idea for something you should blog about and put in other cookbooks first.

The point is that without swat being used to integration test other cookbooks and being widely used in the Chef ecosystem already, it is bad idea to introduce it into cookbooks like this first. If it does not catch on, then it makes this cookbook harder to maintain with a larger barrier to entry for any novice out there, which ultimately makes this cookbook worse. If we had lots of examples of swat-tested cookbooks already and had a userbase that expected to see such tests and already knew how to maintain them, then it would make sense.

That does create a catch-22 since without getting it into core cookbooks it is difficult to get it spread widely, but if its a good idea it'll eventually catch on and happen. Right now though these examples don't offer anything over serverspec or inspec testing, and we've already got those two methodologies are trying to train up our userbase to switch from serverspec to inspec. Adding another framework that they need to know won't help them learn to maintain better cookbooks.

[melezhik]

Hi @lamont-granquist

I'm saying this is a great idea for something you should blog about and put in other cookbooks first.

This is a blog - http://blogs.perl.org/users/melezhik/2016/01/testing-chef-coobooks-using-swat-and-test-kitchen.html

Right now though these examples don't offer anything over serverspec or inspec testing,

I read about inspec. Inspec is good, and I definitely see why opscode moves toward it, but swat still add extra value in compassion with inspec. It is not obvious at all in a such trivial example as for the httpd cookbook test suite and probably such a comparison is beyond this issue , but believe me swat does some extra value, so if you interested - I will gladly share details with you.

it is bad idea to introduce it into cookbooks like this first.

Which cookbooks do I need to start with then? :-) Can't see why not start httpd with? I have chosen it because it relates to web server configuration / installation. Swat is web test specific tool, it could be a great fit ... The only reasons that I can understand that swat is not that convenient for chef eco system as inspec is. BTW I have not seen inspec tests for httpd cookbook yet ...

[iennae]

@melezhik,

Your swat functionality is not going to be added to this cookbook at this time.

• It's functionality has limited use at current time.
• It is functionality that only you have knowledge of at this time, meaning it has fragility in terms of incorporating into core projects. You are a single point of knowledge. (SPOK)
• It's written in perl, which leads to additional complexity in terms of additional folks supporting it. Ruby and shell are both commonly known in the Chef community in writing cookbooks. This may impact it's adoption and use. Personally, I love perl, so it's not a ding against using perl, just the ability to remedy the SPOK situation.

While to you this may seem like the next logical step, as maintainers for this cookbook, we have to consider the costs and benefits to the whole community who depend on this cookbook to just work. This is too costly of an addition to this cookbook at this time.

I encourage you to continue blogging about it, and getting additional support from the community. I also encourage you to create and release quality cookbooks to the community on the supermarket: supermarket.chef.io that include your testing framework. As support grows for your cookbooks, this will lead to additional use.

Once you are not the SPOK on this project, and there is more exposure to the project we can revisit this issue.

Thanks for your understanding.

Jennifer

[melezhik]

@iennae , thanks for reply. Ok, I do see your points. Keep in an eye on swat though 😃 , chef is a great tool, but I believe that other good tools have to go together to make synergy.