Fix #195: Acceptance tests based on Cucumber #205

hferentschik · 2016-05-09T13:29:33Z

To execute the Cucumber tests:

$ bundle install
$ bundle exec rake get_adb
$ bundle exec rake get_cdk
$ bundle exec rake features

Reports in build/features_report.html.

Alternatively one can manually copy the box files to the expected directory (see README)

I switched from vagrant-spec to Cucumber, since the former seems to be lest maintained and the syntax and reporting of Cucumber/Gherkin is much better.

Thoughts?

coolbrg · 2016-05-09T13:48:39Z

README.md

+    $ bundle exec rake -T
+
+<a name="acceptance-tests"></a>
+## Acceptance tests


I think Acceptance Tests would be more appropriate here.

Why? You mean a capital 'T'? I don't think so?

coolbrg · 2016-05-09T14:08:40Z

@hferentschik Implementation looks great 👍 . Need to test it locally.

coolbrg · 2016-05-09T14:10:39Z

README.md


+- [Objective](#objective)


@bexelbie @Preeticp Please have a look at this new TOC with nested feature.

hferentschik · 2016-05-09T19:53:52Z

@budhrg thanks for the review.

I fixed the typo and made some cosmetic changes to the Rakefile and added a features:open_report open report task.

Regarding the Gemfile.lock, I think I still prefer to lock the version for development and mitigate gem file incompatibilities via a CI job and dedicated version checks prior to releases. If we agree on the other changes, we can drop the last commit for now and apply the test changes first in order to move on.

hferentschik · 2016-05-09T19:54:24Z

Implementation looks great 👍 .

thanks

Need to test it locally.

Let me know how it goes.

hferentschik · 2016-05-09T19:57:05Z

@dharmit so what you I need to do now to get access to a CentOS CI server?

hferentschik · 2016-05-11T06:57:05Z

Ok, I removed the checkin of the Gemfile.lock, we can take this discussion later. I also added support for testing against libvirt when available.

Maybe we can get this merged now. After all this is "just" a test framework. We can improve/adjust over time. I just don't won't to have to rebase forever.

coolbrg · 2016-05-11T07:13:48Z

README.md

+_build/boxes/adb-\<provider\>.box_ resp _build/boxes/cdk-\<provider\>.box_. You can
+either copy the box files manually or use the _get_adb_ resp. _get_cdk_ Rake tasks.
+
+Per default the VirtualBox provider is used. However, you can also run against Libvirt


Can we rewrite it as Virtualbox provider is used as default provider.

Why, the default spelling is VirtualBox as far as I know.

I mean to say Let's rewrite this whole sentence as:

VirtualBox is used as a default provider. However, you can also run against Libvirt

As starting sentence using Per default doesn't sound good to me. However, I would like @bexelbie and @Preeticp to review this README since they are mostly handling docs for ADB.

LalatenduMohanty · 2016-05-11T14:13:08Z

README.md

-You can contact us on:
-  * IRC: #atomic and #nulecule on freenode
-  * Mailing List: container-tools@redhat.com
+After cloing the repository, install the [Bundler](http://bundler.io/) gem:


Spell correction s/cloing/cloning/

hferentschik · 2016-05-12T08:27:41Z

For a reference, here is the bug report for running Aruba on Windows - cucumber/aruba#387. Seems the project is still quite active and there is definitely interest to fix this. It might be ok to go with the Cucmber + Aruba approach under the assumption/hope that there will be a fix to the problem soonish.

coolbrg · 2016-05-12T14:37:00Z

README.md

+    # Run tests against CDK using Libvirt
+    $ bundle exec rake features BOX=cdk PROVIDER=libvirt
+
+    # Run against ADB and CDK


probably Run against ADB and/or CDK. As one can give one or both names.

Sure, but the example is for the 'AND' case

Made a small change

hferentschik · 2016-05-12T14:37:47Z

Pushed an update which allows to mix and match which features are run against which provider and box usinf env variables FEATURE, BOX and PROVIDER. See README. Also made IP part of the test scenario

bexelbie · 2016-05-12T14:38:25Z

README.md

+    $ bundle exec rake features:open_report
+
+<a name="builds"></a>
+## Builds


This section can be deleted, imho.

yeah :-) I wanted to ask about this. I do it right away!

bexelbie · 2016-05-12T14:39:11Z

README LGTM with note about possible duplicative section deletion.

coolbrg · 2016-05-12T15:14:07Z

@hferentschik it looks like you took the oldest commit while squashing.

Date of commit is April 25 now.

commit 1e65f76
Author: Hardy Ferentschik hardy@hibernate.org
Date: Mon Apr 25 14:43:26 2016 +0200

coolbrg · 2016-05-12T15:19:50Z

README.md

+    $ bundle exec rake features BOX=cdk PROVIDER=libvirt
+
+    # Run against ADB and CDK (boxes are comma seperated)
+    $ budle exec rake features BOX=cdk,adb


Typo error @hferentschik

coolbrg · 2016-05-13T05:25:58Z

LGTM 👍

➜  vagrant-service-manager git:(hardy_issue_195) ✗ bundle exec rake features BOX=adb,cdk
/home/budhram/.rvm/rubies/ruby-2.2.1/bin/ruby -S bundle exec cucumber 
Using the default and html profiles...
...................--------.--------.........................................-------------------.-------------------.......................----------.----------...........---------...............------.------
18 scenarios (9 skipped, 9 passed)
208 steps (95 skipped, 113 passed)
13m28.174s

LalatenduMohanty · 2016-05-13T05:58:49Z

The CDK tests on libvirt worked fine for me . Trying out it on OSX

[lmohanty@LalatenduM-laptop vagrant-service-manager]$ bundle exec rake features BOX=cdk PROVIDER=libvirt
Your Gemfile lists the gem vagrant-service-manager (>= 0) more than once.
You should probably keep only one of them.
While it's not a problem now, it could cause errors if you change the version of just one of them later.

/usr/bin/ruby -S bundle exec cucumber 
Using the default and html profiles...
.-----------------.........---------.---------------------------------------....................--------------------.---------------------...........-----------.---------...........-------------.......-------

18 scenarios (13 skipped, 5 passed)
208 steps (146 skipped, 62 passed)
4m27.067s

coolbrg · 2016-05-13T06:00:31Z

The CDK tests on libvirt worked fine for me

@LalatenduMohanty Could you run tests with both?

bundle exec rake features BOX=adb,cdk PROVIDER=libvirt

LalatenduMohanty · 2016-05-13T06:03:52Z

Could you run tests with both?

I don't have vagrant box for ADB on my local workstation (the way test wants it) and downloading going to take time :(

coolbrg · 2016-05-13T06:04:33Z

@LalatenduMohanty ok will check with ADB - libvirt combination and merge this PR.

LalatenduMohanty · 2016-05-13T06:20:14Z

Got a ADB 1.8.0 box and it seems to work

[lmohanty@LalatenduM-laptop vagrant-service-manager]$ bundle exec rake features BOX=adb PROVIDER=libvirt
Your Gemfile lists the gem vagrant-service-manager (>= 0) more than once.
You should probably keep only one of them.
While it's not a problem now, it could cause errors if you change the version of just one of them later.

/usr/bin/ruby -S bundle exec cucumber 
Using the default and html profiles...
---------.-----------------.........--------------------.---------------------------------------....................-----------.---------------------...........---------------------------.-------------.......

18 scenarios (14 skipped, 4 passed)
208 steps (157 skipped, 51 passed)
2m46.741s

hferentschik · 2016-05-13T06:30:58Z

Is there a way where I can use aleady installed Vagrant box for the tests? For example I should be able able to use config.vm.box = "projectatomic/adb"

Right now, no. It could be done, but then you need different Vagrantfile (one w/ URL and one w/o) for each box or work out. Makes the setup more complicated and you most likely need to duplicate test code. Hence I opted against it. Effectively you are getting the ADB box from Hashicorp anyways. That's where it is downloaded from, if you use the download task.

I am also a big fan of keeping everything w/i the project, which includes tmp artifacts, tmp files and directories, etc. If I delete pkg and build, there should be nothing left which got created by any development tasks.

This will save some internet bandwidth for me as I dont have to re-download box again

Come on, since when do we care re-downloading something like this. I bet you you have downloaded the box more than once already. So yes, the current approach downloads and keeps the box file local to the project, but until you delete the build (run clean task) directory it will be used from there.

I also think it is in general a bad idea to interact with anything in ~/.vagrant. For example, when experimenting next with moving config into the box Vagrantfile, I will modify the Vagrantfile in ~/.vagrant. If at the same time I would run tests against the CDK using the box in ~/.vagrant I might get unexpected results.

Also definitely a -1 for trying to re-package the box.

- Adding required dependencies - Adding tests - Updating REDAME on how to run tests

LalatenduMohanty · 2016-05-13T07:50:31Z

That's where it is downloaded from, if you use the download task.

It is not exactly the same. Basically if I use atlas I get un-compressed box. But if I use cloud.c.o I get the whole box.

LalatenduMohanty · 2016-05-13T07:53:33Z

bundle install giving error on OS X as I think because gemfile has fog-libvirt

$ bundle install
Your Gemfile lists the gem vagrant-service-manager (>= 0) more than once.
You should probably keep only one of them.
While it's not a problem now, it could cause errors if you change the version of just one of them later.
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 11.1.1


Your user account isn't allowed to install to the system Rubygems.
You can cancel this installation and run:

    bundle install --path vendor/bundle

to install the gems into ./vendor/bundle/, or you can enter your password
and install the bundled gems to Rubygems using sudo.

Password: 
Installing addressable 2.4.0
Using ffi 1.9.10
Using childprocess 0.5.9
Installing contracts 0.14.0
Using builder 3.2.2
Installing gherkin 3.2.0
Installing cucumber-core 1.4.0
Installing cucumber-wire 0.0.1
Using diff-lcs 1.2.5
Using multi_json 1.11.2
Installing multi_test 0.1.2
Installing cucumber 2.3.3
Installing rspec-support 3.4.1
Installing rspec-expectations 3.4.0
Installing thor 0.19.1
Installing aruba 0.14.1
Using bundler 1.8.0
Using unf_ext 0.0.7.2
Using unf 0.1.4
Using domain_name 0.5.20160309
Using erubis 2.7.0
Installing events 0.9.8
Installing excon 0.49.0
Installing formatador 0.2.5
Installing fog-core 1.39.0
Installing fog-json 1.0.2
Using mini_portile2 2.0.0
Using nokogiri 1.6.7.1
Installing fog-xml 0.1.2
Using json 1.8.3

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby extconf.rb 
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
    --with-libvirt-include
    --without-libvirt-include
    --with-libvirt-lib
    --without-libvirt-lib
    --with-libvirt-config
    --without-libvirt-config
    --with-pkg-config
    --without-pkg-config
extconf.rb:73:in `<main>': libvirt library not found in default locations (RuntimeError)


Gem files will remain installed in /var/folders/81/q3z841ds2yb7xblc91xww10m0000gn/T/bundler20160513-5213-1xln3oc/ruby-libvirt-0.6.0/gems/ruby-libvirt-0.6.0 for inspection.
Results logged to /var/folders/81/q3z841ds2yb7xblc91xww10m0000gn/T/bundler20160513-5213-1xln3oc/ruby-libvirt-0.6.0/gems/ruby-libvirt-0.6.0/ext/libvirt/gem_make.out
An error occurred while installing ruby-libvirt (0.6.0), and Bundler cannot continue.
Make sure that `gem install ruby-libvirt -v '0.6.0'` succeeds before bundling.

navidshaikh · 2016-05-13T10:21:13Z

Can we open a separate ticket for issues we are finding for OS X and get this PR merged for Linux platform? This should unblock us to have a CI job first and we keep improving the tests for other platforms as well?

LalatenduMohanty · 2016-05-13T10:21:49Z

@navidshaikh lets do that.

navidshaikh · 2016-05-13T10:24:18Z

Well, I should have merged using command line, this github UI merge will add one more commit in history. :-( Next time!

hferentschik · 2016-05-13T12:19:40Z

LOL, thanks anyway

hferentschik · 2016-05-13T12:20:26Z

This button can be so tempting ;-)

hferentschik force-pushed the issue-195 branch from a8fcd32 to 4ac16a5 Compare May 9, 2016 13:31

coolbrg reviewed May 9, 2016
View reviewed changes

coolbrg changed the title ~~Issue 195 Accepatance tests based on Cucumber~~ Fix #195 Accepatance tests based on Cucumber May 9, 2016

coolbrg changed the title ~~Fix #195 Accepatance tests based on Cucumber~~ Fix #195: Accepatance tests based on Cucumber May 9, 2016

coolbrg reviewed May 9, 2016
View reviewed changes

README.md

- [Objective](#objective)

Copy link

Contributor

coolbrg May 9, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bexelbie @Preeticp Please have a look at this new TOC with nested feature.

hferentschik force-pushed the issue-195 branch from 4ac16a5 to 4e42461 Compare May 9, 2016 19:46

hferentschik force-pushed the issue-195 branch from 4e42461 to 3d81999 Compare May 11, 2016 06:53

hferentschik force-pushed the issue-195 branch 2 times, most recently from 38eb107 to 421327d Compare May 11, 2016 07:10

coolbrg reviewed May 11, 2016
View reviewed changes

hferentschik force-pushed the issue-195 branch from 421327d to 7c9f89b Compare May 11, 2016 10:08

LalatenduMohanty reviewed May 11, 2016
View reviewed changes

hferentschik force-pushed the issue-195 branch from 7c9f89b to b72c8c2 Compare May 11, 2016 18:33

hferentschik changed the title ~~Fix #195: Accepatance tests based on Cucumber~~ Fix #195: Acceptance tests based on Cucumber May 12, 2016

hferentschik force-pushed the issue-195 branch from b72c8c2 to 796cab5 Compare May 12, 2016 14:31

coolbrg reviewed May 12, 2016
View reviewed changes

bexelbie reviewed May 12, 2016
View reviewed changes

hferentschik force-pushed the issue-195 branch 2 times, most recently from 93b17c3 to 1e65f76 Compare May 12, 2016 14:49

coolbrg reviewed May 12, 2016
View reviewed changes

Issue projectatomic#195 Adding Cucumber and Aruba based acceptance tests

d396843

- Adding required dependencies - Adding tests - Updating REDAME on how to run tests

hferentschik force-pushed the issue-195 branch from 1e65f76 to d396843 Compare May 13, 2016 06:35

navidshaikh merged commit b867917 into projectatomic:master May 13, 2016

Fix #195: Acceptance tests based on Cucumber #205

Fix #195: Acceptance tests based on Cucumber #205

Conversation

hferentschik commented May 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coolbrg commented May 9, 2016 • edited

Choose a reason for hiding this comment

hferentschik commented May 9, 2016

hferentschik commented May 9, 2016

hferentschik commented May 9, 2016

hferentschik commented May 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hferentschik commented May 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hferentschik commented May 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bexelbie commented May 12, 2016

coolbrg commented May 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coolbrg commented May 13, 2016 • edited

LalatenduMohanty commented May 13, 2016 • edited

coolbrg commented May 13, 2016

LalatenduMohanty commented May 13, 2016 • edited

coolbrg commented May 13, 2016

LalatenduMohanty commented May 13, 2016 • edited

hferentschik commented May 13, 2016

LalatenduMohanty commented May 13, 2016 • edited

LalatenduMohanty commented May 13, 2016 • edited

navidshaikh commented May 13, 2016

LalatenduMohanty commented May 13, 2016 • edited

navidshaikh commented May 13, 2016

hferentschik commented May 13, 2016

hferentschik commented May 13, 2016

coolbrg commented May 9, 2016 •

edited

coolbrg commented May 13, 2016 •

edited

LalatenduMohanty commented May 13, 2016 •

edited

LalatenduMohanty commented May 13, 2016 •

edited

LalatenduMohanty commented May 13, 2016 •

edited

LalatenduMohanty commented May 13, 2016 •

edited

LalatenduMohanty commented May 13, 2016 •

edited

LalatenduMohanty commented May 13, 2016 •

edited