New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to test recipes that require multiple VMs connected to a chef server #184

Closed
b-dean opened this Issue Aug 14, 2013 · 31 comments

Comments

Projects
None yet
@b-dean

b-dean commented Aug 14, 2013

Some cookbooks, such as opscode-cookbooks/jenkins, have recipes that require more than one chef node. There is no good way to test those with test-kitchen as the combinations of platforms and suites only results in a chef run happening on a single VM (at a time or in parallel).

If one were testing these with a Vagrantfile, then the VMs could all be listed in the Vagrantfile in the correct order to run the tests. But then you lose all the advantages of test-kitchen for running the same suite on a bunch of different platforms.

I don't really have any ideas how to solve this problem but I didn't see any issues specifically addressing it either.

Here's a more concrete scenario. Say you wanted to test the jenkins cookbook's ability to have one chef node be a jenkins master and one be a jenkins slave. There are examples of the run lists for jenkins servers in the jenkins cookbook's .kitchen.yml so I'm not going to type them here but the slave might look something like:

suites:
- name: ssh_slave_node
  run_list:
  - recipe[jenkins::node_ssh]
  attributes:
    jenkins:
      server:
        # IP address for the master
        host: 33.33.33.10

Any thoughts on how to test these kinds of recipes?

@jtimberman

This comment has been minimized.

Show comment
Hide comment
@jtimberman

jtimberman Aug 17, 2013

Contributor

Test Kitchen has always been designed and intended to run a single project's integration tests in an isolated environment. It has never been a goal that test kitchen is used to build a "test cluster" with multiple nodes. I argue that is the purpose of having a non-production environment, and that test-kitchen is used to test that the base functionality and design of a project (cookbook) works. While a lot of people ask about this kind of functionality, I think it is more important that test kitchen remain a test running primitive that doesn't try to be everything to everyone within the core project. Perhaps this is where a community plugin is appropriate, and if the test kitchen core features/API doesn't make it easy to do that, perhaps then we can revisit for the test-kitchen project itself.

@fnichol, @schisamo, or @sethvargo any additional thoughts?

Contributor

jtimberman commented Aug 17, 2013

Test Kitchen has always been designed and intended to run a single project's integration tests in an isolated environment. It has never been a goal that test kitchen is used to build a "test cluster" with multiple nodes. I argue that is the purpose of having a non-production environment, and that test-kitchen is used to test that the base functionality and design of a project (cookbook) works. While a lot of people ask about this kind of functionality, I think it is more important that test kitchen remain a test running primitive that doesn't try to be everything to everyone within the core project. Perhaps this is where a community plugin is appropriate, and if the test kitchen core features/API doesn't make it easy to do that, perhaps then we can revisit for the test-kitchen project itself.

@fnichol, @schisamo, or @sethvargo any additional thoughts?

@damm

This comment has been minimized.

Show comment
Hide comment
@damm

damm Aug 18, 2013

Pardon my unasked for thoughts.

It would be better if you used chef-zero and bootstrapped a jenkins node and then had your chef cookbook use search to find the artifact you had just created for the test environment. To expand on that.

In an ideal world, we would test with the artifacts we create, we would not use external artifacts outside of the testing system. Otherwise you assume 33.33.33.10 is always configured right (and that is not a desirable assumption)

damm commented Aug 18, 2013

Pardon my unasked for thoughts.

It would be better if you used chef-zero and bootstrapped a jenkins node and then had your chef cookbook use search to find the artifact you had just created for the test environment. To expand on that.

In an ideal world, we would test with the artifacts we create, we would not use external artifacts outside of the testing system. Otherwise you assume 33.33.33.10 is always configured right (and that is not a desirable assumption)

@b-dean

This comment has been minimized.

Show comment
Hide comment
@b-dean

b-dean Aug 19, 2013

@damm I was not really intending to assume that 33.33.33.10 was configured correctly. Maybe I wasn't clear in my original post but I wanted 33.33.33.10 and some other VM to both be configured in the same "multi vm platform" or whatever it would be called. The way I've been doing these sort of tests currently is just to not deal with test-kitchen at all and have a Vagrantfile with two VMs defined with the appropriate recipes in each, with :chef_client provisioners connected to a chef-zero server. I don't really like that some tests are that and some are the arguably better, kitchen tests.

@jtimberman I get where you're coming from on this. Ideally each combination of platform and suite would be a single convergence on one VM. But there is a lot of client/server software out there that where it would be very difficult to test how chef sets up the client portion (the jenkins slave in this instance) when the server does not exist. Because of this, cookbooks like the jenkins cookbook only have tests for the server node and all the client node's recipes are untested.

The reason I even ask this is because I have some fixes I'd like to contribute to that opscode-cookbook/jenkins repo but the lack of tests for recipes like jenkins::node_ssh or jenkins::node_jnlp, make me uncomfortable. I'd be interested to know what @schisamo thinks too since he wrote the existing tests for that cookbook.

b-dean commented Aug 19, 2013

@damm I was not really intending to assume that 33.33.33.10 was configured correctly. Maybe I wasn't clear in my original post but I wanted 33.33.33.10 and some other VM to both be configured in the same "multi vm platform" or whatever it would be called. The way I've been doing these sort of tests currently is just to not deal with test-kitchen at all and have a Vagrantfile with two VMs defined with the appropriate recipes in each, with :chef_client provisioners connected to a chef-zero server. I don't really like that some tests are that and some are the arguably better, kitchen tests.

@jtimberman I get where you're coming from on this. Ideally each combination of platform and suite would be a single convergence on one VM. But there is a lot of client/server software out there that where it would be very difficult to test how chef sets up the client portion (the jenkins slave in this instance) when the server does not exist. Because of this, cookbooks like the jenkins cookbook only have tests for the server node and all the client node's recipes are untested.

The reason I even ask this is because I have some fixes I'd like to contribute to that opscode-cookbook/jenkins repo but the lack of tests for recipes like jenkins::node_ssh or jenkins::node_jnlp, make me uncomfortable. I'd be interested to know what @schisamo thinks too since he wrote the existing tests for that cookbook.

@kisoku

This comment has been minimized.

Show comment
Hide comment
@kisoku

kisoku Aug 21, 2013

There are plenty of valid test scenarios inside a library cookbook where multi-vm is a desirable feature to have. Any library cookbook for clustered or clustering services would benefit greatly from having this feature, like pacemaker or elasticsearch or jenkins.

In the jenkins case simply providing a mocked up node for a search to succeed is not sufficient, as some of the node providers requires a functioning jenkins server to register against before the converge can succeed.

kisoku commented Aug 21, 2013

There are plenty of valid test scenarios inside a library cookbook where multi-vm is a desirable feature to have. Any library cookbook for clustered or clustering services would benefit greatly from having this feature, like pacemaker or elasticsearch or jenkins.

In the jenkins case simply providing a mocked up node for a search to succeed is not sufficient, as some of the node providers requires a functioning jenkins server to register against before the converge can succeed.

@sethvargo

This comment has been minimized.

Show comment
Hide comment
@sethvargo

sethvargo Aug 21, 2013

Contributor

Ohai cheffers! We had an internal discussion about this ticket today and I wanted to relay the outcome of that conversation. We don't deny that this feature would be useful in many testing scenarios. Even though Test Kitchen is an integration test for your cookbook, it's actually a unit test in the context of your infrastructure; it's purpose is to test a single unit (cookbook) of your infrastructure. However, given that Test Kitchen is a highly modular and already has a plugin architecture, this functionality should be added via an external, community-maintained plugin. We do not feel comfortable having this functionality in the core of Test Kitchen, as it's not in the original goals of the project.

On possibility would be to use chef-zero and mock data loaded for use with search, etc. I encourage those of you interested in this functionality to collaborate and create a new Test Kitchen plugin. If Opscode can be of any assistance in this process, please feel free to ping me 😄.

Contributor

sethvargo commented Aug 21, 2013

Ohai cheffers! We had an internal discussion about this ticket today and I wanted to relay the outcome of that conversation. We don't deny that this feature would be useful in many testing scenarios. Even though Test Kitchen is an integration test for your cookbook, it's actually a unit test in the context of your infrastructure; it's purpose is to test a single unit (cookbook) of your infrastructure. However, given that Test Kitchen is a highly modular and already has a plugin architecture, this functionality should be added via an external, community-maintained plugin. We do not feel comfortable having this functionality in the core of Test Kitchen, as it's not in the original goals of the project.

On possibility would be to use chef-zero and mock data loaded for use with search, etc. I encourage those of you interested in this functionality to collaborate and create a new Test Kitchen plugin. If Opscode can be of any assistance in this process, please feel free to ping me 😄.

@sethvargo sethvargo closed this Aug 21, 2013

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 23, 2014

Any change of heart in the last 6 months on this topic?

drnic commented Feb 23, 2014

Any change of heart in the last 6 months on this topic?

@sethvargo

This comment has been minimized.

Show comment
Hide comment
@sethvargo

sethvargo Feb 23, 2014

Contributor

@drnic we are working on Chef metal (and then Chef metal integration with TK) that will allow for a solution to this problem in the near future.

Contributor

sethvargo commented Feb 23, 2014

@drnic we are working on Chef metal (and then Chef metal integration with TK) that will allow for a solution to this problem in the near future.

@spheromak

This comment has been minimized.

Show comment
Hide comment
@spheromak

spheromak Feb 23, 2014

yay for chef-metal 👍

spheromak commented Feb 23, 2014

yay for chef-metal 👍

@damm

This comment has been minimized.

Show comment
Hide comment
@damm

damm Feb 23, 2014

hmmm, I wonder if we could use Drone.io in this

damm commented Feb 23, 2014

hmmm, I wonder if we could use Drone.io in this

@coderanger

This comment has been minimized.

Show comment
Hide comment
@coderanger

coderanger Feb 25, 2014

Contributor

Leaving room for other tools like CloudFormation would be good too.

Contributor

coderanger commented Feb 25, 2014

Leaving room for other tools like CloudFormation would be good too.

@simonmcc

This comment has been minimized.

Show comment
Hide comment
@simonmcc

simonmcc Mar 1, 2014

Isn't this what @LordCope's Leibniz is supposed to help with?

simonmcc commented Mar 1, 2014

Isn't this what @LordCope's Leibniz is supposed to help with?

@Atalanta

This comment has been minimized.

Show comment
Hide comment
@Atalanta

Atalanta Mar 3, 2014

Member

All Leibniz does is use the kitchenci API to launch machines, and provide the ability to specify a list of machines, run lists etc in cucmber examples.

I think this is actually a good approach - and encourages people to get on with writing acceptance tests.

How would chef-metal help?

Member

Atalanta commented Mar 3, 2014

All Leibniz does is use the kitchenci API to launch machines, and provide the ability to specify a list of machines, run lists etc in cucmber examples.

I think this is actually a good approach - and encourages people to get on with writing acceptance tests.

How would chef-metal help?

@phundisk

This comment has been minimized.

Show comment
Hide comment
@phundisk

phundisk Dec 9, 2014

+1 on having a feature like this. It would be nice to be able to test a cluster ability with test kitchen.

phundisk commented Dec 9, 2014

+1 on having a feature like this. It would be nice to be able to test a cluster ability with test kitchen.

@felka

This comment has been minimized.

Show comment
Hide comment
@felka

felka Dec 10, 2014

+1 . More and more cookbooks require cluster of nodes - consul, zookeeper, cassandra, kafka and etc.
This feature will really help!

felka commented Dec 10, 2014

+1 . More and more cookbooks require cluster of nodes - consul, zookeeper, cassandra, kafka and etc.
This feature will really help!

@acqant

This comment has been minimized.

Show comment
Hide comment
@acqant

acqant Jan 8, 2015

+1
2015 bump!

acqant commented Jan 8, 2015

+1
2015 bump!

@franklinwise

This comment has been minimized.

Show comment
Hide comment
@franklinwise

franklinwise commented Jan 25, 2015

+1 Yes!

@kisoku

This comment has been minimized.

Show comment
Hide comment
@kisoku

kisoku Jan 26, 2015

If someome could revive Doug Triggs' chef-metal^Wprovisioning provider we
could have this :-)

On Mon, Jan 26, 2015 at 6:21 PM, Stephen Nelson-Smith <
notifications@github.com> wrote:

This is something I'm going to need too, so I'll certainly put some effort
into this.


Reply to this email directly or view it on GitHub
#184 (comment)
.

Mathieu Sauve-Frankel

kisoku commented Jan 26, 2015

If someome could revive Doug Triggs' chef-metal^Wprovisioning provider we
could have this :-)

On Mon, Jan 26, 2015 at 6:21 PM, Stephen Nelson-Smith <
notifications@github.com> wrote:

This is something I'm going to need too, so I'll certainly put some effort
into this.


Reply to this email directly or view it on GitHub
#184 (comment)
.

Mathieu Sauve-Frankel

@acqant

This comment has been minimized.

Show comment
Hide comment
@acqant

acqant Jan 26, 2015

Couldn't you have multiple "suites" with different run_lists?

---
driver:
    name: vagrant
    require_chef_omnibus: 11.16.4-1
    associate_public_ip: true

provisioner:
  name: chef_zero
  environments_path: ../cb-base/test/integration/PROD/environments
  data_bags_path:    ../cb-base/test/integration/PROD/data_bags
  nodes_path:        ../cb-base/test/integration/PRODnodes

platforms:
- name: centos-5.7
  driver:
    box: centos-5.7
    box_url: http://opscode-vm-bento.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.7.box

suites:
  - name: client
    driver:
      vm_hostname: client.domain.com
      network:
      #- ["public_network", {ip: "192.168.1.200"}]
    run_list:
      - recipe[client]
    provisioner:
      client_rb:
        environment: PROD
  - name: server
    driver:
      vm_hostname: server.domain.com
      run_list:
      - recipe[server_stuff
    attributes:
        chef_client:
            config:
                force_logger: true
                log_level: info
    provisioner:
      client_rb:
        environment: PROD

acqant commented Jan 26, 2015

Couldn't you have multiple "suites" with different run_lists?

---
driver:
    name: vagrant
    require_chef_omnibus: 11.16.4-1
    associate_public_ip: true

provisioner:
  name: chef_zero
  environments_path: ../cb-base/test/integration/PROD/environments
  data_bags_path:    ../cb-base/test/integration/PROD/data_bags
  nodes_path:        ../cb-base/test/integration/PRODnodes

platforms:
- name: centos-5.7
  driver:
    box: centos-5.7
    box_url: http://opscode-vm-bento.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.7.box

suites:
  - name: client
    driver:
      vm_hostname: client.domain.com
      network:
      #- ["public_network", {ip: "192.168.1.200"}]
    run_list:
      - recipe[client]
    provisioner:
      client_rb:
        environment: PROD
  - name: server
    driver:
      vm_hostname: server.domain.com
      run_list:
      - recipe[server_stuff
    attributes:
        chef_client:
            config:
                force_logger: true
                log_level: info
    provisioner:
      client_rb:
        environment: PROD

@acqant

This comment has been minimized.

Show comment
Hide comment
@acqant

acqant Jan 26, 2015

More than anything I'd like the these to be git resources or chef server urls

environments_path: ../cb-base/test/integration/environments
data_bags_path: ../cb-base/test/integration/data_bags
nodes_path: ../cb-base/test/integration//prod/nodes

acqant commented Jan 26, 2015

More than anything I'd like the these to be git resources or chef server urls

environments_path: ../cb-base/test/integration/environments
data_bags_path: ../cb-base/test/integration/data_bags
nodes_path: ../cb-base/test/integration//prod/nodes

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Apr 26, 2015

@acqant The problem with your approach would be the interoperability of those isolated nodes.
Maybe be should play around with sharing folders of the chef-zero instance to solve this issue.

@sethvargo How can chef-provisioning help to solve this issue?

ghost commented Apr 26, 2015

@acqant The problem with your approach would be the interoperability of those isolated nodes.
Maybe be should play around with sharing folders of the chef-zero instance to solve this issue.

@sethvargo How can chef-provisioning help to solve this issue?

@Radagaisus

This comment has been minimized.

Show comment
Hide comment
@Radagaisus

Radagaisus May 25, 2015

+1. This would be amazing. And wonderful. And other superlatives.

Radagaisus commented May 25, 2015

+1. This would be amazing. And wonderful. And other superlatives.

@smecsia

This comment has been minimized.

Show comment
Hide comment
@smecsia

smecsia Jun 4, 2015

+1. We would love to see this feature as well!

smecsia commented Jun 4, 2015

+1. We would love to see this feature as well!

@metmajer

This comment has been minimized.

Show comment
Hide comment
@metmajer

metmajer commented Aug 17, 2015

+1

@foulowl

This comment has been minimized.

Show comment
Hide comment
@foulowl

foulowl commented Sep 9, 2015

+1

@builtinnya

This comment has been minimized.

Show comment
Hide comment
@builtinnya

builtinnya commented Oct 5, 2015

+1

@ndamours

This comment has been minimized.

Show comment
Hide comment
@ndamours

ndamours Oct 15, 2015

:( I stumbled on this thread while looking for a way to provide kitchen with multiple suites to test my cluster. I am writing a cookbook for a few node cluster used to run an MPI application.

+1

ndamours commented Oct 15, 2015

:( I stumbled on this thread while looking for a way to provide kitchen with multiple suites to test my cluster. I am writing a cookbook for a few node cluster used to run an MPI application.

+1

@conorsch

This comment has been minimized.

Show comment
Hide comment
@conorsch

conorsch Nov 25, 2015

+1 Great discussion here. Sounds like I'm going to need a different tool to test multihost roles.

conorsch commented Nov 25, 2015

+1 Great discussion here. Sounds like I'm going to need a different tool to test multihost roles.

@mingzhaodotname

This comment has been minimized.

Show comment
Hide comment
@mingzhaodotname

mingzhaodotname Feb 12, 2016

One trick that I used is to create an internal network between those VMs, converge all of them, and then verify all of them. E.g.

  1. kitchen destroy vm1; kitchen destroy vm2
  2. kitchen converge vm1; kitchen converge vm2
  3. kitchen verify vm1; kitchen verify vm2
  4. kitchen destroy vm1; kitchen destroy vm2

mingzhaodotname commented Feb 12, 2016

One trick that I used is to create an internal network between those VMs, converge all of them, and then verify all of them. E.g.

  1. kitchen destroy vm1; kitchen destroy vm2
  2. kitchen converge vm1; kitchen converge vm2
  3. kitchen verify vm1; kitchen verify vm2
  4. kitchen destroy vm1; kitchen destroy vm2
@marutinandanpandya

This comment has been minimized.

Show comment
Hide comment
@marutinandanpandya

marutinandanpandya commented Jun 8, 2016

+1

@alexharv074

This comment has been minimized.

Show comment
Hide comment
@alexharv074

alexharv074 commented Jun 12, 2016

+1

@cheeseplus

This comment has been minimized.

Show comment
Hide comment
@cheeseplus

cheeseplus Jun 12, 2016

Contributor

Hey folks, issue is closed and there is a feature issue for this here #873. Adding +1s after a certain point just becomes noise.

We totally want to add cluster/multi-node support but it's a fairly large feature with lots of fixes in front of it.

Contributor

cheeseplus commented Jun 12, 2016

Hey folks, issue is closed and there is a feature issue for this here #873. Adding +1s after a certain point just becomes noise.

We totally want to add cluster/multi-node support but it's a fairly large feature with lots of fixes in front of it.

@test-kitchen test-kitchen locked and limited conversation to collaborators Jun 13, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.