Execute integration tests in DigitalOcean #144

Merged
merged 1 commit into from Feb 27, 2017

Conversation

Projects
None yet
3 participants
@artem-sidorenko
Member

artem-sidorenko commented Feb 16, 2017

Focus of this PR is only to get the travis-kitchen-DO setup running. I'll fix the failures in the next PRs

## Communication
### GitHub repositories
Much of the issues, goals and ideas are tracked in the respective projects in GitHub. Please use this channel to report bugs and post ideas.
-### Trello

This comment has been minimized.

@artem-sidorenko

artem-sidorenko Feb 16, 2017

Member

I removed Trello as it looks dead, I hope thats fine

@artem-sidorenko

artem-sidorenko Feb 16, 2017

Member

I removed Trello as it looks dead, I hope thats fine

@artem-sidorenko

This comment has been minimized.

Show comment
Hide comment
@artem-sidorenko

artem-sidorenko Feb 16, 2017

Member

If changes in .travis.yml and the .kitchen.yml look reasonable, I will do the same/similar thing for ssh-hardening (with kitchen-dokken)

Member

artem-sidorenko commented Feb 16, 2017

If changes in .travis.yml and the .kitchen.yml look reasonable, I will do the same/similar thing for ssh-hardening (with kitchen-dokken)

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Feb 16, 2017

Coverage Status

Coverage decreased (-47.4%) to 52.632% when pulling 264c066 on do-travis into b774c1b on master.

coveralls commented Feb 16, 2017

Coverage Status

Coverage decreased (-47.4%) to 52.632% when pulling 264c066 on do-travis into b774c1b on master.

@artem-sidorenko

This comment has been minimized.

Show comment
Hide comment
@artem-sidorenko

artem-sidorenko Feb 16, 2017

Member

it looks like supermarket has some issues...

Member

artem-sidorenko commented Feb 16, 2017

it looks like supermarket has some issues...

@chris-rock

Hi @artem-sidorenko Great work, Digital Ocean config looks great. Is there a specific reason why you removed the dokken and vagrant kitchen support? This does not allow me to run integration tests locally anymore.

@@ -1,44 +0,0 @@
----
-driver:

This comment has been minimized.

@chris-rock

chris-rock Feb 16, 2017

Member

I would prefer to have:

  • .kitchen.yml - Docker-based
  • .kitchen.vagrant.yml - vagrant based
  • .kitchen.digitalocean.yml - in digital ocean
    This would allow all users to run the tests locally as well.
@chris-rock

chris-rock Feb 16, 2017

Member

I would prefer to have:

  • .kitchen.yml - Docker-based
  • .kitchen.vagrant.yml - vagrant based
  • .kitchen.digitalocean.yml - in digital ocean
    This would allow all users to run the tests locally as well.

This comment has been minimized.

@artem-sidorenko

artem-sidorenko Feb 17, 2017

Member

well, I did not remove it, I tried to have only one .kitchen.yml:

  • DRY
  • currently kitchen-dokken does not work here

So the idea was to have only one kitchen.yml. Currently you will see locally the vagrant setup and it works without any export KITCHEN_YAML steps. In the CI you have the according DO env vars configured, so it works automatically with DO.

In the end its one .kitchen.yml and not two or three, where some of this files does not get updates usually.

I'm not sure yet how this should/could be implemented for chef-ssh-hardening, as somebody might use kitchen-dokken locally (what leads for my system to problems because of systemd) or some might use kitchen-vagrant locally. But this is a bit another discussion

Can you tell me where exactly do you see a problem with this approach? Do you have any suggestions how we can resolve that but in the same time keep one file?

@artem-sidorenko

artem-sidorenko Feb 17, 2017

Member

well, I did not remove it, I tried to have only one .kitchen.yml:

  • DRY
  • currently kitchen-dokken does not work here

So the idea was to have only one kitchen.yml. Currently you will see locally the vagrant setup and it works without any export KITCHEN_YAML steps. In the CI you have the according DO env vars configured, so it works automatically with DO.

In the end its one .kitchen.yml and not two or three, where some of this files does not get updates usually.

I'm not sure yet how this should/could be implemented for chef-ssh-hardening, as somebody might use kitchen-dokken locally (what leads for my system to problems because of systemd) or some might use kitchen-vagrant locally. But this is a bit another discussion

Can you tell me where exactly do you see a problem with this approach? Do you have any suggestions how we can resolve that but in the same time keep one file?

This comment has been minimized.

@chris-rock

chris-rock Feb 20, 2017

Member

I agree with your aim to merge the kitchen.yml into a more maintainable structure. We could try to use a base configuration and capture the provider in a local kitchen config (KITCHEN_LOCAL_YAML). This would allow us to:

  • kitchen.yml with vagant
  • kitchen.do.local.yml for DigitalOcean

The kitchen local approach does not work well with kitchen-dokken, since nearly everything needs to be overwritten. We could switch to kitchen-docker, but this works by installing ssh into the container while kitchen-dokken is not.

The initial idea of switching to kitchen-dokken was to do all integration tests in travis. Therefore kitchen-dokken was the best solution for all chef projects (except for os-hardening). It works within the travis without external dependencies and is for free. I agree that we need to have another solution for os-hardening since we are testing kernel parameters, but I am not sure we should switch all other projects.

My main goal is to have a self-contained project that can be fully tested locally. I like the digital ocean idea, since it increases test coverage. I do not think we should restrict testing to digital ocean.

@artem-sidorenko Thoughts?

@chris-rock

chris-rock Feb 20, 2017

Member

I agree with your aim to merge the kitchen.yml into a more maintainable structure. We could try to use a base configuration and capture the provider in a local kitchen config (KITCHEN_LOCAL_YAML). This would allow us to:

  • kitchen.yml with vagant
  • kitchen.do.local.yml for DigitalOcean

The kitchen local approach does not work well with kitchen-dokken, since nearly everything needs to be overwritten. We could switch to kitchen-docker, but this works by installing ssh into the container while kitchen-dokken is not.

The initial idea of switching to kitchen-dokken was to do all integration tests in travis. Therefore kitchen-dokken was the best solution for all chef projects (except for os-hardening). It works within the travis without external dependencies and is for free. I agree that we need to have another solution for os-hardening since we are testing kernel parameters, but I am not sure we should switch all other projects.

My main goal is to have a self-contained project that can be fully tested locally. I like the digital ocean idea, since it increases test coverage. I do not think we should restrict testing to digital ocean.

@artem-sidorenko Thoughts?

This comment has been minimized.

@artem-sidorenko

artem-sidorenko Feb 21, 2017

Member

@chris-rock I have the same view more or less. I will have a look how to rework it with kitchen local approach, its a good idea, thanks!

I agree that we need to have another solution for os-hardening since we are testing kernel parameters, but I am not sure we should switch all other projects.

I'm not sure as well, but lets have this discussion outside of this PR

@artem-sidorenko

artem-sidorenko Feb 21, 2017

Member

@chris-rock I have the same view more or less. I will have a look how to rework it with kitchen local approach, its a good idea, thanks!

I agree that we need to have another solution for os-hardening since we are testing kernel parameters, but I am not sure we should switch all other projects.

I'm not sure as well, but lets have this discussion outside of this PR

This comment has been minimized.

@artem-sidorenko

artem-sidorenko Feb 21, 2017

Member

@chris-rock I switched from erb approach to the kitchen local approach. Can you please have a look?

@artem-sidorenko

artem-sidorenko Feb 21, 2017

Member

@chris-rock I switched from erb approach to the kitchen local approach. Can you please have a look?

This comment has been minimized.

.travis.yml
+script:
+ - bundle exec rake prepare_do_env kitchen
+
+matrix:

This comment has been minimized.

@chris-rock

chris-rock Feb 16, 2017

Member

maybe we should use a syntax like:

matrix:
  include:
  - rvm: 2.3.3
    bundler_args: "--without guard tools"
    script: bundle exec rake $SUITE
    env: SUITE=test:integration OS=default-ubuntu-1204 DOCKER=true

reference: https://github.com/chef/inspec/blob/master/.travis.yml#L15-L31

@chris-rock

chris-rock Feb 16, 2017

Member

maybe we should use a syntax like:

matrix:
  include:
  - rvm: 2.3.3
    bundler_args: "--without guard tools"
    script: bundle exec rake $SUITE
    env: SUITE=test:integration OS=default-ubuntu-1204 DOCKER=true

reference: https://github.com/chef/inspec/blob/master/.travis.yml#L15-L31

This comment has been minimized.

@artem-sidorenko

artem-sidorenko Feb 17, 2017

Member

@chris-rock could you maybe explain it? Maybe I miss something: via your suggestion it gets 3x-4x more configuration lines with exactly the same result. The "job definition" is a bit easier to understand, however .travis.yml is always a bit ugly/complicated in my eyes if I compare it for instance with .gitlab-ci.yml job definitions...

I used the approach of chef-client cookbook here: https://github.com/chef-cookbooks/chef-client/blob/master/.travis.yml

@artem-sidorenko

artem-sidorenko Feb 17, 2017

Member

@chris-rock could you maybe explain it? Maybe I miss something: via your suggestion it gets 3x-4x more configuration lines with exactly the same result. The "job definition" is a bit easier to understand, however .travis.yml is always a bit ugly/complicated in my eyes if I compare it for instance with .gitlab-ci.yml job definitions...

I used the approach of chef-client cookbook here: https://github.com/chef-cookbooks/chef-client/blob/master/.travis.yml

This comment has been minimized.

@chris-rock

chris-rock Feb 20, 2017

Member

Agreed. Lets use your approach.

@chris-rock

chris-rock Feb 20, 2017

Member

Agreed. Lets use your approach.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Feb 21, 2017

Coverage Status

Coverage remained the same at 100.0% when pulling 9cfe819 on do-travis into b774c1b on master.

Coverage Status

Coverage remained the same at 100.0% when pulling 9cfe819 on do-travis into b774c1b on master.

@chris-rock

This is awesome work @artem-sidorenko I am seeing some errors in our integration tests, therefore it is good to see this executed during every merge! Thank you @artem-sidorenko

@chris-rock chris-rock merged commit 980b153 into master Feb 27, 2017

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage remained the same at 100.0%
Details

@chris-rock chris-rock deleted the do-travis branch Feb 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment