New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration testing of this cookbook in the CI #142

Closed
artem-sidorenko opened this Issue Feb 13, 2017 · 9 comments

Comments

Projects
None yet
3 participants
@artem-sidorenko
Member

artem-sidorenko commented Feb 13, 2017

Currently we only run unit tests/lints. Its not easily possible to test this cookbook in the same way like we do with chef-ssh-hardening (kitchen-dokken): we change here tonns of OS parameters.

What about to have a proper integration testing via IaaS?

  • I already did it with digitalocean and it works just fine
  • Another option would be maybe the Google Cloud with an advantage - its billed per minute. I do not want to consider the ec2/azure, they are a bit complexer for this simple job (and require a bit more configuration/setup)

My suggested way:

Via this way we get following:

  • integration tests of main repository
  • people with forks can configure their own DO access token in travis and get integration tests too
  • in case of PRs without integration tests, we can repush them to our forks/main repo and get them tested

@atomic111 @chris-rock opinions?

@chris-rock

This comment has been minimized.

Show comment
Hide comment
@chris-rock

chris-rock Feb 14, 2017

Member

I like the idea, especially since not all parameters can be tested properly in a docker container. Do you think we should activate the current kitchen-dokken setup for this cookbook too?
This would allow us to have basic os coverage via containers and extensive testing via digital ocean. In our travis tests, we could make digital ocean optional so that we see the failure, but do not need to merge red PRs.

Member

chris-rock commented Feb 14, 2017

I like the idea, especially since not all parameters can be tested properly in a docker container. Do you think we should activate the current kitchen-dokken setup for this cookbook too?
This would allow us to have basic os coverage via containers and extensive testing via digital ocean. In our travis tests, we could make digital ocean optional so that we see the failure, but do not need to merge red PRs.

@artem-sidorenko

This comment has been minimized.

Show comment
Hide comment
@artem-sidorenko

artem-sidorenko Feb 14, 2017

Member

Do you think we should activate the current kitchen-dokken setup for this cookbook too?

I'm not sure this will work properly: we change a lot of really system related things, which are probably not covered by kernel namespaces (e.g. some sysctl flags related to tcp/ip)

We can try that and see if it works

Member

artem-sidorenko commented Feb 14, 2017

Do you think we should activate the current kitchen-dokken setup for this cookbook too?

I'm not sure this will work properly: we change a lot of really system related things, which are probably not covered by kernel namespaces (e.g. some sysctl flags related to tcp/ip)

We can try that and see if it works

@ehaselwanter

This comment has been minimized.

Show comment
Hide comment
@ehaselwanter

ehaselwanter Feb 14, 2017

Contributor

it would be cool to be able to switch off the kernel related tests anyways. I run the inspec suite against docker images (as the target) and at the moment I switch dem off with a profile.

Contributor

ehaselwanter commented Feb 14, 2017

it would be cool to be able to switch off the kernel related tests anyways. I run the inspec suite against docker images (as the target) and at the moment I switch dem off with a profile.

@chris-rock

This comment has been minimized.

Show comment
Hide comment
@chris-rock

chris-rock Feb 14, 2017

Member

@artem-sidorenko we could add an inspec attribute if required to our baseline tests. Kitchen supports inspec attributes already. Nevertheless, lets split this into two issues:

  • add digitalocean tests
  • add docker tests
Member

chris-rock commented Feb 14, 2017

@artem-sidorenko we could add an inspec attribute if required to our baseline tests. Kitchen supports inspec attributes already. Nevertheless, lets split this into two issues:

  • add digitalocean tests
  • add docker tests
@artem-sidorenko

This comment has been minimized.

Show comment
Hide comment
@artem-sidorenko

artem-sidorenko Feb 14, 2017

Member

@chris-rock It makes definitely sense to add some flag(s) to the chef-os-hardning and to the linux-baseline, which would allow use-case like @ehaselwanter described and enable a partly testing with containers. However I'm not sure about the short-term perspective of this.

What about following suggestion?

  • right now we add DO "full-vm" integration testing support like described (its straight-forward way and does not require architecture changes/new flags within chef-os-hardening and linux-baseline)
  • we release chef-os-hardening 2.0.0
  • for chef-os-hardening 3.0.0 we make the required changes in order to allow it to be more "container-friendly"
Member

artem-sidorenko commented Feb 14, 2017

@chris-rock It makes definitely sense to add some flag(s) to the chef-os-hardning and to the linux-baseline, which would allow use-case like @ehaselwanter described and enable a partly testing with containers. However I'm not sure about the short-term perspective of this.

What about following suggestion?

  • right now we add DO "full-vm" integration testing support like described (its straight-forward way and does not require architecture changes/new flags within chef-os-hardening and linux-baseline)
  • we release chef-os-hardening 2.0.0
  • for chef-os-hardening 3.0.0 we make the required changes in order to allow it to be more "container-friendly"
@chris-rock

This comment has been minimized.

Show comment
Hide comment
@chris-rock

chris-rock Feb 14, 2017

Member

@artem-sidorenko lets go with that approach. Lets have a discussion, what we expect from a bare container. cc @atomic111

Member

chris-rock commented Feb 14, 2017

@artem-sidorenko lets go with that approach. Lets have a discussion, what we expect from a bare container. cc @atomic111

@ehaselwanter

This comment has been minimized.

Show comment
Hide comment
@ehaselwanter

ehaselwanter Feb 15, 2017

Contributor

sounds good to me :-)

Contributor

ehaselwanter commented Feb 15, 2017

sounds good to me :-)

@artem-sidorenko

This comment has been minimized.

Show comment
Hide comment
@artem-sidorenko

artem-sidorenko Feb 16, 2017

Member

@chris-rock @ehaselwanter @atomic111 please have a look and review the #144

Member

artem-sidorenko commented Feb 16, 2017

@chris-rock @ehaselwanter @atomic111 please have a look and review the #144

@artem-sidorenko

This comment has been minimized.

Show comment
Hide comment
@artem-sidorenko

artem-sidorenko Oct 17, 2017

Member

I'll close this as we have now integration tests

Member

artem-sidorenko commented Oct 17, 2017

I'll close this as we have now integration tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment