Test Artifact Fetch Feature #434

Closed
ehaselwanter opened this Issue May 8, 2014 · 8 comments

Comments

Projects
None yet
4 participants
@ehaselwanter

We started to use test-kitchen in a public open source security project. The goal is to provide chef and puppet automation to cover your bases.

In order to re-use integration tests it would be very helpful to be able to include integration tests from an specified artifact location.

  • archive uri
  • git uri

What do you think? Is there already a proposed solution I missed? kitchen should be aware I guess. Is there any hook we could use to trigger a pre verify action.

@fnichol

This comment has been minimized.

Show comment
Hide comment
@fnichol

fnichol May 8, 2014

Member

Okay this makes sense now.

I'm close to finishing up #427 which helps pave the way for features like this. A theme for some of these features (or gaps) is to add more explicit tuneable config to what was previously a directory or naming convention. The Busser and related family of functionality are pretty high on my list, as are generic lifecycle hooks that can be used/abused before/after many of the kitchen actions.

Member

fnichol commented May 8, 2014

Okay this makes sense now.

I'm close to finishing up #427 which helps pave the way for features like this. A theme for some of these features (or gaps) is to add more explicit tuneable config to what was previously a directory or naming convention. The Busser and related family of functionality are pretty high on my list, as are generic lifecycle hooks that can be used/abused before/after many of the kitchen actions.

@ehaselwanter

This comment has been minimized.

Show comment
Hide comment
@ehaselwanter

ehaselwanter May 8, 2014

yep. thought about that. but first ask if it is already on the list of something. an now I have a real world use case. Deutsche Telekom started an effort to OpenSource their security guidelines as well as reference implementations in puppet and chef. so they'll get a batch

verified by test-kitchen :-)

is there a tk bumper sticker 👍

yep. thought about that. but first ask if it is already on the list of something. an now I have a real world use case. Deutsche Telekom started an effort to OpenSource their security guidelines as well as reference implementations in puppet and chef. so they'll get a batch

verified by test-kitchen :-)

is there a tk bumper sticker 👍

@ehaselwanter

This comment has been minimized.

Show comment
Hide comment
@ehaselwanter

ehaselwanter May 15, 2014

so I tried to do something without changing test-kitchen itself but I think that is not quite possible. tried to enrich your thor tasks. ugly implementation:

https://github.com/ehaselwanter/kitchen-sharedtests/blob/master/lib/kitchen/sharedtests_thor_tasks.rb

several issues:

  • there is no place in Kitchen::Config for "global" config.
  • :test_base_path is not write able, must be passed in on creation of the config instance

=> it would be nice to be able to tweak the settings like:

---
tests:
  test_base_path: "./shared_tests"
  test_artifact_uri: "https://github.com/TelekomLabs/tests-os-hardening.git"
    branch: "dev"

or nested under a key global key 'kitchen'

test_artifact_uri should be something with providers I guess (git,hg,http/archive)

not sure if the implementation of this should go into busser, since busser copies over the tests. right?

@fnichol @sethvargo what do you think?

PS: project homepage is up http://telekomlabs.github.io

so I tried to do something without changing test-kitchen itself but I think that is not quite possible. tried to enrich your thor tasks. ugly implementation:

https://github.com/ehaselwanter/kitchen-sharedtests/blob/master/lib/kitchen/sharedtests_thor_tasks.rb

several issues:

  • there is no place in Kitchen::Config for "global" config.
  • :test_base_path is not write able, must be passed in on creation of the config instance

=> it would be nice to be able to tweak the settings like:

---
tests:
  test_base_path: "./shared_tests"
  test_artifact_uri: "https://github.com/TelekomLabs/tests-os-hardening.git"
    branch: "dev"

or nested under a key global key 'kitchen'

test_artifact_uri should be something with providers I guess (git,hg,http/archive)

not sure if the implementation of this should go into busser, since busser copies over the tests. right?

@fnichol @sethvargo what do you think?

PS: project homepage is up http://telekomlabs.github.io

@smurawski

This comment has been minimized.

Show comment
Hide comment
@smurawski

smurawski Feb 4, 2016

Contributor

@ehaselwanter Sorry for the delayed response on this issue. Is this something you still want to accomplish or have you found other workarounds?

Contributor

smurawski commented Feb 4, 2016

@ehaselwanter Sorry for the delayed response on this issue. Is this something you still want to accomplish or have you found other workarounds?

@ehaselwanter

This comment has been minimized.

Show comment
Hide comment
@ehaselwanter

ehaselwanter Feb 4, 2016

@smurawski well, you snapped up my former team mates @chris-rock @arlimus . So it is best to bring them in. As far as I know there is still the requirement to re-use tests throughout different kitchen projects.

Another thing comes to mind too were I do not know whether it is feasible or not: It would be really cool to add multiple testing libraries to one deployment. Like lets say you build an all-in-one machine with a web app + db in it. Then it would be super awesome to be able to reference https://github.com/hardening-io/tests-ssh-hardening https://github.com/hardening-io/tests-nginx-hardening https://github.com/hardening-io/tests-os-hardening und run them against the kitchen.

but is never as easy as that ;-)

@smurawski well, you snapped up my former team mates @chris-rock @arlimus . So it is best to bring them in. As far as I know there is still the requirement to re-use tests throughout different kitchen projects.

Another thing comes to mind too were I do not know whether it is feasible or not: It would be really cool to add multiple testing libraries to one deployment. Like lets say you build an all-in-one machine with a web app + db in it. Then it would be super awesome to be able to reference https://github.com/hardening-io/tests-ssh-hardening https://github.com/hardening-io/tests-nginx-hardening https://github.com/hardening-io/tests-os-hardening und run them against the kitchen.

but is never as easy as that ;-)

@chris-rock

This comment has been minimized.

Show comment
Hide comment
@chris-rock

chris-rock Feb 4, 2016

Contributor

@ehaselwanter @smurawski We will implement such feature with kitchen-inspec. This allows you to load local tests and add additional remote InSpec profiles.

@ehaselwanter This is funny, because the first real implementation was done by you 💯

Contributor

chris-rock commented Feb 4, 2016

@ehaselwanter @smurawski We will implement such feature with kitchen-inspec. This allows you to load local tests and add additional remote InSpec profiles.

@ehaselwanter This is funny, because the first real implementation was done by you 💯

@smurawski

This comment has been minimized.

Show comment
Hide comment
@smurawski

smurawski Feb 4, 2016

Contributor

@ehanselwanter @chris-rock I think this definitely belongs in a verifier plugin (like the inspec one), and should be totally doable there, as you control building the sandbox for transfer to the system under test.

As this is more a concern of the individual verifier (each verifier is responsible for building its own sandbox to transfer), I'm going to close this out as an issue against test-kitchen. If current verifiers (busser, inspec, pester) need this, please open an issue with them and we should be faster about responding (we've recently increased the pool of maintainers for test-kitchen).

Contributor

smurawski commented Feb 4, 2016

@ehanselwanter @chris-rock I think this definitely belongs in a verifier plugin (like the inspec one), and should be totally doable there, as you control building the sandbox for transfer to the system under test.

As this is more a concern of the individual verifier (each verifier is responsible for building its own sandbox to transfer), I'm going to close this out as an issue against test-kitchen. If current verifiers (busser, inspec, pester) need this, please open an issue with them and we should be faster about responding (we've recently increased the pool of maintainers for test-kitchen).

@smurawski

This comment has been minimized.

Show comment
Hide comment
@smurawski

smurawski Feb 4, 2016

Contributor

The mobile github app doesn't auto complete names, sorry for the mistype @ehaselwanter

Contributor

smurawski commented Feb 4, 2016

The mobile github app doesn't auto complete names, sorry for the mistype @ehaselwanter

@smurawski smurawski closed this Feb 4, 2016

@smurawski smurawski changed the title from Test Artifact Fetch Feature to Test Artifact Fetch Feature Feb 4, 2016

@chris-rock chris-rock referenced this issue in inspec/kitchen-inspec Feb 4, 2016

Closed

InSpec Profile support in kitchen-inspec #39

@test-kitchen test-kitchen locked and limited conversation to collaborators Nov 16, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.